Audio decoding method, audio encoding method, apparatus, device, and storage medium

Abstract

The application provides an audio decoding method, an audio encoding method, a device, equipment and a storage medium. The method comprises: performing signal decoding processing on an audio code stream to obtain an audio feature estimation value corresponding to the audio code stream; performing first reconstruction processing on the audio feature estimation value through a main network to obtain a first reconstruction signal corresponding to the audio code stream; performing second reconstruction processing on the audio feature estimation value through a secondary network to obtain a second reconstruction signal corresponding to the audio code stream; wherein the number of convolution layers included in the main network is greater than the number of convolution layers included in the secondary network, and the dimension of the first reconstruction signal is the same as the dimension of the second reconstruction signal; and fusing the first reconstruction signal and the second reconstruction signal to obtain a reconstructed audio signal corresponding to the audio code stream. Through the application, the decoding quality of the audio can be improved.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to an audio decoding method, an audio encoding method, an apparatus, a device, and a storage medium. Background Technology

[0002] Audio encoding and decoding technology is a crucial application in the field of artificial intelligence, and a core technology in communication services, including remote audio and video calls. Simply put, speech coding technology aims to transmit as much speech information as possible with minimal network bandwidth. From the perspective of Shannon's information theory, speech coding is a type of source coding. The goal of source coding is to compress the amount of data we want to transmit as much as possible at the encoding end, removing redundancy, while simultaneously achieving lossless (or near-lossless) reconstruction at the decoding end.

[0003] In related technologies, the signal reconstructed at the decoding end is difficult to recover the most authentic information of the input signal, resulting in poor audio decoding quality. Summary of the Invention

[0004] This application provides an audio decoding method, an audio encoding method, an apparatus, a device, and a storage medium that can improve the decoding quality of audio.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides an audio decoding method, the method comprising:

[0007] The audio bitstream is subjected to signal decoding processing to obtain the estimated audio feature values ​​corresponding to the audio bitstream;

[0008] The estimated audio features are reconstructed using the backbone network to obtain the first reconstructed signal corresponding to the audio bitstream.

[0009] The estimated audio features are reconstructed using a secondary network to obtain the second reconstructed signal corresponding to the audio bitstream.

[0010] Wherein, the number of convolutional layers in the backbone network is greater than the number of convolutional layers in the secondary network, and the dimension of the first reconstructed signal is the same as the dimension of the second reconstructed signal;

[0011] By fusing the first reconstructed signal and the second reconstructed signal, the reconstructed audio signal corresponding to the audio bitstream is obtained.

[0012] In the above scheme, the backbone network includes multiple cascaded first neural network layers; the first reconstruction process performed on the audio feature estimates through the backbone network to obtain the first reconstructed signal corresponding to the audio bitstream includes:

[0013] The input of the nth first neural network layer is used to reconstruct features through the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer, where n is a positive integer that increases sequentially, 0 < n ≤ N, N is the number of the first neural network layers in the backbone network, and the input of the first first neural network layer is the audio feature estimate.

[0014] The feature reconstruction result of the Nth first neural network layer is determined as the first reconstructed signal.

[0015] In the above scheme, the step of performing feature reconstruction on the input of the nth first neural network layer through the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer includes:

[0016] The audio feature estimates are convolved through the first neural network layer to obtain the feature reconstruction result of the first neural network layer.

[0017] The feature reconstruction result of the (m-1)th first neural network layer is decoded through the m-th first neural network layer to obtain the feature reconstruction result of the m-th first neural network layer, where m is a positive integer that increases sequentially, 2 < m < N-1;

[0018] By post-processing the feature reconstruction results of the N-2th first neural network layer through the N-1th first neural network layer, the feature extraction results of the N-1th first neural network layer are obtained.

[0019] The feature reconstruction result of the (N-1)th first neural network layer is obtained by convolution processing through the Nth first neural network layer.

[0020] In the above scheme, the step of decoding the input of the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer includes:

[0021] By using the deconvolution layer included in the nth first neural network layer, the input of the nth first neural network layer is deconvolutionally processed to obtain deconvolution features;

[0022] The deconvolutional features are upsampled through the upsampling layer included in the nth first neural network layer to obtain upsampled features;

[0023] By using the residual blocks included in the nth first neural network layer, feature residual processing is performed on the upsampled features to obtain the feature reconstruction result of the nth first neural network layer.

[0024] In the above scheme, the residual block includes multiple cascaded residual units;

[0025] The step of performing feature residual processing on the upsampled features through the residual blocks included in the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer includes:

[0026] The upsampled features are processed using the first residual unit included in the nth first neural network layer;

[0027] The residual result output by the first residual unit is output to the subsequent cascaded residual units, and the residual processing and residual result output are continued through the subsequent cascaded residual units;

[0028] The residual result output by the last residual unit is used as the feature reconstruction result of the nth first neural network layer.

[0029] In the above scheme, the step of performing feature residual processing on the upsampled features through the first residual unit included in the nth first neural network layer includes:

[0030] The following processing is performed through the first residual unit of the plurality of cascaded residual units:

[0031] The upsampled features are convolved to obtain the convolution result of the first residual unit;

[0032] The convolution result of the first residual unit is added to the residual feature to obtain the residual result output by the first residual unit;

[0033] The process of continuing residual processing and outputting residual results through the subsequently cascaded residual units includes:

[0034] The following processing is performed through the j-th residual unit of the plurality of cascaded residual units:

[0035] The residual result output by the (j-1)th residual unit is convolved to obtain the convolution result of the j-th residual unit;

[0036] The convolution results of the j residual units are added to the residual result output by the (j-1)th residual unit to obtain the residual result output by the jth residual unit.

[0037] Output the residual result output by the j-th residual unit to the (j + 1)-th residual unit;

[0038] where j is a positive integer increasing in sequence, 1 < j < J, and J is the number of the residual units.

[0039] In the above solution, each of the residual units includes a dilated convolution operator;

[0040] Performing convolution processing on the upsampled feature through the first residual unit of the multiple cascaded residual units includes:

[0041] Performing dilated convolution processing on the upsampled feature through the dilated convolution operator included in the first residual unit.

[0042] In the above solution, each of the residual units further includes at least one causal convolution operator;

[0043] After performing dilated convolution processing on the upsampled feature through the dilated convolution operator included in the first residual unit, the method further includes:

[0044] Performing causal convolution processing on the obtained dilated convolution result through at least one causal convolution operator included in the first residual unit, and taking the obtained causal convolution result as the convolution result of the first residual unit.

[0045] In the above solution, the secondary network includes multiple cascaded second neural network layers;

[0046] Performing second reconstruction processing on the audio feature estimate through the secondary network to obtain a second reconstruction signal corresponding to the audio code stream includes:

[0047] Obtain the feature reconstruction result of the g-th first neural network layer in the backbone network, where the feature reconstruction result of the g-th first neural network layer is obtained based on the audio feature estimate, 0 < g < N, and g is a positive integer;

[0048] Perform feature reconstruction on the input of the k-th second neural network layer through the k-th second neural network layer of the secondary network to obtain the feature reconstruction result of the k-th second neural network layer, where k is a positive integer increasing in sequence, 0 < k ≤ K, K is the number of layers of the second neural network layer in the secondary network, and the input of the first second neural network layer is the feature reconstruction result of the g-th first neural network layer;

[0049] Determine the feature reconstruction result of the K-th second neural network layer as the second reconstruction signal.

[0050] An embodiment of the present application provides an audio coding method, and the method includes:

[0051] Feature extraction is performed on the audio signal to obtain the audio features of the audio signal;

[0052] The audio features are processed by signal encoding to obtain the audio bitstream of the audio signal, wherein the audio bitstream is used for decoding based on the audio decoding method provided in the embodiments of this application.

[0053] This application provides an audio decoding device, including:

[0054] The signal decoding module is used to perform signal decoding processing on the audio bitstream to obtain the audio feature estimation value corresponding to the audio bitstream;

[0055] The first reconstruction module is used to perform a first reconstruction process on the audio feature estimate through the backbone network to obtain the first reconstructed signal corresponding to the audio bitstream.

[0056] The second reconstruction module is used to perform a second reconstruction process on the audio feature estimate through a secondary network to obtain the second reconstructed signal corresponding to the audio bitstream;

[0057] Wherein, the number of convolutional layers in the backbone network is greater than the number of convolutional layers in the secondary network, and the dimension of the first reconstructed signal is the same as the dimension of the second reconstructed signal;

[0058] The fusion module is used to fuse the first reconstructed signal and the second reconstructed signal to obtain the reconstructed audio signal corresponding to the audio bitstream.

[0059] This application provides an audio encoding device, including:

[0060] The feature extraction module is used to extract features from the audio signal to obtain the audio features of the audio signal;

[0061] The signal encoding module is used to perform signal encoding processing on the audio features to obtain the audio bitstream of the audio signal, wherein the audio bitstream is used for decoding based on the audio decoding method provided in the embodiments of this application.

[0062] This application provides a method for processing audio bitstreams, wherein the audio bitstreams are decoded based on the audio decoding method provided in this application, or generated according to the audio encoding method provided in this application.

[0063] This application provides an electronic device, the electronic device comprising:

[0064] Memory is used to store executable instructions or computer programs.

[0065] When a processor executes computer-executable instructions or computer programs stored in the memory, it implements the audio encoding method or the audio decoding method provided in the embodiments of this application.

[0066] This application provides a computer-readable storage medium storing a bitstream formed by computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, they implement the audio encoding method or the audio decoding method provided in this application.

[0067] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, they implement the audio encoding method or the audio decoding method provided in this application.

[0068] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the audio encoding method or the audio decoding method provided in this application.

[0069] The embodiments of this application have the following beneficial effects:

[0070] Because the backbone network contains more convolutional layers, it can reconstruct a deeper first reconstructed signal. And because the secondary network has relatively fewer convolutional layers compared to the backbone network, it can reconstruct a shallower second reconstructed signal, which is convenient for capturing shallow details in the audio signal. By fusing the two signals, both deep and shallow information can be used at the same time, so that the reconstructed audio signal contains richer audio information, so as to better recover the audio signal and thus improve the audio decoding quality. Attached Figure Description

[0071] Figure 1 This is a schematic diagram comparing the spectrum at different code rates provided in the embodiments of this application;

[0072] Figure 2 This is a schematic diagram of the audio encoding and decoding system architecture provided in the embodiments of this application;

[0073] Figure 3A This is a first structural schematic diagram of the electronic device provided in an embodiment of this application;

[0074] Figure 3B This is a schematic diagram of the second structure of the electronic device provided in the embodiments of this application;

[0075] Figure 4A This is a first flowchart illustrating the audio encoding method provided in this application embodiment;

[0076] Figure 4B This is a second flowchart illustrating the audio encoding method provided in the embodiments of this application;

[0077] Figure 5 This is a flowchart illustrating the audio decoding method provided in an embodiment of this application;

[0078] Figure 6 This is a schematic diagram of a voice communication link provided in an embodiment of this application;

[0079] Figure 7A This is a first flowchart illustrating the speech encoding and decoding method provided in this application embodiment;

[0080] Figure 7B This is a second flowchart illustrating the speech encoding and decoding method provided in the embodiments of this application; Figure 8A This is a schematic diagram of a common convolutional network provided in an embodiment of this application;

[0081] Figure 8B This is a schematic diagram of the dilated convolutional network provided in an embodiment of this application;

[0082] Figure 9 This is a schematic diagram of the first analysis network provided in an embodiment of this application;

[0083] Figure 10A This is a schematic diagram of the residual block structure used in the coding block provided in the embodiments of this application;

[0084] Figure 10B This is a schematic diagram of the residual unit structure provided in an embodiment of this application;

[0085] Figure 11A This is a first schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in the embodiments of this application;

[0086] Figure 11B This is a network structure diagram of the second long-term connection network provided in the embodiments of this application;

[0087] Figure 11C This is a schematic diagram of an optional structure of the first reconstruction network and the second long-term connection network provided in the embodiments of this application;

[0088] Figure 12A This is a second schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in the embodiments of this application;

[0089] Figure 12B This is a second schematic diagram of an optional structure of the first reconstruction network and the second long-term connection network provided in the embodiments of this application;

[0090] Figure 13A This is a third schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in the embodiments of this application;

[0091] Figure 13B This is a fourth schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in the embodiments of this application;

[0092] Figure 13C This is a third schematic diagram of an optional structure of the first reconstruction network and the second long-term connection network provided in the embodiments of this application;

[0093] Figure 13D This is a fourth schematic diagram of an optional structure of the first reconstruction network and the second long-term connection network provided in the embodiments of this application;

[0094] Figure 13E This is a fourth schematic diagram of an optional structure of the first reconstruction network and the second long-term connection network provided in the embodiments of this application;

[0095] Figure 13F This is a fourth schematic diagram of an optional structure of the first reconstruction network and the second long-term connection network provided in the embodiments of this application.

[0096] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0097] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0098] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0099] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0100] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0101] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0102] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0103] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0104] 1) Neural Network (NN): A mathematical model that mimics the behavioral characteristics of animal neural networks to perform distributed parallel information processing. This type of network relies on the complexity of the system to adjust the relationships between a large number of interconnected nodes, thereby achieving the purpose of information processing.

[0105] 2) Deep Learning (DL): This is a new research direction in the field of Machine Learning (ML). Deep learning learns the inherent patterns and representational layers of sample data. The information gained during this learning process greatly helps in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to have analytical and learning capabilities like humans, and to be able to recognize data such as text, images, and sound.

[0106] 3) Quantization: This refers to the process of approximating a signal's continuous values ​​(or a large number of discrete values) into a finite number (or a small number) of discrete values. Quantization includes vector quantization (VQ) and scalar quantization.

[0107] Vector quantization is an effective lossy compression technique based on Shannon's rate distortion theory. The basic principle of vector quantization is to use the index (also called the quantization value) of the codeword in the code table that best matches the input vector for transmission and storage, while decoding only requires a simple table lookup operation. For example, a vector space can be formed by combining several scalar data points, and this vector space can be divided into several small regions. During quantization, the vector falling into a small region is replaced with its corresponding index as the input vector.

[0108] Scalar quantization is the quantization of a scalar, that is, one-dimensional vector quantization. It divides the dynamic range into several small intervals, each with a representative value (i.e., index). When the input signal falls into a certain interval, the input signal is quantized into that representative value.

[0109] 4) Entropy coding: A lossless coding method that does not lose any information during the encoding process according to the entropy principle. It is also a key module in lossy coding and is located at the end of the encoder. Entropy coding includes Shannon coding, Huffman coding, Exponential Golomb coding, and Arithmetic coding.

[0110] Speech coding technology aims to transmit as much speech information as possible with minimal network bandwidth. Speech codecs can achieve compression ratios of over 10 times, meaning that 10MB of speech data can be compressed to only 1MB for transmission, significantly reducing the bandwidth required for information transmission. For example, for a wideband speech signal with a sampling rate of 16000Hz, using a 16-bit sampling depth (the fineness of speech intensity recording during sampling), the uncompressed version has a bitrate (data transmission rate per unit time) of 256kbps. However, with speech coding technology, even lossy coding, the reconstructed speech signal quality can approach that of the uncompressed version within a bitrate range of 10-20kbps, and may even be perceived as indistinguishable. For services requiring higher sampling rates, such as 32000Hz ultrawideband speech, the bitrate range must be at least 30kbps.

[0111] In communication systems, to ensure smooth communication, the industry deploys standard voice codec protocols, such as standards from international and domestic standards organizations like ITU-T, 3GPP, IETF, AVS, and CCSA, including standards like G.711, G.722, AMR series, EVS, and OPUS. Figure 1A comparative spectral analysis diagram at different bitrates is provided to illustrate the relationship between compression bitrate and quality. Curve 101 represents the spectral curve of the original speech, i.e., the uncompressed signal; curve 102 represents the spectral curve of the OPUS encoder at a bitrate of 20kbps; and curve 103 represents the spectral curve of the OPUS encoding at a bitrate of 6kbps. Figure 1 It can be seen that as the encoding bit rate increases, the compressed signal is closer to the original signal.

[0112] The principle of speech coding is roughly as follows: speech coding can directly encode speech waveform samples one by one; or, based on the principle of human vocalization, relevant low-dimensional features are extracted, the encoding end encodes the features, and the decoding end reconstructs the speech signal based on these parameters.

[0113] The above-mentioned encoding principles are all derived from speech signal modeling, i.e., compression methods based on signal processing, which cannot guarantee the encoding quality of audio. To improve encoding efficiency while ensuring speech quality, embodiments of this application provide an audio encoding method, an audio decoding method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. The following describes exemplary applications of the electronic device provided in this application. The electronic device provided in this application can be implemented as a terminal device, a server, or a combination of both. The following description uses the example of an electronic device implemented as a terminal device.

[0114] For example, see Figure 2 , Figure 2 This is a schematic diagram of the architecture of the audio codec system 100 provided in the embodiments of this application. The audio codec system 100 includes: server 200, network 300, terminal device 400 (i.e., encoding end) and terminal device 500 (i.e., decoding end). The network 300 can be a local area network, a wide area network, or a combination of the two.

[0115] In some embodiments, a client 410 runs on the terminal device 400. The client 410 can be various types of clients, such as instant messaging clients, web conferencing clients, live streaming clients, browsers, etc. In response to an audio acquisition command triggered by the sender (e.g., the initiator of a web conference, a broadcaster, the initiator of satellite communication, etc.), the client 410 calls the microphone built into the terminal device 400 to acquire audio signals, and performs audio encoding processing on the acquired audio signals to obtain an audio bitstream.

[0116] For example, client 410 calls the audio encoding method provided in the embodiments of this application to encode the acquired audio signal, that is, to extract features from the audio signal to obtain audio features of the audio signal; and to perform signal encoding processing on the audio features to obtain the audio bitstream of the audio signal, wherein the audio bitstream is used for decoding based on the audio decoding method provided in the embodiments of this application.

[0117] Client 410 can send the audio stream to server 200 via network 300, so that server 200 can send the audio stream to the terminal device 500 associated with the recipient (such as participants in a web conference, viewers, or receivers of satellite communications).

[0118] After receiving the audio stream sent by the server 200, the client 510 (such as an instant messaging client, web conferencing client, live streaming client, browser, etc.) can perform audio decoding processing on the audio stream to obtain a synthesized audio signal, thereby realizing audio communication.

[0119] For example, client 510 calls the audio decoding method provided in this application embodiment to decode the received audio bitstream, that is, to perform signal decoding processing on the audio bitstream to obtain the audio feature estimation value corresponding to the audio bitstream; to perform a first reconstruction processing on the audio feature estimation value through the backbone network to obtain a first reconstructed signal corresponding to the audio bitstream; to perform a second reconstruction processing on the audio feature estimation value through the secondary network to obtain a second reconstructed signal corresponding to the audio bitstream; wherein, the number of convolutional layers included in the backbone network is greater than the number of convolutional layers included in the secondary network, and the dimension of the first reconstructed signal is the same as the dimension of the second reconstructed signal; and the first reconstructed signal and the second reconstructed signal are fused to obtain the reconstructed audio signal corresponding to the audio bitstream.

[0120] Example, Figure 2 The server 200 shown can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Figure 2 The terminal devices 400 and 500 shown can be smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, in-vehicle terminals, etc., but are not limited to these. The terminal devices (such as terminal devices 400 and 500) and server 200 can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0121] In some embodiments, the terminal device or server 200 can also implement the audio encoding or decoding method provided in this application embodiment by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), that is, a program that needs to be installed in the operating system to run, such as a live streaming APP, a web conferencing APP, or an instant messaging APP; it can also be a mini-program, that is, a program that only needs to be downloaded into a browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin.

[0122] See Figure 3A , Figure 3A This is a schematic diagram of the structure of the electronic device 500 provided in the embodiments of this application. Taking the electronic device 500 as a terminal device as an example, the following explanation is provided. Figure 3A The illustrated electronic device 500 includes at least one processor 520, a memory 550, at least one network interface 530, and a user interface 540. The various components in the electronic device 500 are coupled together via a bus system 550. It is understood that the bus system 550 is used to implement communication between these components. In addition to a data bus, the bus system 550 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 3A The general labeled all buses as Bus System 550.

[0123] The processor 520 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor.

[0124] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices physically located away from the processor 520.

[0125] The memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0126] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0127] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0128] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 530, exemplary network interfaces 530 including Bluetooth, WiFi, and Universal Serial Bus (USB).

[0129] In some embodiments, the audio encoding apparatus provided in this application can be implemented in software. Figure 3A An audio encoding device 555 stored in memory 550 is shown. It can be software in the form of programs and plug-ins, including the following software modules: feature extraction module 5551 and signal encoding module 5552. The feature extraction module 5551 and signal encoding module 5552 are used to implement audio encoding functions. These modules are logical and can therefore be arbitrarily combined or further split according to the functions implemented.

[0130] See Figure 3B , Figure 3B This is a schematic diagram of the structure of the electronic device 600 provided in the embodiments of this application. Taking the electronic device 600 as a terminal device as an example, the following description is provided. Figure 3B The illustrated electronic device 600 includes at least one processor 620, a memory 650, at least one network interface 630, and a user interface 640. The various components of the electronic device 600 are coupled together via a bus system 650. The memory 650 includes an operating system 651 and a network communication module 652. It should be noted that... Figure 3B The function of the structure in Figure 3A The structure in this application functions similarly to the one in the previous one. The audio decoding device provided in this embodiment can be implemented in software. Figure 3BThe figure shows an audio decoding device 655 stored in a memory 650, which may be software in the form of programs and plug-ins, etc., including the following software modules: a signal decoding module 6551, a first reconstruction module 6552, a second reconstruction module 6553, and a fusion module 6554. Among them, the signal decoding module 6551, the first reconstruction module 6552, the second reconstruction module 6553, and the fusion module 6554 are used to implement the audio decoding function. These modules are logical, so they can be combined or further split arbitrarily according to the implemented functions.

[0131] As mentioned above, the audio encoding method provided by the embodiments of the present application can be implemented by various types of electronic devices. Refer to Figure 4A , Figure 4A which is the first process schematic diagram of the audio encoding method provided by the embodiments of the present application. The audio encoding function is implemented through the audio encoding method. The following will be described in conjunction with Figure 4A the steps 101-step 104 shown.

[0132] In step 101, the audio signal is subjected to feature extraction to obtain the audio features of the audio signal.

[0133] Exemplarily, the audio signal can be obtained by real-time recording through a microphone device, or an audio file can be loaded locally as the audio signal. The embodiments of the present application do not limit the specific acquisition method of the audio signal.

[0134] In some embodiments, the feature extraction is implemented through a feature extraction network, and the feature extraction network includes multiple cascaded fourth neural network layers; step 101 can be implemented in the following manner: through the t-th fourth neural network layer of the feature extraction network, the input of the t-th fourth neural network layer is subjected to feature extraction to obtain the feature extraction result of the t-th fourth neural network layer, where t is a positive integer that increases sequentially, 0 < t ≤ T, and T is the number of fourth neural network layers in the feature extraction network. The input of the first fourth neural network layer is the audio signal; the feature extraction result of the T-th fourth neural network layer is determined as the audio feature.

[0135] By performing feature extraction on the audio signal through a feature extraction network constructed by multiple cascaded fourth neural networks, multi-level abstraction and transformation (which can be understood as data compression) can be performed on the original audio signal, so that the vector dimension of the audio features is smaller than that of the audio signal, reducing the data volume and facilitating subsequent encoding and transmission.

[0136] The embodiments of this application are not limited to the form of the fourth neural network layer, such as causal convolutional layers, deep learning networks, etc. As an example, feature extraction is performed on the audio signal through the first fourth neural network layer to obtain the feature extraction result of the first fourth neural network layer. The feature extraction result of the first fourth neural network layer is used as the input of the second fourth neural network layer. The feature extraction result of the first fourth neural network layer is then extracted through the second fourth neural network layer to obtain the feature extraction result of the second fourth neural network layer, and so on, until the feature extraction result of the last fourth neural network layer is determined as the audio feature.

[0137] In some embodiments, feature extraction is performed on the input of the t-th fourth neural network layer through the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer. This can be achieved by performing one of the following processes through the t-th fourth neural network layer of the feature extraction network: performing convolution processing on the input of the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer; performing preprocessing on the input of the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer; or performing encoding processing on the input of the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer.

[0138] For example, see Figure 9 The diagram shows the first analysis network (i.e., the feature extraction network, which includes multiple cascaded fourth neural network layers), where the multiple cascaded fourth neural network layers correspond to... Figure 9 The causal convolutional layer, preprocessing layer, and coding block shown in the figure are as follows: Figure 9As shown, when n = 1, the input to the first fourth neural network layer is subjected to convolutional processing to obtain the feature extraction result of the first fourth neural network layer. That is, by invoking a causal convolutional layer with 16 channels, the input tensor (i.e., the audio signal x(n)) can be expanded into causal convolutional features of 16×320 (i.e., the feature extraction result of the first fourth neural network layer). When n = 2, the input to the second fourth neural network layer is preprocessed to obtain the feature extraction result of the second fourth neural network layer. That is, after expanding the input tensor (i.e., the audio signal) into causal convolutional features of 16×320 (i.e., the feature extraction result of the first fourth neural network layer) by invoking a causal convolutional layer with 16 channels, a convolutional operation is performed on the causal convolutional features of 16×320, along with pooling (Pooling) processing with a factor of 2, and the activation function can be a parametric rectified linear unit (PReLU) to generate a tensor of 16×160 (i.e., the feature extraction result of the second fourth neural network layer). When 2 < n < N, the input to the nth fourth neural network layer is encoded (implemented through an encoding block) to obtain the feature extraction result of the nth fourth neural network layer. When n = N, the input to the Nth fourth neural network layer is subjected to convolutional processing to obtain the first audio feature (i.e., Figure 9 F1(n) in

[0139] In some embodiments, encoding the input to the tth fourth neural network layer to obtain the feature extraction result of the tth fourth neural network layer can be achieved in the following manner: performing residual processing on the input to the tth fourth neural network layer through the residual block included in the tth fourth neural network layer to obtain residual features; performing convolutional processing on the residual features through the convolutional layer included in the tth fourth neural network layer to obtain convolutional features; and performing downsampling processing on the convolutional features through the downsampling layer included in the tth fourth neural network layer to obtain the feature extraction result of the tth fourth neural network layer.

[0140] For example, the convolutional features are obtained by sliding a convolutional kernel (filter) on the feature map of the residual features and calculating the weighted sum of the local regions. Through convolutional processing, local patterns and structural information in the audio signal can be captured, thereby extracting more advanced features.

[0141] The fourth neural network layer includes residual blocks comprising multiple cascaded residual units. The residual block of the t-th fourth neural network layer is used to process the input of the t-th fourth neural network layer to obtain residual features. This can be achieved as follows: the first residual unit of the multiple cascaded residual units processes the input of the t-th fourth neural network layer; the residual result output from the first residual unit is output to subsequent cascaded residual units, and residual processing and output of residual results continue through these subsequent cascaded residual units; the residual result output from the last residual unit is used as the residual feature. The embodiments of this application do not limit the number of residual blocks, which can be any positive integer such as 2, 3, 4, or 5. The embodiments of this application also do not limit the number of residual units in a residual block, which can be any positive integer such as 2, 3, 4, 5, or 6. The number of residual units in multiple residual blocks can be the same or different; for example, one residual block may contain 4 residual units, and another residual block may contain 5 residual units.

[0142] The processing of the residual units in the fourth neural network layer is similar to that in the audio decoding method described below. For details, please refer to the processing of the residual units in the audio decoding method described below.

[0143] For example, the embodiments of this application are not limited to the processing method of the downsampling layer included in the fourth neural network layer. Downsampling can be performed by pooling methods such as max pooling and average pooling, or by methods such as bilinear interpolation and nearest neighbor interpolation. The embodiments of this application do not limit the specific downsampling method.

[0144] It should be noted that step 101 is similar to step 11.

[0145] See Figure 4B , Figure 4B Show Figure 4A Step 101 can be achieved through steps 1011-1013:

[0146] In step 1011, the first feature of the audio signal is extracted through the backbone network to obtain the first audio feature of the audio signal.

[0147] The processing of the backbone network in the encoding end is similar to that of the feature extraction network described above, and will not be repeated here.

[0148] Following step 1011, in step 1012, the audio signal is subjected to second feature extraction through a secondary network to obtain the second audio feature of the audio signal.

[0149] It should be noted that the number of convolutional layers in the secondary network is less than that in the backbone network. Such a design facilitates capturing shallow details in the audio signal, reducing the consumption of computing resources, and improving the processing efficiency.

[0150] In some embodiments, the secondary network includes multiple cascaded fifth neural network layers; obtain the feature extraction result of the a-th fourth neural network layer in the backbone network, where the feature extraction result of the a-th fourth neural network layer is obtained based on the audio signal, 0 < a < N, and a is a positive integer; through the b-th fifth neural network layer of the secondary network, perform feature extraction on the input of the b-th fifth neural network layer to obtain the feature extraction result of the b-th fifth neural network layer, where b is a positive integer that increases sequentially, 0 < b ≤ B, and B is the number of fifth neural network layers in the secondary network, and the input of the first fifth neural network layer is the feature extraction result of the a-th fourth neural network layer; determine the feature extraction result of the B-th fifth neural network layer as the second audio feature.

[0151] Continuing with the above example, performing feature extraction on the input of the b-th fifth neural network layer to obtain the feature extraction result of the b-th fifth neural network layer can be achieved in the following way: the b-th fifth neural network layer of the secondary network performs one of the following processes: perform convolutional processing on the input of the b-th fifth neural network layer to obtain the feature extraction result of the b-th fifth neural network layer; perform encoding processing on the input of the b-th fifth neural network layer to obtain the feature extraction result of the b-th fifth neural network layer.

[0152] Continuing with the above example, performing encoding processing on the input of the b-th fifth neural network layer to obtain the feature extraction result of the b-th fifth neural network layer can be achieved in the following way: through at least one convolutional layer included in the b-th fifth neural network layer, perform at least one convolutional processing on the input of the b-th fifth neural network layer to obtain convolutional features; through the pooling layer included in the b-th fifth neural network layer, perform pooling processing on the convolutional features to obtain the feature extraction result of the b-th fifth neural network layer.

[0153] In some other embodiments, the b-th fifth neural network layer of the secondary network performs feature extraction on the input of the b-th fifth neural network layer to obtain the feature extraction result of the b-th fifth neural network layer, which can be achieved in the following manner: the first fifth neural network layer encodes the feature extraction result of the g-th fourth neural network layer to obtain the feature extraction result of the first fifth neural network layer; the c-th fifth neural network layer encodes the feature extraction result of the (c - 1)-th fifth neural network layer to obtain the feature extraction result of the c-th fifth neural network layer, where x is a positive integer that increases sequentially, 2 < c < B; the B-th fifth neural network layer performs convolution on the feature extraction result of the (B - 1)-th fifth neural network layer to obtain the feature extraction result of the B-th fifth neural network layer.

[0154] It should be noted that step 1012 is similar to step 12.

[0155] Following step 1012, in step 1013, the first audio feature and the second audio feature are fused to obtain a fused audio feature, and the fused audio feature is used as the audio feature for signal encoding.

[0156] It should be noted that the first audio feature and the second audio feature have the same feature dimension. The fused audio feature after fusion contains information at different levels, which can provide a higher-quality feature input for subsequent signal encoding processing, thereby improving the encoding quality of the audio.

[0157] In some embodiments, the backbone network includes a first sub-network and a second sub-network. The first sub-network is used for performing first feature extraction; the first audio feature and the second audio feature are fused to obtain a fused audio feature, which can be achieved in the following manner: the first audio feature and the second audio feature are added together to obtain a third audio feature; the second sub-network performs third feature extraction on the third audio feature to obtain the fused audio feature.

[0158] Here, the specific implementation manner of the third feature extraction can refer to the description of causal convolution in the encoding block above, and will not be elaborated here.

[0159] In some other embodiments, the first audio feature and the second audio feature are added together to obtain a third audio feature; the third audio feature is non-linearly mapped through an activation function to obtain the fused audio feature.

[0160] It should be noted that step 1013 is similar to step 13.

[0161] In step 102, the audio feature is subjected to signal encoding processing to obtain the audio code stream of the audio signal.

[0162] Here, signal encoding processing of audio features refers to converting audio features into a format suitable for storage, transmission or further processing. In this process, audio features are encoded into a specific data stream, namely an audio bitstream, which is more suitable for transmission in network environments with limited bandwidth.

[0163] In some embodiments, before performing signal encoding processing on the audio features to obtain the audio bitstream of the audio signal, the following processing may also be performed: obtaining a target bitrate mode for the audio signal from multiple bitrate modes; correspondingly, step 102 can be implemented in the following way: performing signal encoding processing on the audio features through the target bitrate mode to obtain the audio bitstream of the audio signal.

[0164] Following the example above, the target bitrate mode is used to indicate the use of a target code table for signal encoding processing. The audio bitstream of the audio signal is obtained by encoding audio features using the target bitrate mode, which can be achieved as follows: Quantize the audio features using the target code table to obtain quantized values ​​of the fused audio features; then entropy encode the quantized values ​​using the target bitrate corresponding to the target code table to obtain the audio bitstream of the audio signal.

[0165] For example, the audio features F(n) of an audio signal can be quantized using scalar quantization (each component is quantized individually) and entropy coding. Furthermore, this application does not limit the combination of vector quantization (multiple adjacent components are combined into a vector for joint quantization) and entropy coding techniques. In addition to entropy coding, direct coding can also be used; for example, if a codebook has 8 elements, 3 bits are used for encoding; if a codebook has 16 elements, 4 bits are used for encoding, and so on.

[0166] It should be noted that step 104 is similar to step 14 below.

[0167] As mentioned above, the audio decoding method provided in this application can be implemented by various types of electronic devices. See also Figure 5 , Figure 5 This is a schematic diagram of the first flow of the audio decoding method provided in this embodiment. The audio decoding function is implemented through the audio decoding method. The following describes the process in conjunction with... Figure 5 Steps 201-204 are shown and explained.

[0168] In step 201, the audio bitstream is subjected to signal decoding processing to obtain the audio feature estimation value corresponding to the audio bitstream.

[0169] The audio bitstream is generated according to the audio encoding method provided in the embodiments of this application.

[0170] For example, an audio stream file can be read from a local file system or network storage, or a real-time audio stream (such as an audio stream in a live streaming broadcast or video conference) can be received over a network. This application does not limit the specific method of obtaining the audio stream.

[0171] It's important to note that signal decoding is the inverse process of signal encoding. Since the decoding process at the decoding end is the reverse of the encoding process at the encoding end, the values ​​generated during decoding are estimates relative to the values ​​generated during encoding. For example, the data values ​​representing audio features generated during decoding are estimates relative to the audio features in the encoding process. That is, the former is not necessarily equivalent to the original audio feature data values. For instance, there may be subtle differences due to the encoding and decoding processes, and there may be situations where decoding cannot completely reconstruct the original audio feature data values. Therefore, the former is also called an "audio feature estimate."

[0172] As an example, in response to a decoding request for an audio stream encapsulation, the audio stream encapsulation is obtained, and a target bitrate mode is obtained from the frame header included in the audio stream encapsulation. The audio stream included in the audio stream encapsulation is obtained by audio encoding the audio signal using the target bitrate mode and the audio encoding method provided in this embodiment. The target bitrate mode is obtained from multiple bitrate modes. Here, the decoding request is used to instruct audio decoding processing to be performed on the audio stream.

[0173] The target bitrate mode is used to indicate the use of a target code table for signal decoding of the audio bitstream. The audio feature estimation value corresponding to the audio bitstream is obtained by performing signal decoding on the audio bitstream using the target bitrate mode. This can be achieved as follows: Entropy decoding is performed on the audio bitstream using the target bitrate corresponding to the target code table to obtain the quantized value of the audio bitstream; then, inverse quantization is performed on the quantized value of the audio bitstream using the target code table to obtain the audio feature estimation value corresponding to the audio bitstream.

[0174] For example, for an audio bitstream, the audio bitstream is first entropy decoded using the bitrate corresponding to the target code table to obtain the quantization value corresponding to the audio bitstream. Then, the audio feature estimation value F′(n) corresponding to the audio bitstream is obtained by looking up the target code table.

[0175] In step 202, the audio feature estimates are reconstructed using the backbone network to obtain the first reconstructed signal corresponding to the audio bitstream.

[0176] In some embodiments, the backbone network includes multiple cascaded first neural network layers; performing a first reconstruction process on the audio feature estimate through the backbone network to obtain a first reconstructed signal corresponding to the audio code stream, including: through the nth first neural network layer of the backbone network, performing feature reconstruction on the input of the nth first neural network layer to obtain a feature reconstruction result of the nth first neural network layer, where n is a positive integer increasing in sequence, 0 < n ≤ N, and N is the number of first neural network layers in the backbone network, and the input of the first first neural network layer is the audio feature estimate; determining the feature reconstruction result of the Nth first neural network layer as the first reconstructed signal.

[0177] In some embodiments, through the nth first neural network layer of the backbone network, performing feature reconstruction on the input of the nth first neural network layer to obtain a feature reconstruction result of the nth first neural network layer, including: performing one of the following processes through the nth first neural network layer of the backbone network: performing convolutional processing on the input of the nth first neural network layer to obtain a feature reconstruction result of the nth first neural network layer; performing decoding processing on the input of the nth first neural network layer to obtain a feature reconstruction result of the nth first neural network layer; performing post-processing on the input of the nth first neural network layer to obtain a feature reconstruction result of the nth first neural network layer.

[0178] For example, refer to Figure 11C the schematic diagram of the first reconstruction network shown (i.e., the backbone network at the decoding end, and the backbone network includes multiple cascaded first neural network layers), and the multiple cascaded first neural network layers correspond to Figure 11C the causal convolution, decoding block, and post-processing shown in. When n = 1, performing convolutional processing on the audio feature estimate to obtain a feature reconstruction result of the first first neural network layer, that is, by calling the causal convolution layer, the input tensor (i.e., the audio feature estimate) can be expanded into a causal convolution feature of 256×1. When 1 < n < N - 1, performing decoding processing on the input of the nth first neural network layer (the feature reconstruction result of the previous first neural network layer) to obtain a feature reconstruction result of the nth first neural network layer. When n = N - 1, performing post-processing on the input of the (N - 1)th first neural network layer to obtain a feature reconstruction result of the (N - 1)th first neural network layer ( Figure 11C the 16×320 tensor obtained after post-processing in). When n = N, performing convolutional processing on the input of the Nth first neural network layer (the feature reconstruction result of the (N - 1)th first neural network layer) to obtain the reconstructed audio signal of the Nth first neural network layer ( Figure 11C the 16×320 tensor x′(n) obtained after causal convolution in).

[0179] In some embodiments, decoding the input of the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer includes: performing deconvolution processing on the input of the nth first neural network layer through the deconvolution layer included in the nth first neural network layer to obtain deconvolution features; performing upsampling processing on the deconvolution features through the upsampling layer included in the nth first neural network layer to obtain upsampled features; and performing feature residual processing on the upsampled features through the residual block included in the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer.

[0180] Continuing from the previous example, the residual block includes multiple cascaded residual units. The upsampled features are processed using the residual block included in the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer. This can be achieved as follows: The upsampled features are processed using the first residual unit of the multiple cascaded residual units; the residual result output from the first residual unit is passed to subsequent cascaded residual units for further residual processing and output; the residual result output from the last residual unit is used as the feature reconstruction result of the nth first neural network layer.

[0181] like Figure 11C As shown, the first reconstruction network comprises four cascaded decoding blocks with different upsampling factors (Up_factor). Each decoding block contains a residual block (including at least one residual unit), a convolutional layer, and an upsampling layer. Each residual block includes five dilated convolution-based residual units (the input and output feature dimensions of the residual units remain unchanged). The convolutional layer halves the number of input channels, and the activation function can be PReLU to ensure sufficient data volume and avoid data loss. The upsampling layer is an upsampling operation with an Up_factor (e.g., using a Repeat operation to perform upsampling through repeated padding) to restore the data. Here, the Up_factors of the four decoding blocks are set to 5, 4, 4, and 2, respectively. Therefore, the number of output channels of the four decoding blocks are set to 128, 64, 32, and 16, respectively. After processing through four decoding blocks, the input 256×1 tensor is converted into tensors of 128×5, 64×20, 32×80, and 16×160, respectively. The embodiments of this application are not limited to the number of decoding blocks; it can be any positive integer such as 2, 3, 4, or 5.

[0182] Continuing with the above example, the residual processing of the input to the nth first neural network layer can be achieved through the first residual unit of multiple cascaded residual units in the following way: The first residual unit of multiple cascaded residual units performs the following processing: performing convolutional processing on the input to the nth first neural network layer to obtain the convolutional result of the first residual unit; adding the convolutional result of the first residual unit to the residual feature to obtain the residual result output by the first residual unit.

[0183] In a deep network, during backpropagation of gradients, they may gradually become smaller, making it difficult to update the parameters near the input layer and causing the network training to stagnate. By introducing a skip connection (corresponding to adding the convolutional result of the first residual unit to the residual feature in the above text), the residual unit allows the gradient to directly propagate through the residual path, avoiding the problem of gradients gradually vanishing during multi-layer propagation, making the deep network easier to optimize and its performance not degrade as the number of layers increases.

[0184] Continuing with the above example, the subsequent residual processing and output of the residual result can be achieved through the subsequent cascaded residual units in the following way: The jth residual unit of multiple cascaded residual units performs the following processing: performing convolutional processing on the residual result output by the (j - 1)th residual unit to obtain the convolutional result of the jth residual unit; adding the convolutional result of the jth residual unit to the residual result output by the (j - 1)th residual unit to obtain the residual result output by the jth residual unit; outputting the residual result output by the jth residual unit to the (j + 1)th residual unit; where j is a positive integer that increases sequentially, 1 < j < J, and J is the number of residual units.

[0185] Continuing the example above, each residual unit includes a dilated convolution operator. The input of the nth first neural network layer is convolved through the first residual unit of a cascaded network, which can be achieved as follows: the input of the nth first neural network layer is dilated using the dilated convolution operator included in the first residual unit. That is, the j-th residual unit performs the following processing: the residual result output by the (j-1)-th residual unit is dilated using the dilated convolution operator included in the j-th residual unit, resulting in the dilated convolution result of the j-th residual unit, where j is a sequentially increasing positive integer, 1 < j ≤ J, and J is the number of residual units. It should be noted that each residual unit contains a dilated convolution operator with a specified dilation rate. Using a dilated convolution operator with an asymptotic dilation rate is equivalent to using different receptive fields to extract features of the input at different resolutions, allowing for a more comprehensive analysis of the data. After each residual unit is convolved by the dilated convolution operator with the dilation rate, it is added to the shallow features (i.e., the input of each residual unit) from the jump connection, thereby directly utilizing the shallow feature information, so that the network can make full use of the shallow feature information during the learning process.

[0186] In some embodiments, each residual unit further includes at least one causal convolution operator. After performing dilated convolution processing on the input of the nth first neural network layer using the dilated convolution operator included in the first residual unit, the following processing can be performed: causal convolution processing is applied to the obtained dilated convolution result using the at least one causal convolution operator included in the first residual unit, and the obtained causal convolution result is used as the convolution result of the first residual unit. That is, after performing dilated convolution processing on the residual result output of the (j-1)th residual unit using the dilated convolution operator included in the jth residual unit to obtain the dilated convolution result of the jth residual unit, causal convolution processing is applied to the dilated convolution result of the jth residual unit using the at least one causal convolution operator included in the jth residual unit, and the causal convolution result of the jth residual unit is used as the convolution result of the jth residual unit. It should be noted that each residual unit further includes at least one causal convolution operator, through which local information of the features input to the causal convolution operator is further extracted.

[0187] In some embodiments, grouped convolution can be applied to the convolution operators (including dilated convolution operators and causal convolution operators) of the residual units. Grouped convolution divides the input channels into multiple groups for convolution operations, with the input and output channels only associated within each group. It should be noted that when the input channels are divided into multiple groups, the corresponding output channels are also divided into multiple groups; that is, the number of input channel groups is the same as the number of output channel groups, thus ensuring that after convolution within a group, the input and output channels are associated only within that group. Here, it is assumed that the input channels and output channels of a feature input to a certain convolution operator are 4. If the number of groups is 1, each input channel is associated with all 4 output channels. If the number of groups is 2, the 4 input channels are first divided into two groups, 0-1 and 2-3. Within each group, the input channel is associated with the output channel within that group. For example, input channel 0-1 in the first group is associated with output channel 0-1, and input channel 2-3 in the second group is associated with output channel 2-3.

[0188] For example, when applying grouped convolution to the dilated convolution operator included in the residual unit, dilated convolution processing of the input of the nth first neural network layer can be achieved as follows: the input channels of the input of the nth first neural network layer are grouped to obtain multiple groups, wherein each group includes the first element corresponding to at least two channels in the input of the nth first neural network layer; dilated convolution processing is performed on the first element within each group. Similarly, when applying grouped convolution to the causal convolution operator included in the residual unit, causal convolution processing of the resulting dilated convolution result can be achieved as follows: the input channels of the dilated convolution result are grouped to obtain multiple groups, wherein each group includes the second element corresponding to at least two channels in the dilated convolution result; causal convolution processing is performed on the second element within each group.

[0189] In step 203, the audio feature estimates are reconstructed using a secondary network to obtain the second reconstructed signal corresponding to the audio bitstream.

[0190] The backbone network has a greater number of convolutional layers than the secondary network, and the dimensions of the first reconstructed signal are the same as those of the second reconstructed signal.

[0191] In some embodiments, the secondary network includes a plurality of cascaded second neural network layers; the second reconstruction processing is performed on the audio feature estimation value through the secondary network to obtain a second reconstruction signal corresponding to the audio code stream, which can be achieved in the following manner: obtaining the feature reconstruction result of the g-th first neural network layer in the backbone network, where the feature reconstruction result of the g-th first neural network layer is obtained based on the audio feature estimation value, 0 < g < N, and g is a positive integer; through the k-th second neural network layer of the secondary network, performing feature reconstruction on the input of the k-th second neural network layer to obtain the feature reconstruction result of the k-th second neural network layer, where k is a positive integer that increases sequentially, 0 < k ≤ K, and K is the number of second neural network layers in the secondary network, and the input of the first second neural network layer is the feature reconstruction result of the g-th first neural network layer; determining the feature reconstruction result of the K-th second neural network layer as the second reconstruction signal.

[0192] Continuing with the above example, through the k-th second neural network layer of the secondary network, performing feature reconstruction on the input of the k-th second neural network layer to obtain the feature reconstruction result of the k-th second neural network layer can be achieved in the following manner: performing one of the following processes through the k-th second neural network layer of the secondary network: performing deconvolution processing on the input of the k-th second neural network layer to obtain the feature reconstruction result of the k-th second neural network layer; performing decoding processing on the input of the k-th second neural network layer to obtain the feature reconstruction result of the k-th second neural network layer.

[0193] For example, see Figure 13D , Figure 13D FIG. is a schematic diagram of an optional structure of the first reconstruction network (i.e., the backbone network, which includes a plurality of cascaded second neural network layers) and the second long-term connection network (i.e., the secondary network) provided by an embodiment of the present application. The second neural network layer corresponds to a convolution-copy layer. When K = 4, through the first second neural network, performing convolution-copy processing on the feature extraction result of the first first neural network layer to obtain the feature reconstruction result of the first second neural network layer, that is, through the first second neural network, performing a convolution operation first on the feature reconstruction result of the first first neural network layer ( Figure 13D the 256×1 tensor obtained after causal convolution in ) and an upsampling process with a factor of 5 to generate a 128×5 tensor. Through the second second neural network, performing convolution-copy processing on the feature reconstruction result of the first second neural network layer to obtain a 64×20 tensor. Through the third second neural network, performing convolution-copy processing on the feature reconstruction result of the second second neural network layer to obtain a 32×80 tensor. Through the fourth second neural network, performing convolution-copy processing on the feature reconstruction result of the third second neural network layer to obtain a 16×160 tensor, that is, the second reconstruction signal.

[0194] In some embodiments, decoding the input of the k-th second neural network layer to obtain the feature reconstruction result of the k-th second neural network layer can be achieved by: performing at least one deconvolution process on the input of the k-th second neural network layer through at least one deconvolution layer included in the k-th second neural network layer to obtain deconvolution features; and performing upsampling processing on the convolution features through an upsampling layer included in the k-th second neural network layer to obtain the feature reconstruction result of the k-th second neural network layer.

[0195] like Figure 11B The diagram shows the network structure of the second long-term connection network (i.e., the secondary network). The second neural network layer is implemented through a convolution-repeat layer. The second long-term connection network includes four cascaded convolution-repeat layers with different upsampling factors. Each convolution-repeat layer contains one convolutional layer and one upsampling layer. One convolutional layer is used to halve the number of input channels; one upsampling layer contains a specific up factor to perform upsampling. The number of convolution-repeat layers is not limited in this embodiment and can be any positive integer, such as 2, 3, 4, or 5.

[0196] In some embodiments, the secondary network includes multiple cascaded third neural network layers; the second reconstruction processing of the audio feature estimates through the secondary network to obtain the second reconstructed signal corresponding to the audio bitstream can be achieved in the following way: through the q-th third neural network layer of the secondary network, feature reconstruction is performed on the input of the q-th third neural network layer to obtain the feature reconstruction result of the q-th third neural network layer, where q is a positive integer that increases sequentially, 0 < q ≤ Q, Q is the number of the third neural network layers in the secondary network, and the input of the first third neural network layer is the audio feature estimate; the feature reconstruction result of the Q-th third neural network layer is determined as the second reconstructed signal.

[0197] For example, such as Figure 12B As shown, when the secondary network (i.e., the second long-time connection network, which includes multiple cascaded third neural network layers) is called based on the audio feature estimation value, the audio signal is convolved through the first third neural network layer in the secondary network to obtain the feature reconstruction result of the first third neural network layer. Figure 12B The 128×1 tensor obtained after causal convolution is passed through the second and third neural network layers. The feature reconstruction results of the first and third neural network layers are then processed (convolutional replication) to obtain the feature extraction results of the second and third neural network layers. Figure 12BThe 64×4 tensor obtained after convolutional copying is used in this way. Finally, the feature reconstruction result of the Q-1th third neural network layer is processed through the Qth third neural network layer to obtain the feature reconstruction result of the Qth third neural network layer. Figure 12B The 16×320 tensor obtained after post-processing.

[0198] It should be noted that step 203 is similar to step 22.

[0199] In step 204, the first reconstructed signal and the second reconstructed signal are fused to obtain the reconstructed audio signal corresponding to the audio bitstream.

[0200] In some embodiments, the backbone network includes a first subnetwork and a second subnetwork. The first subnetwork is used to perform a first reconstruction process. The process of fusing the first reconstructed signal and the second reconstructed signal to obtain a reconstructed audio signal corresponding to the audio bitstream includes: adding the first reconstructed signal and the second reconstructed signal to obtain a third reconstructed signal; and performing a third reconstruction process on the third reconstructed signal through the second subnetwork to obtain a reconstructed audio signal.

[0201] For example, see Figure 13D The first reconstruction signal ( Figure 13D The 16×160 tensor after decoding the middle decoding block) and the second reconstructed signal ( Figure 13D The convolutional tensor (and the copied 16×160 tensor) are added together to form the third reconstructed signal. Figure 13D The 16×160 tensor after feature fusion is passed through the second sub-network (corresponding to...) Figure 13D The third reconstruction signal is processed by post-processing and causal convolution to obtain the reconstructed audio signal. Figure 13D (a 1×320 tensor in the text).

[0202] The specific implementation of the third reconstruction process can be found in the description of the first neural network layer above, and will not be repeated here.

[0203] In other embodiments, the first reconstructed signal and the second reconstructed signal are added together to obtain the third reconstructed signal; the third reconstructed signal is then nonlinearly mapped using an activation function to obtain the reconstructed audio signal.

[0204] For example, see Figure 13C The first reconstruction signal ( Figure 13C The 1×320 tensor output by the causal convolution and the second reconstructed signal ( Figure 13C The 1×320 tensor output from the second-time connection network is added to obtain the third reconstructed signal. Then, the third reconstructed signal is nonlinearly mapped through a pre-set activation function (e.g., tanh) to obtain the reconstructed audio signal. Figure 13C (A 1×320 tensor obtained after feature fusion).

[0205] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0206] The embodiments of this application can be applied to various audio scenarios, such as voice calls, instant messaging, remote audio and video calls, and even low-bitrate voice scenarios such as emergency support (i.e., walkie-talkies in emergency scenarios) and satellite calls. The following explanation uses voice calls as an example:

[0207] In related technologies, the principle of speech coding is roughly as follows: speech coding can directly encode speech waveform samples one by one; or, based on the principle of human vocalization, relevant low-dimensional features are extracted, the encoding end encodes the features, and the decoding end reconstructs the speech signal based on these parameters.

[0208] The above encoding principles all originate from speech signal modeling, i.e., signal processing-based compression methods. To improve encoding quality while maintaining speech encoding efficiency compared to signal processing-based compression methods, this application provides a speech encoding / decoding method (i.e., an audio encoding method and an audio decoding method). For an input speech signal, the encoder obtains a first feature vector based on a first analysis network (i.e., the backbone network of the encoder); in parallel, it obtains a first long-time connection feature vector using a first long-time connection network (i.e., the secondary network of the encoder). The first feature vector and the first long-time connection feature vector have the same dimension and format. Finally, the first feature vector and the first long-time connection feature vector are added together, and an activation function (such as tanh) is used to obtain the final feature vector. The final feature vector is then compressed and encoded. Specifically, code tables with different quantization precipitates can be used for quantization and encoding of the feature vector to achieve multi-rate encoding / decoding effects. The neural network employs a "block-based" operation, which reduces algorithm complexity and improves encoding performance. The decoding end decodes the received bitstream to obtain the estimated feature vector values, and then calls the inverse process of the corresponding encoding end to complete the generation of the speech signal.

[0209] The embodiments of this application can be applied to, for example... Figure 6The voice communication link in the satellite communication scenario shown takes a Voice over Internet Protocol (VoIP) satellite communication system as an example. The voice encoding and decoding technologies involved in this application are deployed in the encoding and decoding parts to solve the basic function of voice compression. Generally, both the encoder and decoder are deployed on the client or terminal (such as a mobile phone). The encoder is deployed on the uplink client 601, and the decoder is deployed on the downlink client 602. Voice is acquired through the uplink client and pre-processed, enhanced, encoded, etc. The encoded bitstream is transmitted to the downlink client 602 via the network. The downlink client 602 performs decoding, enhancement, etc., to play back the decoded voice.

[0210] To ensure forward compatibility (i.e., compatibility between the new encoder and existing encoders), transcoders are deployed at relevant nodes in the operator's network to address the interoperability issue between the new and existing encoders. For example, a mobile phone uses the encoder provided in this embodiment to send a very low bitrate stream to a satellite; the satellite may simply forward the stream; after receiving the stream, the ground receiving station converts it into a G.711 stream at the media gateway; considering that G.711 is a widely supported standard, it can be directly forwarded to the receiving end through the terrestrial operator network. On the other hand, the G.711 stream received by the receiving end is converted into a stream generated by the encoder provided in this embodiment (i.e., a very low bitrate stream) at the media gateway; the converted stream is then transmitted to the satellite through the operator; the satellite may simply forward the stream and transmit it to the mobile phone; the mobile phone, equipped with the decoder provided in this embodiment, can play back the audio. Therefore, through the above methods, the voice encoding and decoding technology provided in this embodiment can be deployed on mobile satellite calling and other services provided by operators.

[0211] Before introducing the speech encoding and decoding method provided in the embodiments of this application, we will first introduce dilated convolutional networks.

[0212] See Figure 8A and Figure 8B , Figure 8A This is a schematic diagram of a common convolutional (e.g., causal convolutional) network provided in an embodiment of this application. Figure 8B This is a schematic diagram of the dilated convolutional network provided in an embodiment of this application. Compared to ordinary convolutional networks, dilated convolution can increase the receptive field while maintaining the same feature map size, and can also avoid errors caused by upsampling and downsampling. Although Figure 8A and Figure 8B The kernel sizes shown are all 3×3; however, Figure 8A The receptive field of the ordinary convolution shown is only 3 (801), while Figure 8BThe receptive field of the dilated convolution shown reaches 802, which is 5. That is, for a convolution kernel of size 3×3, Figure 8A The receptive field of the ordinary convolution shown is 3, and the dilation rate (the number of intervals between points in the convolution kernel) is 1; while Figure 8B The receptive field of the dilated convolution shown is 5, and the dilation rate is 2.

[0213] Convolution kernels can also be used in similar ways. Figure 8A or Figure 8B The movement occurs on a plane, which involves the concept of stride rate. For example, if the convolution kernel moves 1 unit at a time, the corresponding stride rate is 1.

[0214] In addition, there is the concept of convolution channel count, which refers to the number of parameters corresponding to the convolution kernels used in convolution analysis. Theoretically, the more channels there are, the more comprehensive the signal analysis and the higher the accuracy; however, the more channels there are, the higher the complexity. For example, a 1×320 tensor can be used with a 24-channel convolution operation, and the output will be a 24×320 tensor.

[0215] It should be noted that the size of the dilated convolution kernel (for example, for speech signals, the kernel size can be set to 1×3), dilation rate, shift rate, and number of channels can be defined according to actual application needs. This application does not impose specific limitations on these aspects.

[0216] The following is combined Figure 7A This application describes the speech encoding and decoding method provided in its embodiments:

[0217] The following processing is performed on the encoding end:

[0218] For the input signal x(n) of the nth frame (i.e., the audio signal mentioned above), the first analysis network is invoked to obtain the first feature vector F1(n) (i.e., the first audio feature mentioned above). Specifically, the dimension of the first feature vector is smaller than the dimension of the input signal (to reduce the amount of data). A specific implementation is that for each frame x(n), a dilated convolutional neural network (Dilated CNN) (encoder part) is invoked to generate a lower-dimensional feature vector F1(n). This application does not limit the neural network structure of the first analysis network, such as an autoencoder, a fully connected (FC) network, a long short-term memory (LSTM) network, a convolutional neural network (CNN) + LSTM, etc. Internally, the neural network employs a "block-based" operation to reduce algorithm complexity.

[0219] For x(n), the first long-term connection network is invoked in parallel to obtain the first long-term connection feature vector F. LoRa (n) (i.e., the second audio feature mentioned above), in particular, the feature dimension of the first long-time connection feature vector is consistent with the feature dimension of the first feature vector.

[0220] Using the feature merging module, the first feature vector and the first long-time connection feature vector are fused to obtain the first fused feature vector (i.e., the fused audio feature mentioned above). Specifically, F1(n) and F... LoRa After adding the corresponding dimensions (n), the final first fused feature vector F(n) is output through the activation function.

[0221] The first fused feature vector is quantized using either vector or scalar methods to obtain an index value. This index value is then directly encoded or entropy encoded to obtain the bitstream (i.e., the audio bitstream mentioned above), which is then transmitted to the decoder. Specifically, for F(n), code tables with different quantization precisions are used for quantization and encoding to achieve multi-rate encoding and decoding effects.

[0222] The following processing is performed on the decoding end:

[0223] The bitstream encoded by the decoding end obtains the estimated value F′(n) of the first fused feature vector (i.e., the audio feature estimate mentioned above).

[0224] The second long-time connection network (i.e., the secondary network at the encoding end) is invoked to decode the bitstream to obtain the second long-time connection feature vector L(n) (i.e., the second reconstructed signal mentioned above).

[0225] The first reconstruction network (i.e., the backbone network at the encoding end) is invoked, combined with the second long-time connection feature vector L(n), to generate the reconstructed signal estimate x′(n) (i.e., the reconstructed speech signal, also known as the reconstructed audio signal). The second long-time connection feature vector, as one of the inputs to the first reconstruction network, can be fused and further processed at intermediate nodes of the first reconstruction network to reconstruct the speech signal. Specifically, the structure of the first reconstruction network can refer to the structure of the first analysis network, similar to its "inverse" process. Furthermore, depending on the actual codeword length in the bitstream, code tables with different quantization precipitates can be used to generate F′(n) of different precisions to achieve multi-rate encoding and decoding effects.

[0226] The speech encoding and decoding method provided in the embodiments of this application will be described in detail below.

[0227] In some embodiments, a narrowband speech signal with a sampling rate of Fs = 8000Hz is taken as an example (it should be noted that the method provided in this application embodiment is also applicable to scenarios with other sampling rates, including but not limited to: 16000Hz, 32000Hz, and 48000Hz). Similarly, this application embodiment does not limit the frame length combination under other sampling rates. For example, the frame length is set to 40ms; for Fs = 8000Hz, this is equivalent to each frame still containing 320 sample points.

[0228] The following is for reference. Figure 7A The flowchart shown describes the encoding and decoding ends in detail.

[0229] The encoding process is as follows:

[0230] Step 11: Call the first analysis network based on x(n).

[0231] As mentioned earlier, the 320 sample points in the nth frame are denoted as x(n). The purpose of the first analysis network is to generate a lower-dimensional feature vector F1(n) based on the signal x(n) by calling a deep network. In this embodiment, x(n) has a dimension of 320, and F1(n) has a dimension of 12. From the perspective of data volume, it is clear that the first analysis network plays a role in "dimensionality reduction," which is essentially a form of data compression.

[0232] See Figure 9 The network structure diagram of the first analysis network is shown below. The process of data compression (i.e., feature extraction) by the first analysis network is explained in detail below:

[0233] First, calling a 16-channel causal convolution can generate a 16×320 tensor from the input tensor (i.e., the feature vector, corresponding to the 1×320 feature vector x(n) in the figure).

[0234] Then, the 16×320 tensor is preprocessed. For example, after performing a convolution operation on the 16×320 tensor, a pooling operation with a factor of 2 is performed, and the activation function can be PReLU, to generate a 16×160 tensor.

[0235] Next, four encoding blocks with different downsampling factors are concatenated. Each encoding block contains a residual block, a convolutional layer, and a pooling layer. Each residual block includes five dilated convolution-based residual units (the input and output feature dimensions of the residual units remain unchanged). The convolutional layer doubles the number of input channels, and the activation function can be PReLU to ensure sufficient data volume and avoid data loss. The pooling layer is a pooling operation with a down factor to perform downsampling and achieve data compression. Here, the down factors of the four encoding blocks are set to 2, 4, 4, and 5, respectively. Therefore, the number of output channels of the four encoding blocks are set to 32, 64, 128, and 256, respectively. After processing by the four encoding blocks, the input 16×160 tensor is transformed into tensors of 32×80, 64×20, 128×5, and 256×1, respectively. In this application embodiment, the number of coding blocks is not limited, and can be any positive integer such as 2, 3, 4, 5, etc. Furthermore, this application embodiment does not limit the number of residual units in a coding block, and can be any positive integer such as 2, 3, 4, 5, 6, etc. The number of residual units in multiple coding blocks can be the same or different; for example, one coding block contains 4 residual units, and another coding block contains 5 residual units.

[0236] This section further introduces the Residual Unit. A Residual Unit is a module in a deep neural network that introduces cross-layer connections, making the neural network easier to optimize during training and avoiding problems such as vanishing or exploding gradients. Its core idea is to perform residual learning on the input within the module, that is, to bypass a portion of the layers and directly pass the input information to the output through a direct path, allowing the network to better utilize shallow feature information during the learning process. Figure 10AThis diagram illustrates the residual block structure used in the encoding block of the first analysis network. This residual block comprises five dilated convolution-based residual units, each containing a dilated convolution block with a specified dilation rate (e.g., dilation rate = 3). In this embodiment, using five dilated convolution blocks with progressively increasing dilation rates is equivalent to using different receptive fields to extract features from the input at different resolutions, allowing for a more comprehensive analysis of the data. After residual processing through the five dilated convolution blocks with specified dilation rates, the output is added to the input from the jump connection to obtain the residual block's output, which is then output to the convolutional layer connected to the residual block.

[0237] Here, further... Figure 10A Describing it using any residual cell, such as Figure 10B As shown. For any residual unit, it contains at least one dilated convolution with a specified dilation rate (for expanding the receptive field), which can be activated using PReLU; in addition, one or more causal convolutions (for extracting local information) can be cascaded, which can also be activated using PReLU. The kernel size of the dilated convolution with the specified dilation rate can be 3, 5, 7, 9, etc., and the kernel size of the causal convolution can be 1, 3, etc. This application and this embodiment do not limit the kernel size of the dilated convolution with the specified dilation rate or the causal convolution. In addition, the causal convolution or dilated convolution in the embodiments of this application can also be implemented by other convolution units with similar or equivalent functions.

[0238] Furthermore, for residual units, a grouped convolution algorithm is introduced to reduce algorithm complexity. Grouped convolution divides the input channels into multiple groups for convolution operations, associating input and output channels only within each group. Here, we assume there are 16 input channels and 32 output channels. If the number of groups is 1, each input channel is associated with all 32 output channels. If the number of groups is 2, the 16 input channels are first divided into two groups: 0-7 and 8-15. Within each group, the input channel is associated with the output channel within that group. For example, input channels 0-7 in the first group are associated with output channels 0-15, and input channels 8-15 in the second group are associated with output channels 16-31. For instance, the 0th output channel is associated only with input channels 0-7 and not with input channels 8-15, and the 25th output channel is associated only with input channels 8-15 and not with input channels 0-7. This comparison shows that introducing grouped convolutions can avoid any correlation between input channels and all output channels, reducing the number of connections and thus lowering complexity. However, since a larger number of groups reduces the correlation between input and output channels, it also affects the encoding performance; therefore, a larger number of groups is not always better. In this embodiment, the dilated convolutions contained in the five residual blocks corresponding to the four coding blocks can use different group number configurations, as shown in Table 1.

[0239] Table 1. Configuration of the number of blocks used by residual units in different coding blocks

[0240] Encoded blocks Number of groups Encoded block (Down_factor=2) 2 Encoded block (Down_factor=4) 4 Encoded block (Down_factor=4) 4 Encoded block (Down_factor=5) 4

[0241] Finally, the 256×1 tensor is further processed by causal convolution and Tahn activation to output a 12-dimensional first feature vector. Specifically, according to the calculations of the first analysis network, each value in the 12-dimensional first feature vector is between [-1, 1]. Furthermore, this embodiment does not limit the use of two or more causal convolutions. For example, the 256×1 tensor can be processed by one causal convolution to output a 128×1 tensor; then, the 128×1 tensor can be processed by another causal convolution to output a 12-dimensional feature vector (i.e., F1(n)). This increases computational complexity only slightly while achieving better feature vector extraction.

[0242] Step 12: Call the first long-term connection network based on x(n).

[0243] The purpose of the first long-term connection network is to generate another lower-dimensional feature vector F based on the signal x(n) by calling a deep network. LoRa (n). Where F LoRa The dimensions of F1(n) are the same as those of F1(n).

[0244] The role of the first long-time connection network is to "transmit" the characteristics of the input signal through a second path via the first long-time feature vector, and then fuse it with the first feature vector output by the first analysis network via the feature fusion module. (Reference) Figure 10A and Figure 10B The residual units in the first analysis network are mostly "single-layer" connections, while the first long-term connection network is "cross-layer" or even cross-multi-layer connections. That is, compared to the first analysis network, the first long-term connection network spans at least one neural network layer in the first analysis network. From the perspective of information transmission, long-term connections are established between the input signal and the feature vector to obtain better feature extraction results.

[0245] In implementation, the topology of the first long-term connection network is similar to that of the first analysis network. However, to control computational complexity, the internal structure of the first long-term connection network is simplified, containing only simple operations such as causal convolution and pooling. Furthermore, the input and output dimensions of each neural network layer in the first long-term connection network are referenced from the corresponding neural network layer in the first analysis network, and are consistent with the input and output dimensions of the corresponding neural network layer in the first analysis network. Therefore, in terms of results, the properties of the first long-term connection feature vector output by the first long-term connection network are consistent with the properties of the first feature vector output by the first analysis network, and the feature dimensions of the first long-term connection feature vector output by the first long-term connection network are consistent with the feature dimensions of the first feature vector output by the first analysis network, making it suitable for subsequent feature fusion operations.

[0246] See Figure 11A , Figure 11A This is a first schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in the embodiments of this application. See below for further details. Figure 11A The diagram shows the network structure of the first long-term connectivity network (which includes multiple cascaded neural network layers), and the process of feature extraction using the first long-term connectivity network is explained in detail below:

[0247] 1) First, a 16-channel causal convolution (i.e., the first neural network layer) is called, which can generate a 16×320 tensor from the input tensor (i.e. the feature vector, corresponding to the 1×320 feature vector x(n) in the figure).

[0248] 2) Next, through the first neural network layer (i.e. Figure 11A The first convolutional-pooling layer (i.e., the second neural network layer) in the first long-term connected network shown preprocesses the 16×320 tensor. For example, after a convolution operation in the convolutional-pooling layer, a pooling operation with a down factor of 2 and an activation function of PReLU is performed to generate a 16×160 tensor.

[0249] 3) Similarly, the second to fifth convolutional-pooling layers (i.e., the third to sixth neural network layers) correspond to the four different downsampling factors of the encoding blocks in the first analysis network. For example, the second convolutional-pooling layer corresponds to the first encoding block in the first analysis network, and the third convolutional-pooling layer corresponds to the second encoding block in the first analysis network. The down_factors of the above four convolutional-pooling layers are set to 2, 4, 4, and 5, respectively, and the number of output channels are set to 32, 64, 128, and 256, respectively. Specifically, the second to fifth convolutional-pooling layers use operations similar to those of the first convolutional-pooling layer. Obviously, the calculation of the convolutional-pooling layers is greatly simplified compared to the encoding blocks in the first analysis network.

[0250] 4) Finally, the 256×1 tensor is passed through a causal convolution (i.e., the last neural network layer) and activated by Tahn, which outputs a 12-dimensional first long-term connection feature vector. Therefore, the first long-term connection feature vector has completely identical attributes and feature dimensions to the aforementioned first feature vector. In this embodiment, the number of causal convolutions used is not limited; two or more causal convolutions can be used. For example, the 256×1 tensor can be passed through a causal convolution to output a 128×1 tensor; then, the 128×1 tensor can be passed through another causal convolution to output a 12-dimensional feature vector (i.e., F). LoRa (n)). In this way, with only a small increase in computational complexity, feature vector extraction can be completed more effectively.

[0251] Step 13: Feature fusion.

[0252] Obtain the first feature vector F1(n) and the first long-term connection feature vector F LoRa After (n), a feature fusion operation is performed. For example, the first feature vector output from the first analysis network is added to the first long-term connection feature vector output from the first long-term connection network along the corresponding dimension, and then tanh activation is performed. In this way, the final feature vector F(n) is output, and each value in the final feature vector F(n) is between [-1, 1].

[0253] Step 14: Quantization encoding.

[0254] For the feature vector F(n), scalar quantization (quantizing each component individually) and entropy coding methods can be used. Furthermore, the embodiments of this application do not limit the combination of vector quantization (combining multiple adjacent components into a single vector for joint quantization) and entropy coding techniques. In addition, besides entropy coding, direct coding can also be used.

[0255] This section focuses on the implementation description of the feature vector F(n). As described above, the 320-point input x(n) of each frame, after passing through the first analysis network, the first long-term connection network, and feature fusion processing, yields a 12-dimensional feature vector. Therefore, for narrowband ultra-low bitrate scenarios, the required bitrate is 1kbps or even lower. This application provides a method based on scalar quantization and entropy coding. For example, code table-1 is [-1.0, -0.66, -0.33, 0, 0.25, 0.5, 0.75, 1.0], corresponding to 3 bits; code table-2 is [-1.0, 0, 0.5, 1.0], corresponding to 2 bits. Taking the 3-bit code table-1 as an example, the required bitrate for each frame is 36 bits, and with 25 frames per second, the bitrate is 25 * 36 = 0.9kbps. Similarly, taking the 2-bit code table-2 as an example, the bitrate is 0.6kbps. Therefore, for the selected actual bit rate, 2-bit or 3-bit encoding can be used directly to complete the quantization and direct encoding of the feature vector.

[0256] In some embodiments, for each frame of the bitstream, 1 bit is first used to represent the bitrate used for encoding (corresponding to the codebook used for encoding), and then the corresponding index value obtained by scalar quantization and direct encoding of the 12-dimensional feature vector is written into the bitstream.

[0257] The decoding process is as follows:

[0258] Step 21, Decode.

[0259] Decoding is the reverse process of encoding. The received bitstream is parsed. After parsing the first bit, the encoding bitrate is obtained, and the corresponding code table (a total of 2 code tables) is selected. Then, based on the bitstream, 12 index values ​​are parsed out, and by looking up the quantization table, the estimated value of the feature vector F′(n) can be obtained.

[0260] The following is a detailed description using a configuration with an encoding rate of 0.9kbps as an example:

[0261] 1) Parse the frame header. The encoding mode bit is 0, which indicates that the bandwidth uses code table -1.

[0262] 2) The code stream is parsed in 12 steps using direct decoding technology. The quantization value F′(n) of the 12-dimensional feature vector can be obtained through code table-1.

[0263] Step 22: Invoke the second long-term connection network.

[0264] The purpose of the second long-term connection network is to take the quantized value F′(n) of the feature vector, call the deep neural network to generate the second long-term feature vector L(n), which is used to process together with the first reconstruction network to reconstruct the speech signal.

[0265] See Figure 11B The network structure diagram of the second long-time connection network is shown below. The network structure of the second long-time connection network is very similar to that of the first long-time connection network. The specific process is as follows:

[0266] 1) Calling a causal convolution can expand the input tensor from 12×1 to 256×1.

[0267] 2) Next, four convolutional-repeat layers with different upsampling factors (Up_factor) are cascaded. Each convolutional-repeat layer contains one convolutional layer and one upsampling layer. One convolutional layer halves the number of input channels; the upsampling layer contains a specific Up_factor to perform upsampling. The Up_factors of the four convolutional-repeat layers are set to 5, 4, 4, and 2, respectively. Therefore, the number of output channels of the four convolutional-repeat layers are set to 128, 64, 32, and 16, respectively. After processing by the four convolutional-repeat layers, the 256×1 tensor is transformed into tensors of 128×5, 64×20, 32×80, and 16×160, respectively. Note that the number of convolutional-repeat layers in this embodiment is not limited and can be any positive integer such as 2, 3, 4, or 5.

[0268] Here, for upsampling layers containing a specific Up_factor, a repeat operation can be used to complete the upsampling operation by repeatedly padding. This saves complexity.

[0269] 3) Post-process the 16×160 tensor output by the cascaded convolution-Repeat layer. For example, perform a factor of 2 Repeat operation on the 16×160 tensor output by the cascaded convolution-Repeat layer to complete upsampling, then perform convolution operation and use the PReLU activation function to generate a 16×320 tensor.

[0270] Other details of the second long-time connection network are the reverse process of the first long-time connection network, and will not be repeated here.

[0271] Step 23: Invoke the first reconstruction network.

[0272] The purpose of the first reconstruction network is to combine the quantized value F′(n) of the feature vector with the output L(n) from the second long-term connection network, call the deep neural network, and generate the estimated value x′(n) of the signal.

[0273] See Figure 11CThe diagram shows the network structure of the first reconstruction network. The network structure of the first reconstruction network is highly similar to that of the first analysis network, for example, it incorporates causal convolution. The post-processing structure in the first reconstruction network is similar to the pre-processing structure in the first analysis network. The decoding block structure is symmetrical to the encoding block on the encoding side. The encoding block on the encoding side first undergoes dilated convolution followed by pooling to achieve downsampling, while the decoding block on the decoding side first undergoes pooling to achieve upsampling, followed by dilated convolution. The specific process is as follows:

[0274] 1) Calling a causal convolution can expand the input tensor from 12×1 to 256×1.

[0275] 2) Next, four decoding blocks with different upsampling factors (Up_factor) are concatenated. Each decoding block contains a convolutional layer, an upsampling layer, and a residual block. The convolutional layer halves the number of input channels; the upsampling layer contains a specific Up_factor to perform upsampling; and the residual block contains five dilated convolution-based residual units. The Up_factors of the four decoding blocks are set to 5, 4, 4, and 2, respectively. Therefore, the number of output channels for the four decoding blocks are set to 128, 64, 32, and 16, respectively. After processing by the four decoding blocks, the 256×1 tensor is converted into tensors of 128×5, 64×20, 32×80, and 16×160, respectively. Note that the number of decoding blocks in this embodiment is not limited and can be any positive integer such as 2, 3, 4, or 5.

[0276] Here, for upsampling modules that contain a specific Up_factor, a repeat operation can be used to complete the upsampling operation by repeatedly filling in the data. This saves complexity.

[0277] Here, the configuration of the five dilated convolution-based residual units at the decoding end is similar to that at the encoding end, including but not limited to the internal structure of the residual units, convolution kernel size, and dilation rate. The number of groups used in the dilated convolution in the decoding block is configured as shown in Table 2. Here, a higher group size of 2 is used in the decoding block to associate more input and output channels, thereby improving the quality of speech reconstruction.

[0278] Table 2. Configuration of the number of groups used by residual units in different decoding blocks

[0279] Decoding block Number of groups Decoding block (Up_factor=5) 4 Decoding block (Up_factor=4) 4 Decoding block (Up_factor=4) 2 Decoding block (Up_factor=2) 2

[0280] 3) Post-process the 16×160 tensor output by the concatenated decoder block. For example, perform a factor-2 Repeat operation on the 16×160 tensor output by the concatenated decoder block to complete upsampling, then perform convolution operation and use the PReLU activation function to generate a 16×320 tensor.

[0281] 4) The 16×320 tensor output from the second long-term connected network is added dimension-by-dimensionally to the 16×320 tensor obtained after post-processing to obtain a new 16×320 tensor through feature fusion. A causal convolution is then applied to convert the input 16×320 tensor into a 1×320 tensor, thus completing signal generation. However, this embodiment is not limited to the feature fusion method. For example, the 16×320 tensor output from the second long-term connected network can also be added dimension-by-dimensionally to the 16×320 tensor obtained after post-processing, and then an activation function (such as PReLU) can be applied to obtain a new 16×320 tensor.

[0282] Other details of the first reconstruction network are the reverse process of the first analysis network mentioned above, and will not be repeated here.

[0283] The following is combined Figure 7B This application describes the speech encoding and decoding method provided in its embodiments:

[0284] For the encoding end, the processing performed by the encoding end is the same as... Figure 7A The processing at the encoding end shown is similar, that is Figure 7B The processing steps 11-14 shown are similar to... Figure 7A The processing steps 11-14 shown are similar.

[0285] For the decoding end, most of the processing performed by the decoding end is similar to... Figure 7A The processing at the decoding end shown is similar, that is Figure 7B The processing steps 21-22 shown are similar to... Figure 7A The processing steps 21-22 shown are similar.

[0286] in, Figure 7B The processing procedure of the first reconstructed network in step 24 shown is similar to... Figure 7A The process of reconstructing the first network in step 23 shown is similar, except that... Figure 7B In step 24 shown, the first reconstruction network does not need to fuse the output of the second long-time connection network, i.e., through Figure 13C The first reconstruction network shown transforms F′(n) into its final output s(n). The specific feature fusion process is achieved through... Figure 7BThe step 25 shown is implemented by fusing the final output s(n) of the first reconstruction network with the output L(n) of step 22 to generate the reconstructed signal estimate x′(n) (i.e., the reconstructed speech signal, also known as the reconstructed audio signal).

[0287] In other embodiments, see Figure 12A , Figure 12A This is a second schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in the embodiments of this application. See below. Figure 12A The network structure shown in the figure will be explained in detail.

[0288] In the preceding embodiment (corresponding to step 12 above), the principle and implementation of the first long-time connection network were introduced; the characteristics of the so-called long-time connection network were explained by comparing it with a general residual unit. Here, another feasible implementation method is given.

[0289] like Figure 12A As shown, in this embodiment, the dimensions of the input and output tensors of the first long-term connection network are completely identical to those of the first analysis network. However, Figure 12A The internal structure of the first long-term connection network in the system has been further simplified (the simplification method can be found in [reference]). Figure 12A and Figure 11A (Differences between them), for example, by modifying the number of convolutional-pooling layers in the first long-term connected network and matching the corresponding Down_factor, ensuring that a 1×320 tensor is mapped to a 12×1 tensor. Figure 12A The structure of the first long-term connection network is more streamlined and has lower computational complexity than the structure of the first analysis network. For example, the number of channels is at most 128 instead of 256.

[0290] Similarly, see Figure 12B , Figure 12B This is a second schematic diagram of an optional structure of the first reconstruction network and the second long-time connection network provided in the embodiments of this application. The second long-time connection network in the decoding end has also been further simplified, mapping the input 12×1 tensor to a 16×320 tensor. In this process, Figure 12B The maximum number of channels in the second long-term connected network shown is also 128, not 256. Then, through feature fusion, the output of the second long-term connected network is fused with the intermediate nodes of the first reconstruction network. For example, after adding the post-processed output of the first reconstruction network to the corresponding dimension of the second long-term connected feature vector, PReLU activation is performed to generate a new 16×320 tensor. Finally, a causal convolution is called to convert the newly generated 16×320 tensor into a 1×320 tensor to complete the signal generation.

[0291] Therefore, the embodiments of this application do not mandate that the configuration parameters of various first long-term connection networks (such as the number of neural network layers, the structure of neural network layers, the Down_factor parameter, etc.) be consistent with the first analysis network, nor do they mandate that the configuration parameters of the second long-term connection network be consistent with the first reconstruction network.

[0292] For details on the specific implementation of the decoding end, please refer to the descriptions of steps 21 to 23 above, which will not be repeated here.

[0293] In other embodiments, the data structures (such as tensor dimensions) for the inputs and outputs of the first and second long-term connection networks can be more flexible. See also Figure 13A as well as Figure 13B , Figure 13A This is a third schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in the embodiments of this application. Figure 13B This is a fourth schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in the embodiments of this application. See below. Figure 13A as well as Figure 13B The network structure shown in the figure will be explained in detail.

[0294] like Figure 13A As shown, the input to the first long-term connection network can be the output of the first causal convolution in the first analysis network, corresponding to a dimension of 16×320. The output of the first long-term connection network is a 12×1 dimensional tensor. Therefore, the output of the first long-term connection network is fused with the final output of the first analysis network. In this configuration, the first long-term connection network can correspond one-to-one with the structures in the first analysis network except for the first causal convolution. Apart from this, the other implementation methods and details are consistent with step 12 above, and will not be repeated here.

[0295] In principle, the first causal convolution of the first analysis network maps the original input to a 16-channel representation, which is equivalent to describing the original signal using multiple neural network channels. Therefore, it is feasible to obtain another form of the first long-term feature vector by using the 16×320 representation as a starting point for the first long-term connection network analysis. This application does not limit other methods of obtaining the first long-term connection network input; for example, it can be based on... Figure 9 The preprocessed 16×160 tensor of the first analysis network is used as the input of the first long-term connection network; correspondingly, the first long-term connection network reduces one convolution-pooling operation. The principle is similar and will not be described in detail here.

[0296] like Figure 13BAs shown, the input of the first long-term connection network is completely identical to that of the first analysis network, i.e., the corresponding dimension is 1×320. However, the output dimension of the first long-term connection network is 128×5. This is achieved through feature fusion with the output of the third encoding block in the first analysis network, followed by further operations in the first analysis network to obtain a 12×1 dimension feature vector. Therefore, it can be shown that the first long-term connection network can perform feature fusion as long as the dimensions of its input and output tensors match those of the neural network layers in the first analysis network. Furthermore, referring to the above-described encoding-side embodiment, this application does not limit the internal structure of the first long-term connection network. Generally, because the first long-term connection network adopts a more streamlined network structure, its computational complexity is less than that of the first analysis network.

[0297] Similarly, see Figure 13C , Figure 13C This is a third schematic diagram of an optional structure of the first reconstruction network and the second long-time connection network provided in the embodiments of this application. The second long-time connection network in the decoding end has also been further simplified, mapping the input 12×1 tensor to a 1×320 tensor. In this process, Figure 13C The second long-term connected network shown takes the decoded 12×1 dimensional feature vector as input and outputs a 1×320 dimensional tensor. Therefore, Figure 13C The second long-term connection network shown is used to generate the final reconstructed speech signal by feature fusion with the final output of the first reconstruction network.

[0298] In other embodiments, see Figure 13D , Figure 13D This is a fourth schematic diagram of an optional structure of the first reconstruction network and the second long-term connection network provided in the embodiments of this application. See below. Figure 13D The network structure shown in the figure will be explained in detail.

[0299] like Figure 13D As shown, the input to the second long-term connection network can be the output of the first causal convolution in the first reconstruction network, which corresponds to a dimension of 256×1. The connection method between the second long-term connection network and the first reconstruction network has been changed (the change can be found in [reference]). Figure 11B and Figure 13D(Differences between them). For example, after the second long-term connected network performs three convolutional-repeat layers, it transforms the 256×1 tensor into a 64×20 tensor. Then, it reuses the intermediate nodes of the first reconstruction network (not the final output), namely the fourth decoding block (Up_factor=2) and the post-processing layer in the first reconstruction network, to generate a 16×160 first tensor. This 16×160 first tensor is then input into the causal convolutional layer of the second long-term connected network to obtain the final output of the second long-term connected network (a 16×160 second tensor). After the first reconstruction network transforms the quantized value F′(n) of the feature vector (i.e., a 12×1 tensor) into a 1×320 tensor, it merges the final output L(n) from the second long-term connected network (i.e., the 16×160 second tensor) to generate the estimated value x′(n) of the signal.

[0300] Therefore, this embodiment of the application realizes the output of the intermediate node of the second long-term connection network ( Figure 13D The 32×80 tensor output by the convolution-pooling layer with Up_factor=4 is input to the intermediate nodes of the first reconstruction network. Figure 13D The fourth coded block in the first reconstructed network, and the output of the intermediate nodes of the first reconstructed network ( Figure 13D The 16×320 tensor output from the post-processing layer is input to the intermediate nodes of the second longest-lived network. Figure 13D The signal is estimated by a causal convolutional layer (in the middle), thereby generating an estimated value x′(n). The specific implementation of the decoding end can be referred to the description of steps 21 to 23 above, and will not be repeated here.

[0301] In other embodiments, see Figure 13E , Figure 13E This is a fifth schematic diagram of an optional structure of the first reconstruction network and the second long-term connection network provided in the embodiments of this application. See below. Figure 13E The network structure shown in the figure will be explained in detail.

[0302] like Figure 13E As shown, the input to the second long-term connection network can be the output of the first causal convolution in the first reconstruction network, which corresponds to a dimension of 256×1. The connection method between the second long-term connection network and the first reconstruction network has been changed (the change can be found in [reference]). Figure 11B and Figure 13E(Differences between them). For example, after the second long-term connection network performs four convolutional-repeat layers, it transforms the 256×1 tensor into a 16×160 tensor. This tensor is then fused with the output of the fourth decoding block (Up_factor=2), an intermediate node of the first reconstruction network (not the final output), to generate a new 16×160 tensor. After this, it undergoes post-processing by the original first reconstruction network's post-processing layer to generate a 16×320 tensor. Following this, the first reconstruction network performs... The remaining operations yield a 1×320 tensor. The first long-term connection network generates a 16×320 tensor as the input to the last causal convolutional layer, converting the 16×320 tensor into a 1×320 tensor to obtain the final output L(n) of the second long-term connection network (i.e., (a 16×320 tensor)). Finally, a second feature fusion is performed based on the final output L(n) of the second long-term connection network and the 1×320 tensor output by the first reconstruction network to obtain the estimated value x′(n) of the signal.

[0303] Therefore, this embodiment of the application implements feature fusion of the output of the intermediate node of the second long-time connection network with the intermediate node of the first reconstruction network, and then performs the remaining operations of the first reconstruction network and the second long-time connection network, as well as a second feature fusion, based on the fused features to obtain the estimated value x′(n) of the signal. The specific implementation of the decoding end can be referred to the description of steps 21 to 23 above, and will not be repeated here.

[0304] As can be seen from the above, the connection between the first reconstruction network (backbone network) and the second long-term connection network (secondary network) can exhibit high flexibility and diversity. Through interaction and fusion at different levels, the feature reconstruction capability of the model is significantly improved. The connection method can be optimized according to specific tasks and data, thereby increasing the flexibility of the model.

[0305] The connection methods between the backbone network and the secondary network include, but are not limited to, the following:

[0306] 1) Independent Extraction and Fusion: The backbone network and secondary networks independently reconstruct features, and then these features are fused in the final stage of the network to achieve more comprehensive information utilization (e.g., Figure 13C ).

[0307] 2) Intermediate Node Connections: The outputs of intermediate nodes in the backbone network are used as inputs to some neural network layers in the secondary network, and the outputs of intermediate nodes in the secondary network can also be used as inputs to some neural network layers in the backbone network (e.g., ...). Figure 13DThe connection method shown in the diagram does not limit the number of inputs and outputs to intermediate nodes of the backbone and secondary networks. This means the secondary network can reuse some neural network layers from the backbone. After the backbone and secondary networks reconstruct their own features, these features are fused to obtain fused features. This feature fusion can occur at the final stage of the network (e.g., ...). Figure 13C The connection method shown in the figure can also be to merge the output of the secondary network with the output of the intermediate nodes of the backbone network (such as...). Figure 13B (The connection method shown in the figure) is used to enhance the expressive power of features.

[0308] 3) Layered fusion: The outputs of the secondary network's neural network layers are fused with the outputs of the intermediate nodes of the backbone network at least once to achieve in-depth feature mining and integration. For example, the outputs of the secondary network's neural network layers are fused with the outputs of the intermediate nodes of the backbone network once, and a second feature fusion is performed at the final output of the backbone network (e.g., ...). Figure 13E (The connection method shown in the figure); the output of the neural network layer of the secondary network and the output of the intermediate node of the backbone network are fused twice.

[0309] Therefore, as can be seen from the above embodiments, the second long-term connection network can be fused as long as the dimensions of the input and output tensors match the neural network layers of the first reconstruction network. Furthermore, referring to the above-described decoding-side embodiments, this application does not limit the internal structure of the second long-term connection network. Generally, because a more streamlined network structure is adopted, the computational complexity of the second long-term connection network is less than that of the first reconstruction network.

[0310] This application embodiment can obtain optimal parameters by jointly training the relevant networks of the encoder and decoder through data collection. Users only need to prepare the data and set the corresponding network structure. After training is completed in the background, the trained model can be put into use.

[0311] Therefore, through the process described in the above embodiments, a low bit-rate speech encoding and decoding scheme based on signal processing and deep learning networks can be completed.

[0312] In summary, this application's embodiments introduce a long-term connection analysis network at the encoding end to generate a more complex encoding network. This more complex encoding network ensures that the feature vectors obtained through complex network analysis retain the most authentic information of the input signal at a very low bit rate, thereby improving encoding performance. Then, by introducing a long-term connection analysis network into the decoding network, higher-quality reconstruction is obtained, reconstructing the most authentic information of the input signal. By introducing a long-term connection analysis network, the end-to-end neural network encoding and decoding technology is further improved, enhancing audio decoding quality while maintaining acceptable neural network model complexity.

[0313] The following description continues to illustrate the exemplary structure of the audio encoding device 555 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 3A As shown, the software modules stored in the audio encoding device 555 of the memory 550 may include:

[0314] The feature extraction module 5551 is used to extract features from the audio signal to obtain the audio features of the audio signal;

[0315] The signal encoding module 5552 is used to perform signal encoding processing on the audio features to obtain the audio bitstream of the audio signal, wherein the audio bitstream is used for decoding based on the audio decoding method provided in the embodiments of this application.

[0316] In some embodiments, the feature extraction is implemented through a feature extraction network, which includes multiple cascaded fourth neural network layers; the feature extraction module 5551 is further configured to extract features from the input of the t-th fourth neural network layer through the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer, where t is a positive integer that increases sequentially, 0 < t ≤ T, T is the number of the fourth neural network layers in the feature extraction network, and the input of the first fourth neural network layer is the audio signal; the feature extraction result of the t-th fourth neural network layer is determined as the audio feature.

[0317] In some embodiments, the feature extraction module 5551 is further configured to perform one of the following processes through the t-th fourth neural network layer of the feature extraction network: performing convolution processing on the input of the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer; performing preprocessing processing on the input of the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer; or performing encoding processing on the input of the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer.

[0318] In some embodiments, the feature extraction module 5551 is further configured to perform residual processing on the input of the t-th fourth neural network layer through the residual block included in the t-th fourth neural network layer to obtain residual features; perform convolution processing on the residual features through the convolutional layer included in the t-th fourth neural network layer to obtain convolutional features; and perform downsampling processing on the convolutional features through the downsampling layer included in the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer.

[0319] The following description continues to illustrate the exemplary structure of the audio decoding device 655 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 3B As shown, the software module stored in the audio decoding device 655 in the memory 650 may include:

[0320] The signal decoding module 6551 is used to perform signal decoding processing on the audio bitstream to obtain the audio feature estimation value corresponding to the audio bitstream;

[0321] The first reconstruction module 6552 is used to perform a first reconstruction process on the audio feature estimate through the backbone network to obtain the first reconstructed signal corresponding to the audio bitstream.

[0322] The second reconstruction module 6553 is used to perform a second reconstruction process on the audio feature estimate through a secondary network to obtain the second reconstructed signal corresponding to the audio bitstream;

[0323] Wherein, the number of convolutional layers in the backbone network is greater than the number of convolutional layers in the secondary network, and the dimension of the first reconstructed signal is the same as the dimension of the second reconstructed signal;

[0324] The fusion module 6554 is used to fuse the first reconstructed signal and the second reconstructed signal to obtain the reconstructed audio signal corresponding to the audio bitstream.

[0325] In some embodiments, the backbone network includes multiple cascaded first neural network layers; the first reconstruction module 6552 is further configured to perform feature reconstruction on the input of the nth first neural network layer through the nth first neural network layer of the backbone network to obtain the feature reconstruction result of the nth first neural network layer, where n is a positive integer that increases sequentially, 0 < n ≤ N, N is the number of the first neural network layers in the backbone network, and the input of the first first neural network layer is the audio feature estimate; the feature reconstruction result of the Nth first neural network layer is determined as the first reconstructed signal.

[0326] In some embodiments, the first reconstruction module 6552 is further configured to perform one of the following processes through the nth first neural network layer of the backbone network: perform convolution processing on the input of the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer; perform decoding processing on the input of the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer; and perform post-processing processing on the input of the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer.

[0327] In some embodiments, the first reconstruction module 6552 is further configured to perform deconvolution processing on the input of the nth first neural network layer through the deconvolution layer included in the nth first neural network layer to obtain deconvolution features; perform upsampling processing on the deconvolution features through the upsampling layer included in the nth first neural network layer to obtain upsampled features; perform feature residual processing on the upsampled features through the residual block included in the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer.

[0328] In some embodiments, the secondary network includes a plurality of cascaded second neural network layers; the second reconstruction module 6553 is further configured to obtain the feature reconstruction result of the gth first neural network layer in the backbone network, where the feature reconstruction result of the gth first neural network layer is obtained based on the audio feature estimation value, 0 < g < N, and g is a positive integer; perform feature reconstruction on the input of the kth second neural network layer through the kth second neural network layer of the secondary network to obtain the feature reconstruction result of the kth second neural network layer, where k is a positive integer that increases sequentially, 0 < k ≤ K, and K is the number of second neural network layers in the secondary network, and the input of the first second neural network layer is the feature reconstruction result of the gth first neural network layer; determine the feature reconstruction result of the Kth second neural network layer as the second reconstruction signal.

[0329] In some embodiments, the second reconstruction module 6553 is further configured to perform one of the following processes through the kth second neural network layer of the secondary network: perform deconvolution processing on the input of the kth second neural network layer to obtain the feature reconstruction result of the kth second neural network layer; perform decoding processing on the input of the kth second neural network layer to obtain the feature reconstruction result of the kth second neural network layer.

[0330] In some embodiments, the second reconstruction module 6553 is further configured to perform at least one deconvolution processing on the input of the kth second neural network layer through at least one deconvolution layer included in the kth second neural network layer to obtain deconvolution features; perform upsampling processing on the convolution features through the upsampling layer included in the kth second neural network layer to obtain the feature reconstruction result of the kth second neural network layer.

[0331] In some embodiments, the secondary network includes multiple cascaded third neural network layers; the second reconstruction module 6553 is further configured to perform feature reconstruction on the input of the q-th third neural network layer through the q-th third neural network layer of the secondary network to obtain the feature reconstruction result of the q-th third neural network layer, where q is a positive integer that increases sequentially, 0 < q ≤ Q, Q is the number of the third neural network layers in the secondary network, and the input of the first third neural network layer is the audio feature estimate; the feature reconstruction result of the q-th third neural network layer is determined as the second reconstructed signal.

[0332] In some embodiments, the backbone network includes a first subnetwork and a second subnetwork, the first subnetwork being used to perform the first reconstruction process; the fusion module 6554 is further used to add the first reconstructed signal and the second reconstructed signal to obtain a third reconstructed signal; and the third reconstructed signal is processed by the second subnetwork to obtain the reconstructed audio signal.

[0333] This application provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the audio encoding or audio decoding method described above in this application.

[0334] This application provides a computer-readable storage medium storing a bitstream formed by computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, they implement the audio encoding or audio decoding method provided in this application. For example, ... Figure 4A The audio encoding method shown or Figure 5 The audio decoding method is shown.

[0335] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0336] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0337] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0338] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0339] In summary, through the embodiments of this application, since the backbone network contains more convolutional layers, it can reconstruct a deeper first reconstructed signal. Since the secondary network has relatively fewer convolutional layers compared to the backbone network, it can reconstruct a shallower second reconstructed signal, which is convenient for capturing shallow details in the audio signal. By fusing the two signals, both deep and shallow information can be utilized simultaneously, so that the reconstructed audio signal contains richer audio information, so as to better recover the audio signal and thus improve the audio decoding quality.

[0340] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An audio decoding method, characterized in that, The method includes: The audio bitstream is subjected to signal decoding processing to obtain the estimated audio feature values ​​corresponding to the audio bitstream; The estimated audio features are reconstructed using the backbone network to obtain the first reconstructed signal corresponding to the audio bitstream. The estimated audio features are reconstructed using a secondary network to obtain the second reconstructed signal corresponding to the audio bitstream. Wherein, the number of convolutional layers in the backbone network is greater than the number of convolutional layers in the secondary network, and the dimension of the first reconstructed signal is the same as the dimension of the second reconstructed signal; By fusing the first reconstructed signal and the second reconstructed signal, the reconstructed audio signal corresponding to the audio bitstream is obtained.

2. The method according to claim 1, characterized in that, The backbone network includes multiple cascaded first neural network layers; The first reconstruction process, which involves performing a first reconstruction on the estimated audio features via a backbone network to obtain the first reconstructed signal corresponding to the audio bitstream, includes: The input of the nth first neural network layer is used to reconstruct features through the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer, where n is a positive integer that increases sequentially, 0 < n ≤ N, N is the number of the first neural network layers in the backbone network, and the input of the first first neural network layer is the audio feature estimate. The feature reconstruction result of the Nth first neural network layer is determined as the first reconstructed signal.

3. The method according to claim 2, characterized in that, The step of performing feature reconstruction on the input of the nth first neural network layer through the backbone network to obtain the feature reconstruction result of the nth first neural network layer includes: One of the following processes is performed through the nth first neural network layer of the backbone network: The input of the nth first neural network layer is convolved to obtain the feature reconstruction result of the nth first neural network layer; The input of the nth first neural network layer is decoded to obtain the feature reconstruction result of the nth first neural network layer; The input of the nth first neural network layer is post-processed to obtain the feature reconstruction result of the nth first neural network layer.

4. The method according to claim 3, characterized in that, The step of decoding the input of the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer includes: By using the deconvolution layer included in the nth first neural network layer, the input of the nth first neural network layer is deconvolutionally processed to obtain deconvolution features; The deconvolutional features are upsampled through the upsampling layer included in the nth first neural network layer to obtain upsampled features; By using the residual blocks included in the nth first neural network layer, feature residual processing is performed on the upsampled features to obtain the feature reconstruction result of the nth first neural network layer.

5. The method according to any one of claims 2 to 4, characterized in that, The secondary network includes multiple cascaded second neural network layers; Performing a second reconstruction process on the audio feature estimation value through the secondary network to obtain a second reconstruction signal corresponding to the audio code stream, including: Obtaining a feature reconstruction result of the g-th first neural network layer in the backbone network, where the feature reconstruction result of the g-th first neural network layer is obtained based on the audio feature estimation value, 0 < g < N, and g is a positive integer; Performing feature reconstruction on the input of the k-th second neural network layer through the k-th second neural network layer of the secondary network to obtain a feature reconstruction result of the k-th second neural network layer, where k is a sequentially increasing positive integer, 0 < k ≤ K, K is the number of second neural network layers in the secondary network, and the input of the first second neural network layer is the feature reconstruction result of the g-th first neural network layer; Determining the feature reconstruction result of the K-th second neural network layer as the second reconstruction signal.

6. The method according to claim 5, characterized in that, Performing feature reconstruction on the input of the k-th second neural network layer through the k-th second neural network layer of the secondary network to obtain a feature reconstruction result of the k-th second neural network layer, including: Performing one of the following processes through the k-th second neural network layer of the secondary network: Performing a transposed convolution process on the input of the k-th second neural network layer to obtain a feature reconstruction result of the k-th second neural network layer; Performing a decoding process on the input of the k-th second neural network layer to obtain a feature reconstruction result of the k-th second neural network layer.

7. The method according to claim 6, characterized in that, Performing a decoding process on the input of the k-th second neural network layer to obtain a feature reconstruction result of the k-th second neural network layer, including: Performing at least one transposed convolution process on the input of the k-th second neural network layer through at least one transposed convolution layer included in the k-th second neural network layer to obtain transposed convolution features; Performing an upsampling process on the convolution features through an upsampling layer included in the k-th second neural network layer to obtain a feature reconstruction result of the k-th second neural network layer.

8. The method according to any one of claims 1 to 4, characterized in that, The secondary network includes multiple cascaded third neural network layers; Performing a second reconstruction process on the audio feature estimation value through the secondary network to obtain a second reconstruction signal corresponding to the audio code stream, including: Performing feature reconstruction on the input of the q-th third neural network layer through the q-th third neural network layer of the secondary network to obtain a feature reconstruction result of the q-th third neural network layer, where q is a sequentially increasing positive integer, 0 < q ≤ Q, Q is the number of third neural network layers in the secondary network, and the input of the first third neural network layer is the audio feature estimation value; Determining the feature reconstruction result of the Q-th third neural network layer as the second reconstruction signal.

9. The method according to any one of claims 1 to 8, wherein The backbone network includes a first sub-network and a second sub-network, and the first sub-network is used for performing the first reconstruction process; Fusing the first reconstruction signal and the second reconstruction signal to obtain a reconstructed audio signal corresponding to the audio code stream, including: The first reconstructed signal and the second reconstructed signal are added together to obtain the third reconstructed signal; The reconstructed audio signal is obtained by performing a third reconstruction process on the third reconstructed signal through the second sub-network.

10. An audio encoding method, characterized in that, The method includes: Feature extraction is performed on the audio signal to obtain the audio features of the audio signal; The audio features are subjected to signal encoding processing to obtain an audio bitstream of the audio signal, wherein the audio bitstream is used for decoding based on the audio decoding method according to any one of claims 1 to 9.

11. The method according to claim 10, characterized in that, The feature extraction is achieved through a feature extraction network, which includes multiple cascaded fourth neural network layers. The step of extracting features from the audio signal to obtain the audio features of the audio signal includes: By using the t-th fourth neural network layer of the feature extraction network, feature extraction is performed on the input of the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer, where t is a positive integer that increases sequentially, 0 < t ≤ T, T is the number of the fourth neural network layer in the feature extraction network, and the input of the first fourth neural network layer is the audio signal; The feature extraction result of the Tth fourth neural network layer is determined as the audio feature.

12. The method according to claim 11, characterized in that, The step of extracting features from the input of the t-th fourth neural network layer through the feature extraction network to obtain the feature extraction result of the t-th fourth neural network layer includes: One of the following processes is performed through the t-th fourth neural network layer of the feature extraction network: The input of the t-th fourth neural network layer is convolved to obtain the feature extraction result of the t-th fourth neural network layer; The input of the t-th fourth neural network layer is preprocessed to obtain the feature extraction result of the t-th fourth neural network layer; The input of the t-th fourth neural network layer is encoded to obtain the feature extraction result of the t-th fourth neural network layer.

13. The method according to claim 12, characterized in that, The process of encoding the input of the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer includes: By using the residual blocks included in the t-th fourth neural network layer, residual processing is performed on the input of the t-th fourth neural network layer to obtain residual features; The residual features are processed by convolution through the convolutional layers included in the t-th fourth neural network layer to obtain convolutional features; The convolutional features are downsampled using the downsampling layer included in the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer.

14. A method for processing audio bitstreams, characterized in that, The audio stream is decoded based on the audio decoding method according to any one of claims 1 to 9, or generated based on the audio encoding method according to any one of claims 10 to 13.

15. An audio decoding device, characterized in that, The device includes: The signal decoding module is used to perform signal decoding processing on the audio bitstream to obtain the audio feature estimation value corresponding to the audio bitstream; The first reconstruction module is used to perform a first reconstruction process on the audio feature estimate through the backbone network to obtain the first reconstructed signal corresponding to the audio bitstream. The second reconstruction module is used to perform a second reconstruction process on the audio feature estimate through a secondary network to obtain the second reconstructed signal corresponding to the audio bitstream; Wherein, the number of convolutional layers in the backbone network is greater than the number of convolutional layers in the secondary network, and the dimension of the first reconstructed signal is the same as the dimension of the second reconstructed signal; The fusion module is used to fuse the first reconstructed signal and the second reconstructed signal to obtain the reconstructed audio signal corresponding to the audio bitstream.

16. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions for a computer; A processor, when executing computer-executable instructions stored in the memory, implements the audio decoding method according to any one of claims 1 to 9, or the audio encoding method according to any one of claims 10 to 13.

17. A computer-readable storage medium storing thereon a code stream formed by computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the audio decoding method according to any one of claims 1 to 9, or the audio encoding method according to any one of claims 10 to 13.

18. A computer-readable storage medium, characterized in that, The device stores a computer program or computer-executable instructions, which, when executed by a processor, implement the audio decoding method according to any one of claims 1 to 9, or the audio encoding method according to any one of claims 10 to 13.

19. A computer program product, characterized in that, It includes computer-executable instructions, which, when executed by a processor, implement the audio decoding method according to any one of claims 1 to 9, or the audio encoding method according to any one of claims 10 to 13.

Images