Audio encoding and decoding method and device

Abstract

The embodiment of the invention provides an audio encoding and decoding method and device, and relates to the field of artificial intelligence. An audio encoding method includes: obtaining a time-frequency spectrum of an audio signal; extracting features of a time domain direction and a frequency domain direction based on the time-frequency spectrum of the audio signal by using a first artificial intelligence AI network; and coding the audio signal based on the characteristics of the time domain direction and the frequency domain direction. Optionally, the method performed by the electronic device may be performed using an artificial intelligence model.

Description

Technical Field

[0001] The present disclosure relates to the field of audio coding and decoding, and in particular, to an audio coding and decoding method, device, electronic device and computer-readable storage medium. Background Art

[0002] Audio coding is becoming increasingly important in an era of rapidly growing data traffic due to network bandwidth and storage limitations. Lossy audio compression aims to reduce the bit rate of an audio signal by compressing the audio into a compact bitstream representation with acceptable distortion.

[0003] In recent years, deep neural networks have been introduced into audio codecs. However, the bitstreams obtained by most current audio codecs based on deep learning still contain a lot of redundant information. Therefore, the efficiency of audio codecs needs to be further improved. Summary of the invention

[0004] The purpose of the present disclosure is to solve at least one of the technical defects in the existing audio coding and decoding methods to better meet the audio coding and decoding requirements. In order to achieve this purpose, the technical solutions proposed in the present disclosure are as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, an audio encoding method is provided, the audio encoding method comprising: obtaining a time-frequency spectrum of an audio signal; using a first artificial intelligence (AI) network to extract features in a time domain and a frequency domain based on the time-frequency spectrum of the audio signal; and encoding the audio signal based on the features in the time domain and the frequency domain.

[0006] Optionally, the first AI network includes at least one convolution module, at least one frequency domain feature extraction module and at least one time domain feature extraction module; using the first AI network, based on the time-frequency spectrum of the audio signal, extracting features in the time domain and frequency domain directions, including: based on the time-frequency spectrum, using the at least one convolution module to perform convolution processing to obtain a first feature of the audio signal; based on the first feature, using the at least one frequency domain feature extraction module to obtain a feature in the frequency domain direction of the audio signal; based on the feature in the frequency domain direction, using the at least one time domain feature extraction module to obtain features in the time domain and frequency domain directions of the audio signal.

[0007] Optionally, based on the time-frequency spectrum, convolution processing is performed using the at least one convolution module to obtain a first feature of the audio signal, including: using the first convolution kernel in the convolution layer of the at least one convolution module to perform convolution operations on the real and imaginary features of the time-frequency spectrum, respectively, to obtain a first convolution feature and a second convolution feature; using the second convolution kernel in the convolution layer to perform convolution operations on the real and imaginary features of the time-frequency spectrum, respectively, to obtain a third convolution feature and a fourth convolution feature; based on the first convolution feature and the fourth convolution feature, obtaining the real feature of the first feature; based on the second convolution feature and the third convolution feature, obtaining the imaginary feature of the first feature.

[0008] Optionally, each of the at least one convolution module is a complex convolution block, and the convolution layer is a complex convolution layer.

[0009] Optionally, based on the first feature, the at least one frequency domain feature extraction module is used to obtain the features of the audio signal in the frequency domain direction, including: using the first self-attention network of the at least one frequency domain feature extraction module to perform feature extraction on the real features and imaginary features of the first feature, respectively, to obtain a first frequency-related feature and a second frequency-related feature; using the second self-attention network of the at least one frequency domain feature extraction module to perform feature extraction on the real features and imaginary features of the first feature, respectively, to obtain a third frequency-related feature and a fourth frequency-related feature; based on the first frequency-related feature and the fourth frequency-related feature, the real features of the features in the frequency domain direction are obtained; based on the second frequency-related feature and the third frequency-related feature, the imaginary features of the features in the frequency domain direction are obtained.

[0010] Optionally, the first self-attention network and the second self-attention network use a multi-head self-attention network.

[0011] Optionally, based on the features in the frequency domain direction, the features in the time domain direction and the frequency domain direction of the audio signal are obtained using the at least one time domain feature extraction module, including: using the first long short-term memory network of the at least one time domain feature extraction module to perform feature extraction on the real features and the imaginary features of the features in the frequency domain direction, respectively, to obtain first time-related features and second time-related features; using the second long short-term memory network of the at least one time domain feature extraction module to perform feature extraction on the real features and the imaginary features of the features in the frequency domain direction, respectively, to obtain third time-related features and fourth time-related features; based on the first time-related features and the fourth time-related features, the real features of the features in the time domain direction and the frequency domain direction are obtained; based on the second time-related features and the third time-related features, the imaginary features of the features in the time domain direction and the frequency domain direction are obtained.

[0012] Optionally, the audio signal is encoded based on the features in the time domain and frequency domain directions, including: quantizing the features in the time domain and frequency domain directions based on a target bit rate to obtain quantized features of the features in the time domain and frequency domain directions; determining an index value corresponding to the quantized feature from a code book; and encoding the index value into a bit stream.

[0013] According to a second aspect of an embodiment of the present disclosure, an audio decoding method is provided, the audio decoding method comprising: acquiring features of an audio signal in a time domain and a frequency domain from a bit stream; using a second artificial intelligence (AI) network to obtain a time-frequency spectrum of the audio signal based on the features in the time domain and the frequency domain; and obtaining the audio signal based on the time-frequency spectrum of the audio signal.

[0014] Optionally, the second AI network includes at least one time domain feature extraction module, at least one transposed convolution module and at least one frequency domain feature extraction module; using the second artificial intelligence AI network, based on the features in the time domain direction and the frequency domain direction, obtaining the time-frequency spectrum of the audio signal, including: based on the features in the time domain direction and the frequency domain direction, using the at least one time domain feature extraction module to obtain the features in the frequency domain direction of the audio signal; based on the features in the frequency domain direction, using the at least one frequency domain feature extraction module to obtain a first feature of the audio signal; based on the first feature, using the at least one transposed convolution module to perform a transposed convolution operation to obtain the time-frequency spectrum of the audio signal.

[0015] Optionally, based on the features in the time domain and frequency domain directions, the features in the frequency domain direction of the audio signal are obtained using the at least one time domain feature extraction module, including: using the first long short-term memory network of the at least one time domain feature extraction module to perform feature extraction on the real features and imaginary features of the features in the time domain and frequency domain directions, respectively, to obtain a first time-related feature and a second time-related feature; using the second long short-term memory network of the at least one time domain feature extraction module to perform feature extraction on the real features and imaginary features of the features in the time domain and frequency domain directions, respectively, to obtain a third time-related feature and a fourth time-related feature; based on the first time-related feature and the fourth time-related feature, the real features of the features in the frequency domain direction are obtained; based on the second time-related feature and the third time-related feature, the imaginary features of the features in the frequency domain direction are obtained.

[0016] Optionally, based on the features in the frequency domain direction, the at least one frequency domain feature extraction module is used to obtain the first feature of the audio signal, including: using the first self-attention network of the at least one frequency domain feature extraction module to perform feature extraction on the real features and imaginary features of the features in the frequency domain direction, respectively, to obtain a first frequency-related feature and a second frequency-related feature; using the second self-attention network of the at least one frequency domain feature extraction module to perform feature extraction on the real features and imaginary features of the features in the frequency domain direction, respectively, to obtain a third frequency-related feature and a fourth frequency-related feature; based on the first frequency-related feature and the fourth frequency-related feature, the real feature of the first feature is obtained; based on the second frequency-related feature and the third frequency-related feature, the imaginary feature of the first feature is obtained.

[0017] Optionally, the first self-attention network and the second self-attention network use a multi-head self-attention network.

[0018] Optionally, based on the first feature, the at least one transposed convolution module is used to perform a transposed convolution operation to obtain the time-frequency spectrum of the audio signal, including: using the first convolution kernel in the convolution layer of the at least one transposed convolution module to perform a transposed convolution operation on the real feature and the imaginary feature of the first feature, respectively, to obtain a first convolution feature and a second convolution feature; using the second convolution kernel in the convolution layer to perform a transposed convolution operation on the real feature and the imaginary feature of the first feature, respectively, to obtain a third convolution feature and a fourth convolution feature; based on the first convolution feature and the fourth convolution feature, obtaining the real feature of the time-frequency spectrum; based on the second convolution feature and the third convolution feature, obtaining the imaginary feature of the time-frequency spectrum.

[0019] Optionally, each of the at least one transposed convolution module is a complex transposed convolution module, and the convolution layer is a complex convolution layer.

[0020] Optionally, obtaining features of an audio signal in a time domain direction and a frequency domain direction from a bit stream includes: obtaining an index value corresponding to a quantized feature of the audio signal from a bit stream; determining the quantized feature from a code book based on the index value; and dequantizing the quantized feature to obtain features of the audio signal in the time domain direction and the frequency domain direction.

[0021] Optionally, the method also includes: using a third artificial intelligence AI network to perform speech enhancement on the obtained audio signal.

[0022] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, prompt the at least one processor to execute a method according to the present disclosure.

[0023] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium storing instructions is provided. When the instructions are executed by at least one processor, the at least one processor is prompted to execute a method according to the present disclosure.

[0024] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0025] By extracting the features of the audio signal in the frequency domain and the time domain, the redundancy of the audio signal features in the time domain and the frequency domain is reduced, thereby reducing the bit rate of the encoded and decoded audio signal and further improving the encoding and decoding efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly and easily illustrate and understand the technical solutions in the embodiments of the present disclosure, the drawings required for describing the embodiments of the present disclosure are briefly introduced below.

[0027] Figure 1 is a flowchart of an audio encoding method according to an embodiment of the present disclosure.

[0028] Figure 2 is a flowchart of an audio decoding method according to an embodiment of the present disclosure.

[0029] Figure 3 is a flowchart of an audio encoding and decoding method according to an embodiment of the present disclosure.

[0030] Figure 4 A schematic structural diagram of an electronic device applicable to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0031] The following description with reference to the accompanying drawings is provided to facilitate a comprehensive understanding of the various embodiments of the present disclosure as defined by the claims and their equivalents. This description includes various specific details to facilitate understanding but should be considered as exemplary only. Therefore, one of ordinary skill in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, for the sake of clarity and conciseness, descriptions of well-known functions and structures may be omitted.

[0032] The terms and expressions used in the following specification and claims are not limited to their dictionary meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present disclosure. Therefore, it should be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustration purposes only and not for the purpose of limiting the present disclosure as defined in the appended claims and their equivalents.

[0033] It should be understood that the singular forms "a", "an", and "the" may also include plural references unless the context clearly indicates otherwise. Thus, for example, reference to "a component surface" includes reference to one or more such surfaces. When we refer to an element as being "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or the one element and the other element may establish a connection relationship through an intermediate element. In addition, "connected" or "coupled" as used herein may include wireless connection or wireless coupling.

[0034] The term "include" or "may include" refers to the presence of the corresponding disclosed functions, operations or components that can be used in various embodiments of the present disclosure, rather than limiting the presence of one or more additional functions, operations or features. In addition, the term "include" or "have" may be interpreted as indicating certain characteristics, numbers, steps, operations, constituent elements, components or combinations thereof, but should not be interpreted as excluding the possibility of the presence of one or more other characteristics, numbers, steps, operations, constituent elements, components or combinations thereof.

[0035] The term "or" used in various embodiments of the present disclosure includes any of the listed terms and all combinations thereof. For example, "A or B" may include A, may include B, or may include both A and B. When describing multiple (two or more) items, if the relationship between the multiple items is not clearly defined, the multiple items may refer to one, multiple, or all of the multiple items. For example, the description of "parameter A includes A1, A2, A3" may be implemented as parameter A including A1 or A2 or A3, or may be implemented as parameter A including at least two of the three items A1, A2, and A3.

[0036] Unless defined differently, all terms (including technical terms or scientific terms) used in the present disclosure have the same meanings as understood by those skilled in the art described in the present disclosure. Common terms as defined in dictionaries are interpreted as having meanings consistent with the context in the relevant technical field, and should not be interpreted ideally or overly formally unless clearly defined in the present disclosure.

[0037] At least some functions of the device or electronic device provided in the embodiments of the present disclosure can be implemented by an AI model, such as at least one module among multiple modules of the device or electronic device can be implemented by an AI model. Functions associated with AI can be performed by a non-volatile memory, a volatile memory, and a processor.

[0038] The processor may include one or more processors. In this case, the one or more processors may be general-purpose processors, such as a central processing unit (CPU), an application processor (AP), etc., or pure graphics processing units, such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an AI-specific processor, such as a neural processing unit (NPU).

[0039] The one or more processors control the processing of input data according to predefined operating rules or artificial intelligence (AI) models stored in non-volatile memory and volatile memory. The predefined operating rules or artificial intelligence models are provided by training or learning.

[0040] Here, providing by learning means obtaining a predefined operating rule or an AI model with desired characteristics by applying a learning algorithm to a plurality of learning data. The learning can be performed in the device or electronic device itself in which the AI ​​according to the embodiment is executed, and / or can be implemented by a separate server / system.

[0041] The AI ​​model may include multiple neural network layers. Each layer has multiple weight values, and each layer performs neural network calculations by calculating between the input data of the layer (such as the calculation results of the previous layer and / or the input data of the AI ​​model) and the multiple weight values ​​of the current layer. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks.

[0042] A learning algorithm is a method of using a plurality of learning data to train a predetermined target device (e.g., a robot) to enable, allow, or control the target device to make a determination or prediction. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0043] The method provided by the present disclosure may involve one or more technical fields such as voice, audio or data intelligence.

[0044] For example, when it comes to the field of voice or audio, in the method performed by an electronic device according to the present disclosure, a voice signal as an analog signal can be received via a voice input device (e.g., a microphone), and the voice signal can be encoded and decoded using the method of the present disclosure.

[0045] For another example, when it comes to the field of intelligent data processing, in the method performed by the electronic device according to the present disclosure, in the inference or prediction stage, an artificial intelligence model can be used to perform predictions by using real-time input data. The processor of the electronic device can perform preprocessing operations on the data to convert it into a form suitable for use as an input to the artificial intelligence model. Inference prediction is a technology for logical reasoning and prediction by determining information, including, for example, knowledge-based reasoning, optimization prediction, preference-based planning or recommendation.

[0046] In the present application, an artificial intelligence model can be obtained by training. Here, "obtained by training" means obtaining a predefined operating rule or artificial intelligence model configured to perform a desired feature (or purpose) by training a basic artificial intelligence model with multiple training data through a training algorithm. The artificial intelligence model may include multiple neural network layers. Each of the multiple neural network layers includes multiple weight values, and the neural network calculation is performed by calculating between the calculation result of the previous layer and the multiple weight values.

[0047] The present disclosure models the time-frequency spectrum features in the frequency domain (frequency) and time domain (time) directions, and introduces a deep complex network with fewer parameters and stronger modeling capabilities into neural audio coding. Specifically, the present disclosure proposes an audio codec structure based on a deep complex network. The present disclosure uses a complex multi-head self-attention (CMHSA) module and a complex long short-term memory (CLSTM) module to model the features extracted by the complex convolution module in the frequency direction and the time direction, respectively, in order to better reduce the redundancy in the time domain and the frequency domain. In addition, an optional time domain enhancer is also designed at the decoding end to reduce the sound quality loss caused by the codec network.

[0048] The following describes several optional embodiments to illustrate the technical solutions of the embodiments of the present disclosure and the technical effects produced by the technical solutions of the present disclosure. It should be noted that the following embodiments can refer to, draw on or combine with each other, and the same terms, similar features and similar implementation steps in different embodiments will not be described repeatedly.

[0049] Figure 1 is a flowchart of an audio encoding method according to an embodiment of the present disclosure.

[0050] Reference Figure 1 In step S101, a time-frequency spectrum of an audio signal is obtained. The audio signal here may be, for example, a speech signal to be processed / encoded, or may be a real-time audio signal.

[0051] The audio signal can be transformed to obtain the time-frequency spectrum of the audio signal.

[0052] As an example, a short-time Fourier transform may be performed on an audio signal to obtain a time-frequency spectrum of the audio signal. After obtaining the time-frequency spectrum, the time-frequency spectrum may be further amplitude compressed to obtain a time-frequency spectrum having a compressed amplitude and an original phase. For example, power-law compression may be used to limit the dynamic range of the amplitude.

[0053] In step S102, a first artificial intelligence AI network is used to extract features in the time domain and frequency domain based on the time-frequency spectrum of the audio signal.

[0054] As an example, the first AI network can be implemented using a complex neural network. For example, the first complex neural network can be used to model the time-frequency spectrum in the frequency direction and the time direction to obtain the complex features of the audio signal in the time domain direction and the frequency domain direction.

[0055] The first AI network may include at least one convolution module, at least one frequency domain feature extraction module, and at least one time domain feature extraction module.

[0056] As an example, based on the time-frequency spectrum, at least one convolution module can be used to perform convolution processing to obtain a first feature of the audio signal; based on the first feature, at least one frequency domain feature extraction module can be used to obtain the frequency domain direction feature of the audio signal; based on the frequency domain direction feature, at least one time domain feature extraction module can be used to obtain the time domain and frequency domain directions features of the audio signal.

[0057] Specifically, in obtaining the first feature of the audio signal, the first convolution kernel in the convolution layer of the at least one convolution module can be used to perform convolution operations on the real and imaginary features of the time-frequency spectrum, respectively, to obtain the first convolution feature and the second convolution feature; the second convolution kernel in the convolution layer can be used to perform convolution operations on the real and imaginary features of the time-frequency spectrum, respectively, to obtain the third convolution feature and the fourth convolution feature; based on the first convolution feature and the fourth convolution feature, the real feature of the first feature is obtained; based on the second convolution feature and the third convolution feature, the imaginary feature of the first feature is obtained. The above-mentioned convolution module can be implemented by a complex convolution module, and the convolution layer can be implemented by a complex convolution layer.

[0058] Taking the time-frequency spectrum with compressed amplitude and original phase as an example, the characteristics of the time-frequency spectrum can be first divided into real characteristics and imaginary characteristics. For example, the real characteristics of the time-frequency spectrum can be obtained according to the following equation (1): and imaginary characteristics

[0059]

[0060]

[0061] Where n and k represent the frame number and frequency point respectively, β represents the compression coefficient (for example, it can be 0.5), θ represents the phase of the time-frequency point, and |S(n,k)| represents the amplitude of the time-frequency point.

[0062] The complex convolution operation can be implemented by two real-valued convolution kernels in a complex convolution layer. The complex convolution operation can be implemented according to the following equation (2):

[0063] Y1 r =X1 r *W r -X1 i *W i

[0064] Y1 i =X1 r *W i +X1 i *W r (2)

[0065] Among them, Y1 r Represents the real part feature of the convolutional layer output, Y1 i Represents the imaginary features of the convolutional layer output, W r represents the first convolution kernel, W i Represents the second convolution kernel, X1 r Represents the real part of the time-frequency spectrum, X1 i Represents the imaginary part of the time-frequency spectrum. The first convolution feature can be X1 r *W r , the second convolution feature can be X1 i *W r , the third convolution feature can be X1 r *W i , the fourth convolutional feature can be X1 i *W i .

[0066] Therefore, the complex features output by the convolutional layer can be represented as Y1 r +jY1 i .

[0067] The complex features output by the convolution layer may be further passed through a normalization layer and a rectified linear unit layer to finally output the first feature. For example, the normalization layer may perform complex batch normalization on the real and imaginary parts of the features output by the convolution layer, respectively, and then the rectified linear unit layer performs a complex nonlinear activation function (such as a complex parametric ReLU (PReLU)) on the output of the normalization layer to obtain the first feature.

[0068] For example, the acquisition of the first feature can be achieved by multiple complex convolution modules. Each complex convolution module may include the convolution layer, normalization layer and rectified linear unit layer described above. The convolution layer may include two different real-valued convolution kernels for simulating complex multiplication operations, the normalization layer may use traditional complex batch normalization, and the rectified linear unit layer may use the PReLU function. In the case of using multiple complex convolution modules, the multiple complex convolution modules can be connected in series and performed according to the above operations to obtain the final first feature.

[0069] In obtaining the features of the audio signal in the frequency domain, a first self-attention network of at least one frequency domain feature extraction module can be used to extract the real features and the imaginary features of the first feature, respectively, to obtain a first frequency-related feature and a second frequency-related feature; a second self-attention network of at least one frequency domain feature extraction module can be used to extract the real features and the imaginary features of the first feature, respectively, to obtain a third frequency-related feature and a fourth frequency-related feature; based on the first frequency-related feature and the fourth frequency-related feature, the real features of the features in the frequency domain are obtained; based on the second frequency-related feature and the third frequency-related feature, the imaginary features of the features in the frequency domain are obtained.

[0070] The first self-attention network and the second self-attention network can be implemented based on a multi-head self-attention network. For example, the output of the multi-head self-attention network can be calculated by the following equation.

[0071] First, the Q, K, and V matrices of each head are calculated. For example, the Q matrix of the hth head is calculated by the following equation (3): h , K h 、V h matrix.

[0072] Q h =XW Q

[0073] K h =XW K

[0074] V h =XW V (3)

[0075] Where X represents the input of the hth head, W Q , WK , W V Represent different learnable linear transformations, Q h , K h 、V h They represent the Q, K, and V matrices onto which X is projected, respectively, and h represents the head index.

[0076] Then, the output of the self-attention value of each head is calculated. For example, the self-attention value of the hth head is calculated by the following equation (4).

[0077]

[0078] Where k represents the projection dimension of the linear transformation.

[0079] Finally, the final output of the multi-head self-attention network is obtained through a feedforward neural network FFN. For example, the final self-attention value is calculated by the following equation (5).

[0080] MHSA(X)=FFN(Concat(Attn 1 ,Attn 2 ,…,Attn h )W M ) (5)

[0081] Among them, Attn 1 ,Attn 2 ,…,Attn h Represents the self-attention value of each head, Concat represents concatenation, and W M Represents a projection.

[0082] Two different multi-head self-attention networks can be used to simulate complex number operation rules to obtain the frequency domain characteristics of the audio signal, as shown in the following equation (6):

[0083] Y2 r =MHSA r (Y1 r )-MHSA i (Y1 i )

[0084] Y2 i =MHSA r (Y1 i )+MHSA i (Y1 r ) (6)

[0085] Among them, MHSA r and MHSA i Represent different multi-head self-attention calculations. The first frequency-related feature can be MHSAr (Y1 r ), the second frequency-related feature may be MHSA r (Y1 i ), the third frequency-related feature may be MHSA i (Y1 r ), the fourth frequency-related feature may be MHSA i (Y1 i ). The above-mentioned method of using a multi-head self-attention network to model frequency correlation is merely exemplary, and the present disclosure is not limited thereto.

[0086] In obtaining the features of the audio signal in the time domain and frequency domain directions, the first long short-term memory network of at least one time domain feature extraction module can be used to extract the real features and imaginary features of the features in the frequency domain direction, respectively, to obtain the first time-related features and the second time-related features; the second long short-term memory network of at least one time domain feature extraction module can be used to extract the real features and imaginary features of the features in the frequency domain direction, respectively, to obtain the third time-related features and the fourth time-related features; based on the first time-related features and the fourth time-related features, the real features of the features in the time domain and the frequency domain directions are obtained; based on the second time-related features and the third time-related features, the imaginary features of the features in the time domain and the frequency domain directions are obtained.

[0087] For example, two different long short-term memory networks can be used to simulate complex number operation rules to obtain the characteristics of the audio signal in the time domain and frequency domain, as shown in the following equation (7):

[0088] Y3 r =LSTM r (Y2 r )-LSTM i (Y2 i )

[0089] Y3 i =LSTM r (Y2 i )+LSTM i (Y2 r ) (7)

[0090] Among them, LSTM r and LSTM i Represent different long and short-term memory networks. The first time-related feature can be LSTM r (Y2 r ), the second time-dependent feature can be LSTM r (Y2 i ), the third time-dependent feature can be LSTM i (Y2 r), the fourth time-dependent feature can be LSTM i (Y2 i ). The above-mentioned method of using long short-term memory network to model time correlation is only exemplary, and the present disclosure is not limited thereto.

[0091] In step S103, the audio signal is encoded based on the characteristics in the time domain and the frequency domain.

[0092] As an example, the features of the audio signal in the time domain and frequency domain directions can be quantized based on the target bit rate to obtain quantized features of the features in the time domain and frequency domain directions; an index value corresponding to the quantized feature is determined from a code book; and the index value is encoded into a bit stream.

[0093] Figure 2 is a flowchart of an audio decoding method according to an embodiment of the present disclosure.

[0094] Reference Figure 2 In step S201, the time domain and frequency domain features of the audio signal are obtained from the bit stream. For example, the time domain and frequency domain features may be in the form of complex features.

[0095] As an example, an index value corresponding to a quantized feature of an audio signal can be obtained from a bit stream; the quantized feature can be determined from a code book based on the index value; the quantized feature can be inversely quantized to obtain complex features of the audio signal in the time domain and frequency domain directions, so as to decode the audio signal based on the complex features.

[0096] In step S202, a second artificial intelligence AI network is used to obtain a time-frequency spectrum of the audio signal based on the characteristics in the time domain and the frequency domain.

[0097] The second AI network can be implemented using a complex neural network. For example, the second complex neural network can be used to model the acquired complex features in the time direction and the frequency direction to obtain the time-frequency spectrum of the audio signal.

[0098] As an example, the second AI network may include at least one time domain feature extraction module, at least one transposed convolution module, and at least one frequency domain feature extraction module. Based on the features in the time domain and frequency domain directions, at least one time domain feature extraction module may be used to obtain the features in the frequency domain direction of the audio signal; based on the features in the frequency domain direction, at least one frequency domain feature extraction module may be used to obtain the first features of the audio signal; based on the first features, at least one transposed convolution module may be used to perform a transposed convolution operation to obtain the time-frequency spectrum of the audio signal. The operation at the decoding end may be regarded as the inverse operation of the operation at the encoding end.

[0099] In obtaining the features of the audio signal in the frequency domain direction, a first long short-term memory network of at least one time domain feature extraction module can be used to extract the real features and imaginary features of the features in the time domain direction and the frequency domain direction, respectively, to obtain a first time-related feature and a second time-related feature; a second long short-term memory network of at least one time domain feature extraction module can be used to extract the real features and imaginary features of the features in the time domain direction and the frequency domain direction, respectively, to obtain a third time-related feature and a fourth time-related feature; based on the first time-related feature and the fourth time-related feature, the real features of the features in the frequency domain direction are obtained; based on the second time-related feature and the third time-related feature, the imaginary features of the features in the frequency domain direction are obtained.

[0100] In obtaining the first feature of the audio signal, the first self-attention network of at least one frequency domain feature extraction module can be used to extract the real and imaginary features of the features in the frequency domain direction, respectively, to obtain the first frequency-related feature and the second frequency-related feature; the second self-attention network of at least one frequency domain feature extraction module can be used to extract the real and imaginary features of the features in the frequency domain direction, respectively, to obtain the third frequency-related feature and the fourth frequency-related feature; based on the first frequency-related feature and the fourth frequency-related feature, the real feature of the first feature is obtained; based on the second frequency-related feature and the third frequency-related feature, the imaginary feature of the first feature is obtained.

[0101] For example, the first self-attention network and the second self-attention network can be implemented using a multi-head self-attention network.

[0102] In obtaining the time-frequency spectrum of the audio signal, the first convolution kernel in the convolution layer of at least one transposed convolution module can be used to perform transposed convolution operations on the real and imaginary features of the first feature, respectively, to obtain the first convolution feature and the second convolution feature; the second convolution kernel in the convolution layer can be used to perform transposed convolution operations on the real and imaginary features of the first feature, respectively, to obtain the third and fourth convolution features; based on the first and fourth convolution features, the real features of the time-frequency spectrum of the audio signal are obtained; based on the second and third convolution features, the imaginary features of the time-frequency spectrum are obtained.

[0103] For example, each of the at least one transposed convolution module may be a complex transposed convolution module, and the convolution layer may be a complex convolution layer. The complex convolution layer may include different real-valued convolution kernels for use with complex multiplication operation rules.

[0104] The structure of the second AI network for decoding is symmetrical to the structure of the first AI network for encoding, and the operation in the second AI network can be understood as the inverse operation of the first AI network.

[0105] In step S203, the audio signal is obtained based on the time-frequency spectrum of the audio signal.

[0106] The obtained time-frequency spectrum can be inversely transformed to obtain an audio signal. As an example, the obtained time-frequency spectrum can be short-time inverse Fourier transformed to obtain an audio signal. In the case where the time-frequency spectrum is an amplitude-compressed time-frequency spectrum, the obtained time-frequency spectrum can be first restored to a time-frequency spectrum with an initial amplitude, and then short-time inverse Fourier transformed.

[0107] According to an embodiment of the present disclosure, a third AI network (such as a multi-scale neural network) can be further used to perform speech enhancement on the obtained audio signal. For example, Wave-U-Net can be used as a time domain enhancer to process the audio signal.

[0108] Figure 3 1 is a flow chart of an audio codec method according to an embodiment of the present disclosure. The following describes the audio codec method of the present disclosure by taking a complex convolutional network and complex features as an example.

[0109] Reference Figure 3 , the audio signal S to be encoded can first be subjected to short-time Fourier transform, and then power-law compression is used to limit the dynamic range of the amplitude of the time-frequency spectrum, so that the time-frequency spectrum with power-law compressed amplitude and original phase is used as the input of the encoder.

[0110] The encoder for encoding the audio signal S may include multiple complex convolution modules, complex multi-head self-attention modules, and complex long short-term memory modules. Specifically, the complex convolution module can model the local correlation of the time-frequency spectrum, while the complex multi-head self-attention module and the complex long short-term memory module can model the time-frequency spectrum in both frequency and time directions to capture the correlation in frequency and time, thereby better removing redundancy.

[0111] A complex convolution module may include a complex two-dimensional convolution layer, a complex batch normalization layer, and a complex ReLU layer. Each complex convolution module uses a complex two-dimensional convolution layer to downsample in the frequency dimension. The complex two-dimensional convolution layer contains two real-valued convolution kernels that are used to simulate the complex multiplication operation rules, thereby controlling the flow of complex information in the entire encoder. The complex convolution operation can be shown as in equation (2) above. The output of the complex two-dimensional convolution layer is processed by batch normalization and a nonlinear activation function, where the corresponding real-valued batch normalization and PReLU process the real and imaginary parts of the output respectively. For example, four stacked complex convolution modules can be used in the encoder.

[0112] The multi-head self-attention module can be applied to model the correlation between frequencies in each frame. For example, the above equation (6) can be used for calculation.

[0113] The complex long short-term memory module is applied to model the temporal dependency. For example, the above equation (7) can be used for calculation.

[0114] Assume that the input feature into the complex convolution module is F in ∈R T×F×2 , where T represents the number of frames, F represents the number of frequency points on the time spectrum, and 2 represents the real and imaginary dimensions of the time spectrum. After passing through the complex convolution module, the input features are downsampled, and the output features of the complex convolution module are F out ∈R T×F’×C , where F' represents the number of down-sampling frequency points and C represents the number of output channels.

[0115] Then, a vector quantizer (not shown) compresses the output features of the encoder at a target bit rate to obtain quantized features, performs feature mapping on the quantized features based on a codebook to obtain quantized index values, and finally encodes the index values ​​to obtain a bitstream.

[0116] The decoder and the encoder have a symmetrical structure and can reconstruct the audio signal from the quantized features. The decoder may include a complex long short-term memory module, a complex multi-head self-attention module, and multiple complex transposed convolution modules. Here, although both the encoder and the decoder include a complex long short-term memory module and a complex multi-head self-attention module, these two modules may have similar network structures in the encoder and decoder, but have different network parameters due to different functions. For example, the decoder first maps the quantization index value in the bit stream to the code vector in the code book to obtain the quantized features, and dequantizes the quantized features to obtain the complex features of the audio signal. After that, it passes through the complex long short-term memory module, then the complex multi-head self-attention module, and then the complex transposed convolution module, and then the decoded audio signal is obtained through the short-time inverse Fourier transform.

[0117] To further improve the audio quality, an optional time domain enhancer can be applied on the decoder side. Wave-U-Net can be used as a time domain enhancer to reduce the encoding and decoding distortion. For example, Wave-U-Net can be composed of 6 downsampling blocks and 6 upsampling blocks, and use skip connections to promote multi-scale feature interaction between the encoder and decoder. Audio signal after short-time inverse Fourier transform Audio enhancement can be performed through Wave-U-Net to obtain audio signals

[0118] Whether to apply the time domain enhancer can be selected according to the computing resources of the decoding end.

[0119] The encoder and decoder can be trained jointly. The first complex neural network described above corresponds to Figure 3The second complex neural network described above can correspond to Figure 3 The decoder.

[0120] Figure 3 The loss of the encoder-decoder network shown may include reconstruction loss and quantization constraint loss, where the reconstruction loss contains two components and They are calculated in the time-frequency domain as follows:

[0121]

[0122]

[0123] in, Respectively represent the real and imaginary parts of the compressed time-frequency spectrum of the input, Respectively represent the real and imaginary parts of the compressed time-frequency spectrum output by decoding, represents the time-frequency spectrum representation of the input, It represents the compressed time-frequency spectrum of the decoded output, n, k represent the frame number and frequency, N and K represent the total number of frames and frequency respectively.

[0124] Quantizing Constraint Loss The L2 loss of the encoder output and its gradient-free quantized value is used, and the codebook is updated using the exponential moving average (EMA) k-means method.

[0125] The total loss of the encoder-decoder network is Where λ is the weight coefficient of the quantized constraint loss, which can be set to 0.1, for example.

[0126] The loss of the time domain enhancer can be a multi-resolution short-time Fourier transform (STFT) loss and the time-domain L1 loss, where the multi-resolution STFT loss can be expressed as follows.

[0127]

[0128] in, represents the STFT loss in the m-th multi-resolution setting, s represents the original audio, N represents the total number of element points on the time spectrum at the window length resolution after the enhanced decoded audio, and M represents the total number of multiple resolutions. For example, in the case of three multiple resolutions, M=3, m is 1, 2 or 3, and the window lengths of the three resolutions can be 512, 1024, and 2048 respectively.

[0129] For each window length resolution, the STFT loss also includes the spectrum convergence loss and log spectral loss

[0130]

[0131]

[0132] in, Represents the enhanced decoded audio N represents the sum of the number of element points on the time-frequency spectrum at the window length resolution.

[0133] The total loss of the time domain enhancer is:

[0134]

[0135] Among them, γ 1 and γ 2 represents the weight coefficient for adjusting the two losses, which can be set to 0.9 and 0.1 respectively, for example. T represents the number of sampling points.

[0136] The parameters of the encoder and decoder are adjusted by minimizing the total loss of the above-mentioned codec network, and the parameters of the time domain enhancer are adjusted by minimizing the total loss of the above-mentioned time domain enhancer.

[0137] Refer to the following Figure 3 A specific audio codec example is described in detail.

[0138] The encoded audio has a sampling rate of 44100, and the input size of the encoder is (B, 1, S), where B represents the batch size and S represents the number of samples. After the STFT with a frame length of 40ms, a frame shift of 10ms, and an FFT size of 2048, the tensor size is (B, 2, F, T), where 2 represents the dimension of the real and imaginary parts, F represents the number of frequency points on the time-frequency spectrum, and T represents the number of frames. The STFT spectrum is then amplitude compressed, and the real and imaginary parts of the compressed STFT spectrum are: Where n and k represent the frame number and frequency point, β represents the compression coefficient, which is 0.5, and θ represents the phase of the time-frequency point. Then, four complex two-dimensional causal convolutional layers are passed, where the convolution kernel size of the convolutional layer is (7, 2) and the convolution step size is {(1, 1), (4, 1), (4, 1), (4, 1)}, which means that the time-frequency spectrum is downsampled in the frequency direction without changing the time resolution. The output channels are {16, 32, 64, 96}, where each convolutional layer is followed by a complex batch normalization layer and a complex parameterized rectified linear unit layer (PReLU), and the output size is (B, C, F, T), where C represents the number of channels in the current last convolution layer, and F represents the number of frequency points after downsampling. Then, two layers of complex multi-head self-attention layers are passed, where the number of heads of the complex multi-head self-attention is 8 and the model size is 96. The same frequency modeling is performed on each input frame without changing the input tensor size. Then, two layers of complex unidirectional long short-term memory network layers are passed through, in which the number of hidden units is 64, and the same time series modeling is performed for each frequency point of the input without changing the input tensor size. The shape of the tensor is then rearranged to (B, T, C*F / N, N), where N represents the number of groups of group quantization, and each group of features (B, T, C*F / N) is quantized to the index value of the code book. At the decoder end, the quantization index value of each group in the bit stream is first mapped to the code vector in the code book to obtain the quantized feature (B, T, C*F / N, N), which is then rearranged to (B, C, F, T). Then, two layers of complex unidirectional long short-term memory network layers are passed through, in which the number of hidden units is 64, and the same time series modeling is performed for each frequency point of the input without changing the input tensor size. Then it passes through two layers of complex multi-head self-attention layers, where the number of heads of the complex multi-head self-attention is 8 and the model size is 96. The same frequency modeling is performed on each frame of the input without changing the input tensor size. Then it passes through four complex two-dimensional causal transposed convolution layers, where the convolution kernel size of the convolution layer is (7, 2) and the convolution step size is {(4, 1), (4, 1), (4, 1), (1, 1)}, which means that the time-frequency spectrum is upsampled in the frequency direction without changing the time resolution. The output channels are {64, 32, 16, 2}. Each transposed convolution layer is followed by a complex batch normalization layer and a complex parameter corrected linear unit layer, and the output size is (B, 2, F, T). Finally, it is transformed to the time domain input (B, 1, S) through the ISTFT corresponding to the encoder end.

[0139] If the decoder has a time domain enhancer Wave-U-Net, the above decoder output (B, 1, S) is first passed through six one-dimensional convolutional layers, where the convolution kernel size of the convolutional layer is 15, the convolution step size is 1, and the number of output channels is {24, 48, 72, 96, 120, 144}, to obtain the output (B, C, S), where C represents the output channel of the last convolutional layer, and S represents the number of samples after downsampling. Then it passes through an intermediate convolutional layer, where the convolution kernel size is 15, the step size is 1, and the output channel remains unchanged at 144. Then upsampling is performed through six one-dimensional convolutions, where the convolution kernel size of the convolutional layer is 5. Since there is a jump connection with the previous convolutional layer, the number of output channels is {240, 192, 144, 96, 48, 1}, and the final output size is (B, 1, S). The above examples are merely exemplary, and the present disclosure is not limited to this.

[0140] According to an embodiment of the present disclosure, frequency-time modeling is performed using a complex multi-head self-attention module and a complex long short-term memory module. The complex multi-head self-attention module is applied to model the correlation between frequencies in each frame, while the complex long short-term memory module is applied to model time dependencies. The multi-head self-attention mechanism has excellent performance in sequence modeling. The present disclosure extends the multi-head self-attention structure to a complex-valued network. The output features of the complex convolution module on the frequency scale and the time scale are modeled respectively by the complex multi-head self-attention module and the complex long short-term memory module, and then the features extracted by the encoder are mapped to indexes based on the code book and written into the code stream. The symmetrical decoder also includes a complex multi-head self-attention module and a complex long short-term memory module, followed by a complex transposed convolution module for reconstructing the STFT spectrum.

[0141] In addition, in order to further improve the audio quality, the present disclosure also proposes to apply an optional time domain enhancer at the decoding end. Wave-U-Net can be used as a time domain enhancer to reduce coding distortion. Thanks to the above training strategy, it is possible to choose whether to apply the enhancer based on the computing resources of the decoding end.

[0142] Although the above embodiments are described based on complex neural networks and complex features, according to the inventive concept of the embodiments of the present disclosure, real-valued neural networks and real-valued features can also be used, and feature extraction and encoding and decoding of audio signals in the time domain and frequency domain can be implemented in a similar manner to the above.

[0143] An embodiment of the present disclosure also provides an electronic device, which includes a processor and, optionally, may also include at least one transceiver and / or at least one memory coupled to the at least one processor, wherein the at least one processor is configured to execute the steps of the method provided in any optional embodiment of the present disclosure.

[0144] Figure 4 A schematic diagram of the structure of an electronic device applicable to an embodiment of the present invention is shown in FIG. Figure 4 As shown, Figure 4 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, each of the processor 4001, the memory 4003 and the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present disclosure. Optionally, the electronic device may be a network node.

[0145] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0146] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0147] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0148] The memory 4003 is used to store computer programs or executable instructions for executing the embodiments of the present disclosure, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer programs or executable instructions stored in the memory 4003 to implement the steps shown in the above method embodiments.

[0149] An embodiment of the present disclosure provides a computer-readable storage medium having a computer program or instructions stored thereon. When the computer program or instructions are executed by at least one processor, the steps and corresponding contents of the aforementioned method embodiment can be executed or implemented.

[0150] The embodiments of the present disclosure also provide a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiments when executed by a processor.

[0151] The terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than that shown or described in the text.

[0152] It should be understood that, although the flowchart of the embodiment of the present disclosure indicates each operation step by arrows, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present disclosure, the implementation steps in each flowchart can be executed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times. In scenarios with different execution times, the execution order of these sub-steps or stages can be flexibly configured according to demand, and the embodiment of the present disclosure does not limit this.

[0153] The above text and drawings are provided only as examples to help readers understand the present disclosure. They are not intended and should not be interpreted as limiting the scope of the present disclosure in any way. Although certain embodiments and examples have been provided, based on the contents disclosed herein, it is obvious to those skilled in the art that the embodiments and examples shown can be changed without departing from the scope of the present disclosure, and other similar implementation means based on the technical ideas of the present disclosure are adopted, which also fall within the protection scope of the embodiments of the present disclosure.

Claims

1. An audio encoding method, characterized in that: The audio encoding method comprises: Obtaining the time-frequency spectrum of the audio signal; Using a first artificial intelligence AI network, extracting features in the time domain and frequency domain based on the time-frequency spectrum of the audio signal; The audio signal is encoded based on the characteristics in the time domain and the frequency domain.

2. The audio encoding method according to claim 1, characterized in that The first AI network includes at least one convolution module, at least one frequency domain feature extraction module, and at least one time domain feature extraction module; Using the first AI network, based on the time-frequency spectrum of the audio signal, extracting features in the time domain and the frequency domain, including: Based on the time-frequency spectrum, using the at least one convolution module to perform convolution processing to obtain a first feature of the audio signal; Based on the first feature, using the at least one frequency domain feature extraction module, obtaining a feature of the audio signal in the frequency domain direction; Based on the features in the frequency domain, the features in the time domain and the frequency domain of the audio signal are obtained using the at least one time domain feature extraction module.

3. The audio encoding method according to claim 2, characterized in that: Based on the time-frequency spectrum, using the at least one convolution module to perform convolution processing to obtain a first feature of the audio signal, including: Using a first convolution kernel in a convolution layer of the at least one convolution module to perform convolution operations on the real feature and the imaginary feature of the time-frequency spectrum respectively, to obtain a first convolution feature and a second convolution feature; Using the second convolution kernel in the convolution layer to perform convolution operations on the real part feature and the imaginary part feature of the time-frequency spectrum respectively, to obtain a third convolution feature and a fourth convolution feature; Based on the first convolution feature and the fourth convolution feature, obtaining a real feature of the first feature; An imaginary feature of the first feature is obtained based on the second convolution feature and the third convolution feature.

4. The audio encoding method according to claim 3, characterized in that: Each of the at least one convolution module is a complex convolution block, and the convolution layer is a complex convolution layer.

5. The audio encoding method according to claim 2, characterized in that: Based on the first feature, using the at least one frequency domain feature extraction module to obtain a feature of the audio signal in the frequency domain direction includes: Using the first self-attention network of the at least one frequency domain feature extraction module to extract the real feature and the imaginary feature of the first feature respectively, to obtain a first frequency-related feature and a second frequency-related feature; Using the second self-attention network of the at least one frequency domain feature extraction module to extract the real feature and the imaginary feature of the first feature respectively, to obtain a third frequency-related feature and a fourth frequency-related feature; Based on the first frequency-related feature and the fourth frequency-related feature, obtaining a real part feature of the feature in the frequency domain direction; Based on the second frequency-related feature and the third frequency-related feature, an imaginary feature of the feature in the frequency domain direction is obtained.

6. The audio encoding method according to claim 5, characterized in that: The first self-attention network and the second self-attention network use a multi-head self-attention network.

7. The audio encoding method according to claim 2, characterized in that: Based on the features in the frequency domain, using the at least one time domain feature extraction module, obtaining the features in the time domain and the frequency domain of the audio signal, including: Using a first long short-term memory network of the at least one time domain feature extraction module to extract the real part feature and the imaginary part feature of the feature in the frequency domain direction respectively, to obtain a first time-related feature and a second time-related feature; Using the second long short-term memory network of the at least one time domain feature extraction module to extract the real part feature and the imaginary part feature of the feature in the frequency domain direction respectively, to obtain a third time-related feature and a fourth time-related feature; Based on the first time-related feature and the fourth time-related feature, obtain the real part features of the features in the time domain direction and the frequency domain direction; Based on the second time-related feature and the third time-related feature, the imaginary part features of the features in the time domain direction and the frequency domain direction are obtained.

8. The audio encoding method according to claim 1, characterized in that: Encoding the audio signal based on the characteristics in the time domain and the frequency domain, comprising: quantizing the features in the time domain direction and the frequency domain direction based on a target bit rate to obtain quantized features of the features in the time domain direction and the frequency domain direction; Determine an index value corresponding to the quantized feature from a code book; The index value is encoded into a bitstream.

9. An audio decoding method, characterized in that: The audio decoding method comprises: Acquire the characteristics of the audio signal in the time domain and the frequency domain from the bit stream; Using a second artificial intelligence AI network, based on the characteristics in the time domain and the frequency domain, obtaining a time-frequency spectrum of the audio signal; The audio signal is obtained based on the time-frequency spectrum of the audio signal.

10. The audio decoding method according to claim 9, characterized in that: The second AI network includes at least one time domain feature extraction module, at least one transposed convolution module, and at least one frequency domain feature extraction module; Using a second artificial intelligence AI network, based on the characteristics in the time domain and the frequency domain, obtaining a time-frequency spectrum of the audio signal, including: Based on the features in the time domain and frequency domain, using the at least one time domain feature extraction module, obtaining features in the frequency domain of the audio signal; Based on the feature in the frequency domain direction, using the at least one frequency domain feature extraction module, obtaining a first feature of the audio signal; Based on the first feature, the at least one transposed convolution module is used to perform a transposed convolution operation to obtain a time-frequency spectrum of the audio signal.

11. The audio decoding method according to claim 10, characterized in that: Based on the features in the time domain and the frequency domain, using the at least one time domain feature extraction module to obtain the features in the frequency domain of the audio signal, comprising: Using the first long short-term memory network of the at least one time domain feature extraction module to extract the real features and the imaginary features of the features in the time domain direction and the frequency domain direction, respectively, to obtain a first time-related feature and a second time-related feature; Using the second long short-term memory network of the at least one time domain feature extraction module to extract the real features and the imaginary features of the features in the time domain direction and the frequency domain direction, respectively, to obtain a third time-related feature and a fourth time-related feature; Based on the first time-related feature and the fourth time-related feature, obtain a real part feature of the feature in the frequency domain direction; Based on the second time-related feature and the third time-related feature, an imaginary feature of the feature in the frequency domain direction is obtained.

12. The audio decoding method according to claim 10, characterized in that: Based on the feature in the frequency domain direction, using the at least one frequency domain feature extraction module, obtaining a first feature of the audio signal includes: Using the first self-attention network of the at least one frequency domain feature extraction module to extract the real part features and the imaginary part features of the features in the frequency domain direction respectively, to obtain a first frequency-related feature and a second frequency-related feature; Using the second self-attention network of the at least one frequency domain feature extraction module to extract the real part features and the imaginary part features of the features in the frequency domain direction respectively, to obtain a third frequency-related feature and a fourth frequency-related feature; Based on the first frequency-related feature and the fourth frequency-related feature, obtaining a real feature of the first feature; An imaginary feature of the first feature is obtained based on the second frequency-related feature and the third frequency-related feature.

13. The audio decoding method according to claim 12, characterized in that: The first self-attention network and the second self-attention network use a multi-head self-attention network.

14. The audio decoding method according to claim 10, characterized in that: Based on the first feature, using the at least one transposed convolution module to perform a transposed convolution operation to obtain a time-frequency spectrum of the audio signal, including: Using a first convolution kernel in a convolution layer of the at least one transposed convolution module to perform a transposed convolution operation on the real feature and the imaginary feature of the first feature, respectively, to obtain a first convolution feature and a second convolution feature; Using the second convolution kernel in the convolution layer to perform transposed convolution operations on the real feature and the imaginary feature of the first feature, respectively, to obtain a third convolution feature and a fourth convolution feature; Based on the first convolution feature and the fourth convolution feature, obtaining a real part feature of the time-frequency spectrum; Based on the second convolution feature and the third convolution feature, an imaginary part feature of the time-frequency spectrum is obtained.

15. The audio decoding method according to claim 14, characterized in that: Each of the at least one transposed convolution module is a complex transposed convolution module, and the convolution layer is a complex convolution layer.

16. The audio decoding method according to claim 9, characterized in that: The time domain and frequency domain features of the audio signal are obtained from the bitstream, including: Obtaining an index value corresponding to a quantized feature of the audio signal from a bit stream; Determining the quantitative feature from a codebook based on the index value; The quantized features are inversely quantized to obtain features of the audio signal in the time domain and frequency domain directions.

17. The audio decoding method according to claim 9, further comprising: The obtained audio signal is subjected to speech enhancement using a third artificial intelligence (AI) network.

18. An electronic device, comprising: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to perform the method of any one of claims 1 to 17.

19. A computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by at least one processor, the at least one processor is prompted to perform the method according to any one of claims 1 to 17.

Images