Decoding method compatible with low-bitrate encoder, ai vocoder and related apparatus

Abstract

The application discloses a decoding method compatible with a low-code-rate encoder, an AI vocoder and related devices, wherein the method comprises adding noise reduction before the encoder, performing preprocessing and normalization operation on the audio coding parameters extracted from the bit stream of the low-code-rate, sequentially performing convolution, upsampling processing, feature extraction and nonlinear transformation processing, and converting the data obtained after the feature extraction and nonlinear transformation processing into an audio time domain signal. The application also provides an AI vocoder (NAED) compatible with multiple low-code-rate encoders. The vocoder can be compatible with AMBE, NVOC, Codec2, MELPe and other encoders. The application improves the sound quality through upsampling processing, and uses causal convolution instead of ordinary convolution, thereby reducing the additional delay while ensuring the field of view. The vocoder in the application can also spread the output of the low-code-rate encoder to realize 16K sampling rate audio synthesis.

Description

Technical Field

[0001] This application relates to the field of low bit rate audio encoding and decoding technology, and in particular to a decoding method compatible with low bit rate encoders, an AI vocoder, and related devices. Background Technology

[0002] A low bitrate audio vocoder is a set of technical devices and algorithms that extract and encode the core feature parameters of an audio signal to achieve audio transmission and reconstruction at extremely low data rates (usually below 16kbps, and even below 1kbps).

[0003] Traditional low-bitrate audio vocoders include various types, such as AMBE (Advanced Multi-Band Excitation), NVOC (Narrowband Voice Coder), Codec2, and MELPe (Mixed Excitation Linear Prediction, enhanced). While traditional low-bitrate audio vocoders can meet the needs of low-bitrate scenarios, they generally suffer from poor synthesized audio quality and a strong mechanical feel in the synthesized audio. Summary of the Invention

[0004] In view of the above problems, this application provides a decoding method, AI vocoder, and related apparatus compatible with low bitrate encoders to improve the quality of synthesized audio. The specific solution is as follows:

[0005] The first aspect of this application provides a decoding method compatible with low bit rate encoders, comprising:

[0006] The front-end module performs preprocessing and normalization operations on the audio encoding parameters extracted from the low bitrate audio bitstream. The audio encoding parameters include at least one of the following: the fundamental frequency of the current audio frame, the number of sub-band spectra of the current audio frame, the audio / non-audio decision of each sub-band spectra, and the amplitude of each sub-band spectra.

[0007] The synthesizer model built on a neural network convolves the audio coding parameters after preprocessing and normalization. The data processed during the convolution includes the data of the current audio frame and the data of the adjacent audio frames, which are located after the current audio frame.

[0008] The synthesizer model upsamples the data obtained after convolution in the audio frame dimension.

[0009] The synthesizer model performs feature extraction and nonlinear transformation on the data obtained after upsampling.

[0010] The synthesizer model converts the data obtained after feature extraction and nonlinear transformation into audio time-domain signals.

[0011] In one possible implementation, the synthesizer model includes an input layer, an intermediate layer, and an output layer, wherein the intermediate layer includes at least one convolutional next-generation module layer ConvNeXt Block, and each ConvNeXt Block includes a depthwise convolutional layer, a first pointwise convolutional layer, an activation layer, a second pointwise convolutional layer, and a weight parameter layer.

[0012] The synthesizer model performs feature extraction and nonlinear transformation on the data obtained after upsampling, including:

[0013] The deep convolutional layer uses causal convolution to perform spatial convolution on the data obtained after upsampling;

[0014] The first pointwise convolutional layer performs pointwise convolution on the data obtained after spatial convolution;

[0015] The activation layer performs a non-linear transformation on the data obtained after pointwise convolution;

[0016] The second pointwise convolutional layer performs pointwise convolution on the data obtained after nonlinear transformation;

[0017] The weight parameter layer multiplies the data obtained after pointwise convolution with the learnable weight parameters to obtain a product.

[0018] In one possible implementation, the front-end module performs preprocessing and normalization operations on the audio encoding parameters extracted from the low-bitrate audio bitstream, including:

[0019] The front-end module fills the subband spectrum of each audio frame in the audio encoding parameters extracted from the low bitrate audio bitstream into the target quantity, and fills the amplitude of the filled subband spectrum with a preset value. It then uses logarithms to perform dynamic range compression on the amplitude of the subband spectrum of each audio frame and the fundamental frequency of each audio frame in the filled audio encoding parameters, and normalizes the compressed data.

[0020] And / or, the front-end module generates the linear frequency amplitude spectrum of each audio frame based on the audio encoding parameters extracted from the low bitrate audio bitstream using a sine wave model, filters the linear frequency amplitude spectrum through a Mel filter bank, takes the logarithm of the filtered energy to obtain Mel spectral features, and normalizes the Mel spectral features.

[0021] In one possible implementation, the front-end module generates a linear frequency amplitude spectrum for each audio frame based on audio coding parameters extracted from a low-bitrate audio bitstream, using a sine wave model, including:

[0022] The front-end module generates an ideal impulse sequence based on the fundamental frequency F0 extracted from the low bit rate audio bitstream, and convolves it with the Sinc kernel to obtain the voiced amplitude spectrum.

[0023] The front-end module shapes the white noise based on the amplitude M of the subband spectrum extracted from the low bit rate audio bitstream to obtain the unvoiced amplitude spectrum.

[0024] The front-end module adds the voiced amplitude spectrum to the unvoiced amplitude spectrum to obtain the linear frequency amplitude spectrum of each audio frame.

[0025] One possible implementation also includes:

[0026] The audio signal obtained by the microphone is denoised using a neural network-based noise reduction module, and the denoised audio signal is then input into the encoder.

[0027] The training data used when training the synthesizer model includes the data denoised by the neural network-based denoising module.

[0028] In one possible implementation, the low bitrate audio bitstream is an audio bitstream generated by any one of the AMBE encoder, NVOC encoder, Codec2 encoder, and MELPe encoder.

[0029] A second aspect of this application provides an AI vocoder, comprising: a front-end module and a synthesizer model built on a neural network, wherein the AI ​​vocoder is configured to perform a decoding method of a low-bitrate encoder compatible with the first aspect or any implementation thereof.

[0030] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the decoding method of a compatible low bit rate encoder as described in the first aspect or any implementation thereof.

[0031] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0032] The memory is used to store computer programs;

[0033] The processor is used to execute the computer program so that the electronic device can implement the decoding method of the compatible low bit rate encoder of the first aspect or any implementation thereof.

[0034] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs that, when executed by an electronic device, enable the electronic device to implement a decoding method compatible with a low bit rate encoder, as described in the first aspect or any implementation thereof.

[0035] Using the above technical solutions, the decoding method, AI vocoder, and related devices compatible with low bitrate encoders provided in this application preprocess and normalize the audio encoding parameters extracted from the low bitrate audio bitstream, and then sequentially perform convolution, upsampling, feature extraction, and nonlinear transformation processing. The data obtained after feature extraction and nonlinear transformation processing is then converted into an audio time-domain signal. This application upsamples the data obtained after convolution along the audio frame dimension, resulting in a larger number of audio frames per unit time, thereby improving sound quality and reducing mechanical sounds. The data processed during convolution includes the current frame and the next frame, eliminating the need to pad the right side during feature extraction and nonlinear transformation processing, effectively reducing the additional latency caused by padding the right side. Furthermore, this application is compatible with traditional encoders such as AMBE encoders, NVOC encoders, Codec2 encoders, and MELPe encoders. Attached Figure Description

[0036] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0037] Figure 1 A flowchart illustrating a decoding method compatible with a low bit rate encoder, provided for an embodiment of this application;

[0038] Figure 2 This is a schematic diagram of the process for feature extraction and nonlinear transformation of data obtained after upsampling, provided in an embodiment of this application.

[0039] Figure 3 This is a flowchart illustrating the preprocessing and normalization of audio encoding parameters extracted from a low-bitrate audio bitstream, as provided in an embodiment of this application.

[0040] Figure 4 This is a schematic diagram illustrating another process for preprocessing and normalizing audio encoding parameters extracted from a low-bitrate audio bitstream, provided as an embodiment of this application.

[0041] Figure 5 This is a schematic diagram of the structure of an AI vocoder provided in an embodiment of this application;

[0042] Figure 6 This is a schematic diagram of another AI vocoder provided in an embodiment of this application;

[0043] Figure 7 A schematic diagram of the structure of another AI vocoder provided in the embodiments of this application;

[0044] Figure 8 This is a schematic diagram of the structure of another AI vocoder provided in an embodiment of this application;

[0045] Figure 9 A schematic diagram of the structure of a ConvNeXt Block provided in an embodiment of this application;

[0046] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0047] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0048] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0049] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0050] like Figure 1 As shown in the embodiments of this application, a decoding method compatible with low bit rate encoders may include:

[0051] S100, the front-end module performs preprocessing and normalization operations on the audio encoding parameters extracted from the low bitrate audio bitstream.

[0052] Optionally, the decoding method compatible with low bit rate encoders provided in this application embodiment can be executed by an AI vocoder, which may include a front-end module and a synthesizer model.

[0053] The low bitrate audio bitstream can be an audio bitstream generated by any of the AMBE encoder, NVOC encoder, Codec2 encoder, or MELPe encoder.

[0054] The audio coding parameters include at least one of the following: the fundamental frequency of the current audio frame, the number of sub-band spectra of the current audio frame, the audio / non-audio decision for each sub-band spectra, and the amplitude of each sub-band spectra. For example, the audio coding parameters of AMBE include F0, L, UV, and M, where F0 is the fundamental frequency of the current audio frame (generally the angular frequency in AMBE); L is the number of sub-band spectra of the current audio frame, ranging from 0 to 55; UV is the audio / non-audio decision for each sub-band spectra, with UV=0 for unvoiced (non-speech) and UV=1 for voiced (speech). The audio / non-audio decision for each sub-band spectra can be represented as UV1, UV2...UV L M represents the amplitude of each sub-band spectrum, and the amplitude of each sub-band spectrum can be represented as M1, M2, ..., M. L For example, Codec2's audio encoding parameters include F0, L, UV, and LogM. The meanings of F0, L, and UV are the same as those in AMBE's audio encoding parameters, but Codec2's L ranges from 0 to 80; LogM is the amplitude of each subband's spectrum. MELPe's audio encoding parameters include F0, L, UV, fs_mag, and lpc. The meanings of F0, L, and UV are the same as those in AMBE's audio encoding parameters. Optionally, MELPe's L value is fixed at 10. The fs_mag (Fourier series amplitude) parameter is a set of parameters in the MELPe vocoder used for fine-tuning the amplitude of voiced harmonics, similar to spectral amplitude; lpc belongs to linear predictive coding and is used for linear filtering of the generated excitation signal.

[0055] The preprocessing in this application can transform and remap the extracted audio coding parameters to generate feature representations that are more suitable for subsequent processing steps.

[0056] Optionally, this application may use different preprocessing methods for the extracted audio encoding parameters of different encoding mechanisms, which will be described in detail in subsequent embodiments. If the audio encoding parameters of a certain encoding mechanism also include other parameters, these other parameters can also be used as features input to the synthesizer model.

[0057] Normalization adjusts the distribution of input data to a stable and standardized range, which facilitates subsequent processing steps.

[0058] S200. The synthesizer model based on the neural network convolves the audio coding parameters after preprocessing and normalization. The data processed during convolution includes the data of the current audio frame and the data of the adjacent audio frames, which are located after the current audio frame.

[0059] Steps S200 to S500 can be executed using a synthesizer model built based on a neural network, such as... Figure 5 As shown, the synthesizer model 200 includes an input layer 210, an intermediate layer 220, and an output layer 230. The input layer 210 includes a first convolutional layer 211 and a transposed convolutional layer 212. The first convolutional layer 211 executes step S200, the transposed convolutional layer 212 executes step S300, the intermediate layer 220 executes step S400, and the output layer 230 executes step S500.

[0060] The first convolutional layer can be as follows: Figure 8 The 3×1Conv1d shown can be followed by LayerNorm and ReLU layers, with the 3×1Conv1d having a right view frame.

[0061] In step S200, convolution operations can be used to capture short-term dynamic patterns in the audio frame sequence. By analyzing the joint information of the current frame and its subsequent frames, a more stable feature representation containing richer contextual information is generated, thereby improving the quality and naturalness of the final synthesized audio. Step S200 can utilize convolution operations performed through a neural network.

[0062] This application processes data from both the current audio frame and the next audio frame during convolution, effectively adding a right-side view to capture more complete short-term contextual information. This significantly improves the accuracy and stability of parameter estimation for the current frame, ultimately enhancing the quality of the output audio. This is an effective strategy that trades minimal processing latency for a substantial improvement in audio quality. Furthermore, it eliminates the need to pad the right-side data (i.e., the audio frames following the current one) during feature extraction and nonlinear transformation, effectively reducing the additional latency caused by padding the right-side data.

[0063] Optionally, the data processed during convolution in this application includes the data of the current audio frame and the data of the N audio frames following the current audio frame, where N is not less than 2. As N increases, this application can obtain more complete short-term contextual information and achieve higher output audio quality, but it also leads to a linear increase in system overhead and complexity. The value of N can be set according to actual needs in practical applications.

[0064] The S300 synthesizer model upsamples the data obtained after convolution along the audio frame dimension.

[0065] The transposed convolutional layer can be, for example, as follows: Figure 8 The diagram shows a 4×1 TransposeConv, which can be followed by LayerNorm and ReLU layers. TransposeConv is primarily used for upsampling in the frame dimension.

[0066] In this application, the step size is set to 2 and the kernel size is set to 4 (set to 4 to avoid the chessboard effect).

[0067] Step S300, by upsampling along the audio frame dimension, can effectively improve temporal resolution (from low frame rate to high frame rate), thereby improving sound quality. For example, the data obtained after convolution is one frame every 20ms along the audio frame dimension. For speech, some vowels change rapidly, and one frame every 20ms is insufficient to represent the rapid changes of some vowels; therefore, by using the upsampling method, the frame length can be shortened to one frame every 10ms, thus improving temporal resolution.

[0068] The S400 synthesizer model performs feature extraction and nonlinear transformation on the data obtained after upsampling.

[0069] Step S400 can process the high frame rate coarse-grained feature sequence obtained after upsampling in step S300 to improve the quality of the feature sequence. Specifically, step S400 can significantly improve the discriminative ability and expressiveness of the features while maintaining high temporal resolution, enabling the subsequent waveform generation module to synthesize more fidelity and natural speech.

[0070] Optional, such as Figure 5 and Figure 6 As shown, the synthesizer model includes: an input layer 210, an intermediate layer 220, and an output layer 230. The intermediate layer 220 includes at least one convolutional next-generation module layer, ConvNeXt Block (e.g., ...). Figure 6 The three convolutional next-generation module layers shown are the first, second, and third convolutional next-generation module layers. Each ConvNeXt Block includes: a depthwise convolutional layer, a first pointwise convolutional layer, an activation layer, a second pointwise convolutional layer, and a weight parameter layer, as shown. Figure 2 As shown, step S400 may specifically include S410 to S450:

[0071] S410, the deep convolutional layer uses causal convolution to perform spatial convolution on the data obtained after upsampling.

[0072] Optionally, the spatial convolution described above can be depthwise convolution. The S410 can perform depthwise convolution using a two-dimensional convolution kernel; specifically, each input channel is independently convolved with a dedicated convolution kernel. Assuming the input data size is [Batch, Time, Channels], the depthwise convolution kernel does not sum across channels, and the number of output channels is the same as the number of input channels. Causal convolution ensures that the convolution kernel only covers the current time point and previous time points, guaranteeing that the output does not depend on future information, which is crucial for real-time streaming processing (such as real-time audio streaming).

[0073] Step S410 uses spatial convolution to independently extract short-term, localized temporal dynamic patterns (such as subtle pitch variations and energy fluctuations in speech) from the feature sequences of each channel. Because no channel mixing is performed, the computational efficiency is extremely high.

[0074] S420, the first pointwise convolutional layer performs pointwise convolution on the data obtained after spatial convolution.

[0075] Pointwise convolution can be performed using a 1×1 kernel. Pointwise convolution can mix and combine information from all channels of the spatial convolution output (linear projection). Step S420 can increase, decrease, or maintain the number of channels and learn the optimal weighted combination of features from different channels.

[0076] Step S420 can fuse cross-channel information to achieve feature projection. After spatial convolution, although the features of each channel contain temporal information, they are independent of each other. The role of pointwise convolution is to break this independence. By calculating the weighted sum of all channels, new channel features are generated, thereby fusing information from different sources and greatly enhancing the representational power of the features.

[0077] The combination of steps S410 and S420 (spatial convolution + pointwise convolution) significantly reduces the number of parameters and computational cost while maintaining strong feature extraction capabilities.

[0078] S430, the activation layer performs a nonlinear transformation on the data obtained after pointwise convolution.

[0079] Step S430 independently transforms each value in the feature map of the pointwise convolution output using a function, such as a non-linear activation function like ReLU, GELU, or Swish. Step S430 can introduce non-linearity to enhance the expressive power of the synthesizer model, enabling the network to simulate complex patterns in the real world.

[0080] S440, the second pointwise convolutional layer performs pointwise convolution on the data obtained after nonlinear transformation.

[0081] Step S440 also uses a 1×1 convolution kernel for convolution operations. Step S440 can further refine and compress the features after non-linear activation to further fuse features and adjust the number of channels to match the final output target dimension. Step S440 can increase the depth of the network, thereby improving its learning ability.

[0082] S450, the weight parameter layer multiplies the data obtained after pointwise convolution with the learnable weight parameters to obtain the product.

[0083] Step S450 multiplies each channel of the feature map output from step S440 by a learnable scalar weight. The network automatically learns a weight vector during training that evaluates the importance of each channel's feature. Important feature channels are amplified, while less important feature channels are suppressed. This allows the network to adaptively emphasize information-rich features and suppress less information-rich features, thereby optimizing the quality of the final output.

[0084] The first, second, and third convolutional next-generation module layers have the same structure and initial parameters, such as... Figure 6 As shown, the input data of the first convolutional next-generation module layer is the data output by the input layer, the input data of the second convolutional next-generation module layer is the data output by the first convolutional next-generation module layer, and the input data of the third convolutional next-generation module layer is the data output by the second convolutional next-generation module layer.

[0085] The first convolutional next-generation module layer, the second convolutional next-generation module layer, and the third convolutional next-generation module layer are... Figure 8 The three ConvNeXt Blocks shown are modified versions of the standard ConvNeXt architecture, and their detailed structures are as follows: Figure 9 As shown. Figure 9 As shown, the ConvNeXt Block of this application has an inverted bottleneck structure in terms of dimensions, that is, from narrow (256) to wide (768) and then back to narrow (256).

[0086] like Figure 9As shown, the ConvNeXt Block of this application includes one depthwise convolution (i.e., the depthwise convolutional layer mentioned above) and two pointwise convolutions (i.e., the first and second pointwise convolutional layers mentioned above); the depthwise convolution dimension is 256×3×1; in the standard ConvNeXt structure, kernel=7 and pad=3, but in this application, kernel=3 and pad=0 (i.e., causal convolution). Setting pad=0 avoids the additional latency caused by the pad. Since a right view is added in the first convolutional layer, it is not necessary to look further at this point; the two convolutions after the depthwise convolution are both pointwise convolutions. A LayerNorm layer is inserted between the depthwise convolution and the first pointwise convolution (i.e., the first pointwise convolutional layer mentioned above) for normalization and to stabilize the numerical range. An activation layer is inserted between the two pointwise convolutions. In the standard ConvNeXt architecture, Gelu is used, but since this activation function is not conducive to practical inference implementation, especially in embedded devices, it is modified to a Relu function in this application. The learnable weight parameter gamma (with a length of 256) used after the second pointwise convolution is multiplied with the convolution output. Finally, a residual (Res) connection is made between the output and the input.

[0087] Optionally, the convolution dimension of the depthwise convolutional layer can be 256×3×1, the convolution dimension of the first pointwise convolutional layer can be 768×1×1, and the convolution dimension of the second pointwise convolutional layer can be 256×1×1.

[0088] Optionally, a normalization layer may be provided between the depth convolutional layer and the first pointwise convolutional layer, and / or the activation function used by the activation layer may be the ReLU function.

[0089] The output data of the first, second, and third convolutional next-generation module layers are all the sum of the product and the input data.

[0090] S410 to S450 sequentially perform a series of operations: timing modeling, channel fusion, nonlinear activation, secondary refinement, and importance calibration. This transforms the coarse features obtained from upsampling into a high-quality driving signal that is temporally smooth, information-rich, and weighted by importance, laying the foundation for generating high-fidelity audio waveforms in the backend. It maximizes computational efficiency while maintaining performance, making it ideal for real-time audio synthesis tasks.

[0091] The S500 synthesizer model converts the data obtained after feature extraction and nonlinear transformation into audio time-domain signals.

[0092] Step S500 can first map the data obtained after feature extraction and nonlinear transformation to the dimension corresponding to the number of frequency domain points, and convert it into a frequency domain signal, and then convert the frequency domain signal into an audio time domain signal.

[0093] Optionally, this application may send the audio time-domain signal to a speaker for playback.

[0094] The decoding method compatible with low-bitrate encoders provided in this application upsamples the data obtained after convolution in the audio frame dimension, resulting in a larger number of audio frames per unit time, thereby improving sound quality and reducing mechanical sound. The data processed during convolution in this application includes the current frame and the next frame, eliminating the need to pad the right side of the data during feature extraction and nonlinear transformation, effectively reducing the additional latency caused by padding the right side. Simultaneously, this application can process the audio encoding parameters extracted from the low-bitrate audio bitstream to obtain the audio time-domain signal. Therefore, it can process the audio bitstream output by traditional encoders to obtain the audio time-domain signal, making this application compatible with traditional encoders. Thus, when upgrading a traditional codec requires upgrading the traditional decoder, only the traditional decoder needs to be upgraded to a decoder using the decoding method compatible with low-bitrate encoders provided in this application, without needing to upgrade the traditional encoder itself.

[0095] Optional, such as Figure 3 As shown, in another decoding method compatible with low bit rate encoders provided in this application embodiment, step S100 may specifically include:

[0096] S111: The front-end module fills the subband spectrum of each audio frame in the audio encoding parameters extracted from the low bitrate audio bitstream into the target quantity, and fills the amplitude of the filled subband spectrum with preset values.

[0097] In various audio coding methods, the number of subband spectra L per frame is not fixed. Therefore, this application first pads the different subband spectra to the same number. For example, the AMBE standard specifies a maximum number of 56, and this application uniformly pads the number of subbands to 56. The specific padding method is as follows:

[0098] ;

[0099] in, For the UV decision of the i-th subband extracted after dequantization, The value is the filled value. In the standard AMBE parameters, a UV decision result of 0 indicates a voiceless sound (non-speech), and a UV decision result of 1 indicates a voiced sound (speech). Therefore, for the filled UV, this application uses -1 as the marker instead of directly filling it with 0.

[0100] Accordingly, this application also fills the amplitude of the filled subband spectrum with a preset value to avoid the amplitude of the filled subband spectrum being empty. The preset value can be 0.

[0101] For example, the amplitude of the filled AMBE standard subband spectrum can be filled using a preset value (such as 0), as shown in the formula below:

[0102] .

[0103] in, The amplitude of the spectrum of the i-th sub-band after padding. L represents the amplitude of the i-th subband spectrum before padding, and L is the number of subband spectra.

[0104] By filling the subband spectrum quantity and amplitude of the subband spectrum as described above, this application can fill the subband spectrum quantity of each audio frame to the same quantity and fill it with a preset value, thereby standardizing non-fixed-length and irregular input data into fixed-length and regular data, thus providing a stable and consistent input format for subsequent steps.

[0105] S112. The front-end module uses the amplitude of the subband spectrum of each audio frame and the fundamental frequency of each audio frame after the audio coding parameters are filled with logarithmic pairs to perform dynamic range compression.

[0106] Step S112 can map audio parameters with exponential or power-law distribution characteristics to a linear range that better matches the characteristics of human auditory perception, thereby significantly improving the statistical distribution of parameters and enhancing the numerical stability and quantization efficiency of subsequent processing.

[0107] The fundamental frequency F0 and spectral amplitude M of audio typically fluctuate significantly (e.g., M may range from 0.0 to 1000.0). Directly inputting these as features into the synthesizer model would lead to unstable training and highly volatile values, hindering training. Therefore, this application employs dynamic range compression. Specifically, since the human ear's perception of audio energy is generally logarithmically proportional, logarithmic compression is used. For example, this application can compress the spectral amplitude M using the following formula:

[0108] .

[0109] in, The spectral amplitude of the k-th subband after compression. This represents the spectral amplitude of the k-th subband before compression.

[0110] The formula for compressing the fundamental frequency F0 is similar and will not be repeated here.

[0111] Optionally, after the above dynamic range compression process, the obtained features can be combined into a vector Fea=[F0,L,UV1,UV2,...UV 56 ,M1,M2,...M 56 The length of this vector is 114.

[0112] S113. The front-end module performs normalization processing on the compressed data.

[0113] Step S113 can further standardize the log-compressed data to a normal distribution with a mean of zero and a standard deviation of one, thereby eliminating systematic biases caused by different speakers, recording conditions and audio content, providing standardized input for all subsequent processing modules, and ensuring the consistency and robustness of the system.

[0114] Optionally, step S113 can also normalize the vector before feeding it into the synthesizer model using the following formula:

[0115] ;

[0116] Where mean is the vector's mean, std is the vector's standard deviation, and the dimension is 114. It is a vector after normalization. It is the vector before normalization.

[0117] Steps S111 to S113 can directly utilize the audio encoding parameters extracted from the low bitrate audio bitstream to transform them into a normalized feature vector suitable for neural network processing, avoiding the introduction of new errors or information loss during feature transformation. The overall scheme has high coupling and a simple design. Figure 3 The steps shown preserve the audio generation model information inherent in the original vocoder parameters to the greatest extent possible. The entire process involves only simple operations such as padding, logarithmizing, and normalization, making it suitable for ultra-low latency real-time communication scenarios (such as voice calls). It has low requirements for processor computing power and is easy to deploy in embedded devices or mobile terminals. Furthermore, its output is a fixed-length vector (e.g., 114-dimensional), with compact data volume, reducing the computational burden on subsequent neural networks, lowering model complexity and memory usage, and facilitating lightweight and efficient inference.

[0118] Optional, such as Figure 4 As shown, in another decoding method compatible with low bit rate encoders provided in this application embodiment, step S100 may specifically include:

[0119] S121. The front-end module generates the linear frequency amplitude spectrum of each audio frame based on the audio encoding parameters extracted from the low bit rate audio bit stream using a sine wave model.

[0120] The sine wave model is a strongly constrained model based on acoustic principles. During the reconstruction process, it uses physical rules (such as harmonic structures) to generate a well-structured spectrum, which has a certain corrective and smoothing effect on the quantization noise and extraction error of audio coding parameters. It helps to reduce artificial noise in synthesized audio and improve the robustness of the system under low bit rate or channel error conditions.

[0121] Optionally, step S121 may specifically include S121a, S121b, and S121c. These are explained in detail below:

[0122] S121a. Generate an ideal impulse sequence based on the fundamental frequency F0 extracted from the low bit rate audio bitstream, and convolve it with the Sinc kernel to obtain the voiced amplitude spectrum.

[0123] The ideal impact sequence can be generated according to the following formula. :

[0124] k=0,1,2...N / 2.

[0125] Where k is the spectral index corresponding to the 512-point FFT, ranging from 0 to 256; N is the number of FFT points, 512; and F0 is the fundamental frequency of the audio frame. For harmonic indexing; For the first The spectral amplitude of each harmonic, UV is the speech / non-speech decision; For the first Speech / non-speech decision based on harmonics Let fs / N be the sampling frequency. For an 8kHz sampling rate, a 512-point FFT... .

[0126] The reason for convolving the ideal impulse sequence with the Sinc kernel in this application is that the spectrum of voiced tones consists of a series of discrete harmonic peaks, the shape of which is affected by a window function (or frame truncation) and presents the shape of the Sinc function. Therefore, multiplying by the Sinc kernel can more realistically simulate the truncation effect.

[0127] Optionally, the formula for the Sinc convolution kernel can be as follows:

[0128] ;

[0129] in, Here, is the Sinc function (also known as the Singer function), and m is the sampling point index corresponding to each time domain point of the convolution kernel. As shown in the formula above, m is symmetrically distributed around 0, ranging from -25 to 25, for a total of 51. The kernel length is generally set to an odd number, and in the formula above, the kernel length is set to 51.

[0130] The specific formula for calculating convolution is as follows:

[0131] ;

[0132] in, This is the amplitude spectrum of voiced sounds.

[0133] S121b: The white noise is shaped based on the sub-band amplitude M extracted from the low bit rate audio bitstream to obtain the unvoiced amplitude spectrum.

[0134] This application can first generate white noise that follows a standard normal distribution and has a sequence length N of 256. Then, the noise within the corresponding subband at the specified frequency point is calculated using the following formula. And it is used as the unvoiced amplitude spectrum:

[0135] ;

[0136] Where k is the frequency index corresponding to the 512-point FFT spectrum, and its value ranges from 0 to 256; n is a random number from 0 to 256. This is the nth sampling point in a standard normal white noise sequence of length 256. For the first The spectral amplitude of each harmonic.

[0137] S121c: Add the amplitude spectrum of voiced sounds to the amplitude spectrum of unvoiced sounds to obtain the linear frequency amplitude spectrum of each audio frame.

[0138] This application can use the following formula to add the amplitude spectrum of voiced sounds to the amplitude spectrum of unvoiced sounds:

[0139] ;

[0140] Among them, X k The frequency amplitude spectrum is linear, and N is the number of FFT points.

[0141] S122. The front-end module filters the linear frequency amplitude spectrum through a Mel filter bank and takes the logarithm of the filtered energy to obtain the Mel spectrum characteristics.

[0142] Mel spectrum is a rich acoustic representation of speech content, containing a wealth of information such as phase coherence, harmonic structure, and spectral envelope details. Mel spectrum provides neural networks with a richer and more accurate feature representation of the original speech waveform, reducing the difficulty of learning the mapping from parameters to waveforms and enabling the synthesis of more natural and higher-fidelity speech. Mel spectrum is a general acoustic feature, not tied to a specific vocoder (such as AMBE). This allows decoders / vocoders using the low-bit-rate encoder-compatible decoding method described in this application to be easily adapted to other encoders, vocoders, or different audio sources, improving scalability and flexibility and leaving room for future upgrades or compatibility with multiple coding standards.

[0143] Optionally, this application can use the following formula for filtering:

[0144] ;

[0145] in, This represents the Mel filter bank; m is the number of the Mel filter bank. Optionally, an 80-dimensional Mel filter bank can be used, resulting in 80 Mel filter banks. Let be the energy of the output of the m-th Mel filter bank.

[0146] Furthermore, in order to conform to the human hearing curve, step S122 also takes the logarithm of the filtered energy. Specifically, the logarithm can be taken using the following formula: ;

[0147] in, It is a very small constant. To avoid taking the logarithm of 0, The spectral characteristics are obtained by taking the logarithm.

[0148] S123. The front-end module normalizes the Mel spectrum features.

[0149] The normalization process performed in step S123 is the same as that performed in step S113, and will not be described again.

[0150] pass Figure 4 The steps shown allow Mel-spectral features to be used as input features for the synthesizer model. Since the dimensions of the Mel-spectral features are fixed (e.g., fixed at 80 dimensions), they can be directly used as input features for the synthesizer model without padding or alignment. Furthermore, Figure 4 The steps shown do not directly use abstract vocoder parameters, but instead convert them into a more general and acoustically detailed Mel-spectral representation, providing a more intuitive input to the neural network, which can significantly improve the quality of audio synthesis and enhance robustness.

[0151] Optionally, another decoding method compatible with low bit rate encoders provided in this application embodiment may further include:

[0152] The audio signal obtained by the microphone is denoised using a neural network-based noise reduction module, and the denoised audio signal is then input into the encoder.

[0153] The training data used when training the synthesizer model includes the data denoised by the neural network-based denoising module.

[0154] Since this application extracts audio coding parameters from a low bitrate audio bitstream, the accuracy of the extracted audio coding parameters can be effectively improved by performing noise reduction processing on the audio signal obtained by the microphone.

[0155] Optionally, this application may use a neural network-based denoising model for denoising. Because neural networks have strong learning capabilities, they can effectively eliminate noise while minimizing audio distortion. Furthermore, during training, the denoising model can be jointly trained with the denoised data and the synthesizer model, allowing the synthesizer model to learn how to reconstruct and synthesize the denoised audio, further improving the quality of the synthesized audio.

[0156] Optionally, the loss functions used when training the synthesizer model include: multi-scale short-time Fourier transform loss function, adversarial loss function, and feature matching loss function. The total loss value when training the synthesizer model is: the weighted sum of the first loss value calculated by the multi-scale short-time Fourier transform loss function, the second loss value calculated by the adversarial loss function, and the third loss value calculated by the feature matching loss function. The weight of the first loss value is greater than the weight of the second loss value, and the weight of the first loss value is greater than the weight of the third loss value.

[0157] Among them, the adversarial loss function includes: multi-period discriminator and multi-scale reconstruction loss discriminator.

[0158] Optionally, the total loss value during the training of the synthesizer model is a weighted sum of the first loss value, the second loss value, the third loss value, and the fourth loss value. The first loss value is the loss value of the multi-scale short-time Fourier transform loss function, the second loss value is the loss value of the feature matching loss, the third loss value is the loss value of the multi-period discriminator, and the fourth loss value is the loss value of the multi-scale reconstruction loss discriminator.

[0159] This application allows for training the synthesizer model using SFT loss + GAN. The main loss function is multi-SFT loss; its loss function is as follows:

[0160] ;

[0161] in, Main loss value, The value can be one of 1, 2, or 3; These correspond to STFT transforms with different FFT point counts and hop sizes. In this application, multi-scale STFTs with FFT point counts and window lengths of 128, 256, and 512 are used, corresponding to hop sizes of 40, 64, and 80. The specific configuration can be adjusted according to the sampling rate; the default here is 8k. m is the index of the STFT configuration, and M is the total number of STFT configurations involved in the loss calculation. This is authentic audio. For generator audio.

[0162] The adversarial loss function can include a multi-period discriminator (MPD) and a multi-scale reconstruction loss discriminator (MRD). The MPD discriminator uses a standard configuration (periods of 2, 3, 5, 7, and 11), while the MRD uses multiple STFT-scale discriminators with FFT points and window lengths of 128, 256, and 512. Its loss function is the least squares loss function (LSGAN), used for stable adversarial training. Its formula is as follows:

[0163] ;

[0164] .

[0165] in, For discriminator loss, The generator loss is represented by K, where K is the total number of discriminators involved in the loss calculation, and k is the index of a single discriminator. For the k-th discriminator to the real audio The output value, For the k-th discriminator to generate audio The output value, This is authentic audio. For generator audio.

[0166] To stabilize the adversarial training process, this application also introduces a feature matching loss (feature map loss), as shown in the following formula:

[0167] ;

[0168] in, For feature matching loss, D represents the generator. k Represents the k-th discriminator. For the expectation operator, The total number of intermediate feature layers in the discriminator. For feature layer index, Let be the total number of elements in the feature map of the i-th layer, and s be the conditional signal. This represents the feature output of the i-th layer of the k-th discriminator for the generated samples. This represents the feature output of the i-th layer of the k-th discriminator for the real sample, where k is the discriminator index.

[0169] The total loss during the training of the synthesizer model is:

[0170] Total loss = 10 × main loss value + generator loss + feature matching loss;

[0171] That is, the main loss has a weight of 10, and the weights of the other losses are 1.

[0172] like Figure 7 As shown, the AI ​​vocoder may also include a forward error correction decoding module.

[0173] The decoding method compatible with low bit rate encoders provided in this application may further include:

[0174] The forward error correction decoding module is configured to perform forward error correction processing on the audio bitstream output by the encoder.

[0175] The input to the forward error correction decoding module can be the audio bitstream output by the encoder. Optionally, the bit rate of the audio bitstream can be 3600bps, the frame rate can be 50 frames / second, and the data volume of each frame bitstream is 72 bits, of which 49 bits are valid data and the other 23 bits are FEC (Forward Error Correction) error correction code.

[0176] For example, the forward error correction decoding module can adopt the forward error correction decoding (FEC decoding) method in the standard AMBE, which mainly performs FEC error correction on the received bit stream to prevent errors during transmission.

[0177] like Figure 7 As shown, the output layer may include: a fully connected layer, a frequency domain computation layer, and a time domain transformation layer.

[0178] The fully connected layer is configured to map the data output from the last ConvNeXt Block in the intermediate layer to the dimension corresponding to the number of frequency domain points;

[0179] Optionally, the number of IFFT points set in this embodiment can be 512, so the layer dimension corresponds to 514×256. The reason for 514 is that frequency domain signals usually include the highest frequency points.

[0180] The frequency domain computation layer is configured to compute on the data output by the fully connected layer to obtain the frequency domain signal;

[0181] After passing through the fully connected layer, the corresponding frequency domain signal can be calculated using the following formula:

[0182] ;

[0183] ;

[0184] .

[0185] in, The output is a fully connected layer with dimensions of 1×514×2T; For the corresponding frequency point amplitude, For the corresponding frequency point phase; For the final frequency domain signal, The imaginary unit, This is the output value at position k+257 in the fully connected layer output.

[0186] The time-domain conversion layer is configured to convert frequency-domain signals into audio time-domain signals.

[0187] This application can utilize a time-domain transformation layer (STFT) to transform frequency-domain signals to the time domain:

[0188] .

[0189] in, Represents the time-domain audio waveform. Representing the inverse short-time Fourier transform, Y represents the set of complex signals in the frequency domain. Representing the Hanning window. These represent complex signals at frequency points 0, 1, ..., 256, respectively. A Hanning window of length 512 is used (to prevent spectral leakage), and the hop size is 80 (80 for 8kHz, 160 for 16kHz). Since one frame of AMBE encoding / decoding corresponds to 20ms, in the synthesizer model of this application, after upsampling, it becomes one frame of 10ms, thus corresponding to 80 points at an 8kHz sampling rate.

[0190] Wherein, if the audio time-domain signal is a 16K sampling rate signal, the dimension of the input data of the fully connected layer is 1×384×2T, the 384 dimensions include 256 dimensions corresponding to low-frequency spectrum data from 0 to 4K and 128 dimensions corresponding to high-frequency spectrum data from 4K to 8K, the output frame shift of the output layer is 160 sampling points, the convolution dimension of the deep convolutional layer is 384×3×1, the convolution dimension of the first pointwise convolutional layer is 1152×1×1, and the convolution dimension of the second pointwise convolutional layer is 384×1×1.

[0191] If the audio time-domain signal is an 8K sampling rate signal, then the dimension of the input data of the fully connected layer is 1×256×2T, the output frame shift of the output layer is 80 sampling points, the convolution dimension of the depthwise convolutional layer is 256×3×1, the convolution dimension of the first pointwise convolutional layer is 768×1×1, and the convolution dimension of the second pointwise convolutional layer is 256×1×1.

[0192] For the 16K version of the vocoder, the dimensionality needs to be increased so that the synthesizer model can better reconstruct high frequencies. Optionally, the model structure of the synthesizer model in the 16K version of the vocoder is different from... Figure 8 The structure shown is consistent, but the dimensions of the intermediate layers are expanded from 256 to 384, the output hop size is adjusted from 80 to 160, and the number of FFT points remains unchanged. To avoid the synthesizer model focusing too much on high-frequency details and ignoring low-frequency components (because low frequencies determine speech intelligibility and basic listening experience, while high frequencies determine speech fullness), the fully connected layers in the output layer need to be grouped. That is, the intermediate layer dimensions corresponding to the spectrum are divided according to a low-frequency:high-frequency ratio of 2:1: the 256-dimensional fully connected layer corresponds to the low-frequency 0-4k spectrum, and the 128-dimensional fully connected layer corresponds to the high-frequency 4k-8k spectrum.

[0193] Corresponding to the decoding method for low-bit-rate encoders provided in the embodiments of this application above, the present invention also provides an AI vocoder that performs the above-described decoding method for low-bit-rate encoders. This AI vocoder is a Neural Enhanced Advanced Econder Decoder codec (NAED) compatible with multiple low-bit-rate encoders.

[0194] like Figure 5 As shown in the embodiment of this application, an AI vocoder may include: a front-end module 100 and a synthesizer model 200. The synthesizer model 200 includes: an input layer 210, an intermediate layer 220 and an output layer 230. The input layer 210 includes: a first convolutional layer 211 and a transposed convolutional layer 212.

[0195] The front-end module 100 is configured to preprocess and normalize the audio encoding parameters extracted from the low-bitrate audio bitstream.

[0196] The first convolutional layer 211 is configured to convolve the audio encoding parameters after preprocessing and normalization operations of the front-end module. The data processed by the first convolutional layer during convolution includes the data of the current audio frame and the data of the adjacent audio frames of the current audio frame, which are located after the current audio frame.

[0197] The intermediate layer 220 is configured to perform feature extraction and nonlinear transformation on the data obtained after upsampling the transposed convolutional layer.

[0198] Output layer 230 is configured to convert the data output from the intermediate layer into an audio time-domain signal.

[0199] Optionally, the intermediate layer 220 may include at least one convolutional next-generation module layer ConvNeXt Block, each ConvNeXt Block including: a depthwise convolutional layer, a first pointwise convolutional layer, an activation layer, a second pointwise convolutional layer, and a weight parameter layer;

[0200] The deep convolutional layers are configured to spatially convolve the input data using causal convolution.

[0201] The first pointwise convolutional layer is configured to perform pointwise convolution on the data output from the depthwise convolutional layer;

[0202] The activation layer is configured to perform a non-linear transformation on the data output by the first pointwise convolutional layer;

[0203] The second pointwise convolutional layer is configured to perform pointwise convolution on the data output by the activation layer;

[0204] The weight parameter layer has learnable weight parameters, which are configured to be multiplied with the data output from the second pointwise convolutional layer to obtain a product.

[0205] Optional, such as Figure 6 As shown, the intermediate layer 220 may include a first convolutional next-generation module layer, a second convolutional next-generation module layer, and a third convolutional next-generation module layer. The first convolutional next-generation module layer, the second convolutional next-generation module layer, and the third convolutional next-generation module layer have the same structure and initial parameters. The input data of the first convolutional next-generation module layer is the data output from the input layer, the input data of the second convolutional next-generation module layer is the data output from the first convolutional next-generation module layer, and the input data of the third convolutional next-generation module layer is the data output from the second convolutional next-generation module layer.

[0206] The output data of the first convolutional next-generation module layer, the second convolutional next-generation module layer, and the third convolutional next-generation module layer are all the sum of the product and the input data;

[0207] The output layer is specifically configured to convert the data output by the third convolutional next-generation module layer into an audio time-domain signal.

[0208] Optionally, the front-end module 100 may include: a first processing module and / or a second processing module;

[0209] The first processing module is configured to fill the subband spectrum of each audio frame in the audio encoding parameters extracted from the low bitrate audio bitstream to the target number, fill the amplitude of the filled subband spectrum with a preset value, use logarithms to perform dynamic range compression on the amplitude of the subband spectrum of each audio frame and the fundamental frequency of each audio frame in the filled audio encoding parameters, and normalize the compressed data.

[0210] The second processing module is configured to generate the linear frequency amplitude spectrum of each audio frame based on the audio coding parameters extracted from the low bit rate audio bit stream using a sine wave model, filter the linear frequency amplitude spectrum through a Mel filter bank, take the logarithm of the filtered energy to obtain the Mel spectral features, and then normalize the Mel spectral features.

[0211] Optionally, the second processing module, based on the audio coding parameters extracted from the low-bitrate audio bitstream, generates the linear frequency amplitude spectrum of each audio frame using a sine wave model, and is specifically configured as follows:

[0212] The second processing module generates an ideal impulse sequence based on the fundamental frequency F0 extracted from the low bit rate audio bit stream, and convolves it with the Sinc kernel to obtain the voiced amplitude spectrum; the second processing module shapes the white noise based on the subband amplitude M extracted from the low bit rate audio bit stream to obtain the unvoiced amplitude spectrum; the second processing module adds the voiced amplitude spectrum and the unvoiced amplitude spectrum to obtain the linear frequency amplitude spectrum of each audio frame.

[0213] Optional, Figure 5 The AI ​​vocoder shown may also include: a neural network-based noise reduction module, configured to perform noise reduction processing on the audio signal obtained by the microphone, and input the noise-reduced audio signal into the encoder;

[0214] The training data used when training the synthesizer model includes the data denoised by the neural network-based denoising module.

[0215] Optionally, the loss functions used when training the synthesizer model include: multi-scale short-time Fourier transform loss function, adversarial loss function, and feature matching loss function. The total loss value when training the synthesizer model is: the weighted sum of the first loss value calculated by the multi-scale short-time Fourier transform loss function, the second loss value calculated by the adversarial loss function, and the third loss value calculated by the feature matching loss function. The weight of the first loss value is greater than the weight of the second loss value, and the weight of the first loss value is greater than the weight of the third loss value.

[0216] Among them, the adversarial loss function includes: multi-period discriminator and multi-scale reconstruction loss discriminator.

[0217] Optionally, the total loss value during the training of the synthesizer model is a weighted sum of the first loss value, the second loss value, the third loss value, and the fourth loss value. The first loss value is the loss value of the multi-scale short-time Fourier transform loss function, the second loss value is the loss value of the feature matching loss, the third loss value is the loss value of the multi-period discriminator, and the fourth loss value is the loss value of the multi-scale reconstruction loss discriminator.

[0218] like Figure 7 As shown, Figure 5 The AI ​​vocoder shown may also include a forward error correction decoding module.

[0219] The forward error correction decoding module is configured to perform forward error correction processing on the audio bitstream output by the encoder;

[0220] For example, the forward error correction decoding module can adopt the forward error correction decoding (FEC decoding) method in the standard AMBE, which mainly performs FEC error correction on the received bit stream to prevent errors during transmission.

[0221] like Figure 7 As shown, the output layer may include: a fully connected layer, a frequency domain computation layer, and a time domain transformation layer.

[0222] The fully connected layer is configured to map the data output from the last ConvNeXt Block in the intermediate layer to the dimension corresponding to the number of frequency domain points;

[0223] The frequency domain computation layer is configured to compute on the data output by the fully connected layer to obtain the frequency domain signal;

[0224] The time-domain conversion layer is configured to convert frequency-domain signals into audio time-domain signals.

[0225] This application also provides an electronic device in its embodiments. (See reference...) Figure 10 The diagram illustrates a structural schematic of an electronic device suitable for implementing the decoding method of the low-bit-rate encoder compatible with the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 10 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0226] like Figure 10As shown, the electronic device may include a processing unit 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory 602 or a program loaded from a storage device 608 into a random access memory 603. When the electronic device is powered on, the random access memory 603 also stores various programs and data required for the operation of the electronic device. The processing unit 601, the read-only memory 602, and the random access memory 603 are interconnected via a bus 604. An input / output interface 605 is also connected to the bus 604.

[0227] Typically, the following devices can be connected to the input / output interface 605: input devices 606 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 608 including, for example, memory card, hard disk, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 10 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0228] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the decoding methods for a low-bit-rate encoder compatible with this application.

[0229] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the decoding methods compatible with low bit rate encoders provided in this application.

[0230] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0231] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0232] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0233] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A decoding method compatible with low bit rate encoders, characterized in that, include: The front-end module performs preprocessing and normalization operations on the audio encoding parameters extracted from the low bitrate audio bitstream. The audio encoding parameters include: the fundamental frequency of the current audio frame, the number of sub-band spectra of the current audio frame, the audio / non-audio decision of each sub-band spectra, and the amplitude of each sub-band spectra. The synthesizer model built on a neural network convolves the audio coding parameters after preprocessing and normalization. The data processed during the convolution includes the data of the current audio frame and the data of the adjacent audio frames, which are located after the current audio frame. The synthesizer model upsamples the data obtained after convolution in the audio frame dimension. The synthesizer model performs feature extraction and nonlinear transformation on the data obtained after upsampling. The synthesizer model converts the data obtained after feature extraction and nonlinear transformation into an audio time-domain signal. The synthesizer model includes an input layer, an intermediate layer, and an output layer. The intermediate layer includes at least one convolutional next-generation module layer, ConvNeXt Block. Each ConvNeXt Block includes a depthwise convolutional layer, a first pointwise convolutional layer, an activation layer, a second pointwise convolutional layer, and a weight parameter layer. The synthesizer model performs feature extraction and nonlinear transformation on the data obtained after upsampling, including: The deep convolutional layer uses causal convolution to perform spatial convolution on the data obtained after upsampling; The first pointwise convolutional layer performs pointwise convolution on the data obtained after spatial convolution; The activation layer performs a non-linear transformation on the data obtained after pointwise convolution; The second pointwise convolutional layer performs pointwise convolution on the data obtained after nonlinear transformation; The weight parameter layer multiplies the data obtained after pointwise convolution with the learnable weight parameters to obtain a product.

2. The decoding method compatible with low bit rate encoders according to claim 1, characterized in that, The front-end module performs preprocessing and normalization operations on the audio encoding parameters extracted from the low-bitrate audio bitstream, including: The front-end module fills the subband spectrum of each audio frame in the audio encoding parameters extracted from the low bitrate audio bitstream into the target quantity, and fills the amplitude of the filled subband spectrum with a preset value. It then uses logarithms to perform dynamic range compression on the amplitude of the subband spectrum of each audio frame and the fundamental frequency of each audio frame in the filled audio encoding parameters, and normalizes the compressed data. And / or, the front-end module generates the linear frequency amplitude spectrum of each audio frame based on the audio encoding parameters extracted from the low bitrate audio bitstream using a sine wave model, filters the linear frequency amplitude spectrum through a Mel filter bank, takes the logarithm of the filtered energy to obtain Mel spectral features, and normalizes the Mel spectral features.

3. The decoding method compatible with low bit rate encoders according to claim 2, characterized in that, The front-end module generates a linear frequency amplitude spectrum for each audio frame based on audio encoding parameters extracted from a low-bitrate audio bitstream, using a sine wave model, including: The front-end module generates an ideal impulse sequence based on the fundamental frequency F0 extracted from the low bit rate audio bitstream, and convolves it with the Sinc kernel to obtain the voiced amplitude spectrum. The front-end module shapes the white noise based on the amplitude M of the subband spectrum extracted from the low bit rate audio bitstream to obtain the unvoiced amplitude spectrum. The front-end module adds the voiced amplitude spectrum to the unvoiced amplitude spectrum to obtain the linear frequency amplitude spectrum of each audio frame.

4. The decoding method compatible with low bit rate encoders according to claim 1, characterized in that, Also includes: The audio signal obtained by the microphone is denoised using a neural network-based noise reduction module, and the denoised audio signal is then input into the encoder. The training data used when training the synthesizer model includes the data denoised by the neural network-based denoising module.

5. The decoding method compatible with low bit rate encoders according to claim 1, characterized in that, The low bitrate audio bitstream is an audio bitstream generated by any one of the AMBE encoder, NVOC encoder, Codec2 encoder, and MELPe encoder.

6. An AI vocoder, characterized in that, The AI ​​vocoder includes a front-end module and a synthesizer model built on a neural network, and is configured to perform the decoding method of the low bitrate encoder compatible with any one of claims 1 to 5.

7. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the decoding method for a low-bit-rate encoder as described in any one of claims 1 to 5.

8. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the decoding method of the low bit rate encoder as described in any one of claims 1 to 5.

9. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the decoding method compatible with a low bit rate encoder as described in any one of claims 1 to 5.

Images