Speech coding method and decoding method based on time sequence modeling and related devices

Abstract

The embodiment of the invention provides a voice coding method and decoding method based on time sequence modeling and a related device. The voice coding method comprises the following steps: acquiring target voice; extracting voice features corresponding to the voice waveform of the target voice through an input cavity convolution extraction module; performing down-sampling processing on the voice features through a down-sampling module to obtain a down-sampling sequence; performing long time sequence modeling on the down-sampling sequence through a long and short time memory neural network added with exponential activation and matrix operation to obtain a coding sequence; and converting the feature dimension of the coding sequence into 128 dimensions through the first convolutional layer to obtain a target code stream. According to the embodiment of the invention, the feature extraction module, the down-sampling module, the long and short term memory neural network and the output cavity convolution layer code the target voice, so that the compression of the voice with the extremely low bit rate can be realized, and the reconstruction of the voice with the extremely low bit rate and high quality under 150bps is realized.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech encoding method, decoding method and related devices based on time series modeling. Background Art

[0002] In the field of deep learning, existing speech codec models primarily optimize their structure to model long speech sequences and extract their most essential features, or optimize the learnable codebook training strategy for the quantizer. However, these approaches struggle to achieve high-quality speech encoding and compression at around 150 bits per second. Therefore, in areas such as narrowband low-orbit satellite communications, military security communications, and emergency rescue satellite communications, where bandwidth resources are limited and channel costs are expensive, there is an urgent need for extremely low-bit-rate speech transmission solutions. Summary of the Invention

[0003] The main purpose of this application is to provide a speech encoding method, decoding method and related devices based on time series modeling, aiming to solve the technical problem that existing speech encoding methods cannot achieve extremely low bit rate speech encoding and compression.

[0004] To achieve the above objectives, the present application provides a speech coding method, comprising: Get the target voice; Extracting speech features corresponding to the speech waveform of the target speech by inputting a dilated convolution module; Downsampling the speech features by a downsampling module to obtain a downsampling sequence; Performing long-term temporal modeling on the downsampled sequence by adding a long short-term memory neural network with exponential activation and matrix calculation to obtain an encoding sequence; The dimension of the coding sequence is converted to 128 dimensions through the first convolutional layer to obtain a target code stream.

[0005] Optionally, extracting speech features corresponding to the speech waveform of the target speech by inputting a dilated convolution module includes: Extract shallow speech features through one-dimensional causal convolutional layer; The shallow features of the speech are input into the hollow causal convolution layer to extract the speech features corresponding to the speech waveform of the target speech.

[0006] Optionally, performing downsampling processing on the speech features by a downsampling module to obtain a downsampling sequence includes: Processing the speech features through an activation layer; Superimposing the speech feature with the output of the activation layer to obtain a feature to be sampled; The features to be sampled are downsampled and compressed by different multiples through the dilated causal convolution to obtain a downsampled sequence.

[0007] Optionally, the long short-term memory neural network with added exponential activation and matrix calculation is used to perform long time series modeling on the downsampled sequence to obtain an encoded sequence, comprising: Performing long-term temporal modeling on the downsampled sequence through a long short-term memory neural network in an exponentially activated form to obtain an encoded sequence, or; The down-sampled sequence is modeled into a long short-term memory neural network based on a self-attention mechanism to obtain a coding sequence.

[0008] Optionally, the method further includes: The target code stream is subjected to step-by-step residual quantization by a residual vector quantizer to obtain a quantized target code stream.

[0009] In addition, to achieve the above objectives, the present application provides a speech decoding method, comprising: Get the target stream; Performing convolution processing on the target code stream through a second convolution layer to obtain a decoding sequence; The encoding sequence is modeled and restored by adding a long short-term memory neural network with exponential activation and matrix calculation to obtain an upsampled sequence; Performing upsampling processing on the upsampling sequence by different multiples through an upsampling module to obtain speech features; The speech features are convolved through an output transposed convolution layer to obtain the target speech.

[0010] In addition, to achieve the above-mentioned purpose, the present application provides a speech coding and decoding model, which includes a speech coding module for implementing the above-mentioned speech coding method, a quantization module, and a speech decoding module for implementing the above-mentioned speech decoding method; The speech coding module includes an input hole convolution feature extraction module, a downsampling module, a long short-term memory neural network and a first convolution layer; The speech decoding module includes a second convolutional layer, a long short-term memory neural network, an upsampling module, and an output transposed convolutional layer.

[0011] Optionally, the input dilated convolution module includes one one-dimensional convolution layer and one dilated causal convolution layer, and the first convolution layer includes one one-dimensional convolution layer; The second convolutional layer includes a one-dimensional convolutional layer and a void causal convolutional layer, and the output transposed convolutional layer includes a one-dimensional convolutional layer and a transposed causal convolutional layer; The speech coding model further includes a discriminator module, wherein the discriminator module includes a first discriminator, a second discriminator and a third discriminator; Processing the first time-frequency spectrum feature of the target speech by a first discriminator to obtain a first time-frequency spectrum graph; Processing the second time-frequency spectrum feature of the target speech by a second discriminator to obtain a second time-frequency spectrum graph; Processing the third time-frequency spectrum feature of the target speech by a third discriminator to obtain a third time-frequency spectrum graph; Aggregating the first time-spectrogram, the second time-spectrogram, and the third time-spectrogram to obtain a speech waveform; The first time-frequency spectrum feature, the second time-frequency spectrum feature, and the third time-frequency spectrum feature are of different types.

[0012] In addition, to achieve the above-mentioned purpose, the present application provides a training method based on a speech coding and decoding model, characterized in that the training method is applied to the above-mentioned speech coding and decoding model, using a codebook-enhanced generation segmented training strategy, wherein the speech coding and decoding model is a generator, and the discriminator uses the above-mentioned discriminator model. The training method includes: In the first training stage, the speech encoding module and the speech decoding module are trained using training samples; the reconstruction loss functions of the speech encoding module and the speech decoding module include the time domain loss function and the frequency domain loss function of the generator; In the second training phase, the speech encoding module, speech decoding module, and quantization module are trained using training samples; the loss functions of the speech encoding module and the speech decoding module include the commitment loss function of the quantization module. In the third training phase, the speech decoding module is trained using training samples.

[0013] Optionally, the commitment loss function includes a negative entropy regularization term, which is used to calculate the Euclidean distance matrix between the input features and the embedding matrix, generate a probability distribution function through a softmax function, and use a soft allocation strategy to enable the gradient of the commitment regularization term to be propagated in a certain direction.

[0014] Compared with the prior art, the embodiments of the present application have the following beneficial effects: The embodiment of the present application provides a speech encoding method, decoding method and related devices based on time series modeling. The above-mentioned speech encoding method includes: obtaining a target speech; extracting the speech features corresponding to the speech waveform of the target speech by inputting a void convolution module; performing downsampling processing on the speech features by different multiples through a downsampling module to obtain a downsampling sequence; performing long time series modeling on the downsampling sequence by adding a long short-term memory neural network with exponential activation and matrix calculation to obtain a coding sequence; performing convolution processing on the coding sequence through a first convolution layer to obtain a target code stream. In this embodiment, the target speech is encoded by the above-mentioned feature extraction module, downsampling module, long short-term memory neural network with exponential activation and matrix calculation and first convolution layer, which can achieve compression for extremely low bit rate speech and reconstruction of high-quality speech at an extremely low bit rate of 150bps. In addition, the model can be deployed to terminal devices such as mobile phones, and can provide a feasible solution for voice communication in satellite-ground integrated networks and 6G networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the solutions in this application, a brief introduction will be given below to the drawings required for use in the description of the embodiments of this application. Obviously, the drawings described below are some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0016] Figure 1 This is a schematic diagram of the architecture of the speech coding and decoding model provided in the embodiment of the present application; Figure 2 This is a flow chart of the speech coding method provided by an embodiment of the present application; Figure 3 This is one of the schematic diagrams of the long short-term memory neural network provided in the embodiments of the present application; Figure 4 This is the second schematic diagram of the long short-term memory neural network provided in the embodiment of the present application; Figure 5 This is a flowchart of the speech decoding method provided by an embodiment of the present application; Figure 6 This is a flowchart of a training method based on a speech coding model provided in an embodiment of the present application; Figure 7 is a Mel-spectrogram of English audio provided in an embodiment of the present application; Figure 8 It is a mel-spectrogram generated in the first training phase when the training sample in the embodiment of the present application is English audio; Figure 9 It is a mel-spectrogram generated in the second training phase when the training sample in the embodiment of the present application is English audio; Figure 10It is a mel-spectrogram generated in the third training phase when the training sample in the embodiment of the present application is English audio; Figure 11 This is a Mel-spectrogram of Chinese audio provided in an embodiment of the present application; Figure 12 It is a mel-spectrogram generated in the first training phase when the training sample in the embodiment of the present application is Chinese audio; Figure 13 It is a mel-spectrogram generated in the second training phase when the training sample in the embodiment of the present application is Chinese audio; Figure 14 This is the mel-spectrogram generated in the third training stage when the training sample is Chinese audio in the embodiment of the present application. DETAILED DESCRIPTION

[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0018] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0019] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0020] See also Figure 1 , Figure 1 This is a schematic diagram of the architecture of the speech coding and decoding model provided in the embodiment of the present application. Figure 1 As shown, the speech coding and decoding model provided in the embodiment of the present application includes a speech coding module, a quantization module and a speech decoding module; The speech coding module includes an input hole convolution module, a downsampling module, a long short-term memory neural network with increased exponential activation and matrix calculation, and a first convolution layer; The speech decoding module includes a second convolutional layer, a long short-term memory neural network with increased exponential activation and matrix calculation, an upsampling module, and an output transposed convolutional layer.

[0021] Optionally, the input dilated convolution module includes 1 one-dimensional convolution layer and 1 dilated causal convolution layer, and the first convolution layer includes 1 one-dimensional convolution layer.

[0022] Optionally, the second convolutional layer includes 1 one-dimensional convolutional layer and 1 void causal convolutional layer, and the third convolutional layer includes 1 one-dimensional convolutional layer and 1 transposed causal convolutional layer.

[0023] like Figure 1 As shown, the speech coding module includes an input hole convolution module, a downsampling module, a long short-term memory neural network with increased exponential activation and matrix calculation, and a first convolution layer, wherein the above-mentioned input hole convolution feature extraction module includes 1 one-dimensional convolution layer and 1 hole causal convolution layer, the above-mentioned downsampling module includes 4 residual convolution layers, and the above-mentioned first convolution layer includes 1 one-dimensional convolution layer.

[0024] The above-mentioned speech decoding module includes a second convolutional layer, a long short-term memory neural network with increased exponential activation and matrix calculation, an upsampling module, and an output transposed convolutional layer. The above-mentioned second convolutional layer includes 1 one-dimensional convolutional layer and 1 void causal convolutional layer. The above-mentioned third convolutional layer includes 1 one-dimensional convolutional layer and 1 one-dimensional transposed convolutional layer. The above-mentioned upsampling module includes 4 residual convolutional layers.

[0025] It should be noted that the above-mentioned speech encoding module can execute the speech encoding method in subsequent embodiments, and the above-mentioned speech decoding module can execute the speech decoding method in subsequent embodiments.

[0026] It is easy to understand that since the speech encoding module can execute the speech encoding method in the subsequent embodiments, and the speech decoding module can execute the speech decoding method in the subsequent embodiments, the speech encoding and decoding model provided in this embodiment has all the technical effects of the subsequent speech encoding method and speech decoding method.

[0027] See also Figure 2 , Figure 2 This is a flow chart of the speech coding method provided by the embodiment of the present application. Figure 2 As shown, the speech encoding method provided in the embodiment of the present application includes: S110, acquiring target speech; S120, extracting speech features corresponding to the speech waveform of the target speech by inputting a dilated convolution module; S130, performing downsampling processing on the speech features by different multiples through a downsampling module to obtain a downsampling sequence; S140, performing long-term time series modeling on the downsampled sequence by adding a long short-term memory neural network with exponential activation and matrix calculation to obtain a coding sequence; S150: Convert the dimension of the coding sequence to 128 through the first convolutional layer to obtain a target code stream.

[0028] The speech coding method provided in an embodiment of the present application includes: obtaining a target speech; extracting speech features corresponding to the speech waveform of the target speech through an input void convolution module; wherein the input void convolution module includes 1 one-dimensional convolution layer and 1 void causal convolution layer, and the speech features are downsampled by different multiples through the downsampling module to obtain a downsampling sequence; wherein the downsampling module includes 4 residual convolution layers; long-term time series modeling of the downsampling sequence is performed by adding a long short-term memory neural network with exponential activation and matrix calculation to obtain a coding sequence; the dimension of the coding sequence is converted to 128 dimensions through the first convolution layer to obtain a target code stream, wherein the first convolution layer includes 1 one-dimensional convolution layer.

[0029] In this embodiment, the target speech is encoded through the aforementioned input dilated convolution module, downsampling module, long short-term memory neural network, and first convolutional layer. This allows for compression of extremely low-bitrate speech and reconstruction of high-quality speech at 150 bps. Furthermore, the model can be deployed on mobile phones and other terminal devices, providing a feasible solution for voice communications in integrated satellite-ground networks and 6G networks.

[0030] Optionally, the speech coding and decoding model further includes a discriminator module, wherein the discriminator module includes a first discriminator, a second discriminator and a third discriminator; Processing the first time-frequency spectrum feature of the target speech by a first discriminator to obtain a first time-frequency spectrum graph; Processing the second time-frequency spectrum feature of the target speech by a second discriminator to obtain a second time-frequency spectrum graph; Processing the third time-frequency spectrum feature of the target speech by a third discriminator to obtain a third time-frequency spectrum graph; Aggregating the first time-spectrogram, the second time-spectrogram, and the third time-spectrogram to obtain a speech waveform; The first time-frequency spectrum feature, the second time-frequency spectrum feature, and the third time-frequency spectrum feature are features of different types.

[0031] The discriminator module in this embodiment is a multi-scale STFT discriminator. By running three discriminators with different STFT parameters in parallel, the authenticity of audio is judged from multi-dimensional time-frequency spectrum features, significantly improving the ability to distinguish generated audio artifacts. Each discriminator independently processes different STFT time-frequency spectra of the same audio, and finally outputs a multi-scale discrimination result, which is then aggregated through a loss function. Through different STFT parameters, time-frequency spectrograms of different resolutions are generated. Through two layers of normalized convolution extraction modules, the time-frequency features are initially extracted, and then through two layers of frequency downsampling convolution modules, the frequency axis size is gradually compressed to cover more frequency bins and extract the global frequency pattern. The last layer of convolution adjusts the convolution kernel from (3,9) to (3,3), integrates the high-level features of time and frequency, and finally outputs the discrimination layer, maps the high-level features to the discrimination results of 1 channel, and outputs the "authenticity score" of each time and frequency position.

[0032] Optionally, extracting speech features corresponding to the speech waveform of the target speech by inputting a dilated convolution module includes: Extract shallow speech features through one-dimensional causal convolutional layer; The shallow features of the speech are input into the hollow causal convolution layer to extract the speech features corresponding to the speech waveform of the target speech.

[0033] In this embodiment, the input dilated convolution module includes a one-dimensional causal convolution layer and a dilated causal convolution layer. The one-dimensional causal convolution layer is used to extract the shallow speech features of the target speech, and the dilated causal convolution layer is used to extract the speech features corresponding to the speech waveform of the target speech.

[0034] Optionally, performing downsampling processing on the speech features by a downsampling module to obtain a downsampling sequence includes: Processing the speech features through an activation layer; Superimposing the speech feature with the output of the activation layer to obtain a feature to be sampled; The features to be sampled are downsampled and compressed by different multiples through the dilated causal convolution to obtain a downsampled sequence.

[0035] In this embodiment, the process of downsampling the speech features by the downsampling module is as follows: first, an exponential linear activation function ELU is used to alleviate the problem of gradient disappearance in long sequence speech during training, and convolution activation layers with different convolution sizes and opposite input and output channels are superimposed. The input and output of the convolution activation layer are then superimposed, and the features are downsampled by different multiples through void causal convolution, that is, the residual convolution downsampling process is completed, and the residual convolution downsampling layers with different downsampling factors are used to downsample and compress the speech features.

[0036] Optionally, the long short-term memory neural network with added exponential activation and matrix calculation is used to perform long time series modeling on the downsampled sequence to obtain an encoded sequence, comprising: Performing long-term temporal modeling on the downsampled sequence through a long short-term memory neural network in an exponentially activated form to obtain an encoded sequence, or; The down-sampled sequence is modeled into a long short-term memory neural network based on a self-attention mechanism to obtain a coding sequence.

[0037] An optional implementation is to perform temporal modeling on the down-sampled sequence through a long short-term memory neural network in an exponentially activated form to obtain a coding sequence.

[0038] like Figure 3 As shown in Figure 1, the exponentially activated long short-term memory (ALSTM) neural network changes the input and forget gate storage formats, replacing the sigmoid activation function with an exponential activation function. Furthermore, an auxiliary state is introduced into the memory cell to track the cumulative effect of the gate, regularizing the memory cell. The exponential activation function expands the dynamic range of the gate value, allowing the ALSTM to more flexibly focus and adjust the update strength of the memory cell.

[0039] Another optional implementation is to perform temporal modeling on the down-sampled sequence through a long short-term memory neural network based on a self-attention mechanism to obtain a coding sequence.

[0040] like Figure 4 As shown in the figure, the self-attention-based matrix ALSTM module uses matrix multiplication to process the input of the candidate memory cells of the LSTM. The parallelization of the qkv matrix multiplication is due to the high degree of independence of its operations. The input tensor undergoes three independent linear transformations to obtain the Q (query), K (key), and V (value) matrices. These transformations use different weights, but the calculation of each row is independent and has no dependencies, so they can be calculated in parallel, greatly enhancing the model's memory capacity and parallel processing capabilities, allowing it to capture more complex data relationships and patterns within a single time step.

[0041] In this embodiment, LSTM time series modeling based on self-attention is implemented at the encoding and decoding end. Compared with traditional LSTM, the long short-term memory neural network in this embodiment can also perform parallel operations, which improves the training and reasoning speed.

[0042] Optionally, the method further includes: The target code stream is subjected to step-by-step residual quantization by a residual vector quantizer to obtain a quantized target code stream.

[0043] Optionally, the quantization module includes a residual vector quantizer.

[0044] In this embodiment, a residual vector quantizer is used to perform step-by-step residual quantization. A vector quantizer is used to iteratively quantize the quantization residual of the upper level and map the 128-dimensional output of the speech coding module to N dimensions in the codebook. Then use The codebook of , the compression factor of the quantizer is times.

[0045] See also Figure 5 , Figure 5 This is a flow chart of the speech decoding method provided by the embodiment of the present application. Figure 5 As shown, the speech decoding method provided in the embodiment of the present application includes: S210, obtaining a target bitstream; S220, performing convolution processing on the target bitstream through a second convolution layer to obtain a decoding sequence; S230, modeling and restoring the encoded sequence by adding a long short-term memory neural network with exponential activation and matrix calculation to obtain an upsampled sequence; S240, performing upsampling processing on the upsampled sequence by different multiples through an upsampling module to obtain speech features; S250, convolving the speech features through an output transposed convolution layer to obtain a target speech.

[0046] The speech decoding method provided in the embodiment of the present application includes: obtaining a target code stream; performing convolution processing on the target code stream through a second convolution layer to obtain a decoding sequence; wherein the above-mentioned second convolution layer includes 1 one-dimensional convolution layer and 1 void causal convolution layer. The coding sequence is modeled and restored by a long short-term memory neural network with increased exponential activation and matrix calculation to obtain an upsampling sequence; the upsampling sequence is upsampled by different multiples through an upsampling module to obtain speech features; wherein the upsampling module includes 4 residual convolution layers. The speech features are convolved through a third convolution layer to obtain a target speech; wherein the transposed convolution layer includes 1 one-dimensional convolution layer and 1 transposed causal convolution layer.

[0047] In this embodiment, the target speech is decoded from the target bitstream using the aforementioned second convolutional layer, a long short-term memory neural network with increased exponential activation and matrix computation, an upsampling module, and a transposed convolutional layer. This achieves high-quality speech reconstruction at an extremely low bitrate of 150 bps. Furthermore, the model can be deployed on mobile phones and other terminal devices, providing a feasible solution for voice communications in integrated satellite-ground networks and 6G networks.

[0048] See also Figure 6 , Figure 6This is a flowchart of a training method based on a speech codec model provided in an embodiment of the present application. It should be understood that the training method based on a speech codec model provided in an embodiment of the present application can be applied to the above-mentioned speech codec model. The training method based on a speech codec model provided in an embodiment of the present application uses a codebook-enhanced generation segmentation training strategy, wherein the speech codec model is a generator and the discriminator uses the discriminator described above.

[0049] like Figure 6 As shown, the training method based on the speech codec model provided in the embodiment of the present application includes: S310, in a first training phase, training a speech encoding module and a speech decoding module using training samples; the reconstruction loss functions of the speech encoding module and the speech decoding module include a time domain loss function and a frequency domain loss function of a generator; S320, in a second training phase, using training samples to train a speech encoding module, a speech decoding module, and a quantization module; the loss functions of the speech encoding module and the speech decoding module include a commitment loss function of the quantization module; S330: In the third training phase, the speech decoding module is trained using the training samples.

[0050] In this embodiment, the training of the speech coding and decoding model includes three training stages.

[0051] In the first training phase, the model is trained using a generative adversarial network. The entire speech codec model serves as the generative network, and the discriminator module uses a multi-scale short-time Fourier transform (MS-STFT) discriminator. In the first training phase, training is performed with a small number of samples. The model training process does not include codebook training for the quantization module. Training of the MS-STFT discriminator involves feature loss. The loss functions used in the first training phase include both time-domain and frequency-domain loss functions.

[0052] In the second training phase, the quantization module codebook is trained and its commitment loss is incorporated into the loss function, forcing the quantized discrete representation to maintain consistency with the input residual. Finally, the reconstruction loss function and the commitment loss function are forward-propagated to stabilize the codebook update process. The commitment loss function in the second phase is the output of the speech coding module and consists of the 1-norm loss of the quantizer output at each layer. While minimizing the commitment loss function, the quantization module gradient calculation is stopped and the gradients of the speech decoding module are back-propagated to the speech coding module, achieving end-to-end training.

[0053] Optionally, the commitment loss function includes a negative entropy regularization term, which is used to calculate the Euclidean distance matrix between the input features and the embedding matrix, generate a probability distribution function through a softmax function, and use a soft allocation strategy to enable the gradient of the commitment regularization term to be propagated in a certain direction.

[0054] A negative entropy regularization term is added to the commitment loss to form a joint optimization objective. The negative entropy regularization term calculates the Euclidean distance matrix between the input features and the embedding matrix. A softmax function is then used to generate a probability distribution function. A soft allocation strategy ensures that the gradient of the commitment regularization term propagates in different directions. By maximizing the entropy of the codeword usage distribution, the codewords in the codebook are uniformly utilized, preventing some codewords from being frequently used while others are ignored.

[0055] The commitment loss function is as follows:

[0056] In the third training phase, the speech encoding module is fixed, and the speech encoding module and quantization module are not trained. Instead, the speech decoding module and the MS-STFT discriminator are jointly trained to enhance the speech decoding module's ability to reconstruct waveforms. The discriminator classifies the real waveform as 1 and the generated waveform as 0.

[0057] Optionally, during the three-stage training process, a balancer is referenced to perform reverse gradient updates to stabilize the entire training process. The balancer design refers to the Encodec model training process design, and mainly uses exponential moving average to dynamically adjust the loss weight.

[0058] This embodiment proposes a segmented adversarial training strategy, which trains each module separately at different stages, greatly achieving the optimization of each module. At the same time, in the training process of the quantization module, negative entropy regularization is introduced to stabilize the training of the quantizer, while improving the utilization of the codebook and the generalization ability of the model.

[0059] To illustrate the technical effects of the speech encoding method and speech decoding method provided in the embodiments of the present application, the following describes relevant experimental results: The quality of the reconstructed audio was evaluated using both subjective and objective evaluation metrics. The experiment used the LibriTTS English dataset, training a large number of noise-free English training sets. The speech quality of the speech codec model was verified on both a clean English test set and a noisy Chinese test set to assess the model's generalization ability. Since practical applications are mostly for domestic communications, it is necessary to verify the model's generalization ability on the Chinese training set. Therefore, the Aishell Mandarin Chinese open source speech database was used. The recordings in this dataset cover 11 areas, including smart homes, autonomous driving, and industrial production. The recordings were made in a quiet indoor environment using three different devices: a high-fidelity microphone, an Android phone, and an iOS phone, to verify the model's ability to reconstruct speech on various mainstream devices.

[0060] (1) Objective evaluation indicators VISQOL is used to evaluate the objective quality of synthesized speech. It generates a MOS score by comparing the spectro-temporal similarity between reference and test audio. MOS scores range from 1 (worst) to 5 (best). Experiments have shown that, particularly at 150 bits per second (bps), the reconstructed speech scores 4.2, far exceeding the required speech transmission quality in narrowband satellite communications and emergency rescue scenarios. This means that the speech encoding and decoding methods provided in this embodiment not only achieve speech reconstruction at extremely low bit rates, but also ensure intelligibility.

[0061] (2) Mel spectrum comparison in the segmented generative adversarial training stage The audio Mel spectrograms of the three training stages are compared at 150bps, as shown in the following example: Figure 7-10 As shown in the figure, when the training dataset is the English test dataset of LibriTTS, the improvement in the quality of speech reconstruction by the generator using adversarial training can be compared across three stages. In the first stage of training, since the quantization module is not involved in training, the speech encoding module and the speech decoding module are jointly trained, giving the speech decoding module a preliminary ability to reconstruct audio. In the second stage, the quantization module quantizes and compresses the output of the speech encoding module. Since the gradient of the quantization module is not involved in the update, the training uses a negative entropy regularized commitment loss to maximize the approximation of the embedding vector of the codebook mapping to the output of the speech encoding module, while also improving codebook utilization. In the third stage, the speech encoding module is fixed, focusing on improving the quality of audio reconstruction by the speech decoding module.

[0062] In addition, training is also conducted on the Chinese training set in the hope of improving the generalization ability of the model, which is expected to be deployed in narrowband satellite communications, emergency rescue, military communications and other fields. Figure 11-14 As shown in Figure 3, the model can strongly capture the characteristics of audio at each stage.

[0063] (3) Speech encoding module and speech decoding module parameter test The real-time factor (RTF) was tested on a 13th Gen Intel(R) Core(TM) i5-13400 CPU thread. The test results are shown in Table 1 below. RTF is defined as the ratio between the duration of the input audio and the time required to encode or decode it using the model. When RTF > 1, the model processing speed is faster than real time, as shown in Table 1 below. Therefore, the speech encoding and decoding methods provided in this embodiment are suitable for real-time applications.

[0064] Table 1: Bitrate (bps) Params(enc) Params(dec) RTF(enc) RTF(dec) VISQOL 150 23.13MB 30.11MB 16.9 14.0 4.06±0.26 300 23.13MB 30.11MB 10.9 8.8 4.24±0.17 580 23.13MB 30.11MB 14.0 13.5 4.20±0.21 800 22.63MB 29.61MB 11.8 11.5 4.20±0.21 1000 22.57MB 29.54MB 10.6 10.9 4.23±0.19 2000 22.57MB 29.54MB 4.4 4.4 4.34±0.15 3340 22.25MB 29.23MB 4.6 4.7 4.32±0.17 Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware online platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of this application.

[0065] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0066] Obviously, the embodiments described above are only some of the embodiments of the present application, rather than all of the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.

Claims

1. A speech coding method, characterized in that: include: Get the target voice; Extracting speech features corresponding to the speech waveform of the target speech by inputting a dilated convolution module; Downsampling the speech features by a downsampling module to obtain a downsampling sequence; Performing long-term temporal modeling on the downsampled sequence by adding a long short-term memory neural network with exponential activation and matrix calculation to obtain an encoding sequence; The dimension of the coding sequence is converted to 128 dimensions through the first convolutional layer to obtain a target code stream.

2. The method according to claim 1, characterized in that The extracting speech features corresponding to the speech waveform of the target speech by inputting a dilated convolution module includes: Extract shallow speech features through one-dimensional causal convolutional layer; The shallow features of the speech are input into the hollow causal convolution layer to extract the speech features corresponding to the speech waveform of the target speech.

3. The method according to claim 1, characterized in that The downsampling process is performed on the speech features by a downsampling module to obtain a downsampling sequence, including: Processing the speech features through an activation layer; Superimposing the speech feature with the output of the activation layer to obtain a feature to be sampled; The features to be sampled are downsampled and compressed by different multiples through the dilated causal convolution to obtain a downsampled sequence.

4. The method according to claim 1, wherein The long short-term memory neural network with added exponential activation and matrix operation is used to perform long-term time series modeling on the down-sampled sequence to obtain a coding sequence, including: Performing long-term temporal modeling on the downsampled sequence through a long short-term memory neural network in an exponentially activated form to obtain an encoded sequence, or; The down-sampled sequence is modeled into a long short-term memory neural network based on a self-attention mechanism to obtain a coding sequence.

5. The method according to claim 1, wherein The method further comprises: The target code stream is subjected to step-by-step residual quantization by a residual vector quantizer to obtain a quantized target code stream.

6. A speech decoding method, characterized in that: include: Get the target stream; Performing convolution processing on the target code stream through a second convolution layer to obtain a decoding sequence; The encoding sequence is modeled and restored by adding a long short-term memory neural network with exponential activation and matrix calculation to obtain an upsampled sequence; Performing upsampling processing on the upsampling sequence by different multiples through an upsampling module to obtain speech features; The speech features are convolved through an output transposed convolution layer to obtain the target speech.

7. A speech coding and decoding model, characterized in that: The speech coding and decoding model includes a speech coding module and a quantization module for implementing the speech coding method according to any one of claims 1 to 5, and a speech decoding module for implementing the speech decoding method according to claim 6; The speech coding module includes an input hole convolution module, a downsampling module, a long short-term memory neural network and a first convolution layer; The speech decoding module includes a second convolutional layer, a long short-term memory neural network, an upsampling module, and an output transposed convolutional layer.

8. The speech coding and decoding model according to claim 7, characterized in that The input dilated convolution module includes a one-dimensional convolution layer and a dilated causal convolution layer, and the first convolution layer includes a one-dimensional convolution layer; The second convolutional layer includes a one-dimensional convolutional layer and a void causal convolutional layer, and the output transposed convolutional layer includes a one-dimensional convolutional layer and a transposed causal convolutional layer; The speech coding model further includes a discriminator module, wherein the discriminator module includes a first discriminator, a second discriminator and a third discriminator; Processing the first time-frequency spectrum feature of the target speech by a first discriminator to obtain a first time-frequency spectrum graph; Processing the second time-frequency spectrum feature of the target speech by a second discriminator to obtain a second time-frequency spectrum graph; Processing the third time-frequency spectrum feature of the target speech by a third discriminator to obtain a third time-frequency spectrum graph; Aggregating the first time-spectrogram, the second time-spectrogram, and the third time-spectrogram to obtain a speech waveform; The first time-frequency spectrum feature, the second time-frequency spectrum feature, and the third time-frequency spectrum feature are of different types.

9. A training method based on a speech coding and decoding model, characterized in that: The training method is applied to the speech coding model according to any one of claims 7 to 8, using a codebook-enhanced generation segment training strategy, wherein the speech coding model is a generator, and the discriminator adopts the discriminator according to claim 7, and the training method includes: In the first training stage, the speech encoding module and the speech decoding module are trained using training samples; the reconstruction loss functions of the speech encoding module and the speech decoding module include the time domain loss function and the frequency domain loss function of the generator; In the second training phase, the speech encoding module, the speech decoding module and the quantization module are trained using the training samples; the loss functions of the speech encoding module and the speech decoding module include the commitment loss function of the quantization module; In the third training stage, the speech decoding module is trained using training samples.

10. The method according to claim 9, characterized in that The commitment loss function includes a negative entropy regularization term, which is used to calculate the Euclidean distance matrix between the input features and the embedding matrix, generate a probability distribution function through a softmax function, and use a soft allocation strategy to make the gradient of the commitment regularization term propagate in a certain direction.

Images