Speech compression method and system based on multi-scale back-projection feature fusion

Abstract

The present invention discloses a speech compression method and system based on multi-scale back-projection feature fusion, which belongs to the field of speech synthesis technology. The method calls multiple multi-scale back-projection feature fusion layers in an encoder to encode a speech signal to be synthesized to obtain speech features; and calls multiple multi-scale back-projection feature fusion layers in a decoder to decode the speech features to obtain synthesized speech. The multi-scale back-projection feature fusion layer encodes or decodes the input features, wherein the multi-scale back-projection feature fusion layer encodes or decodes the input features, including: using convolution kernels of different scales to cross-learn the input features to obtain multi-scale features; back-projecting the multi-scale features to obtain back-projection features; fusing the back-projection features and multi-scale features, and then fusing them with the features of the input multi-scale back-projection feature fusion layer to obtain the output features of the multi-scale back-projection feature fusion layer. This method improves the quality of speech synthesis and solves the technical problem of limited speech synthesis quality in current methods.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of speech signal processing technology, and in particular to a speech compression method and system based on multi-scale back-projection feature fusion. Background Art

[0002] Low-rate speech coding technology has wide application demands in many key fields such as satellite communication, shortwave communication, underwater acoustic communication and confidential communication.

[0003] The current method mainly synthesizes the original speech through a speech coding model, which includes an encoding end and a decoding end; the encoding end downsamples the original speech through a convolutional neural network to complete the feature extraction of the speech signal, and quantizes and dequantizes the features extracted by the encoding end to obtain the dequantized speech. The decoding end uses a convolutional neural network to upsample the dequantized speech, restore the input speech, and obtain the speech synthesis result.

[0004] Speech signals are complex, containing a wealth of information such as pitch, rhythm, and timbre. Speech signals also possess multi-scale characteristics, encompassing speech information across different time and frequency ranges. Current methods for speech synthesis, however, rely solely on convolutional neural networks to capture speech signal features. These often fail to fully capture these complex multi-scale features, leading to a loss of detailed speech information. Furthermore, current methods lack mechanisms to compensate for these details after upsampling and downsampling, limiting the quality of the resulting synthesized speech. Summary of the Invention

[0005] The present invention provides a speech compression method and system based on multi-scale back-projection feature fusion, which can improve the quality of speech synthesis. The technical solution is as follows:

[0006] On the one hand, a speech compression method based on multi-scale back-projection feature fusion is provided, including:

[0007] Call multiple multi-scale back-projection feature fusion layers in the encoder to perform multi-layer feature encoding on the synthesized speech signal to obtain speech features;

[0008] Call multiple multi-scale back-projection feature fusion layers in the decoder to perform multi-layer feature decoding on the speech features to obtain synthesized speech;

[0009] Among them, the process of encoding or decoding the input features by the multi-scale back-projection feature fusion layer includes: using convolution kernels of different scales to cross-learn the input features to obtain multi-scale features, back-projecting the extracted multi-scale features respectively to obtain back-projection features, fusing the back-projection features and the multi-scale features to obtain a fusion result, fusing the fusion result with the features of the input multi-scale back-projection feature fusion layer to obtain the output features of the multi-scale back-projection feature fusion layer.

[0010] Optionally, the multi-scale back-projection feature fusion layer uses convolution kernels of different scales to perform multi-path convolution processing on the input features, splices the features after all path convolution processing to obtain splicing features of multiple paths; and performs feature extraction on each splicing feature to obtain multi-scale features.

[0011] Optionally, the multi-scale back-projection feature fusion layer first downsamples each multi-scale feature and then upsamples it to obtain an upsampled feature, calculates the residual between the multi-scale feature and its upsampled feature, and performs weighted fusion on the multi-scale feature and the residual to obtain a fusion result; all fusion results are spliced ​​to obtain a spliced ​​feature, and the spliced ​​feature is residually connected with the feature of the input multi-scale back-projection feature fusion layer to obtain the output feature of the multi-scale back-projection feature fusion layer.

[0012] Optionally, the weights used in weighted fusion of multi-scale features and residuals are obtained by calculating the attention vectors of the multi-scale features and the residuals and normalizing the two attention vectors.

[0013] Optionally, after performing multi-layer feature encoding on the speech signal to be synthesized, the multiple multi-scale back-projection feature fusion layers in the encoder input the signal into a residual convolution feedforward network, and the output features of the residual convolution feedforward network are subjected to convolution processing to output speech features;

[0014] After multiple multi-scale back-projection feature fusion layers in the decoder perform multi-layer feature decoding on the speech features, the features are input into the residual convolution feedforward network. The output features of the residual convolution feedforward network are convolutionally processed to output the synthesized speech.

[0015] Among them, the residual convolution feedforward network processes the features input to the network including:

[0016] The features input to the network are convolved and randomly inactivated to obtain expanded features, which are then subjected to depthwise separable convolution to obtain channel-enhanced features. The channel-enhanced features are added to the expanded features to obtain residual-enhanced features. The residual-enhanced features are convolved and then residually connected with the features of the input residual convolution feedforward network to obtain speech features.

[0017] Optionally, calling a convolutional layer in the encoder to extract basic temporal features of the speech signal to be synthesized;

[0018] Afterwards, multiple layers of multi-scale back-projection feature fusion layers are used to perform multi-layer feature extraction on the basic time series features to obtain speech features.

[0019] On the other hand, a speech compression system based on multi-scale back-projection feature fusion is provided, including:

[0020] An encoding module is used to call multiple multi-scale back-projection feature fusion layers in the encoder to perform multi-layer feature encoding on the speech signal to be synthesized to obtain speech features;

[0021] The decoding module is used to call multiple multi-scale back-projection feature fusion layers in the decoder to perform multi-layer feature decoding on the speech features to obtain synthesized speech;

[0022] Among them, the process of encoding or decoding the input features by the multi-scale back-projection feature fusion layer includes: the multi-scale back-projection feature fusion layer uses convolution kernels of different scales to cross-learn the input features to obtain multi-scale features, back-projects the extracted multi-scale features respectively to obtain back-projection features, fuses the back-projection features and the multi-scale features to obtain a fusion result, and fuses the fusion result with the features of the input multi-scale back-projection feature fusion layer to obtain the output features of the multi-scale back-projection feature fusion layer.

[0023] In another aspect, a computer device is provided, comprising:

[0024] a processor adapted to execute a computer program;

[0025] A computer-readable storage medium having a computer program stored therein, wherein when the computer program is executed by the processor, the speech compression method based on multi-scale back-projection feature fusion provided in the above aspect is implemented.

[0026] On the other hand, a computer-readable storage medium is also provided, which stores a computer program, and the computer program is suitable for being loaded by a processor and executing the speech compression method based on multi-scale back-projection feature fusion provided by the above aspect.

[0027] On the other hand, a computer program product is also provided, which includes a computer program. When the computer program is executed by a processor, it implements the speech compression method based on multi-scale back-projection feature fusion provided by the above aspect.

[0028] The embodiments of the present application provide a speech compression method and system based on multi-scale back-projection feature fusion. The method uses a multi-scale back-projection feature fusion layer in the process of encoding a speech signal through an encoder to obtain speech features, and decoding the speech features through a decoder to obtain synthesized speech. The multi-scale back-projection feature fusion layer uses convolution kernels of different scales to cross-learn the input features, captures speech detail features of different scales through cross-learning, obtains multi-scale features, back-projects the extracted multi-scale features respectively to obtain back-projection features, compresses, reconstructs and performs residual feedback on the captured multi-scale features using a back-projection mechanism to obtain finer features, fuses the back-projection features and the multi-scale features to obtain a fusion result, fuses the fusion result with the features of the input multi-scale back-projection feature fusion layer to obtain the output features of the multi-scale back-projection feature fusion layer, and solves the problem of inconsistent information levels in multi-scale feature fusion through feature fusion, thereby improving the quality of speech synthesis.

[0029] In addition, the method proposed in the embodiment of the present application also adds a residual convolution feedforward network to the encoder and decoder, introduces depth-separable convolution and residual connection in the residual convolution feedforward network, improves the model's local modeling ability and the temporal expression ability of speech, enhances detail information and reduces information redundancy, and further improves the synthesis quality of speech by combining the residual convolution feedforward network with the multi-scale back-projection feature fusion layer. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0031] Figure 1 A schematic diagram of the overall flow of a speech compression method based on multi-scale back-projection feature fusion provided in an embodiment of the present application;

[0032] Figure 2 A flow chart of speech feature quantization in a speech compression method based on multi-scale back-projection feature fusion provided in an embodiment of the present application;

[0033] Figure 3 A structural block diagram of a speech synthesis model in a speech compression method based on multi-scale back-projection feature fusion provided in an embodiment of the present application;

[0034] Figure 4 A structural block diagram of a multi-scale back-projection feature fusion layer in a speech compression method based on multi-scale back-projection feature fusion provided in an embodiment of the present application;

[0035] Figure 5 A structural block diagram of the back-projection mechanism provided in an embodiment of the present application;

[0036] Figure 6 This is a structural diagram of the feature fusion block provided in the embodiment of the present application;

[0037] Figure 7 This is a block diagram of the residual convolution feedforward network structure in the speech compression method based on multi-scale back-projection feature fusion provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0039] Low-rate speech coding technology has widespread application in key areas such as satellite communications, shortwave communications, underwater acoustic communications, and secure communications. The demand for speech synthesis technology is particularly increasing in bandwidth-constrained or extremely harsh environments. For example, satellite communications often require traversing the atmosphere and space, where signal transmission bandwidth is extremely limited. In these situations, the use of low-rate speech coding can significantly save transmission bandwidth and improve communication efficiency. In encrypted communications, low-rate speech synthesis technology not only compresses data but also enhances speech security through encryption and coding, preventing data theft or tampering. In extremely harsh mountainous environments, where communication equipment must withstand extreme temperatures and strong winds while maintaining stable operation under limited energy supplies, the coding rate for speech synthesis is often extremely low. As the coding rate decreases, speech synthesis quality degrades. Therefore, ensuring speech synthesis quality at low bit rates has become a key issue in current research on speech coding and synthesis technology.

[0040] The current method mainly synthesizes the original speech through the speech coding model. The process of synthesizing speech using the speech coding model includes:

[0041] (1) Encoding side: Use convolutional neural network to downsample the original speech so that the main feature information can be retained while compressing and the acoustic information can be extracted.

[0042] (2) Quantization end: A codebook is designed to quantize the features extracted by the encoding end, and the codebook is updated by calculating the Euclidean distance between the input vector and the initialized codebook; the codeword with the closest Euclidean distance to the input vector is calculated and replaced with the codebook index for transmission.

[0043] (3) Dequantization end: Dequantize the quantized features generated by the quantization end according to the quantization index, and restore the acoustic features according to the codebook.

[0044] (4) Decoding end: Use convolutional neural network to upsample the dequantized speech and restore the input speech.

[0045] (5) Discriminator: The discriminator is responsible for evaluating the generated speech and encourages the generator to generate more realistic speech through adversarial training, thereby improving the quality of the reconstructed speech.

[0046] Speech signals are complex, containing a rich array of information, such as pitch, rhythm, and timbre. Speech signals also possess multi-scale features, encompassing speech information across different time and frequency ranges. However, current methods often fail to fully capture these complex multi-scale features, resulting in loss of detailed speech information. Alternatively, they suffer from information inconsistency and redundancy when fusing multi-scale information, limiting speech synthesis quality. Furthermore, the encoders and decoders in current methods lose high-frequency details during upsampling and downsampling, resulting in poor synthesized speech quality. During downsampling, especially for low-bitrate speech coding, excessive compression of details distorts the speech signal, making it difficult for the decoder to restore the original clear, natural speech. While upsampling can restore features to their original resolution, the decoder often blurs and fails to fully restore details due to information loss. Therefore, current methods lack a mechanism to compensate for the details of up- and down-sampled speech, limiting the quality of the resulting synthesized speech.

[0047] In order to improve the quality of speech synthesis, the embodiment of the present application proposes a speech compression method based on multi-scale back-projection feature fusion. A multi-scale back-projection feature fusion layer and a residual convolution feedforward network are introduced in the encoder and decoder to improve the quality of synthesized speech. Among them, the multi-scale back-projection feature fusion layer uses convolution kernels of different scales to cross-learn the features input to the layer, captures more complete speech feature details, obtains multi-scale features, and back-projects the extracted multi-scale features to obtain back-projection features. The back-projection mechanism is used to compress, reconstruct and provide residual feedback on the captured multi-scale features to obtain more refined features; the feature fusion block adaptively adjusts the weights from different speech branches and fuses them to solve the problem of inconsistent information levels in multi-scale feature fusion; secondly, depthwise separable convolution and residual connection are introduced in the residual convolution feedforward network to improve the model's local modeling ability and the temporal expression ability of speech, enhance detailed information and reduce information redundancy. By combining the residual convolution feedforward network with the multi-scale back-projection feature fusion layer, the synthesis quality of speech is further improved.

[0048] The following is a detailed description of the speech compression method based on multi-scale back-projection feature fusion provided in the embodiment of the present application.

[0049] The speech compression method based on multi-scale back-projection feature fusion provided in the embodiment of the present application is as follows: Figure 1-Figure 7 Shown, including:

[0050] Call multiple multi-scale back-projection feature fusion layers in the encoder to perform multi-layer feature encoding on the synthesized speech signal to obtain speech features;

[0051] Call multiple multi-scale back-projection feature fusion layers in the decoder to perform multi-layer feature decoding on the speech features to obtain synthesized speech;

[0052] Among them, the process of encoding or decoding the input features by the multi-scale back-projection feature fusion layer includes: using convolution kernels of different scales to cross-learn the input features to obtain multi-scale features, back-projecting the extracted multi-scale features respectively to obtain back-projection features, fusing the back-projection features and the multi-scale features to obtain a fusion result, fusing the fusion result with the features of the input multi-scale back-projection feature fusion layer to obtain the output features of the multi-scale back-projection feature fusion layer.

[0053] In one possible implementation, the multi-scale back-projection feature fusion layer uses convolution kernels of different scales to perform multi-path convolution on the input features. The convolution features of all paths are then concatenated to obtain concatenated features from multiple paths. Feature extraction is then performed on each concatenated feature to obtain multi-scale features. By using convolution kernels of different scales for cross-learning, more complete speech feature details can be captured.

[0054] In one possible implementation, the multi-scale back-projection feature fusion layer first downsamples each multi-scale feature and then upsamples it to obtain the upsampled feature, calculates the residual between the multi-scale feature and its upsampled feature, and performs weighted fusion on the multi-scale feature and the residual to obtain a fusion result; all the fusion results are spliced ​​to obtain a spliced ​​feature, and the spliced ​​feature is residually connected with the feature of the input multi-scale back-projection feature fusion layer to obtain the output feature of the multi-scale back-projection feature fusion layer.

[0055] The weights used in the weighted fusion of multi-scale features and residuals are obtained by calculating the attention vectors of the multi-scale features and residuals and normalizing the two attention vectors. By adaptively adjusting the weights from different speech branches and fusing them, the problem of inconsistent information levels in multi-scale feature fusion is resolved.

[0056] In one possible implementation, multiple multi-scale back-projection feature fusion layers in the encoder perform multi-layer feature encoding on the speech signal to be synthesized, and then input the encoded data into a residual convolutional feedforward network. The output features of the residual convolutional feedforward network are convolutionally processed to output speech features.

[0057] After multiple multi-scale back-projection feature fusion layers in the decoder perform multi-layer feature decoding on the speech features, the features are input into the residual convolution feedforward network. The output features of the residual convolution feedforward network are convolutionally processed to output the synthesized speech.

[0058] Among them, the residual convolution feedforward network processes the features input to the network including:

[0059] The features input to the network are convolved and randomly inactivated to obtain expanded features, and the expanded features are subjected to depthwise separable convolution to obtain channel enhanced features. The channel enhanced features are added to the expanded features to obtain residual enhanced features. The residual enhanced features are convolved and then residually connected with the features of the input residual convolution feedforward network to obtain the output features of the residual convolution feedforward network.

[0060] By introducing depthwise separable convolution and residual connection, the local modeling capability and the temporal expression capability of speech are improved to enhance detail information and reduce information redundancy.

[0061] In one possible implementation, a convolutional layer in an encoder is called to extract basic temporal features of the speech signal to be synthesized;

[0062] Afterwards, multiple layers of multi-scale back-projection feature fusion layers are used to perform multi-layer feature extraction on the basic time series features to obtain speech features.

[0063] The embodiment of the present application provides a speech compression method based on multi-scale back-projection feature fusion. At the encoding end, the encoder uses a multi-scale back-projection feature fusion layer to extract features from the speech signal, and then performs back-projection mapping on the extracted multi-scale features. The extracted multi-scale features are corrected by feedback residual information, and the extracted multi-scale features are further refined. The feature fusion block is used to fuse them to improve the model's understanding of features at different scales. After the encoder performs downsampling, the features are enhanced by a convolutional feedforward network, and the feature context information is learned to obtain a latent vector. At the quantization end, the latent vector is quantized using multi-level vector quantization, and the encoded index is packaged into a binary byte stream for transmission. At the decoding end, the decoder obtains the quantized vector of the codebook based on the index, and then synthesizes speech through upsampling, a multi-scale back-projection feature fusion layer, and a residual convolutional feedforward network. In addition, a discriminator is introduced during the training process to distinguish the authenticity of the synthesized speech. Through adversarial training, the generator is prompted to generate more realistic speech, thereby improving the quality of the reconstructed speech.

[0064] like Figure 1 As shown, the specific embodiment process of the speech compression method based on multi-scale back-projection feature fusion provided in the embodiment of the present application includes:

[0065] (1) Obtain the speech signal to be synthesized, wherein the speech signal to be synthesized is a speech signal sampled at 8KHz. The speech signal can be obtained by express, is the number of speech channels, T is the total number of speech sample points, , where d is the duration of the speech signal, is the sampling rate of the speech signal.

[0066] (2) At the encoding end, multiple multi-scale back-projection feature fusion layers in the encoder are called to perform multi-layer feature encoding on the compressed speech signal to obtain speech features.

[0067] like Figure 3 As shown in the figure, the encoder consists of two convolutional layers, four multi-scale back-projection feature fusion layers, and a residual convolution feedforward network. The core of the encoder is the introduction of the multi-scale back-projection feature fusion layer and the residual convolution feedforward network. Among them, the two convolutional layers are one-dimensional convolutions.

[0068] The encoder encodes the features of the synthesized speech signal. The process of obtaining speech features includes:

[0069] First, the speech signal to be synthesized input to the encoder is convolved through one of the convolutional layers to obtain the basic temporal features of the speech signal to be synthesized; this convolutional layer is a one-dimensional convolution with 32 channels and a convolution kernel size of 7.

[0070] Afterwards, the basic temporal features are encoded with multi-layer features through four multi-scale back-projection feature fusion layers. For the first three multi-scale back-projection feature fusion layers, the output features of each multi-scale back-projection feature fusion layer are used as the input features of the next multi-scale back-projection feature fusion layer, and the output features of the last multi-scale back-projection feature fusion layer are used as the input features of the residual convolution feed-forward network.

[0071] Each multi-scale back-projection feature fusion layer consists of three parts: multi-scale convolution block, back-projection mechanism and feature fusion block.

[0072] like Figure 4As shown in the figure, the multi-scale convolution block processes the features of the input multi-scale back projection feature fusion layer separately through two different receptive field paths. One path convolves the feature x through a convolution layer with a convolution kernel size of 3 to obtain the first convolution feature; the other path convolves the feature x through a convolution layer with a convolution kernel size of 5 to obtain the second convolution feature. The convolution features extracted by the two paths are spliced ​​in the channel dimension to obtain two spliced ​​features. Subsequently, one-dimensional convolution features are extracted from the two spliced ​​features through convolution layers (3 and 5) of matching sizes. Then, one-dimensional convolution is performed on the features extracted from the two spliced ​​features to reduce the features back to the original dimension to generate multi-scale features x1 and multi-scale features x2.

[0073] The multi-scale features x1 and x2 are processed through the back-projection mechanism to achieve the correction of the detail features in the multi-scale features. Figure 5 As shown in the figure, taking the multi-scale feature x1 as an example, the back-projection mechanism is explained. A convolutional layer with normalization and causal convolution with a step size of 2 is used to downsample the multi-scale feature x1, compress the time dimension length, and obtain the down-sampled result of the multi-scale feature x1. Subsequently, the corresponding transposed convolutional layer is used to upsample the down-sampled result of the multi-scale feature x1, and try to restore it to a time series length close to the original fusion feature to obtain the up-sampled feature. Then, the residual between the multi-scale feature x1 and the up-sampled feature is calculated, and the residual is input into the residual block to further model the error signal, and nonlinear correction is performed to obtain the residual R1.

[0074] Then the multi-scale feature x1 and the residual R1 corrected by the back-projection residual are sent to the feature fusion block, which dynamically assigns different weights to adjust the importance of the multi-scale features. Specifically, Figure 6As shown in the figure, the multi-scale feature x1 and the residual R1 are first added together as the main input feature U, and then the two features are stacked along the last dimension into a new four-dimensional tensor [B, C, T, 2], where B is the batch size, C is the number of channels, T is the time dimension, and 2 represents two branches. The main input feature U is subjected to global average pooling and global maximum pooling in the channel dimension, extracting global semantic features and local salient features. The last dimension is expanded to a shape of [B, C, 1], obtaining average pooling features and maximum pooling features. These are then concatenated and reduced to their original dimensions through a one-dimensional convolution. The features are then mapped using a Pre-Reduced Linear Unit (PReLU) activation to form a latent vector z. Attention vectors are then calculated for each branch through the corresponding convolutional layer. The attention vectors of each branch are then concatenated along the branch dimension to form a tensor of [B, 2, C]. The attention vectors of the two branches are then softmax-normalized so that the sum of the weights between branches is 1, enhancing discriminability. This results in multi-scale feature weights and residual weights. Finally, the multi-scale feature x1 and the residual R1 are weighted and summed according to the above weights to generate the fusion result T1. The same steps are followed for the multi-scale feature x2 and the residual R2 to obtain the fusion result T2. The fusion results T1 and T2 are then concatenated in the channel dimension and reduced back to the original dimension through one-dimensional convolution. This yields feature T. A skip connection is used to obtain the identity mapping I of the input multi-scale back-projection feature fusion layer. Next, a residual connection is made between T and the identity mapping I to obtain the output features of the multi-scale back-projection feature fusion layer.

[0075] Each multi-scale back-projection feature fusion layer in the encoder is followed by a one-dimensional convolution with downsampling, with a stride of S and a kernel twice the stride S. The number of channels is doubled during the downsampling process. Where S = (1, 4, 8, 10).

[0076] The output features of the last multi-scale back-projection feature fusion layer in the encoder are downsampled by a one-dimensional convolution and then fed into a residual convolutional feedforward network. This network further performs nonlinear modeling and local receptive field enhancement on the encoded speech features. Its architecture primarily consists of two layers of 1×1 convolutions, a depthwise separable convolution, an activation function, and a random dropout mechanism. Residual connections are introduced to enhance training stability. Assuming the input feature B to the residual convolutional feedforward network is C, and the time step is T, the input tensor size of the residual convolutional feedforward network is [B, C, T].

[0077] like Figure 7 As shown in Figure 2, the residual convolution feedforward network processes feature B as follows:

[0078] First, feature B is convolved through a 1×1 convolutional layer, expanding the number of channels of feature B from C to 2C, achieving a dimensionality increase in the channel dimension. Dropout is added after the convolution to prevent overfitting, resulting in the expanded feature B1. Subsequently, a depthwise separable convolution is performed on the expanded feature B1 with a kernel size of 3, a padding of 1, and a number of groups equal to the number of channels (i.e., each channel is independently convolved). This enhances the local perception of each channel, resulting in the channel-enhanced feature B*. A residual enhancement strategy is then employed to add the expanded feature B1 and the channel-enhanced feature B* to obtain the residual-enhanced feature B2. The nonlinear representation of the residual-enhanced feature B2 is then enhanced using the GELU activation function. A 1×1 convolution is then performed to restore the number of channels back to the original dimension C, completing the channel compression mapping. Finally, dropout is applied again and a residual connection is made with the input feature B to obtain the output feature Y of the residual convolution feedforward network. The design parameters of the residual convolutional feedforward network are as follows: input channels C = 512, expansion factor = 2, corresponding to 2C = 1024 intermediate channels, and dropout ratio of 0.2. The speech features output by the encoder are then obtained through a one-dimensional convolutional layer with a kernel size of 7 and a channel size of 512.

[0079] The speech features output by the encoder are quantized at the quantization end.

[0080] The quantization end includes a multi-level vector quantization system, which contains M codebooks, M=256, the vector dimension C of each codebook is 512, and the number of potential frame vectors N is 25. Each codebook can encode To ensure the quantization performance and convergence speed, the K-Means clustering algorithm is used to divide the potential vector space after the encoder is encoded to obtain the initialized codebook set , the speech feature set input to the quantization end is , corresponding to the low-dimensional feature representation obtained by the encoder processing for each frame. The embodiment of the present application uses a multi-level quantizer to quantize the speech features output by the encoder. The multi-level quantizer consists of several levels, each level quantizes the quantization residual of the previous level, and gradually approaches the original vector. Each level of the quantizer uses the Euclidean distance metric as the similarity index to quickly and accurately locate the most matching codebook vector, thereby minimizing the quantization error. The first level quantizer finds the quantization vector closest to the speech feature z in the codebook set C. , the second-stage quantizer quantizes the first-stage vector The residual is quantized to obtain the second-level quantization vector , ; The third-level quantizer continues to quantize the remaining residual, that is, the third-level quantizer quantizes the second-level quantization vector and the first-level quantization vector The residual is quantized to obtain the second-level quantization vector , The remaining quantizers are iterated in sequence until all quantization is completed. The quantized vector is stored as an index; the obtained quantized vector index is compressed and transmitted to the decoding end.

[0081] Preferably, the quantized speech features are packaged into a binary byte stream and transmitted to the decoding end.

[0082] At the decoding end, the dequantized features are obtained in the multi-level quantizer according to the index, and the final potential frame vector is obtained by adding them in sequence. .

[0083] The dequantized speech features The input is fed into the decoder to synthesize speech. The decoder's structure is symmetrical to the encoder's. It first undergoes a one-dimensional convolution with a kernel size of 7 and 512 channels. This is followed by four multi-scale back-projection feature fusion layers. Unlike the encoder, each multi-scale back-projection feature fusion layer is followed by a one-dimensional transposed convolution with a stride of E and a kernel size twice that of stride E. The number of channels is doubled during upsampling, where E = (10, 8, 4, 1). Local features are then enhanced using a residual convolutional feedforward network, followed by a one-dimensional convolution with a kernel size of 7 and 32 channels. The resulting reconstructed speech, or synthesized speech, is then obtained.

[0084] To further improve the speech synthesis quality of the generator, during the training phase, the embodiments of the present application use a multi-scale STFT discriminator (MS-STFT) and a multi-period discriminator (MPD) to judge the quality of the synthesized speech. These discriminators are only used during the training phase. The generator refers to an overall network model consisting of an encoder, a quantizer, and a decoder. Specifically, the main function of the discriminator is to determine the quality of the generator's generation and to train the generator to better generate speech signals. The input of the discriminator is the input speech of the encoder and the synthesized speech output by the decoder. The input speech is the original speech. By predicting the authenticity of the synthesized speech, the discriminator learns how to distinguish between the original speech and the synthesized speech, and feeds the prediction results of the synthesized speech back to the generator, competing and collaborating with each other in an adversarial training manner. The MS-STFT discriminator consists of multiple sub-discriminators with the same structure, each acting on the complex STFT representation at different scales. Specifically, it employs five sub-discriminators, corresponding to five short-time Fourier transform (STFT) window lengths: 2048, 1024, 512, 256, and 128, with sliding steps of 512, 256, 128, 64, and 32, respectively. Each sub-discriminator operates at a different time scale and frequency bandwidth to fully extract both local details and global structural information from the speech signal. After the input speech signal undergoes the STFT transform, the real and imaginary parts of its complex spectrum are extracted and concatenated along the channel dimension to form a two-dimensional input tensor with the shape [B, 2, F, T], where F is the frequency dimension and T is the number of time frames. This complex spectrum is then fed into a two-dimensional convolutional network with the same structure for discrimination. The 2D convolutional network architecture consists of a 3×8 kernel and 32 channels; several 2D convolutional layers with varying dilation rates D in the time dimension and stride 2 in the frequency dimension; and a final 2D convolutional layer (kernel size 3×3, stride 1×1) for authenticity judgment. The discriminator at each scale outputs a set of true and false classifications and intermediate feature maps. The results of all sub-discriminators are combined for training.

[0085] MPD is a hybrid architecture consisting of multiple sub-discriminators, each focused on modeling speech patterns with a specific period p, thereby improving the perception of different periodic features. Specifically, the sub-discriminators capture distinct implicit structures by examining different parts of the input audio. The period p is set to [2, 3, 5, 7, 11] to avoid overlap and structural redundancy. The one-dimensional original speech of length T is first reshaped into a two-dimensional matrix of height T / p and width p. Two-dimensional convolutions are then applied to the reshaped two-dimensional matrix, with each convolution followed by a LeakyReLU activation function. All convolutional layers operate on the time dimension, with the kernel size on the width axis constrained to 1 to independently process periodic samples. Each sub-discriminator outputs a feature map of the real and generated samples after each convolution layer. The outputs of all sub-discriminators are combined for training.

[0086] In summary, MS-STFT and MPD form a complementary discrimination mechanism in the frequency and time domains, providing the decoder with more stable and multi-angle training signals, which helps improve the naturalness and clarity of synthesized speech.

[0087] The speech compression method based on multi-scale back-projection feature fusion provided in the embodiments of the present application introduces multi-scale back-projection feature fusion technology and residual convolution feedforward network technology into the encoder and decoder, using multi-scale feature extraction and back-projection residual correction to learn richer and more complete features, and using the residual convolution feedforward network to locally enhance the up- and down-sampled information to improve the quality of synthesized speech. To achieve the above objectives, a multi-scale back-projection feature fusion layer and a residual convolution feedforward network are proposed. The multi-scale back-projection feature fusion layer mainly includes three parts: multi-scale feature extraction, back-projection correction mechanism, and selective feature fusion. The multi-scale convolution captures speech detail features at different scales through cross-learning; the back-projection mechanism compresses, reconstructs, and provides residual feedback on the captured detail features to obtain finer features; the feature fusion block adaptively adjusts the weights from different branches and fuses them. The residual convolution feedforward network improves the model's local modeling capabilities and the temporal expression capabilities of speech by introducing depthwise separable convolution and residual connections in the feedforward structure, enhancing detail information and reducing information redundancy. This scheme improves the quality of speech synthesis by introducing two technologies: multi-scale back-projection feature fusion and residual convolution feedforward network.

[0088] The embodiment of the present application further provides a speech compression system based on multi-scale back-projection feature fusion, including:

[0089] An encoding module is used to call multiple multi-scale back-projection feature fusion layers in the encoder to perform multi-layer feature encoding on the speech signal to be synthesized to obtain speech features;

[0090] The decoding module is used to call multiple multi-scale back-projection feature fusion layers in the decoder to perform multi-layer feature decoding on the speech features to obtain synthesized speech;

[0091] Among them, the process of encoding or decoding the input features by the multi-scale back-projection feature fusion layer includes: the multi-scale back-projection feature fusion layer uses convolution kernels of different scales to cross-learn the input features to obtain multi-scale features, back-projects the extracted multi-scale features respectively to obtain back-projection features, fuses the back-projection features and the multi-scale features to obtain a fusion result, and fuses the fusion result with the features of the input multi-scale back-projection feature fusion layer to obtain the output features of the multi-scale back-projection feature fusion layer.

[0092] The present application also provides a computer device, which includes:

[0093] a processor adapted to execute a computer program;

[0094] A computer-readable storage medium having a computer program stored therein, wherein the computer program, when executed by the processor, implements the speech compression method based on multi-scale back-projection feature fusion provided in an embodiment of the present application.

[0095] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. The computer program is suitable for being loaded by a processor and executing the speech compression method based on multi-scale back-projection feature fusion provided in an embodiment of the present application.

[0096] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the speech compression method based on multi-scale back-projection feature fusion provided in the embodiment of the present application.

[0097] The method disclosed in the embodiments of the present application can be directly implemented as a hardware processor, or can be implemented using a combination of hardware and software modules in the processor. The software module can be located in a storage medium that is well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, it will not be described in detail here.

[0098] Those skilled in the art will appreciate that the units and algorithm steps of the various examples described in conjunction with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0099] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A speech compression method based on multi-scale back-projection feature fusion, characterized in that: include: Call multiple multi-scale back-projection feature fusion layers in the encoder to perform multi-layer feature encoding on the synthesized speech signal to obtain speech features; Call multiple multi-scale back-projection feature fusion layers in the decoder to perform multi-layer feature decoding on the speech features to obtain synthesized speech; Each multi-scale back-projection feature fusion layer in the encoder is followed by a one-dimensional convolution with downsampling; Each multi-scale back-projection feature fusion layer in the decoder is followed by a one-dimensional transposed convolution; Among them, the process of encoding or decoding the input features by the multi-scale back-projection feature fusion layer includes: using convolution kernels of different scales to cross-learn the input features to obtain multi-scale features, back-projecting the extracted multi-scale features respectively to obtain back-projection features, fusing the back-projection features and the multi-scale features to obtain a fusion result, fusing the fusion result with the features of the input multi-scale back-projection feature fusion layer to obtain the output features of the multi-scale back-projection feature fusion layer.

2. The speech compression method based on multi-scale back-projection feature fusion according to claim 1, characterized in that: The multi-scale back-projection feature fusion layer uses convolution kernels of different scales to perform multi-path convolution processing on the input features, splices the features after all path convolution processing to obtain the splicing features of multiple paths; and extracts features from each splicing feature to obtain multi-scale features.

3. The speech compression method based on multi-scale back-projection feature fusion according to claim 1, wherein: The multi-scale back-projection feature fusion layer first downsamples each multi-scale feature and then upsamples it to obtain the upsampled feature, calculates the residual between the multi-scale feature and its upsampled feature, and performs weighted fusion on the multi-scale feature and the residual to obtain the fusion result; all the fusion results are spliced ​​to obtain the spliced ​​feature, and the spliced ​​feature is residually connected with the feature of the input multi-scale back-projection feature fusion layer to obtain the output feature of the multi-scale back-projection feature fusion layer.

4. The speech compression method based on multi-scale back-projection feature fusion according to claim 3, characterized in that: The weights used in the weighted fusion of multi-scale features and residuals are obtained by calculating the attention vectors of multi-scale features and residuals and normalizing the two attention vectors.

5. The speech compression method based on multi-scale back-projection feature fusion according to claim 1, wherein: The encoder uses multiple multi-scale back-projection feature fusion layers to perform multi-layer feature encoding on the synthesized speech signal, which is then input into the residual convolution feedforward network. The output features of the residual convolution feedforward network are then convolved to output speech features. After multiple multi-scale back-projection feature fusion layers in the decoder perform multi-layer feature decoding on the speech features, the features are input into the residual convolution feedforward network. The output features of the residual convolution feedforward network are convolutionally processed to output the synthesized speech. Among them, the residual convolution feedforward network processes the features input to the network including: The features input to the network are convolved and randomly inactivated to obtain expanded features, and the expanded features are subjected to depthwise separable convolution to obtain channel enhanced features. The channel enhanced features are added to the expanded features to obtain residual enhanced features. The residual enhanced features are convolved and then residually connected with the features of the input residual convolution feedforward network to obtain the output features of the residual convolution feedforward network.

6. The speech compression method based on multi-scale back-projection feature fusion according to claim 1, characterized in that: Call the convolutional layer in the encoder to extract the basic temporal features of the speech signal to be synthesized; Afterwards, multiple layers of multi-scale back-projection feature fusion layers are used to perform multi-layer feature extraction on the basic time series features to obtain speech features.

7. A speech compression system based on multi-scale back-projection feature fusion, characterized in that: include: An encoding module is used to call multiple multi-scale back-projection feature fusion layers in the encoder to perform multi-layer feature encoding on the speech signal to be synthesized to obtain speech features; The decoding module is used to call multiple multi-scale back-projection feature fusion layers in the decoder to perform multi-layer feature decoding on the speech features to obtain synthesized speech; Each multi-scale back-projection feature fusion layer in the encoder is followed by a one-dimensional convolution with downsampling; Each multi-scale back-projection feature fusion layer in the decoder is followed by a one-dimensional transposed convolution; Among them, the process of encoding or decoding the input features by the multi-scale back-projection feature fusion layer includes: the multi-scale back-projection feature fusion layer uses convolution kernels of different scales to cross-learn the input features to obtain multi-scale features, back-projects the extracted multi-scale features respectively to obtain back-projection features, fuses the back-projection features and the multi-scale features to obtain a fusion result, and fuses the fusion result with the features of the input multi-scale back-projection feature fusion layer to obtain the output features of the multi-scale back-projection feature fusion layer.

8. An electronic device, characterized in that: The device comprises: a processor adapted to execute a computer program; A computer-readable storage medium having a computer program stored therein, wherein the computer program, when executed by the processor, implements the speech compression method based on multi-scale back-projection feature fusion according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is suitable for being loaded by a processor and executing the speech compression method based on multi-scale back-projection feature fusion according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, it implements the speech compression method based on multi-scale back-projection feature fusion according to any one of claims 1 to 6.

Images