Speech Encoding and Decoding Methods Based on Principal Component Analysis and Multi-Scale Deep Attention

Abstract

This invention discloses a speech encoding / decoding method based on principal component analysis and multi-scale deep attention, relating to the field of speech signal processing technology. The method includes: performing multiple multi-scale deep attention convolution operations on the speech signal to obtain a first feature; wherein features are extracted separately through multiple parallel, depth-separable convolutions with different depth kernels, and after feature fusion, the fused features are multiplied channel-by-channel according to channel attention weights; the first feature is projected based on a projection matrix and the feature mean to obtain a second feature; wherein principal component analysis is performed on the first feature of the training set to determine the feature mean, and the top feature vectors with the highest cumulative variance contribution rate are selected to form the projection matrix; the second feature is sequentially subjected to multi-level vector quantization to obtain a reconstructed vector; the reconstructed vector is then decoded and the reconstructed speech is output. This achieves high-quality speech communication with ultra-low bitrates.

Description

Technical Field

[0001] This invention relates to the field of speech signal processing technology, and in particular to a speech encoding and decoding method based on principal component analysis and multi-scale deep attention. Background Technology

[0002] Low bit-rate speech coding technology is widely used in satellite communications, secure communications, underwater acoustic communications, and VoIP (Voice over Internet Protocol) in congested network environments. Because these specialized applications are generally subject to strict limitations on channel bandwidth (typically below 1 kbps) and the complexity of the transmission environment, achieving high-definition, highly natural speech transmission under extremely low bit-rate constraints remains a key challenge in the field of speech signal processing. For example, in ocean operations or disaster relief, communication equipment often relies on extremely narrowband channels, where speech coding efficiency directly affects the accuracy and timeliness of command transmission.

[0003] With the rapid development of deep learning technology, end-to-end speech codecs based on neural networks have gradually replaced traditional linear predictive coding techniques. A typical neural audio codec architecture usually consists of an encoder composed of convolutional neural networks, a residual vector quantizer (RVQ), and a decoder. The encoder is responsible for downsampling the high-sampling-rate temporal speech waveform and mapping it into compact latent acoustic features; the quantizer discretizes the continuous feature vectors into bitstream indices through a multi-level codebook to achieve compressed transmission; after receiving the bitstream, the decoder recovers the features by looking up a table and reconstructs the speech waveform using a deconvolutional network. In addition, to improve the listening quality of synthesized speech, a discriminator from a Generative Adversarial Network (GAN) is usually introduced during the training phase for auxiliary supervision.

[0004] However, although existing neural codecs perform excellently at medium to high bitrates (above 3kbps), they still face technical bottlenecks when dealing with ultra-low bitrates (below 1kbps, such as 0.8kbps) and lightweight edge deployments.

[0005] First, there is a serious contradiction between the high-dimensional redundancy of the feature space and the low bit budget. Current mainstream models typically set the dimensionality of the latent feature space to a high level to preserve sufficient semantic and acoustic details. Under ultra-low bitrate conditions, the quantizer can only use a very limited number of codebooks to represent high-dimensional features. Since high-dimensional features often exhibit strong linear correlations between their dimensions and contain a large amount of non-critical background noise information, limited bit resources are wasted on fitting this redundant information. This sparsity of feature distribution makes it difficult for the quantizer to effectively cluster core semantics, resulting in low codebook utilization and ultimately leading to severe spectral deficiencies, ambiguity, and unnatural mechanical noise in the decoded speech.

[0006] Secondly, the computational complexity of the network architecture is mismatched with the limitations of edge computing power. To capture long-term dependencies and intricate frequency domain structures in speech signals, existing high-performance models often tend to stack deep standard convolutional networks or introduce computationally intensive self-attention mechanisms. While this design improves feature extraction capabilities, it also leads to a surge in the number of model parameters (typically exceeding tens of millions) and extremely high floating-point operations (FLOPs). For mobile terminals, wearable devices, or IoT edge nodes with limited battery capacity and computing power, this translates to a huge power consumption burden and unacceptable inference latency, severely hindering the practical application of the technology.

[0007] Furthermore, directly training low-dimensional models faces optimization challenges. To address the feature redundancy problem, some studies have attempted to directly construct network structures with low-dimensional bottleneck layers. However, without effective prior knowledge guidance, end-to-end training struggles to automatically discover the optimal low-dimensional feature manifold. The network often gets trapped in local optima due to the narrow information bottleneck, leading to significant loss of speech details and even training divergence. Current solutions largely rely on increasing model depth to compensate for the decline in expressive power, but this exacerbates computational complexity. Summary of the Invention

[0008] To address the aforementioned issues, this invention proposes a speech encoding and decoding method based on principal component analysis and multi-scale deep attention. By introducing a statistical prior guidance mechanism based on principal component analysis, the method maximizes the coding efficiency of residual vector quantization under limited bit resources, eliminating ambiguity and distortion in reconstructed speech. A lightweight backbone network based on multi-scale deep separable convolution and channel attention mechanisms is designed, which significantly reduces the number of model parameters and computational complexity while maintaining efficient capture of multi-scale time-frequency features, achieving high-quality speech communication with ultra-low bitrate.

[0009] In a first aspect, the present invention provides a speech encoding and decoding method based on principal component analysis and multi-scale deep attention, comprising:

[0010] After performing multiple multi-scale deep attention convolution operations on the acquired speech signal, the first feature is obtained. Each operation includes: extracting features through multiple parallel separable convolutions with different depth kernels, fusing the features, multiplying the obtained fused features channel by channel according to the channel attention weights, and then proceeding to the next operation.

[0011] The first feature is projected using the projection matrix and the feature mean to obtain the second feature. Specifically, principal component analysis is performed on the first feature of the training set to determine the feature mean, and the features with the highest cumulative variance contribution rate are selected. The eigenvectors constitute the projection matrix;

[0012] By determining the nearest neighbor codeword in the codebook, the second feature is sequentially subjected to multi-level vector quantization to obtain the reconstructed vector;

[0013] The reconstructed vector is processed by inverse projection and then decoded to output the reconstructed speech.

[0014] As an alternative implementation, the depthwise convolution kernel sizes of the multiple parallel depthwise separable convolutions are set to 3, 7, 11, and 15, respectively.

[0015] As an alternative implementation method, the calculation process for channel attention weights is as follows:

[0016] The fused features are compressed into channel descriptors using global average pooling. The calculation formula is: ;

[0017] Then, the weight coefficients of each channel are learned through two fully connected layers. : ;

[0018] in, For the fusion feature of the first The feature channel in the first Feature response values ​​at each time step; For the first The global average feature value of each channel; L represents the total length of the current layer feature map in the time dimension; It is the ReLU activation function. It is the Sigmoid activation function. and These are the weights for the fully connected layer.

[0019] As an alternative implementation, the process of constructing the projection matrix includes: determining the eigenvalue covariance matrix after principal component analysis, performing singular value decomposition on the covariance matrix to obtain the eigenvalue sequence and the corresponding eigenvector sequence; defining the first... The variance contribution rate of the i-th eigenvector is the i-th eigenvalues The proportion of the sum of all eigenvalues, the cumulative variance contribution rate is the sum of the variance contribution rates of the eigenvectors, and the first The cumulative variance contribution rate of each eigenvector is: = D represents the total number of eigenvectors.

[0020] As an alternative implementation method, the projection process is as follows: Inverse projection processing is ;in, This is the second characteristic; The first feature; The characteristic mean; The projection matrix; For reconstructing features.

[0021] As an alternative implementation, the vector quantization process includes: quantizing the input residuals of the q-th level vector quantization. According to the codebook Find the nearest neighbor code Thus, the reconstructed vector is obtained. And update the next level residual. The final output is the sum of the reconstruction vectors at all levels. Among them, the input residual of the first-level vector quantization The second feature is N, where N is the size of each level of the codebook.

[0022] Secondly, the present invention provides a speech encoding and decoding system based on principal component analysis and multi-scale deep attention, comprising:

[0023] The encoding module is configured to perform multiple multi-scale deep attention convolution operations on the acquired speech signal to obtain the first feature; wherein each operation includes: extracting features through multiple parallel and depth-separable convolutions with different depth kernels, fusing the features, multiplying the obtained fused features channel by channel according to the channel attention weights, and then proceeding to the next operation.

[0024] The projection module is configured to project the first feature based on the projection matrix and the feature mean to obtain the second feature; wherein, principal component analysis is performed on the first feature of the training set to determine the feature mean, and the features with the highest cumulative variance contribution rate are selected. The eigenvectors constitute the projection matrix;

[0025] The quantization module is configured to perform multi-level vector quantization on the second feature sequentially by determining the nearest neighbor codeword in the codebook to obtain the reconstructed vector.

[0026] The decoding module is configured to perform inverse projection processing on the reconstructed vector before decoding and outputting the reconstructed speech.

[0027] Thirdly, the present invention provides an electronic device including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in the first aspect.

[0028] Fourthly, the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in the first aspect.

[0029] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0030] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0031] To address the problems of high redundancy of high-dimensional features leading to decreased quantization accuracy, large number of network model parameters making it unsuitable for edge deployment, and difficulty in convergence when directly training low-dimensional models in ultra-low bitrate applications, this invention proposes a speech encoding and decoding method based on principal component analysis and multi-scale deep attention. In terms of network structure, a lightweight backbone network based on multi-scale deep attention convolution is designed. Through parallel multi-scale deep convolution and channel attention mechanisms, the number of model parameters and computational complexity are significantly reduced while ensuring effective capture of acoustic features.

[0032] In terms of quantization strategy, a principal component subspace adaptation mechanism is proposed. By introducing a statistical prior guidance mechanism based on principal component analysis (PCA), after performing PCA on the pre-trained features, a subspace projection layer based on principal component initialization is constructed. This losslessly compresses the high-dimensional acoustic features into the most informative low-dimensional positive tangent form for residual vector quantization, thereby improving the coding efficiency of residual vector quantization with limited bit resources and eliminating the ambiguity and distortion of the reconstructed speech.

[0033] During training, this invention employs a hierarchical learning rate fine-tuning strategy to ensure stable model adaptation to low-dimensional subspaces, thus solving the problem of training instability during the adaptation process of low-dimensional feature spaces. This invention can achieve high-quality speech reconstruction and low computational complexity under ultra-low bitrate (below 1kbps) conditions, and the model is lightweight and efficient, making it suitable for application scenarios with strict limitations on bandwidth and computing power, such as satellite communication, emergency narrowband communication, IoT voice transmission, and mobile edge computing.

[0034] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0036] Figure 1 This is an overall framework diagram of the speech encoding and decoding method based on principal component analysis and multi-scale deep attention provided in Embodiment 1 of the present invention;

[0037] Figure 2 This is a schematic diagram of the multi-scale deep attention convolution principle provided in Embodiment 1 of the present invention;

[0038] Figure 3 A schematic diagram illustrating the principle of multi-scale parallel depth convolution provided in Embodiment 1 of the present invention;

[0039] Figure 4 This is a schematic diagram of a lightweight feedforward network provided in Embodiment 1 of the present invention;

[0040] Figure 5 This is a schematic diagram of the channel attention module provided in Embodiment 1 of the present invention. Detailed Implementation

[0041] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0042] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0043] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. Furthermore, it should be understood that the terms “comprising” and “including”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0044] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0045] Example 1

[0046] This embodiment proposes a speech encoding and decoding method based on principal component analysis and multi-scale deep attention, which mainly includes:

[0047] After performing multiple multi-scale deep attention convolution operations on the acquired speech signal, the first feature is obtained. Each operation includes: extracting features through multiple parallel separable convolutions with different depth kernels, fusing the features, multiplying the obtained fused features channel by channel according to the channel attention weights, and then proceeding to the next operation.

[0048] The first feature is projected using the projection matrix and the feature mean to obtain the second feature. Specifically, principal component analysis is performed on the first feature of the training set to determine the feature mean, and the features with the highest cumulative variance contribution rate are selected. The eigenvectors constitute the projection matrix;

[0049] By determining the nearest neighbor codeword in the codebook, the second feature is sequentially subjected to multi-level vector quantization to obtain the reconstructed vector;

[0050] The reconstructed vector is processed by inverse projection and then decoded to output the reconstructed speech.

[0051] The following is combined Figure 1 The specific implementation process of the method in this embodiment will be described.

[0052] In this embodiment, the sampling rate of the input speech is set to... =8000Hz, setting frame shift and segment length, used for small-batch training and streaming inference, typically with a basic granularity of 20ms for reading and writing.

[0053] After obtaining the training set of speech signals, the data is preprocessed, including amplitude normalization, removal of DC components, random cropping to fixed-length segments, and Gaussian white noise or room reverberation is added with a certain probability to enhance the robustness of the model.

[0054] In this embodiment, a lightweight encoder based on multi-scale deep attention convolution is constructed to process the preprocessed speech signal. The input is fed into the encoder, and after undergoing multiple multi-scale depth attention convolution operations, the first feature is obtained. ;in, For batch size, For time step.

[0055] like Figure 1As shown, the encoder mainly consists of a one-dimensional strided convolutional layer and several stacked multi-scale deep attention convolutional layers. The input speech first passes through the front-end one-dimensional strided convolutional layer for preliminary feature extraction and downsampling, then enters the stacked multi-scale deep attention convolutional layers for deep feature extraction, and finally passes through the rear one-dimensional strided convolutional layer to adjust the dimension and output a high-dimensional latent representation.

[0056] The processing flow of one-dimensional strided convolution is as follows: The input feature sequence is set as... The convolution kernel is , bias is The stride is set to ( The convolution kernel moves in steps along the time axis. Perform a sliding window operation. For the first... Each time step, the output is calculated as the sum of the dot products of the features within the corresponding local window in the input sequence and the weights of the convolution kernel, i.e. , The convolution kernel is indexed; through this processing flow, not only are local acoustic features extracted, but the length of the time dimension is also reduced to its original value. This achieves temporal compression (downsampling) of features. In specific implementations, hierarchical compression of speech signals is achieved by configuring different step sizes (such as 10, 8, 4, 1 in this embodiment).

[0057] like Figure 2 As shown, the internal computation flow of each multi-scale deep attention convolution is as follows:

[0058] (1) Input tensor ( The current layer feature length is split into four parallel depthwise separable convolution branches, i.e., first performing channel-wise depthwise convolution, then pointwise convolution; the outputs of each branch are concatenated, fused by a 1×1 convolution, and then connected to the original input residual to obtain the fused feature. .

[0059] To simultaneously capture transient changes in the speech signal (such as consonant bursts) and long-term periodic structures (such as fundamental frequency and harmonics), the depthwise convolution kernel sizes of the four branches are set to [sizes to be filled in]. =3、 =7、 =11 and =15, such as Figure 3 As shown.

[0060] (2) Fusion characteristics The input is fed into the channel attention (Squeeze-Excitation, SE) module via a lightweight feedforward network.

[0061] like Figure 4 As shown, the lightweight feedforward network consists of sequentially connected layer normalization, 1×1 convolution, GELU activation function, and 1×1 convolution. To reduce computational complexity, the channel expansion ratio of the lightweight feedforward network is set to [value missing]. =2.0 (i.e., the intermediate hidden layer dimension is) Compared to traditional design ( =4.0) Reduces the number of parameters by about 50%.

[0062] (3) In the channel attention module, such as Figure 5 As shown, the fused features are first compressed into channel descriptors using global average pooling. The calculation formula is: ; Input feature map In the middle, the first The feature channel in the first Feature response values ​​at each time step; For the first The global average feature value of each channel represents the global importance level of that feature channel throughout the entire time segment; L represents the total length of the current layer feature map in the time dimension.

[0063] Then, the weight coefficients of each channel are learned through two fully connected layers. : ;in, It is the ReLU activation function. It is the Sigmoid activation function. and These are the weights for the fully connected layer.

[0064] Finally, the fused features are multiplied with the channel attention weights channel by channel to recalibrate the original feature channels. The features output by the channel attention module are then concatenated with the fused feature residuals before proceeding to the next operation.

[0065] Following the one-dimensional strided convolutional layer at the end of the encoder, there are two lightweight Long Short-Term Memory (SLSTM) networks to extract global temporal dependencies, resulting in a final output temporal length of [missing information]. High-dimensional latent tensor ,in =256 is the high-dimensional feature dimension.

[0066] In this embodiment, the encoder and decoder adopt a mirror-symmetric network structure, and a high-dimensional residual vector quantizer (High-Dim RVQ) is constructed between the encoder and decoder. This quantizer operates at... The first stage of high-dimensional feature preprocessing is performed on 256 dimensions.

[0067] Without introducing a PCA projection layer and low-dimensional constraints, waveform reconstruction loss is utilized. and multi-resolution spectral loss The aforementioned backbone network was trained end-to-end.

[0068] Waveform reconstruction loss Defined as input waveform With reconstructed waveform L1 distance between them: .

[0069] Multi-resolution spectral loss Includes several different Fourier transform parameter configurations For example, (512,128), (1024,256), and (2048,512), calculate the spectral convergence loss and the log-magnitude loss respectively.

[0070] The goal of this stage is to enable the model to learn from time-domain waveforms to high-dimensional acoustic feature spaces. The stable mapping is obtained, and the compactness of the high-dimensional RVQ constraint feature distribution is utilized to provide a high-quality data foundation for PCA analysis.

[0071] In this embodiment, the pre-trained encoder parameters are frozen, the entire training dataset is traversed, and the first feature set corresponding to all audio segments is extracted. ,in The total number of feature samples, and for the set Perform principal component analysis; specifically:

[0072] First, calculate the characteristic mean. :

[0073] ;

[0074] And calculate the characteristic covariance matrix :

[0075] ;

[0076] Then, the covariance matrix... Perform singular value decomposition to obtain the eigenvalue sequence. and the corresponding feature vector sequence .

[0077] Select the top contributors with the highest cumulative variance contribution rate The eigenvectors constitute the projection matrix. .

[0078] In principal component analysis, eigenvalues The magnitude of directly reflects the degree of dispersion of the data along the j-th eigenvector (principal component), i.e., variance. The larger the variance, the more information that dimension contains. Define the j-th eigenvector... The variance contribution rate of each eigenvector is the proportion of that eigenvalue to the sum of all eigenvalues. The cumulative variance contribution rate is the sum of the variance contribution rates of the eigenvectors. The formula for calculating the sum of the variance contribution rates of each principal component is: = .

[0079] In this embodiment, a selection is made that maximizes the contribution rate of cumulative variance. The highest (i.e., close to 1, for example, reaching over 99%) The projection matrix is ​​composed of eigenvectors. This means that only eigenvectors are used. One dimension (such as 40 dimensions) can preserve the original This allows for the extraction of most of the effective information from multiple dimensions (such as 256 dimensions), thus achieving lossless compression.

[0080] This step uses mathematical statistical methods to explicitly extract the orthogonal directions (principal components) with the most information in the feature space, eliminating dimensions containing noise and redundant correlations.

[0081] Therefore, based on the principal component analysis above, the pre-trained encoder and decoder parameters are retained, while the original high-dimensional RVQ is removed;

[0082] Insert a linear projection layer (input dimension) at the encoder output. Output dimension Its weight matrix Initialize as projection matrix bias Initialized as negative feature mean projection ;

[0083] Insert a linear inverse projection layer at the decoder input (input dimension) Output dimension Its weight matrix Initialize to bias Initialize to the feature mean ;

[0084] Simultaneously, a new low-dimensional residual vector quantizer (Low-Dim RVQ) is created and placed after the linear projection layer. The input dimension of this quantizer is... =40.

[0085] With this initialization, the positive transformation formula for the feature is: That is, the PCA projection after removing the mean was performed;

[0086] The inverse transformation formula is This means that a PCA refactoring was performed.

[0087] This ensures that, at the initial moment of model adaptation training, while the features are forcibly compressed into a 40-dimensional orthogonal subspace, the original acoustic information is preserved to the greatest extent.

[0088] In addition, layer normalization is introduced after the linear projection layer to standardize the 40-dimensional features and stabilize the input distribution of the quantizer.

[0089] Based on this, the first feature, after being processed by the linear projection layer, yields a 40-dimensional low-dimensional second feature. ,Will The input is fed into a new low-dimensional RVQ module for low-dimensional residual vector quantization.

[0090] The quantification process is composed of Composed of cascaded vector quantizers, the q-th quantizer Has an independent codebook ,in The size of each codebook layer is =256, the dimension of the quantized feature and the dimension of the low-dimensional subspace. =40 Strict alignment.

[0091] For input residuals (Initial residual) Find the nearest neighbor codeword: The reconstructed vector is obtained. And update the next level residual. The final quantization output is the sum of the reconstructed vectors at all levels: .

[0092] Since the feature dimension has been reduced to 40 dimensions and redundancy has been removed through PCA, only 4 layers of quantizers are needed to achieve high-precision feature approximation, effectively avoiding the "codebook collapse" phenomenon in high-dimensional space.

[0093] In this embodiment, the second stage of hierarchical fine-tuning is performed. Pre-trained weights and PCA-initialized weights are loaded, the RVQ module is inserted, and end-to-end fine-tuning is conducted. To protect the statistical prior introduced by PCA from being compromised, a hierarchical learning rate strategy is adopted:

[0094] The encoder, linear projection layer, linear inverse projection layer, and normalization layer are divided into fine-tuning group (Group A), and given an extremely low learning rate. It is only allowed to make minor adjustments near the subspace manifold defined by PCA;

[0095] The RVQ quantizer, decoder, and discriminator are divided into a training group (Group B), and given a normal learning rate. This allows it to quickly adapt to reconstruct waveforms from highly compressed 40-dimensional features.

[0096] In addition, to improve the perceptual quality of synthesized speech, a multi-domain joint adversarial training framework was introduced during the training phase, which includes a multi-scale STFT discriminator and a multi-cycle discriminator.

[0097] Multi-Scale Short-Time Fourier Transform Discriminator (MS-STFTD): Discriminates the real and imaginary parts of the complex spectrum at different FFT (Fast Fourier Transform) scales, and is used to constrain the fine structure and spectral envelope of the frequency domain.

[0098] Multi-Period Discriminator (MPD): This function divides a one-dimensional speech waveform into different prime number periods (e.g., ...). It is reshaped into a two-dimensional texture map for discrimination, which is used to constrain the periodic patterns and phase continuity in the time domain.

[0099] Total loss function Defined as:

[0100]

[0101] in, To help the generator combat loss, For feature matching loss, To quantize the commitment loss (sg represents stopping gradient operations); , , , and These are the weighting coefficients for each loss.

[0102] In this embodiment, at the decoding end, the quantized low-dimensional features are first processed through a linear inverse projection layer. Mapping back to the original high-dimensional space yields a reconstructed high-dimensional latent representation. Subsequently, Input decoder. The decoder structure is symmetrical to the encoder. It performs upsampling through transposed convolutions and then gradually recovers temporal waveform details through cascaded multi-scale deep attention convolutions. The final output is reconstructed speech. .

[0103] This embodiment's method maps the time-domain speech waveform to a high-dimensional latent representation containing rich acoustic information through an encoder; it uses a linear projection layer to linearly map the high-dimensional representation to a low-dimensional orthogonal subspace with the largest variance using statistical priors to remove feature redundancy; it uses residual vector quantization to perform multi-level discretization encoding of features in the low-dimensional subspace to generate a low bit-rate bitstream; it uses linear inverse projection to restore the quantized low-dimensional features to the original high-dimensional space; and it uses a decoder to reconstruct a high-quality time-domain speech waveform from the restored high-dimensional features.

[0104] Compared with existing low bitrate speech codecs, the method in this embodiment designs a three-stage training framework of "pre-training-analysis-adaptation", which combines an ultra-lightweight network architecture with a statistical prior-guided quantization strategy to achieve high-quality speech communication at ultra-low bitrates.

[0105] First, an ultra-lightweight backbone network design based on multi-scale deep attention blocks is proposed. To address the bottleneck of limited computing power in mobile terminals and IoT edge nodes, the basic computational unit of the codec is redesigned. This unit abandons the traditional standard large convolutional or Transformer architecture, simultaneously capturing transient changes and long-term periodic structures of speech through parallel multi-scale deep separable convolutional branches; a channel attention mechanism is introduced to dynamically recalibrate feature weights under extremely compressed channel numbers; and a simplified feedforward network with a channel expansion rate of only 2.0 is adopted, significantly reducing computational overhead. The codec built based on this unit can still maintain strong acoustic feature extraction capabilities with fewer than 10M parameters, while significantly reducing floating-point operations. This achieves lightweight model and accelerated inference, overcoming the deployment limitations of edge devices.

[0106] Secondly, existing technologies directly and forcibly quantize high-dimensional features, leading to severe feature aliasing and dead codes under limited codebook budgets, often resulting in metallic sounds or spectral defects in decoded speech. Therefore, this embodiment proposes a subspace adaptation strategy based on principal component analysis (PCA) statistical priors. Addressing the low efficiency of high-dimensional feature quantization at low bitrates, this strategy avoids relying on the network to blindly learn low-dimensional mappings. Instead, it utilizes the high-dimensional features obtained from the first-stage pre-training for PCA analysis to construct an optimal orthogonal subspace. By initializing the weights of the projection layer after the encoder and the inverse projection layer before the decoder using PCA principal component vectors, the features are forcibly compressed losslessly to a low-dimensional manifold (e.g., 40 dimensions). This explicit statistical guidance removes noise and redundant correlations in the feature space, constructs a low-dimensional positive tangent with extremely high information density, and enables residual vector quantization in a pure subspace. This allows the residual vector quantizer to achieve high-precision feature approximation with limited bit resources (such as 0.8kbps), accurately representing the core acoustic semantics of speech. It effectively eliminates the ambiguity and high-frequency distortion in traditional low-bit-rate coding, solves the problems of codebook collapse and sound quality ambiguity, and significantly improves the clarity and naturalness of speech reconstruction.

[0107] Furthermore, a hierarchical learning rate fine-tuning training strategy was designed. To address the potential training instability issues arising from the introduction of the PCA projection layer, the model parameters were divided into different optimization groups. The encoder and projection layer, which incorporate PCA priors, were given extremely low learning rates, allowing fine-tuning only near the subspace manifold; the quantizer and decoder were given normal, higher learning rates, enabling them to quickly adapt to the reconstruction of low-dimensional features. Combined with joint adversarial training using a multi-scale STFT discriminator and a multi-period discriminator, this ensured smooth migration and stable convergence of the model from high-dimensional space to low-dimensional subspace.

[0108] Directly training low-dimensional bottleneck layers typically faces challenges such as cold start difficulties and a tendency to get trapped in local optima. This embodiment employs a three-stage strategy of "pre-training-analysis-adaptation," combined with a hierarchical learning rate mechanism, cleverly utilizing pre-trained features in high-dimensional space as guidance. By locking the encoder's main feature extraction capabilities, only the quantizer and decoder are allowed to quickly adapt to the low-dimensional subspace defined by PCA. This training method avoids the oscillations and divergences of end-to-end training, enabling the model to quickly converge to the optimal working range and significantly shortening the model development cycle.

[0109] Finally, thanks to the dynamic feature selection capability of the channel attention mechanism in the network structure and the modeling of global temporal dependencies by the long short-term memory network, the adaptability to complex acoustic environments is enhanced. Compared with traditional models that rely solely on local convolution, the method in this embodiment can more effectively separate speech components from background interference, maintaining stable coding quality even when processing noisy or reverberant speech signals. This embodiment achieves clear and natural speech reconstruction at an extremely low bitrate of 0.8 kbps by jointly optimizing a lightweight network structure and a subspace quantization strategy. It is suitable for neural vocoders and voice communication systems with strict limitations on bandwidth and computing power.

[0110] Example 2

[0111] This embodiment provides a speech encoding and decoding system based on principal component analysis and multi-scale deep attention, including:

[0112] The encoding module is configured to perform multiple multi-scale deep attention convolution operations on the acquired speech signal to obtain the first feature; wherein each operation includes: extracting features through multiple parallel and depth-separable convolutions with different depth kernels, fusing the features, multiplying the obtained fused features channel by channel according to the channel attention weights, and then proceeding to the next operation.

[0113] The projection module is configured to project the first feature based on the projection matrix and the feature mean to obtain the second feature; wherein, principal component analysis is performed on the first feature of the training set to determine the feature mean, and the features with the highest cumulative variance contribution rate are selected. The eigenvectors constitute the projection matrix;

[0114] The quantization module is configured to perform multi-level vector quantization on the second feature sequentially by determining the nearest neighbor codeword in the codebook to obtain the reconstructed vector.

[0115] The decoding module is configured to perform inverse projection processing on the reconstructed vector before decoding and outputting the reconstructed speech.

[0116] It should be noted that the above modules correspond to the steps described in Embodiment 1, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0117] In further embodiments, the following is also provided:

[0118] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in Embodiment 1. For brevity, further details are omitted here.

[0119] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0120] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0121] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in Embodiment 1.

[0122] The method in Example 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0123] A computer program product includes a computer program that, when executed by a processor, implements the method described in Embodiment 1.

[0124] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which execute in a device on a target real or virtual processor to perform the processes / methods described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0125] The computer program code used to implement the methods of the present invention may be written in one or more programming languages. This computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the computer or other programmable data processing device, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0126] In the context of this invention, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and the like. Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0127] Those skilled in the art will recognize that the units and algorithm steps described in connection with the various examples of this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0128] It should be noted that all data acquisition is conducted in accordance with laws and regulations and with user consent, and the data is used legally.

[0129] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A speech encoding and decoding method based on principal component analysis and multi-scale deep attention, characterized in that, include: After performing multiple multi-scale deep attention convolution operations on the acquired speech signal, the first feature is obtained. Each operation includes: extracting features through multiple parallel separable convolutions with different depth kernels, fusing the features, multiplying the obtained fused features channel by channel according to the channel attention weights, and then proceeding to the next operation. The first feature is projected using the projection matrix and the feature mean to obtain the second feature. Specifically, principal component analysis is performed on the first feature of the training set to determine the feature mean, and the features with the highest cumulative variance contribution rate are selected. The eigenvectors constitute the projection matrix; By determining the nearest neighbor codeword in the codebook, the second feature is sequentially subjected to multi-level vector quantization to obtain the reconstructed vector; The reconstructed vector is processed by inverse projection and then decoded to output the reconstructed speech.

2. The speech encoding and decoding method based on principal component analysis and multi-scale deep attention as described in claim 1, characterized in that, The depthwise kernel sizes of multiple parallel depthwise separable convolutions are set to 3, 7, 11, and 15, respectively.

3. The speech encoding and decoding method based on principal component analysis and multi-scale deep attention as described in claim 1, characterized in that, The calculation process for channel attention weights is as follows: The fused features are compressed into channel descriptors using global average pooling. The calculation formula is: ; Then, the weight coefficients of each channel are learned through two fully connected layers. : ; in, For the fusion feature of the first The feature channel in the first Feature response values ​​at each time step; For the first The global average feature value of each channel; L represents the total length of the current layer feature map in the time dimension; It is the ReLU activation function. It is the Sigmoid activation function. and These are the weights for the fully connected layer.

4. The speech encoding and decoding method based on principal component analysis and multi-scale deep attention as described in claim 1, characterized in that, The process of constructing the projection matrix includes: determining the eigenvalue covariance matrix after principal component analysis; performing singular value decomposition on the covariance matrix to obtain the eigenvalue sequence and the corresponding eigenvector sequence; defining the first... The variance contribution rate of the i-th eigenvector is the i-th eigenvalues The proportion of the sum of all eigenvalues, the cumulative variance contribution rate is the sum of the variance contribution rates of the eigenvectors, and the first The cumulative variance contribution rate of each eigenvector is: = D represents the total number of eigenvectors.

5. The speech encoding and decoding method based on principal component analysis and multi-scale deep attention as described in claim 1, characterized in that, Projection processing is as follows: Inverse projection processing is ;in, This is the second characteristic; The first feature; The characteristic mean; The projection matrix; For reconstructing features.

6. The speech encoding and decoding method based on principal component analysis and multi-scale deep attention as described in claim 1, characterized in that, The vector quantization process includes: quantizing the input residual at the q-th level vector. According to the codebook Find the nearest neighbor code Thus, the reconstructed vector is obtained. And update the next level residual. The final output is the sum of the reconstruction vectors at all levels. Among them, the input residual of the first-level vector quantization The second feature is N, where N is the size of each level of the codebook.

7. A speech encoding and decoding system based on principal component analysis and multi-scale deep attention, characterized in that, include: The encoding module is configured to perform multiple multi-scale deep attention convolution operations on the acquired speech signal to obtain the first feature; wherein each operation includes: extracting features through multiple parallel and depth-separable convolutions with different depth kernels, fusing the features, multiplying the obtained fused features channel by channel according to the channel attention weights, and then proceeding to the next operation. The projection module is configured to project the first feature based on the projection matrix and the feature mean to obtain the second feature; wherein, principal component analysis is performed on the first feature of the training set to determine the feature mean, and the features with the highest cumulative variance contribution rate are selected. The eigenvectors constitute the projection matrix; The quantization module is configured to perform multi-level vector quantization on the second feature sequentially by determining the nearest neighbor codeword in the codebook to obtain the reconstructed vector. The decoding module is configured to perform inverse projection processing on the reconstructed vector before decoding and outputting the reconstructed speech.

8. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the method described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the method described in any one of claims 1-6.

Images