Speech coding method, apparatus, device, and medium

Abstract

This invention relates to the field of artificial intelligence technology and can be applied to fintech and medical technology. It discloses a speech encoding and decoding method, apparatus, device, and medium. The method includes: acquiring an original speech signal and extracting its spectral features; encoding the signal using an encoder trained with a first-stage mirror architecture to obtain a continuous vector; segmenting the continuous vector into sub-vectors, matching them with corresponding sub-encoding dictionaries using a quantizer to obtain sub-indices, and combining them to generate a global index; parsing the global index to obtain sub-indexes, retrieving sub-discrete vectors and concatenating them into a discrete vector; reconstructing the target speech spectral features from the discrete vectors using a decoder trained with a second-stage non-mirror architecture, and then converting the target speech signal. This invention employs a two-stage training process: the first-stage mirror architecture ensures encoding stability, while the second-stage non-mirror architecture enhances decoding flexibility. Combined with product quantization technology, it improves speech reconstruction quality while balancing computational efficiency and storage overhead.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology and can be applied to the fields of financial technology and medical technology. In particular, it relates to a voice encoding and decoding method, apparatus, device and medium. Background Technology

[0002] Speech codecs are technologies used for compressing, encoding, decompressing, and reconstructing speech signals. Their core objective is to reduce the bit rate of data transmission or storage while maintaining speech quality. They are widely used in communications, speech synthesis, and audio storage. With the development of artificial intelligence and deep learning, speech codecs are gradually shifting from traditional signal processing schemes to neural network implementations. This type of technology, known as neural speech codecs, leverages the powerful modeling capabilities of neural networks to generate more natural, clearer, and higher-quality speech at lower bit rates. However, existing neural speech codecs have significant shortcomings: Firstly, mirror-architecture technologies employ a symmetrical design for the codec, which, while simple to train and convergent, suffers from poor decoder flexibility and difficulty in effectively capturing complex speech context, especially limiting sound quality at low bit rates. Secondly, while non-mirror-architecture technologies improve the ability to restore speech details through flexible decoders, the lack of effective training signals for discrete coding dictionaries leads to insufficient coding accuracy and a significant decrease in reconstruction quality at low bit rates. In scenarios such as remote medical consultations and voice underwriting in the insurance industry, these technical deficiencies may affect the accurate transmission of speech information, potentially leading to risks. Therefore, how to balance the architecture design of the codec to ensure both encoding accuracy and speech reconstruction quality at low bit rates has become an urgent problem to be solved in current neural speech coding and decoding technology. Summary of the Invention

[0003] This invention provides a speech encoding and decoding method, apparatus, computer equipment, and medium to solve the problem that existing speech encoding and decoding architectures are unable to simultaneously guarantee encoding accuracy and speech reconstruction quality at low bit rates.

[0004] In a first aspect, a speech encoding and decoding method is provided, characterized by comprising:

[0005] The original speech signal is acquired and input into a pre-trained encoder to obtain a continuous vector. A discrete vector matching the continuous vector is obtained from the encoding dictionary through a quantizer according to the encoding dictionary, and an index corresponding to the discrete vector is generated. The encoder is trained in the first stage using a mirror architecture.

[0006] The target speech signal is obtained by reconstructing the target speech signal by retrieving the corresponding discrete vector from the encoding dictionary according to the index by a pre-trained decoder, wherein the decoder is trained in the second stage using a non-mirror architecture, and the second stage uses the encoder of the first stage.

[0007] Secondly, a speech codec apparatus is provided, including a unit for performing the above-described method.

[0008] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0009] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described method.

[0010] This invention provides a speech encoding / decoding method, apparatus, device, and medium. The method includes: acquiring an original speech signal; extracting features from the original speech signal to obtain corresponding original speech spectral features; encoding the original speech spectral features into a continuous vector using an encoder trained in a first stage using a mirror architecture, wherein the network structures of the encoder and decoder in the mirror architecture are symmetrical; dividing the continuous vector into multiple sub-vectors; performing similarity matching between the sub-vectors and corresponding discrete sub-vectors in a sub-encoding dictionary using a quantizer to obtain sub-indexes corresponding to the matching discrete sub-vectors; and combining and calculating the sub-indexes according to the size of each sub-encoding dictionary to generate discrete vectors matching the continuous vectors. The overall index corresponding to the quantity is obtained; the overall index is parsed by performing an inverse operation on the overall index according to the size of each sub-encoding dictionary to obtain each sub-index; the corresponding sub-discrete vector is obtained from each sub-encoding dictionary corresponding to each sub-index; the obtained multiple sub-discrete vectors are concatenated to obtain a discrete vector that matches the continuous vector; the discrete vector is reconstructed to obtain the target speech spectrum features by the decoder trained on the second stage using a non-mirror architecture; and the target speech spectrum features are converted into a target speech signal. In this case, the network structures of the encoder and decoder in the non-mirror architecture are asymmetrical. The encoder in the second stage uses the encoder in the first stage, and the structure of the decoder in the second stage is different from that of the decoder in the first stage. This invention employs a two-stage training architecture. In the first stage, a mirrored encoder and decoder are used to achieve symmetric encoding and preliminary reconstruction of features, ensuring the stability and feature consistency of the encoding process. In the second stage, a non-mirrored architecture is used, which optimizes the decoder based on the encoder of the first stage, improving decoding flexibility and speech reconstruction capabilities. At the same time, a product quantization mechanism combining sub-vector segmentation and sub-encoding dictionary matching is used to significantly reduce computation and storage overhead. Meanwhile, the mapping between the overall index and sub-index ensures accurate reconstruction of discrete vectors. Ultimately, this invention achieves synergistic optimization of efficiency, storage, and reconstruction quality in the speech encoding and decoding process, improving the reconstruction accuracy and naturalness of the speech signal, and enhancing the performance of speech encoding and decoding. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating the speech encoding and decoding method according to an embodiment of the present invention;

[0013] Figure 2 This is a schematic diagram of the first-stage training steps of the speech encoding and decoding method according to an embodiment of the present invention;

[0014] Figure 3 This is a schematic diagram of the second-stage training steps of the speech encoding and decoding method according to an embodiment of the present invention;

[0015] Figure 4 yes Figure 1 A flowchart illustrating the sub-steps of step S150.

[0016] Figure 5 yes Figure 1 A flowchart illustrating another sub-step of step S150;

[0017] Figure 6 yes Figure 1 A flowchart illustrating another sub-step of step S150.

[0018] Figure 7 This is a schematic block diagram of the speech encoding and decoding device according to an embodiment of the present invention;

[0019] Figure 8 This is a schematic diagram of the structure of a computer device in an embodiment of the present invention;

[0020] Figure 9 This is another structural schematic diagram of the computer device in an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] The speech encoding and decoding method provided in this invention can be applied to a client or a server. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0023] Please see Figure 1 As shown, Figure 1 A flowchart of a speech encoding and decoding method provided in an embodiment of the present invention includes the following steps: S110-S120.

[0024] S110. Obtain the original speech signal, and perform feature extraction on the original speech signal to obtain the corresponding original speech spectrum features;

[0025] In this embodiment, the raw speech signal refers to the unprocessed speech waveform signal acquired from external sources, which can be obtained through audio acquisition devices such as microphones and is presented in the form of a time-domain waveform. The raw speech spectral characteristics refer to the feature representation obtained after time-frequency conversion of the raw speech signal. These characteristics reflect the energy distribution of the speech signal across different frequencies and time dimensions, serving as the input basis for subsequent encoding processes. Specifically, acquiring the raw speech signal can be achieved by recording speech in real-time through audio acquisition devices or by reading existing speech files from storage media. When extracting features from the original speech signal to obtain the corresponding original speech spectrum features, the original speech signal is first preprocessed, including removing DC components and high-frequency noise, and boosting the energy of high-frequency signals through pre-emphasis. Then, a sliding window is used to segment the preprocessed speech signal into multiple consecutive speech frames, each typically 20-30 milliseconds long, with some overlap between adjacent frames to ensure feature continuity. Next, a Fourier transform is performed on each speech frame to convert the time-domain signal into a frequency-domain signal, obtaining the spectrum of the speech frame. Finally, the spectrum is filtered using a Mel filter bank to convert it into a Mel spectrum, and the logarithm of the Mel spectrum is taken to obtain the original speech spectrum features. By converting the original speech signal in the time domain into spectral features in the frequency domain through feature extraction, not only is the data dimensionality reduced, and the computational load of subsequent processing decreased, but key acoustic information in the speech signal, such as pitch, timbre, and semantic features, is also preserved, providing a more easily processed input format for the subsequent encoding process.

[0026] S120. The original speech spectrum features are encoded into a continuous vector by an encoder trained based on the mirror architecture in the first stage, wherein the network structures of the encoder and decoder in the mirror architecture are symmetrical.

[0027] In this embodiment, the first stage refers to the initial stage of model training, focusing on building the basic encoding and decoding capabilities of the encoder and decoder. A symmetrical structure is used to achieve bidirectional feature learning and collaborative parameter optimization. The mirror architecture refers to the encoder and decoder employing a symmetrical network structure design, maintaining consistency in the number of network layers, the dimensions of each layer, and the type of core computational modules. The encoder refers to the neural network used to compress and encode the input features, consisting of an input layer, a convolutional downsampling layer, a residual attention layer, and a fully connected mapping layer. The continuous vector refers to the high-dimensional real-valued vector output by the encoder, which is a compressed representation of the original speech spectral features and contains the core information of the speech. Specifically, the encoder's structure is as follows: The input layer receives the original speech spectrum features and adjusts the dimensions to a preset size; the convolutional downsampling layer contains multiple convolutional blocks, each consisting of a convolution operation, batch normalization, and an activation function, achieving feature downsampling through convolutions with a stride of 2, gradually reducing the feature space dimension; the residual attention layer consists of residual units and channel attention mechanisms. The residual units alleviate the gradient vanishing problem in deep networks through skip connections, while the channel attention mechanism enhances the expression of important features by learning the weights of each feature channel; the fully connected mapping layer maps the processed features into a continuous vector of fixed dimensions through linear transformation. During encoding, the original speech spectrum features are first input to the input layer for dimensionality adaptation; then, they pass through each convolutional downsampling layer, where each layer performs local feature extraction and dimensionality compression; next, they enter the residual attention layer, where the residual units perform nonlinear transformations on the features, and the attention mechanism performs weighted optimization of the feature channels; finally, the fully connected mapping layer outputs a continuous vector. The advantage of using an encoder trained with a mirror architecture in the first stage lies in the fact that the symmetrical structure ensures a high degree of consistency in the parameter update paths of the encoder and decoder, significantly reducing training complexity and achieving rapid model convergence. Simultaneously, the bidirectional constraint mechanism of the symmetrical network (the errors in the encoder's compressed features and the decoder's reconstructed features feedback each other) effectively avoids bias in unidirectional training, enabling the encoder to more accurately capture key features of the speech spectrum during the learning process, resulting in high encoding accuracy. Furthermore, the parameter co-optimization characteristic of the mirror architecture ensures the stability of the encoding process and reduces the interference of training fluctuations on feature representation. Therefore, the encoder trained using a mirror architecture can fully learn the key information in the original speech spectrum features, achieving effective feature compression and abstraction through multi-layer network processing. The output continuous vector retains the core features of speech while having low dimensionality, laying the foundation for subsequent quantization. At the same time, the symmetry of the mirror architecture ensures the stability, accuracy, and reliability of the feature representation in the encoding process.

[0028] S130. The continuous vector is divided into multiple sub-vectors. The sub-vectors are matched with the corresponding discrete vectors in the sub-encoding dictionary by a quantizer to obtain the sub-index corresponding to the discrete vector that matches the sub-vector. The sub-indexes are combined and calculated according to the size of each sub-encoding dictionary to generate the overall index corresponding to the discrete vector that matches the continuous vector.

[0029] In this embodiment, Product Quantization is an efficient high-dimensional vector quantization technique. Its core idea is to decompose a high-dimensional continuous vector into multiple low-dimensional sub-vectors, quantize each sub-vector individually (i.e., match it with its corresponding sub-encoding dictionary), and then represent the complete vector through a combination of sub-indices, thereby reducing the computational complexity and storage overhead of high-dimensional vector quantization. A sub-vector refers to multiple low-dimensional vectors obtained by dividing a continuous vector according to preset rules; each sub-vector contains some feature information of the continuous vector. A quantizer is a module used to convert continuous vectors into discrete vectors, achieving the mapping from continuous features to discrete indices through similarity matching. A sub-encoding dictionary refers to the set of discrete vectors corresponding to each sub-vector; each sub-encoding dictionary contains multiple sub-discrete vectors with fixed dimensions. A sub-discrete vector refers to a discrete vector in a sub-encoding dictionary; a sub-index refers to the position identifier of a sub-discrete vector within its respective sub-encoding dictionary; and the overall index refers to a global identifier that uniquely identifies a discrete vector, obtained by combining the sub-indices. Specifically, when dividing a continuous vector into multiple sub-vectors, the vector is uniformly divided according to its dimension and a preset number of divisions. For example, a 1024-dimensional continuous vector is divided into four 256-dimensional sub-vectors. When the quantizer performs similarity matching between the sub-vectors and their corresponding discrete sub-vectors in the sub-encoding dictionary, firstly, a corresponding sub-encoding dictionary is assigned to each sub-vector, with the same dimension as the sub-vector. Then, the quantizer calculates the Euclidean distance between the sub-vector and each discrete sub-vector in the sub-encoding dictionary; the smaller the distance, the higher the similarity. The discrete sub-vector with the smallest distance is selected as the matching result, and its position in the sub-encoding dictionary is recorded to obtain the sub-index. When generating the overall index by combining the sub-indexes based on the size of each sub-encoding dictionary, the first step is to determine the number of discrete vectors contained in each sub-encoding dictionary, i.e., the size of the sub-encoding dictionary. Next, the sub-indexes are combined based on their sizes to obtain the overall index. The core logic of this combination calculation is to construct weight coefficients using the size of the sub-encoding dictionary, ensuring that each sub-index occupies an independent numerical range within the overall index. This guarantees that different combinations of sub-indexes correspond to a unique overall index. Specifically, assuming there are k vector segments, with corresponding sub-encoding dictionary sizes S1, S2, ..., S... k (S n(This represents the number of sub-discrete vectors contained in the nth sub-encoded dictionary), where the sub-indices are I1, I2, ..., I... k (0≤I n <S n The overall index is calculated as follows: Overall index = I1 × (S2 × S3 × … × S k )+I2×(S3×…×S k )+…+I k-1 ×S k +I k In this way, each sub-index is assigned a weight related to the product of the sizes of all subsequent sub-encoded dictionaries. This preserves the original information of each sub-index while achieving a unique mapping from the sub-index to the overall index. Throughout the encoding process, the sub-encoded dictionaries logically combine to form a virtual super-large encoded dictionary with a total capacity of S1×S2×…×S k This is equivalent to a very large dictionary containing that number of discrete vectors. By segmenting continuous vectors using product quantization and quantizing them separately using sub-encoding dictionaries, the size of a single encoding dictionary is significantly reduced, decreasing the computational and storage overhead during quantization. At the same time, the combined calculation of sub-indexes simulates the function of a virtual very large encoding dictionary. While ensuring computational efficiency, it retains the ability of a large-size encoding dictionary to represent continuous vectors in a fine manner, improving quantization accuracy and the richness of representation of discrete vectors.

[0030] For example, if a continuous vector is divided into three segments, with corresponding sub-encoded dictionary sizes of S1 = 100, S2 = 200, and S3 = 300, and the quantized sub-indices are I1 = 10, I2 = 50, and I3 = 20, then the overall index is calculated as follows: First, calculate the weight coefficients: S2 × S3 = 200 × 300 = 60000, S3 = 300; then substitute them into the formula: overall index = 10 × 60000 + 50 × 300 + 20 = 600000 + 15000 + 20 = 615020. This overall index uniquely corresponds to this set of sub-index combinations, and the total capacity of the virtual super-large encoded dictionary is 100 × 200 × 300 = 6,000,000, which is equivalent to a super-large dictionary containing 6 million discrete vectors. This avoids the overhead of storing 6 million discrete vectors and achieves the same representation capability through segmented quantization and combined indexes.

[0031] S140. Perform an inverse operation on the overall index according to the size of each sub-encoding dictionary to parse out each sub-index, obtain the corresponding sub-discrete vector from each sub-encoding dictionary corresponding to each sub-index, and concatenate the obtained multiple sub-discrete vectors to obtain a discrete vector that matches the continuous vector.

[0032] In this embodiment, the inverse operation refers to the calculation process of decomposing the overall index into individual sub-indexes based on the size of the sub-encoding dictionary; it is the reverse operation of the sub-index combination calculation. A discrete vector refers to a complete vector formed by concatenating multiple discrete sub-vectors. It has the same dimension as a continuous vector and is a discrete representation of a continuous vector after quantization. Specifically, when parsing sub-indexes, the size of each sub-encoding dictionary, S1, S2, ..., S..., is first determined. n Then, perform the inverse operation on the overall index: taking a dictionary with 3 sub-encoded words as an example, the overall index is token. First, calculate the value of S2×S3. Divide token by this value to get the quotient I1, and the remainder is token1=token%(S2×S3); then divide token1 by S3 to get the quotient I2, and the remainder is I3. For example, taking three sub-encoded dictionaries (S1 = 100, S2 = 200, S3 = 300, overall index = 615020), the first step is to calculate S2 × S3 = 200 × 300 = 60000. Divide the overall index by 60000; the quotient is 10 (i.e., I1 = 10), and the remainder is 615020%60000 = 15020. The second step is to divide the remainder 15020 by S3 = 300; the quotient is 50 (i.e., I2 = 50), and the remainder is 15020%300 = 20 (i.e., I3 = 20). When obtaining the sub-discrete vectors, the sub-discrete vectors with matching positions are retrieved from the corresponding sub-encoded dictionaries based on the parsed sub-indexes. When concatenating the discrete vectors, all the sub-discrete vectors are sequentially connected according to the original order of sub-vector segmentation to form a complete discrete vector with the same dimension as the continuous vector. By accurately parsing each sub-index through inverse operations, the correct mapping from the overall index to the sub-discrete vectors is ensured. The complete discrete vector is obtained by concatenating the sub-discrete vectors, realizing the reversibility of the continuous vector quantization process and providing accurate discrete feature input for subsequent speech reconstruction. At the same time, the process is simple and efficient to calculate, and can quickly complete the reconstruction of the discrete vector.

[0033] S150. The discrete vector is reconstructed using the decoder trained in the second stage using a non-mirror architecture to obtain the target speech spectrum features, and the target speech spectrum features are converted into a target speech signal. The network structures of the encoder and decoder in the non-mirror architecture are asymmetrical. The encoder in the second stage uses the encoder in the first stage, and the structure of the decoder in the second stage is different from that of the decoder in the first stage.

[0034] In this embodiment, the second stage refers to the optimization stage of model training, used to improve the speech reconstruction capability of the decoder. The non-mirror architecture refers to the encoder and decoder using asymmetric network structures, differing in network depth and module composition. Specifically, the encoder trained in the first stage using a mirror architecture is denoted as A1, and the decoder as B1. In the second stage, the encoder uses A1, while the decoder is trained after adjusting its structure based on B1, resulting in an optimized decoder denoted as B2. The decoder (specifically B2) refers to the neural network used to reconstruct speech spectral features from discrete vectors; the target speech spectral features refer to the reconstruction result output by the decoder corresponding to the original speech spectral features; and the target speech signal refers to the audio signal that converts the target speech spectral features back to the time domain. Specifically, after switching to the non-mirror architecture in the second stage, the encoder structure remains consistent with A1, and its parameters are frozen and no longer trained. Only the decoder structure is adjusted (making it different from the encoder structure), and training continues with the parameters of B1 as the initial values ​​to obtain B2. When reconstructing the target speech spectral features using decoder B2, the decoder first receives discrete vectors and performs feature mapping, transforming them into a dimensional space matching the speech spectral features. Then, it gradually recovers the detailed information of the spectral features, including frequency resolution and temporal continuity, through multi-layer nonlinear transformations. Finally, it outputs the target speech spectral features. When converting the target speech spectral features into the target speech signal, the inverse Fourier transform is first performed on the target speech spectral features to convert the frequency domain features into a time domain frame signal. Then, the time domain frame signal is overlapped and added to eliminate the boundary effects caused by frame division. Finally, a playable target speech signal is obtained through digital-to-analog conversion. The non-mirror architecture gives the second-stage decoder a high degree of design flexibility: B2 can break through the structural symmetry limitations of A1 and flexibly add different modules to adjust the structure (such as a sound quality enhancement auxiliary module), thereby enhancing decoding capabilities and improving decoding sound quality. It can freely adjust the network structure according to speech reconstruction needs (such as improving sound quality, enhancing temporal coherence, optimizing the performance of specific frequency bands, etc.), for example, adding a temporal modeling module for long speech, strengthening the recovery layer of high-frequency features, or introducing noise reduction-related feature processing units, without being limited by the symmetrical structure of the first-stage encoder. Meanwhile, the second stage uses the same encoder A1 as the first stage, preserving the high encoding accuracy brought by the mirror architecture training, ensuring the consistency of feature representation, and avoiding feature fragmentation problems. This combination of "fixed encoder + flexible decoder" (i.e., A1 encoding, B2 decoding) not only retains the basic encoding capabilities of the first stage, but also significantly improves the quality, naturalness, and scene adaptability of speech reconstruction through targeted optimization of the decoder, thus taking into account the dual advantages of the high encoding accuracy of the mirror architecture and the flexible decoding of the non-mirror architecture.

[0035] To better understand the method of the embodiments of the present invention, the following description will be provided in conjunction with two example scenarios.

[0036] For example, in medical scenarios, clear communication between doctors and patients during remote consultations requires speech encoding and decoding technology. When using the solution of this invention, the doctor's original speech signal (e.g., "The patient needs to monitor blood sugar daily") is first acquired, and the original speech spectral features are extracted to preserve the frequency details of the speech. The spectral features are then encoded using an encoder trained with a first-stage mirror architecture to generate a continuous vector containing the core information of the speech. The mirror architecture ensures the stability of the encoding process and avoids the loss of key medical instructions. Subsequently, based on product quantization technology, the continuous vector is divided into sub-vectors, matched with a sub-encoding dictionary to generate a sub-index, and combined into a whole index, significantly compressing the data volume for efficient transmission. The receiving end parses the whole index to obtain sub-discrete vectors and concatenates them into a complete discrete vector. Finally, the second-stage non-mirror architecture decoder reconstructs the speech, specifically optimizing the speech clarity of medical terms (e.g., enhancing the spectral features of keywords such as "blood sugar") to generate the target speech signal, ensuring that the patient accurately understands the medical instructions and reducing communication errors.

[0037] For example, in an insurance scenario, an intelligent underwriting system needs to process users' voice-based insurance application information (such as "applying for health insurance, with no prior critical illnesses"). When applying this invention, the original spectral features of the user's voice are first extracted and encoded into a continuous vector using a first-stage mirror architecture encoder. The symmetrical structure ensures the integrity of key time, status, and other features in the insurance information. Product quantization is used to segment the continuous vector into sub-vectors, and the generated overall index can be efficiently stored in the system, reducing storage overhead. In the decoding stage, the second-stage non-mirror architecture decoder can flexibly design a temporal modeling module, focusing on optimizing the coherence of long sentences (such as clearly distinguishing the negative meaning of "no critical illnesses"), while simultaneously enhancing the accuracy of voice reconstruction of numbers, dates, and other information. The generated target voice signal is used for secondary system verification or manual review to ensure the accuracy of the insurance application information and improve underwriting efficiency.

[0038] In summary, this invention addresses the challenge of maintaining both coding accuracy and speech reconstruction quality at low bit rates by innovatively combining a two-stage training architecture with product quantization technology. The first stage utilizes a mirrored encoder and decoder to achieve symmetric feature encoding, ensuring basic coding accuracy. Simultaneously, product quantization divides high-dimensional continuous vectors into sub-vectors, which are then matched with sub-encoding dictionaries to generate sub-indexes and combined into a unified index, significantly reducing the bit rate. The second stage employs a non-mirrored architecture. While reusing the encoder from the first stage to ensure feature consistency, it allows the decoder to flexibly design the network structure according to speech reconstruction needs, specifically optimizing reconstruction quality. Ultimately, this achieves both high coding accuracy and excellent speech reconstruction results while maintaining low bit rate transmission and storage.

[0039] In one embodiment, the training steps of the encoder and decoder obtained by training in this embodiment specifically include: inputting training data into a one-stage encoder and a one-stage decoder of a mirror architecture for a first stage of training to iteratively update the parameters of the one-stage encoder and the one-stage decoder, to obtain the trained one-stage encoder and the one-stage decoder; inputting training data into a two-stage encoder and a two-stage decoder of a non-mirror architecture for a second stage of training to iteratively update the parameters of the two-stage encoder and the two-stage decoder, to obtain the finally trained encoder and the decoder; wherein, the two-stage encoder uses the one-stage encoder and freezes the parameters of the one-stage encoder during the second stage of training, so that the encoder obtained by the final training is the one-stage encoder; the two-stage decoder uses the parameters of the one-stage decoder as the initial parameters for the second stage of training, so that the two-stage decoder obtained by the second stage of training is the finally trained decoder.

[0040] In this embodiment, the training process of the encoder and decoder is divided into two phases. First, the first phase of training is conducted. Prepared training data is input into the mirrored one-stage encoder and decoder. Through model learning from the training data, the parameters of the one-stage encoder and decoder are iteratively updated until the model reaches the preset training effect, resulting in the trained one-stage encoder and decoder. Next, the second phase of training is performed using a non-mirrored two-stage encoder and decoder, also with input training data. The two-stage encoder directly uses the one-stage encoder trained in the first phase, and during the second phase, the parameters of the one-stage encoder are frozen and no longer updated. Therefore, the encoder obtained after training is the one-stage encoder. The two-stage decoder uses the parameters of the one-stage decoder trained in the first phase as its initial parameters. Based on this, iteratively updates its own parameters through the second phase of training. After training is completed, the two-stage decoder is the final trained decoder. Through this two-stage training, the mirror architecture in the first stage, with its symmetrical design of the encoder and decoder, establishes a stable feature mapping relationship in the early stages of training, ensuring consistency between the encoding and initial decoding processes and providing the model with stable and high-precision basic encoding capabilities. The non-mirror architecture in the second stage breaks the constraint of structural symmetry, allowing the decoder to perform targeted optimizations for specific speech reconstruction needs (such as improving sound quality and optimizing temporal coherence) while inheriting the basic parameters from the first stage. Simultaneously, the stable encoding characteristics of the first stage are preserved due to the frozen encoder parameters. Thus, it leverages both the advantage of the mirror architecture in ensuring feature consistency during the basic training stage and the flexibility of the non-mirror architecture in the optimization stage, ultimately achieving a synergistic improvement in encoding accuracy and speech reconstruction quality, avoiding the limitations of a single architecture in low-bit-rate scenarios where both are difficult to balance.

[0041] In one embodiment, such as Figure 2 As shown, the steps of the first stage of training include: S201-S205.

[0042] S201. Construct a mirrored architecture of a one-stage encoder and a one-stage decoder, wherein the one-stage encoder and the one-stage decoder are symmetrical.

[0043] S202. Obtain the training speech signal, input the training speech signal into the first-stage encoder for encoding, and obtain the first training continuous vector.

[0044] S203. The training continuous vector is discretized into training discrete vectors in the encoding dictionary by the quantizer, and the corresponding first training index is generated.

[0045] S204. Obtain the corresponding training discrete vector from the encoding dictionary according to the first training index, and reconstruct the first training target speech signal by the first-stage decoder based on the training discrete vector;

[0046] S205. Based on the reconstruction error and quantization loss of the training speech signal and the first training target speech signal, update the parameters of the first-stage encoder, the first-stage decoder, the quantizer and the encoding dictionary until training converges to complete the first stage of training.

[0047] In this embodiment, the mirror architecture refers to the one-stage encoder and one-stage decoder employing a completely symmetrical network structure, maintaining consistency in terms of network layers, number of neurons per layer, and activation functions. The one-stage encoder is the neural network module used in the first stage of training to convert speech spectral features into continuous vectors. The one-stage decoder is the neural network module used in the first stage of training to reconstruct speech signals from discrete vectors, and is symmetrical to the one-stage encoder. The training speech signal refers to the raw speech data used for model training, which can cover speech from various scenarios, speeds, and intonations. The first training continuous vector is the undiscreteized feature vector output by the one-stage encoder after encoding the spectral features of the training speech signal. The quantizer is the module that implements the conversion from continuous vectors to discrete vectors, responsible for finding matching discrete vectors in the encoding dictionary. The encoding dictionary is a set of multiple discrete vectors, which are the basic units used to represent speech features during model training. The training discrete vector is the discretized feature vector in the encoding dictionary that matches the first training continuous vector. The first training index is the position identifier of the training discrete vector in the encoding dictionary. The first training target speech signal is the speech signal reconstructed by the one-stage decoder based on the training discrete vector. Reconstruction error is the difference between the training speech signal and the first training target speech signal, used to measure the reconstruction effect. Quantization loss is the difference between a continuous vector and its corresponding discrete vector during quantization, used to optimize the accuracy of the encoding dictionary.

[0048] In the first stage of training, a mirrored one-stage encoder and decoder are constructed to ensure symmetry in network layer structure and parameter scale, guaranteeing consistency in feature processing. Next, training speech signals are acquired, encompassing speech data from different genders, ages, accents, and environments. These signals are input into the one-stage encoder, which uses multi-layer neural network feature extraction and transformation, such as convolution and recurrent processing, to encode the spectral features of the training speech signals into a first continuous training vector containing key speech features. Subsequently, a quantizer discretizes this first continuous training vector, calculating the distance (e.g., Euclidean distance, cosine similarity) between the continuous vector and each discrete vector in the encoding dictionary. The discrete vector with the smallest distance is selected as the training discrete vector, and a first training index is generated for this vector. Finally, the one-stage decoder reconstructs the speech signal. Based on the first training index, the decoder retrieves the corresponding training discrete vector from the encoding dictionary and performs reverse feature transformation using a network structure symmetrical to the one-stage encoder, reconstructing the discrete vector into the first training target speech signal. Finally, based on the reconstruction errors (such as mean square error, absolute error, etc.) between the training speech signal and the first training target speech signal, as well as the quantization loss generated during quantization, the gradients of each parameter are calculated using the backpropagation algorithm. An optimizer (such as Adam, SGD, etc.) is then used to update the weights and biases of the first-stage encoder, the weights and biases of the first-stage decoder, the quantization parameters of the quantizer, and the discrete vectors in the encoding dictionary according to the gradients. This process of encoding, quantization, reconstruction, loss calculation, and parameter update is iterated repeatedly until the total loss of the model (the weighted sum of reconstruction error and quantization loss) no longer decreases significantly, reaching a convergent state, thus completing the first stage of training. This process provides a stable learning signal for the encoding dictionary through the symmetry of the mirror architecture, which helps improve the accuracy of the discretized representation. Simultaneously, it enables the first-stage encoder and first-stage decoder to cooperate well in collaborative training, laying a reliable foundation for the subsequent second stage of training.

[0049] In one embodiment, such as Figure 3 As shown, the steps of the second stage of training include: S206-S210.

[0050] S206. Construct a two-stage encoder and a two-stage decoder with a non-mirror architecture, wherein the two-stage encoder uses the same encoder as the one-stage encoder, and an audio quality enhancement auxiliary module is inserted before the input of the two-stage decoder. The parameters of the one-stage decoder used by the two-stage decoder are used as initial parameters. During the second stage training, the parameters of the one-stage encoder, the quantizer, and the encoding dictionary are frozen. The audio quality enhancement auxiliary module includes one of a converter module, a long short-term memory module, and a root mean square normalization module.

[0051] S207. Acquire training speech signal, input the training speech signal into the first-stage encoder for encoding, and obtain second training continuous vector;

[0052] S208. The second training continuous vector is discretized into the training discrete vector in the encoding dictionary by the quantizer, and the corresponding second training index is generated.

[0053] S209. Obtain the corresponding training discrete vector from the encoding dictionary according to the second training index, and input the training discrete vector into the sound quality enhancement auxiliary module and the two-stage decoder in sequence for processing to reconstruct the second training target speech signal;

[0054] S210. Based on the reconstruction error between the training speech signal and the second training target speech signal, update the parameters of the two-stage decoder and the sound quality enhancement auxiliary module until the training converges to complete the second stage of training.

[0055] In this embodiment, the non-mirror architecture refers to the asymmetric network structure design of the two-stage encoder and the two-stage decoder, which may differ in terms of the number of network layers and module composition. The audio quality enhancement auxiliary module is inserted before the input of the two-stage decoder to improve the quality of speech reconstruction. It includes one of the following: a transformer module (for capturing long-distance dependencies), a long short-term memory module (LSTM for capturing sequence information), and a root mean square normalization module (RMSNorm for improving numerical stability). The two-stage encoder is a network module that uses the first-stage encoder in the second stage of training, and the two-stage decoder is a network module that uses the parameters of the first-stage decoder as initial parameters in the second stage of training. The second training continuous vector is the feature vector output by the first-stage encoder after encoding the training speech signal. The second training index is the position identifier of the training discrete vector in the encoding dictionary. The second training target speech signal is the speech signal reconstructed by the two-stage decoder in conjunction with the audio quality enhancement auxiliary module.

[0056] In the second phase of training, firstly, a two-stage encoder and decoder using a non-mirror architecture are constructed. The second-stage encoder directly uses the first-stage encoder trained in the first phase. A sound quality enhancement auxiliary module is inserted before the input of the second-stage decoder, and the parameters of the second-stage decoder are used as initial parameters. During this process, the parameters of the first-stage encoder, quantizer, and encoding dictionary trained in the first phase must be frozen to ensure that they are not updated in the second phase of training. Next, the training speech signal is acquired and input into the first-stage encoder for encoding processing. Through feature extraction and transformation by the encoder, a second training continuous vector is obtained. Subsequently, the quantizer discretizes the second training continuous vector, calculates the similarity between the continuous vector and each discrete vector in the encoding dictionary, selects the discrete vector with the highest matching degree, and generates the corresponding second training index. Next, the corresponding training discrete vectors are obtained from the encoding dictionary according to the second training index. These training discrete vectors are then sequentially input into the audio quality enhancement auxiliary module and the two-stage decoder for processing. The audio quality enhancement auxiliary module performs targeted processing on the training discrete vectors (e.g., the converter module captures long-distance dependencies between discrete vectors, the long short-term memory module mines the temporal information of the discrete vector sequence, and the root mean square normalization module adjusts the discrete vector features numerically to improve stability). The second training target speech signal is then reconstructed by the two-stage decoder. Finally, based on the reconstruction error between the training speech signal and the second training target speech signal, the gradients of each parameter in the two-stage decoder and the audio quality enhancement auxiliary module are calculated using the backpropagation algorithm. The optimizer updates the parameters of these two parts according to the gradients. The above encoding, quantization, discrete vector processing, reconstruction, and parameter update processes are iterated repeatedly until the reconstruction error of the model tends to stabilize, reaching a training convergence state, thus completing the second stage of training.

[0057] This embodiment achieves several advantages. First, by using the mature encoder trained in the first stage and freezing its parameters, it retains the effective coding capabilities learned in the first stage, ensuring the stability and consistency of the coding process, while also preventing the invalidation of previous training results due to changes in encoder parameters. Second, the second-stage decoder breaks free from the constraints of the mirror architecture, allowing for flexible design of the network structure based on speech reconstruction needs. Combined with the targeted optimization of discrete vectors by the audio quality enhancement auxiliary module (such as strengthening the capture of speech detail features and improving the ability to model temporal coherence), it significantly improves the quality of speech reconstruction, making the target speech signal closer to the original speech in terms of naturalness and clarity. Simultaneously, the training method of updating only the decoder and audio quality enhancement auxiliary module parameters reduces the dimensionality of parameter optimization, lowers training complexity, and improves training efficiency. Furthermore, this architectural design allows the system to focus more on optimizing the reconstruction performance at the decoding end while maintaining coding efficiency, thereby achieving a balance between coding efficiency and speech quality in low bit-rate transmission scenarios, enhancing the system's practicality and adaptability.

[0058] In one embodiment, such as Figure 4 As shown, step S150 further includes: S151-S152.

[0059] S151. Input the discrete vector into the converter module, and perform cross-dimensional correlation modeling on the discrete vector through the multi-head self-attention mechanism of the converter module to capture the long-distance dependency relationship between different feature dimensions and generate discrete vector features that integrate global correlation information.

[0060] S152. Input the discrete vector features that integrate global correlation information into the two-stage decoder, and perform multi-level feature transformation on the discrete vector features through the two-stage decoder to generate target speech spectrum features.

[0061] In this embodiment, the converter module is a neural network module based on a self-attention mechanism. Its core is a multi-head self-attention mechanism, which can simultaneously perform parallel correlation calculations on multiple feature dimensions of discrete vectors. Cross-dimensional correlation modeling refers to the correlation analysis and fusion of features at different positions and in different dimensions of discrete vectors. Discrete vector features that integrate global correlation information refer to feature representations that integrate the global dependencies of speech features. Multi-level feature transformation refers to the two-stage decoder processing and adjusting the dimensions of the input features step by step through multiple network layers. Specifically, during processing, the discrete vector is first input to the converter module. The multi-head self-attention mechanism of the converter module first maps the discrete vector to multiple subspaces, calculates the attention weights between features in each subspace, and captures the long-distance dependencies of features in different dimensions (such as the frequency and time features of speech). Then, the correlation information of multiple subspaces is fused by weighted summation to generate discrete vector features containing global feature correlations. Then, this feature is input to the two-stage decoder. The decoder performs nonlinear transformations on the feature through multi-level structures such as convolutional layers and residual blocks, gradually recovering the detailed information of the speech spectrum, and finally generating the target speech spectrum features. The multi-head self-attention mechanism of the converter module can effectively capture the correlation of long-distance features in discrete vectors, which is especially suitable for encoding and decoding long speech segments (such as meeting minutes). It can ensure that semantically related content (such as pronouns and referents) in speech remains consistent during reconstruction, thereby improving the overall coherence and semantic integrity of long speech.

[0062] In one embodiment, such as Figure 5 As shown, step S150 further includes: S153-S154.

[0063] S153. The discrete vector is input into the Long Short-Term Memory (LSTM) module in the order of speech time sequence. The LSM module's gating unit performs temporal feature accumulation and key information filtering on the discrete vector, retains the dynamic change features of the speech sequence, and generates discrete vector features that integrate temporal correlation information.

[0064] S154. Input the discrete vector features of the fused temporal correlation information into the two-stage decoder, and perform temporal feature alignment and hierarchical mapping on the discrete vector features through the two-stage decoder to generate target speech spectrum features.

[0065] In this embodiment, the Long Short-Term Memory (LSTM) module is a recurrent neural network module capable of capturing the temporal dependencies of sequential data. Its gating units (including input gate, forget gate, and output gate) are used to selectively retain or discard temporal information. Temporal feature accumulation refers to the gradual superposition and updating of features of discrete vectors in chronological order. Discrete vector features incorporating temporal correlation information refer to feature representations containing the dynamic changes of speech sequences. Temporal feature alignment refers to the two-stage decoder calibrating the temporal dimension of the input features to ensure consistency with the temporal order of the original speech. Specifically, during processing, discrete vectors are first input into the LSM module in chronological order of speech. The module's forget gate filters and discards redundant historical features (such as background noise components in speech), while the input gate incorporates the key features of the current discrete vector (such as fundamental frequency changes in speech) into the memory unit. The output gate extracts effective information from the memory unit according to the current task requirements. Through this process, temporal features are accumulated and filtered, generating discrete vector features incorporating temporal correlation information. The feature is then input into a two-stage decoder. The decoder adjusts the time step of the feature through a temporal alignment mechanism, and then converts the feature into target speech spectral features that match the temporal sequence of the original speech through hierarchical mapping. The gating mechanism of the long short-term memory module can accurately capture the dynamic changes of the speech sequence, which is especially suitable for scenarios containing key temporal information. It can ensure that the time-series related content in the reconstructed speech is not disordered, and improve the reconstruction accuracy of time-sensitive speech.

[0066] In one embodiment, such as Figure 6 As shown, step S150 further includes: S155-S156.

[0067] S155. Input the discrete vector into the root mean square normalization module, calculate the mean square of the elements of the discrete vector through the root mean square normalization module and take the square root to obtain the root mean square value, perform a ratio operation between each element of the discrete vector and the root mean square value to achieve numerical scaling, and perform correction in combination with the learnable offset parameter to generate a discrete vector feature with stable numerical distribution.

[0068] S156. Input the discrete vector features with stable numerical distribution into the two-stage decoder, and perform feature mapping and spectral feature generation on the discrete vector features through the two-stage decoder to obtain the target speech spectral features.

[0069] In this embodiment, the root mean square (RMS) normalization module is a module that normalizes features using the root mean square (RMS) value. The element-square mean refers to the arithmetic mean of the squares of all elements in the discrete vector, and the RMS value is its square root. The learnable offset parameter refers to a correction parameter that can be adjusted during training to compensate for feature biases that may be lost during normalization. Numerically stable discrete vector features refer to feature representations where the numerical ranges of each dimension tend to be consistent and have small fluctuations. Feature mapping refers to the two-stage decoder performing a fine-tuning transformation on stable features to restore spectral details. Specifically, during processing, the discrete vector is first input to the RMS normalization module. The module first calculates the squares of all elements in the discrete vector, takes their arithmetic mean, and then takes the square root to obtain the RMS value. Then, each element of the discrete vector is compared with this RMS value to achieve numerical scaling of the features, making the magnitudes of the features in each dimension tend to be consistent. Finally, the learned offset parameter is used to fine-tune the scaled features, correcting any feature biases that may be caused by normalization, and generating numerically stable discrete vector features. The feature is then input into a two-stage decoder, which performs precise feature mapping based on a stable feature distribution, gradually generating detailed target speech spectral features. The root mean square normalization module effectively reduces the impact of outliers or extreme features in discrete vectors, making the feature distribution more stable. This is particularly suitable for speech encoding and decoding in noisy environments (such as conversations in the subway), reducing the interference of noise-induced feature fluctuations on reconstruction and improving speech clarity in complex environments.

[0070] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0071] This invention also provides a speech encoding / decoding method apparatus, which corresponds one-to-one with the speech encoding / decoding methods described in the above embodiments. For example... Figure 7 As shown, the speech encoding / decoding method apparatus 300 includes: a feature extraction unit 301, an encoding unit 302, a segmentation unit 303, a parsing unit 304, and a decoding unit 305. Detailed descriptions of each functional unit are as follows:

[0072] Feature extraction unit 301 is used to acquire the original speech signal and perform feature extraction on the original speech signal to obtain the corresponding original speech spectrum features;

[0073] Encoding unit 302 encodes the original speech spectral features into a continuous vector using an encoder trained on a mirror architecture in the first stage, wherein the network structures of the encoder and decoder in the mirror architecture are symmetrical.

[0074] The segmentation unit 303 is used to segment the continuous vector into multiple sub-vectors, perform similarity matching between the sub-vectors and the corresponding discrete vectors in the sub-encoding dictionary through a quantizer to obtain the sub-index corresponding to the discrete vector that matches the sub-vector, and perform combined calculation on each sub-index according to the size of each sub-encoding dictionary to generate an overall index corresponding to the discrete vector that matches the continuous vector.

[0075] The parsing unit 304 is used to perform an inverse operation on the overall index according to the size of each sub-encoding dictionary to parse each sub-index, obtain the corresponding sub-discrete vector from each sub-encoding dictionary corresponding to each sub-index, and concatenate the obtained multiple sub-discrete vectors to obtain a discrete vector that matches the continuous vector.

[0076] Decoding unit 305 is used to reconstruct the target speech spectrum features from the discrete vector using a decoder trained on a non-mirror architecture in the second stage, and to convert the target speech spectrum features into a target speech signal. The network structures of the encoder and decoder in the non-mirror architecture are asymmetrical. The encoder in the second stage uses the encoder in the first stage, and the structure of the decoder in the second stage is different from that of the decoder in the first stage.

[0077] In one embodiment, the speech encoding / decoding method apparatus includes a training unit. The training unit is used to input training data into a mirrored architecture one-stage encoder and one-stage decoder for a first-stage training to iteratively update the parameters of the one-stage encoder and one-stage decoder, resulting in a trained one-stage encoder and one-stage decoder; and to input training data into a non-mirrored architecture two-stage encoder and two-stage decoder for a second-stage training to iteratively update the parameters of the two-stage encoder and two-stage decoder, resulting in a finally trained encoder and decoder. The two-stage encoder uses the parameters of the one-stage encoder, but freezes the parameters of the one-stage encoder during the second-stage training, so that the finally trained encoder is the one-stage encoder. The two-stage decoder uses the parameters of the one-stage decoder as the initial parameters for the second-stage training, so that the two-stage decoder after the second-stage training is the finally trained decoder.

[0078] In one embodiment, the training unit is further configured to: construct a mirror-architectured one-stage encoder and one-stage decoder, wherein the one-stage encoder and one-stage decoder are symmetrical; acquire a training speech signal, input the training speech signal into the one-stage encoder for encoding to obtain a first training continuous vector; discretize the training continuous vector into a training discrete vector in the encoding dictionary through the quantizer, and generate a corresponding first training index; obtain the corresponding training discrete vector from the encoding dictionary according to the first training index, and reconstruct a first training target speech signal through the one-stage decoder based on the training discrete vector; update the parameters of the one-stage encoder, the one-stage decoder, the quantizer, and the encoding dictionary based on the reconstruction error and quantization loss between the training speech signal and the first training target speech signal, until training converges to complete the first stage of training.

[0079] In one embodiment, the training unit is further configured to: construct a non-mirror-image architecture two-stage encoder and a two-stage decoder, wherein the two-stage encoder uses the same parameters as the one-stage encoder, and a sound quality enhancement auxiliary module is inserted before the input of the two-stage decoder; the two-stage decoder uses the parameters of the one-stage decoder as initial parameters; during the second stage of training, the parameters of the one-stage encoder, the quantizer, and the encoding dictionary are frozen; the sound quality enhancement auxiliary module includes one of a converter module, a long short-term memory module, and a root mean square normalization module; acquire a training speech signal, and input the training speech signal into the one-stage encoder for further processing. The second training continuous vector is obtained by row encoding; the second training continuous vector is discretized into training discrete vectors in the encoding dictionary by the quantizer, and a corresponding second training index is generated; the corresponding training discrete vector is obtained from the encoding dictionary according to the second training index, and the training discrete vector is sequentially input into the sound quality enhancement auxiliary module and the two-stage decoder for processing to reconstruct the second training target speech signal; based on the reconstruction error between the training speech signal and the second training target speech signal, the parameters of the two-stage decoder and the sound quality enhancement auxiliary module are updated until the training converges to complete the second stage of training.

[0080] In one embodiment, the decoding unit 305 is further configured to: input the discrete vector to the converter module, perform cross-dimensional correlation modeling on the discrete vector through the multi-head self-attention mechanism of the converter module, capture long-distance dependencies between different feature dimensions, and generate discrete vector features that integrate global correlation information; input the discrete vector features that integrate global correlation information to the two-stage decoder, and perform multi-level feature transformation on the discrete vector features through the two-stage decoder to generate target speech spectrum features.

[0081] In one embodiment, the decoding unit 305 is further configured to: input the discrete vector into the long short-term memory module in the order of speech temporal sequence; perform temporal feature accumulation and key information filtering on the discrete vector through the gating unit of the long short-term memory module, retain the dynamic change features of the speech sequence, and generate discrete vector features that integrate temporal correlation information; input the discrete vector features that integrate temporal correlation information into the two-stage decoder; and perform temporal feature alignment and hierarchical mapping on the discrete vector features through the two-stage decoder to generate target speech spectrum features.

[0082] In one embodiment, the decoding unit 305 is further configured to: input the discrete vector to the root mean square normalization module, calculate the mean square of the elements of the discrete vector and take the square root to obtain the root mean square value, perform a ratio operation between each element of the discrete vector and the root mean square value to achieve numerical scaling, and perform correction in combination with a learnable offset parameter to generate discrete vector features with stable numerical distribution; input the discrete vector features with stable numerical distribution to the two-stage decoder, and perform feature mapping and spectral feature generation on the discrete vector features through the two-stage decoder to obtain the target speech spectral features.

[0083] Specific limitations regarding the speech encoding / decoding method apparatus can be found in the limitations of the speech encoding / decoding method described above, and will not be repeated here. Each module in the aforementioned speech encoding / decoding method apparatus can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0084] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a voice encoding / decoding method on the server side.

[0085] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 9As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the functions or steps of a voice encoding / decoding method on the client side.

[0086] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described speech encoding and decoding method.

[0087] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described speech encoding / decoding method.

[0088] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0089] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0090] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0091] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A speech encoding and decoding method, characterized in that, include: Obtain the original speech signal, and extract the corresponding original speech spectrum features from the original speech signal; The original speech spectral features are encoded into a continuous vector by an encoder trained using a mirror architecture in the first stage, wherein the network structures of the encoder and decoder in the mirror architecture are symmetrical. The continuous vector is divided into multiple sub-vectors. The sub-vectors are matched with the corresponding discrete vectors in the sub-encoding dictionary by a quantizer to obtain the sub-index corresponding to the discrete vector that matches the sub-vector. The sub-indexes are combined and calculated according to the size of each sub-encoding dictionary to generate the overall index corresponding to the discrete vector that matches the continuous vector. Based on the size of each sub-encoding dictionary, the overall index is parsed by performing an inverse operation to obtain each sub-index. The corresponding sub-discrete vector is obtained from each sub-encoding dictionary corresponding to each sub-index. The multiple obtained sub-discrete vectors are concatenated to obtain a discrete vector that matches the continuous vector. The discrete vector is reconstructed using a decoder trained in the second stage using a non-mirror architecture to obtain the target speech spectral features, and the target speech spectral features are converted into a target speech signal. The network structures of the encoder and decoder in the non-mirror architecture are asymmetrical. The encoder in the second stage uses the encoder in the first stage, and the structure of the decoder in the second stage is different from that of the decoder in the first stage. The training steps for the trained encoder and decoder include: inputting training data into a mirrored architecture's one-stage encoder and decoder for a first-stage training to iteratively update the parameters of the one-stage encoder and decoder, resulting in a trained one-stage encoder and decoder; inputting training data into a non-mirrored architecture's two-stage encoder and decoder for a second-stage training to iteratively update the parameters of the two-stage encoder and decoder, resulting in a finally trained encoder and decoder; wherein the two-stage encoder uses the same parameters as the one-stage encoder, and freezes the parameters of the one-stage encoder during the second-stage training, so that the finally trained encoder is the same as the one-stage encoder; the two-stage decoder uses the parameters of the one-stage decoder as the initial parameters for the second-stage training, so that the two-stage decoder after the second-stage training is the finally trained decoder.

2. The method according to claim 1, characterized in that, The first phase of training includes the following steps: Construct a mirrored architecture with a one-stage encoder and a one-stage decoder, where the one-stage encoder and the one-stage decoder are symmetrical. Acquire training speech signals, input the training speech signals into the first-stage encoder for encoding, and obtain a first training continuous vector; The quantizer discretizes the first continuous training vector into discrete training vectors in the encoding dictionary and generates the corresponding first training index. The corresponding training discrete vector is obtained from the encoding dictionary according to the first training index, and the first training target speech signal is obtained by the first-stage decoder based on the training discrete vector. Based on the reconstruction error and quantization loss between the training speech signal and the first training target speech signal, the parameters of the first-stage encoder, the first-stage decoder, the quantizer, and the encoding dictionary are updated until training converges to complete the first stage of training.

3. The method according to claim 2, characterized in that, The steps of the second phase of training include: Construct a non-mirror architecture for a two-stage encoder and a two-stage decoder, wherein the two-stage encoder uses the same encoder as the one-stage encoder, and an audio quality enhancement auxiliary module is inserted before the input of the two-stage decoder. The two-stage decoder uses the parameters of the one-stage decoder as initial parameters. During the second stage of training, the parameters of the one-stage encoder, the quantizer, and the encoding dictionary are frozen. The audio quality enhancement auxiliary module includes one of a converter module, a long short-term memory module, and a root mean square normalization module. Acquire training speech signals, input the training speech signals into the first-stage encoder for encoding, and obtain a second training continuous vector; The second training continuous vector is discretized into the training discrete vector in the encoding dictionary by the quantizer, and a corresponding second training index is generated. The corresponding training discrete vector is obtained from the encoding dictionary according to the second training index, and the training discrete vector is sequentially input into the sound quality enhancement auxiliary module and the two-stage decoder for processing to reconstruct the second training target speech signal; Based on the reconstruction error between the training speech signal and the second training target speech signal, the parameters of the two-stage decoder and the sound quality enhancement auxiliary module are updated until the training converges to complete the second stage of training.

4. The method according to any one of claims 1-3, characterized in that, The decoder trained in the second stage using a non-mirror architecture includes a converter module and a two-stage decoder. The step of reconstructing the target speech spectral features from the discrete vector using the decoder trained in the second stage using a non-mirror architecture includes: The discrete vector is input into the converter module, and the multi-head self-attention mechanism of the converter module is used to perform cross-dimensional correlation modeling on the discrete vector, capture the long-distance dependency relationship between different feature dimensions, and generate discrete vector features that integrate global correlation information. The discrete vector features that integrate global correlation information are input into the two-stage decoder, and the two-stage decoder performs multi-level feature transformation on the discrete vector features to generate target speech spectrum features.

5. The method according to any one of claims 1-3, characterized in that, The decoder trained in the second stage using a non-mirror architecture includes a long short-term memory module and a two-stage decoder. The step of reconstructing the target speech spectral features from the discrete vector using the decoder trained in the second stage using a non-mirror architecture includes: The discrete vectors are input into the Long Short-Term Memory (LSTM) module in the order of speech time sequence. The LSM module's gating unit performs temporal feature accumulation and key information filtering on the discrete vectors, retaining the dynamic change features of the speech sequence and generating discrete vector features that integrate temporal correlation information. The discrete vector features with fused temporal correlation information are input into the two-stage decoder. The two-stage decoder performs temporal feature alignment and hierarchical mapping on the discrete vector features to generate target speech spectral features.

6. The method according to any one of claims 1-3, characterized in that, The decoder trained in the second stage using a non-mirror architecture includes a root mean square normalization module and a two-stage decoder. The step of reconstructing the target speech spectral features from the discrete vector using the decoder trained in the second stage using a non-mirror architecture includes: The discrete vector is input to the root mean square normalization module, which calculates the square mean of the elements of the discrete vector and takes the square root to obtain the root mean square value. The ratio of each element of the discrete vector to the root mean square value is calculated to achieve numerical scaling. The learnable offset parameter is used for correction to generate a discrete vector feature with stable numerical distribution. The discrete vector features with stable numerical distribution are input into the two-stage decoder. The two-stage decoder performs feature mapping and spectral feature generation on the discrete vector features to obtain the target speech spectral features.

7. A voice encoding / decoding device, characterized in that, Includes a unit for performing the method according to any one of claims 1-6.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images