Speech processing method, apparatus, device, medium, and product

Abstract

The application provides a speech processing method, device, equipment, medium and product. The method comprises the following steps: obtaining to-be-processed speech data, and extracting a target speech feature vector of the to-be-processed speech data; performing quantization processing on the target speech feature vector to obtain a quantized speech feature vector; and obtaining a processing result of the to-be-processed speech data based on the quantized speech feature vector. The application determines the speech processing result of the to-be-processed speech data based on the quantized speech feature vector mapped into a limited space, can learn the global essential features of the speech signal, and improves the quality of the speech data processing result.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to speech processing methods, apparatus, equipment, media and products. Background Technology

[0002] In existing AI-powered speech processing tasks (such as speech enhancement, speech conversion, and speech separation), the speech data to be processed is input into a neural network model. The model then extracts and processes features from the speech data, ultimately outputting the processed speech data. Current mainstream speech processing models are mostly based on deep learning architectures, including Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Transformers. These models progressively mine the temporal and frequency domain features of speech signals through multi-layered network structures. For example, in speech enhancement tasks, the model needs to separate clean speech components from noisy speech; in speech conversion tasks, it needs to convert speaker features while preserving the speech content; and in speech separation tasks, it needs to separate independent speech streams from different speakers from mixed speech. The core of these tasks relies on the accurate extraction and effective processing of speech features; the quality of feature representation directly determines the performance of the final processing result.

[0003] In existing technologies, when using neural networks for speech processing, feature extraction of speech data maps the data to an infinite continuous space. During training, each iteration of the learning process seeks the features most suitable for the current training batch within this continuous space. However, speech data, compared to other data, possesses harmonic characteristics. This means speech data has informational features at a global scale. For example, different speakers' voices have unique fundamental frequency ranges and formant distributions; the same speaker's voice maintains a stable harmonic structure in different contexts; even in noisy environments, the harmonic characteristics of the speech signal persist, becoming the core basis for distinguishing speech from noise. Speech data features should be confined to a finite space. The current method of mapping speech data to an infinite continuous space is too divergent, failing to learn the essential global features of the speech signal in information processing, resulting in low-quality speech data processing results. Summary of the Invention

[0004] This application provides speech processing methods, apparatus, equipment, media, and products to address the shortcomings of low quality in speech data processing results in the prior art, and to achieve the technical effect of improving the quality of speech data processing results.

[0005] This application provides a speech processing method, including: Acquire the speech data to be processed, and extract the target speech feature vector from the speech data to be processed; The target speech feature vector is quantized to obtain a quantized speech feature vector; The processing result of the speech data to be processed is obtained based on the quantized speech feature vector.

[0006] According to a speech processing method provided in this application, the target speech feature vector is an attention mechanism vector; the extraction of the target speech feature vector from the speech data to be processed includes: The speech data to be processed is input into the speech processing model, and the target speech feature vector is obtained through the attention mechanism vector extraction module in the speech processing model.

[0007] According to a speech processing method provided in this application, the target speech feature vector includes key-value vector pairs in the attention mechanism vector.

[0008] According to a speech processing method provided in this application, the step of obtaining the processing result of the speech data to be processed based on the quantized speech feature vector includes: Based on the target speech feature vector and the attention mechanism, attention features are obtained; Based on the attention features, the processing result of the speech data to be processed is obtained.

[0009] According to a speech processing method provided in this application, the step of quantizing the target speech feature vector to obtain a quantized speech feature vector includes: The target speech feature vector is input into the codec to obtain the reconstructed speech feature vector output by the codec; The reconstructed speech feature vector is quantized to obtain the quantized speech feature vector.

[0010] According to a speech processing method provided in this application, the step of quantizing the target speech feature vector to obtain a quantized speech feature vector includes: Replace the target speech feature vector with a preset, specified quantized speech feature vector; The specified quantized speech feature vector is obtained by training multiple times based on multiple samples of speech data to be processed. Each training process includes: The sample speech data to be processed is input into the feature extraction module in the speech processing model to obtain the sample target speech feature vector of the sample speech data to be processed; The sample target speech feature vector is quantized to obtain the sample quantized speech feature vector; The quantized speech feature vector of the sample is input into the subsequent processing module in the speech processing model to obtain the sample speech processing result corresponding to the sample speech data to be processed. The speech processing model is trained based on the sample speech processing results; The quantized speech feature vectors of the samples during the training process are statistically analyzed, and the specified quantized speech feature vectors are determined based on the statistical results.

[0011] This application also provides a voice processing device, including: The feature extraction module is used to acquire the speech data to be processed and extract the target speech feature vector of the speech data to be processed. The quantization module is used to quantize the target speech feature vector to obtain a quantized speech feature vector; The processing result output module is used to obtain the processing result of the speech data to be processed based on the quantized speech feature vector.

[0012] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described voice processing methods.

[0013] This application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech processing method described above.

[0014] This application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described speech processing methods.

[0015] The speech processing method, apparatus, device, medium, and product provided in this application extract features from the speech data to be processed to obtain a target speech feature vector, and then quantize the target speech feature vector to obtain a quantized speech feature vector. The quantization process maps the feature vector to a finite space. Therefore, the quantized speech feature vector is not a vector in an infinite continuous space, but a feature vector in a finite dictionary. In this way, the speech processing result of the speech data to be processed is determined based on the quantized speech feature vector mapped to the finite space. The global essential features of the speech signal can be learned, thereby improving the quality of the speech data processing result. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the speech processing method provided in this application.

[0018] Figure 2 This is a schematic diagram of FSQ in one implementation of the speech processing method provided in this application.

[0019] Figure 3 This is a diagram of the existing attention mechanism's computational process.

[0020] Figure 4 This is a diagram illustrating the attention mechanism operation process in one implementation of the speech processing method of this application.

[0021] Figure 5 This is a schematic diagram of embedding the speech processing method provided in this application into an existing speech processing model. Figure 1 .

[0022] Figure 6 This is a schematic diagram of embedding the speech processing method provided in this application into an existing speech processing model. Figure 2 .

[0023] Figure 7 This is a schematic diagram of the speech processing device provided in this application.

[0024] Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0027] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0028] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0029] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0030] The following is combined with Figures 1-6 Describe the speech processing method provided in this application. For example... Figure 1 As shown, the speech processing method includes the following steps: S110. Obtain the speech data to be processed and extract the target speech feature vector of the speech data to be processed. S120. Quantize the target speech feature vector to obtain the quantized speech feature vector; S130. Obtain the processing result of the speech data to be processed based on the quantized speech feature vector.

[0031] In the speech processing method provided in this application, the speech data to be processed is the speech data that needs to be processed. The processing tasks for the speech data to be processed in the speech processing method provided in this application can include various speech processing tasks, such as speech enhancement and speech conversion. The speech data to be processed can be the speech signal obtained by converting user speech received through a microphone or other sound receiving device. The sources of the speech data to be processed are diverse, including not only real-time acquired speech signals, but also historical speech data stored locally or in the cloud, and speech data streams transmitted over the network.

[0032] After acquiring the speech data to be processed, the target speech feature vector is extracted from the speech data. This extraction can be achieved through the feature extraction module in the speech processing model. In one possible implementation, the speech processing model can adopt the architecture of an existing speech processing task model. Different speech processing model architectures can be used depending on the specific speech processing task. For example, for speech conversion tasks, the architecture of seed-VC (an open-source speech conversion model based on the SEED-TTC architecture) can be used as the architecture of the speech processing model in the speech processing method provided in this application; for speech separation tasks, the architecture of TF-GridNet (a speech separation model) can be used as the architecture of the speech processing model in the speech processing method provided in this application, and so on. Different speech processing tasks have different core requirements; therefore, the choice of model architecture needs to be tailored accordingly. Taking speech conversion tasks as an example, the seed-VC model, based on the SEED-TTC architecture, possesses efficient speaker feature conversion capabilities. Its core lies in extracting content and speaker features from the speech through a Transformer encoder, and then generating the target speaker's speech through a decoder. The method in this application can be embedded into the attention mechanism of its Transformer encoder to quantize key-value vector pairs, improving the accuracy of speaker feature conversion. For speech separation tasks, the TF-GridNet model employs a spatiotemporal-frequency three-dimensional grid structure, effectively capturing the spatiotemporal-frequency features of speech signals, making it suitable for separating mixed speech from multiple speakers. The method provided in this application can be embedded into its self-attention module to constrain the feature space and improve separation performance. Furthermore, for speech enhancement tasks, architectures such as DeepFilterNet and the DNS series models can be used. These models exhibit excellent noise suppression performance, and embedding them into the method provided in this application can further improve the clean speech restoration accuracy.

[0033] In another possible implementation, the speech processing model can be composed of functional modules from existing neural network models, including fully connected layers, convolutional layers, pooling layers, and residual modules. This modular approach offers greater flexibility, allowing the model architecture to be customized to specific task requirements.

[0034] In the speech processing method provided in this application, a new quantization module is added after the feature extraction module of the speech processing model used for the speech processing task. This quantization module is used to quantize the extracted target speech feature vector to obtain a quantized speech feature vector, which is then input into the subsequent processing module in the speech processing model. It is worth noting that the speech processing model can have multiple feature extraction modules. Quantization can be performed on the target speech feature vector extracted by one of the feature extraction modules, or on the target speech feature vectors extracted by multiple or all of the feature extraction modules. The embedding position and number of quantization modules can be flexibly adjusted according to the model architecture and task requirements. For models with a single feature extraction module, the quantization module is directly embedded after that module; for models with multiple feature extraction modules, the output of key feature extraction modules (such as the attention mechanism module or deep feature extraction module) can be quantized, or the output of all feature extraction modules can be quantized. For example, in Transformer-based models, each encoder and decoder layer includes an attention mechanism module. Quantization can be performed on key-value vector pairs from all attention mechanism modules to comprehensively optimize feature representation. In hybrid CNN and Transformer architectures, only the output of the Transformer attention module can be quantized, focusing on optimizing the representation of long-range dependent features. The quantization module is embedded in a "plug-and-play" manner, requiring no modification to the core structure of the original model. Only a quantization operation step needs to be added between the feature extraction module and subsequent processing modules, reducing the difficulty of model modification.

[0035] The speech processing method provided in this application extracts features from the speech data to be processed to obtain a target speech feature vector, and then quantizes the target speech feature vector to obtain a quantized speech feature vector. The quantization process maps the feature vector to a finite space. Therefore, the quantized speech feature vector is not a vector in an infinite continuous space, but a feature vector in a finite dictionary. In this way, the speech processing result of the speech data to be processed is determined based on the quantized speech feature vector mapped to the finite space. The global essential features of the speech signal can be learned, thereby improving the quality of the speech data processing result.

[0036] In one possible implementation, the feature extraction module for extracting the target speech feature vector in the speech processing method provided in this application is the attention mechanism vector extraction module in the speech processing model. That is, the target speech feature vector is an attention mechanism vector, and extracting the target speech feature vector from the speech data to be processed includes: The speech data to be processed is input into the speech processing model, and the target speech feature vector is obtained through the attention mechanism vector extraction module in the speech processing model.

[0037] The attention mechanism includes three types of vectors: Q (Query) vector, K (Key) vector, and V (Value) vector. The attention mechanism determines the attention weights by calculating the similarity between the query and key vectors, and then performs a weighted sum of the value vectors to obtain the final output. The attention mechanism is a crucial component of the Transformer architecture, a vital structure in neural networks. It dynamically models the dependencies at any position in the speech signal through the attention mechanism, significantly improving word error rate performance. Furthermore, compared to other sequential computational structures, the Transformer supports fully parallel processing, resulting in higher efficiency and lower latency.

[0038] In deep learning, attention mechanisms mimic the selective attention humans exhibit when processing information. This allows models to dynamically adjust their attention weights as they process input data, highlighting important information and ignoring less important information. Because attention mechanisms allow models to focus on key parts of information and ignore irrelevant information, they can improve processing efficiency and accuracy. Compared to traditional sequence processing models, such as recurrent neural networks (RNNs) and long short-term memory networks (LSTMs), which are prone to losing information from earlier inputs as sequence length increases and thus failing to capture long-distance dependencies, attention mechanisms allow models to establish direct connections between different positions in a sequence, effectively capturing the dependencies between them regardless of their distance. There are also variations of attention mechanisms, such as multi-head attention and cross-attention.

[0039] In speech processing tasks, the attention mechanism allows models to learn to "selectively listen." For example, in speech enhancement tasks, during training, the model is exposed to a large amount of noisy audio data. It learns to assign higher weights to frequency bands and time points that may contain human voices, while suppressing parts that appear as background noise. The attention mechanism provides flexibility; it does not rely on fixed sound templates but dynamically constructs the association between sound features and text based on the actual audio input. The ability to "selectively listen" is the core application value of the attention mechanism in speech processing. Its essence is to achieve accurate extraction of target information by dynamically adjusting attention weights. In speech enhancement tasks, the model can learn to identify the feature differences between speech signals and noise. For example, speech signals have obvious harmonic structures, while noise is mostly random signals. Therefore, the model will assign higher attention weights to frequency bands with prominent harmonic features, thus focusing on retaining information from these frequency bands and suppressing noise frequency bands in subsequent processing. In speech recognition tasks, the attention mechanism can dynamically associate speech features with text features. For example, when there is unclear pronunciation in the speech signal, the model will adjust the attention weights and combine them with contextual text information to correct errors. This flexibility allows the model to adapt to different speech scenarios and input conditions without the need for predefined fixed feature templates, significantly improving the model's generalization ability. Furthermore, in multi-speaker speech separation tasks, the attention mechanism can distinguish the speech features of different speakers, assigning unique attention weights to each speaker, thus achieving accurate separation of the speech streams from different speakers.

[0040] The speech processing method provided in this application does not require major modifications to existing attention mechanism modules. It only requires adding a quantization step to the output attention mechanism vector. This allows it to be embedded into various existing speech processing models, exhibiting high compatibility, lower application difficulty, and lower application cost. Compatibility is one of the key advantages of this application, its core being the "non-intrusive" design of the quantization step. Existing speech processing models typically output three vectors: Q, K, and V. This application only needs to obtain the target vector, quantize it, and then feed the quantized vector back to the subsequent computation of the original model, without modifying the internal structure and computational logic of the attention mechanism module. This design enables the application to quickly adapt to various speech processing models based on Transformer, CNN+Attention, and other architectures, such as seed-VC, TF-GridNet, DeepFilterNet, and Wav2Vec, reducing the difficulty of technology migration and application. For existing deployed models, performance improvement can be achieved simply by adding a quantization step to the model inference process, without retraining the entire model (or only requiring minor fine-tuning), significantly reducing application costs. In addition, the quantization process involves relatively little computation and will not significantly affect the overall processing efficiency of the model, thus ensuring the engineering practicality of the solution.

[0041] In attention mechanisms applied to speech processing tasks, query vectors often reflect the features of noisy speech, while key and value vectors often reflect the features of clean speech (noise-free speech). Noise-free speech is clean speech, possessing the characteristic of informational features of speech data at a global scale. Therefore, in some possible embodiments, the target speech feature vector includes the key or value vector in the attention mechanism vector. Further, in some possible embodiments, the target speech feature vector includes key-value pairs in the attention mechanism vector, thus mapping the features of clean speech across multiple dimensions to a finite space, further improving the quality of speech data processing results provided by the speech processing method of this application. In speech processing tasks, the physical meanings of Q, K, and V vectors are clearly distinguishable. Taking speech enhancement tasks as an example, the query vector Q is usually generated from noisy speech features and is used to "query" information related to clean speech; the key vector K and value vector V are generated from clean speech features (or clean speech representations learned by the model), with K providing the feature attributes of clean speech and V providing the content information of clean speech. Therefore, K and V vectors better reflect the global essential features of speech signals (such as harmonic structure and formant distribution). Quantizing them as target speech feature vectors allows for more precise constraint of the feature space, highlighting the global features of clean speech. Compared to using K or V vectors alone, choosing a KV vector pair can simultaneously preserve both the attributes and content information of clean speech. The quantized vector pair can more comprehensively represent the features of clean speech, providing more accurate input for subsequent attention operations and thus improving the quality of the processing results. For example, in speech conversion tasks, the K vector can represent the target speaker's speech attributes (such as fundamental frequency range and formant position), while the V vector can represent the content features of the speech. The quantized KV vector pair can more stably transmit the target speaker's attribute information, making the converted speech closer to the target speaker's timbre.

[0042] Compared to traditional attention mechanisms, the speech processing method provided in this application employs a quantization structure before the key and value vectors participate in the computation, making them finite key-value pairs. This serves as a constraint, enabling the attention mechanism to learn global key-value pairs, potentially improving the system's generalization ability and the quality of the speech processing results. In traditional attention mechanisms, the K and V vectors are vectors in continuous space, with their number matching the length of the speech sequence. This results in an infinite number of KV pairs, making it difficult for the model to learn universally applicable global features, thus limiting generalization ability. This application maps the continuous KV vectors to a finite discrete space through quantization, forming a finite number of KV pairs. These KV pairs constitute a globally shared feature dictionary that reflects the common global features of different speech samples. During training, the model no longer needs to learn dedicated continuous space features for each training batch; instead, it learns the mapping relationship between the KV pairs in the finite dictionary and the speech processing results, thereby improving the model's generalization ability. For example, when faced with unfamiliar noisy environments or new speakers, the key-value pairs in a finite dictionary can provide more stable feature references, enabling the model to quickly adapt to new scenarios. Furthermore, the constraint of finite key-value pairs can reduce the risk of overfitting, allowing the model to maintain good processing performance even when there are differences in the distribution of training and test data, further improving the quality and stability of speech processing results.

[0043] The target speech feature vector is quantized to obtain a quantized speech feature vector. This can be achieved using existing quantization methods, such as FSQ (Finite Scalar Quantization) and VQ-VAE (Vector Quantized-Variational Autoencoder). Taking FSQ as an example... Figure 2 As shown, FSQ is a finite-scale quantization method. Its principle is to compress information onto discrete points of a high-order cube, thereby achieving quantization.

[0044] like Figure 3 As shown, in existing attention mechanisms, the Q (Query) vector is first multiplied by the K (key) vector (...). Figure 3 After the "Matmul" operation, a scaled dot product is then performed. Figure 3 The "Scale" operation is performed, followed by normalization. Figure 3The "Softmax" operation in the query vector and key vector is used to perform matrix multiplication with the V (value) vector. In the speech processing method provided in this application, such as... Figure 4 As shown, in the speech processing method provided in this application, before performing operations on the key vector, value vector, and query vector, the key vector and value vector are first quantized. Figure 4 The image shows the processing using the FSQ quantization method.

[0045] In the speech processing method provided in this application, after the K and V vectors are processed by FSQ, the resulting K' and V' form key-value pairs. Furthermore, because FSQ uses the nearest neighbor method, the finite K' and V' pairs only form a finite number of key-value pairs. Compared to the infinite number of key-value pairs obtained from the original K and V, this provides a finite codebook. Moreover, during the inference phase, the key-value pair codebook remains fixed and does not change with variations in samples.

[0046] Furthermore, in one possible implementation, the target speech feature vector is quantized to obtain a quantized speech feature vector, including: The target speech feature vector is input into the codec to obtain the reconstructed speech feature vector output by the codec; The reconstructed speech feature vector is quantized to obtain the quantized speech feature vector.

[0047] An encoder-decoder consists of an encoder and a decoder. By encoding first and then decoding, the original target speech feature vector can be reconstructed into a compact, useful, and structured feature representation, enabling the learning of deeper features and thus more accurate global information features. The core function of the encoder-decoder is feature compression and reconstruction. Essentially, the encoder maps the high-dimensional, redundant original feature vector to a low-dimensional latent space, removing irrelevant information and retaining core features. Then, the decoder reconstructs the low-dimensional latent space features into a high-dimensional feature vector, achieving feature optimization and structuring. The encoder typically uses multiple convolutional layers, fully connected layers, or Transformer encoders. By progressively reducing the feature dimensionality, it extracts high-level semantic features. For example, in speech processing, the encoder can extract core global features such as the fundamental frequency, formants, and harmonic structure of the speech, removing redundant information such as noise and distortion. The decoder uses a network structure symmetrical to the encoder, such as transposed convolutional layers, unpooling layers, or Transformer decoders, to upsample the low-dimensional latent space features to the original feature dimension, reconstructing a speech feature vector with a clearer structure and more accurate representation. Through codec processing, the structure of the target speech feature vector is significantly improved, and the global features are more prominent, providing a higher quality input for subsequent quantization processing. For example, for the KV vector of noisy speech, after reconstruction by the codec, the noise component is effectively suppressed, and the global features of clean speech are clearer. Further quantization processing can further improve the quality of quantized features and reduce the impact of quantization errors on subsequent processing.

[0048] In the speech processing method provided in this application, the quantization of the target speech feature vector can be implemented through a quantization module embedded in the speech processing model and applied to various tasks such as speech enhancement and speech conversion. Specifically, the step of quantizing the target speech feature vector in the speech processing method provided in this application can be applied to any one or more attention mechanism modules in the speech processing model. For one or more attention mechanism modules in the speech processing model, after generating the key vector and value vector, before the key vector and value vector participate in the calculation, the key vector and value vector are first quantized as the target speech feature vector to obtain the quantized speech feature vector. The quantized key vector and value vector participate in the calculation according to the original attention mechanism principle and further participate in subsequent processing steps.

[0049] For example Figure 5 and Figure 6As shown, taking TF-GridNet (a common speech enhancement model architecture) as an example, the network's input, output, and target are noisy speech, estimated speech, and target speech, respectively. The network's attention structure can be replaced with functional modules that implement the speech processing method provided in this application, or the attention mechanism can be connected in a concatenated manner to the quantization module that implements the quantization processing in the speech processing method provided in this application. Figure 5 and Figure 6 The diagrams show the block and attention structures of TF-Gridnet, respectively. Figure 6 As shown (the red part indicates the embedding location of the quantization module in the speech processing method provided in this application), the quantization module is placed at the end of the K and V matrices and at the beginning of Matmul. The module implementing the speech processing method provided in this application can be partially replaced. Figure 5 The self-attention module shown can also be concatenated as an additional finite dictionary attention module. Figure 5 The latter part of the self-attention module is shown in the figure.

[0050] In the speech processing method provided in this application, the embedding of the quantization module does not affect the training and inference methods of the speech processing model. Taking the speech conversion task as an example, information such as the mel spectrum, fundamental frequency F0, and speaker embedding of the original and target speech are extracted to form the condition and mel spectrum. Then, the model is trained using flow-matching and Diffusion Transformer (DiT, a fusion architecture that combines transformer and diffusion model). During testing, the converted mel spectrum is solved by the ode-solver of flow-matching, and the converted speech is obtained by a pre-trained vocoder.

[0051] The speech processing method provided in this application is applied to a trained speech processing model. The attention mechanism vector extraction module in the speech processing model extracts the target speech feature vector of the speech data to be processed. The quantization module in the speech processing model performs vectorization processing on the target speech feature vector. The processing result of the speech data to be processed based on the quantized speech feature vector is the subsequent processing module in the speech processing model. The subsequent processing module includes at least the attention mechanism vector operation module. The subsequent processing module may also include other modules in the existing speech processing task model. For example, when the speech processing method provided in this application is implemented by embedding the quantization module into the existing speech processing task model, the subsequent processing module includes the subsequent module of the attention mechanism vector extraction module in the existing speech processing task model.

[0052] The training process of the quantization model of the speech processing method provided in this application is described below. This quantization module, as part of the speech processing model, is trained together with other modules in the speech processing model. The training process includes: Input the sample speech data to be processed into the speech processing model, and obtain the sample speech processing results output by the speech processing model; The training loss is obtained based on the sample speech processing results, and the speech processing model is updated based on the training loss.

[0053] The specific implementation of the training process for a speech processing model with embedded quantization can be found in the training methods of existing speech processing task models, and will not be elaborated here.

[0054] As explained above, speech data has features at a global scale. In one possible implementation of the speech processing method provided in this application, the target speech feature vector is quantized to obtain a quantized speech feature vector, including: Replace the target speech feature vector with a pre-defined, specified quantized speech feature vector; The specified quantized speech feature vector is obtained by training multiple times based on multiple samples of speech data to be processed. Each training process includes: The sample speech data to be processed is input into the feature extraction module in the speech processing model to obtain the sample target speech feature vector of the sample speech data to be processed; The sample target speech feature vector is quantized to obtain the sample quantized speech feature vector; The sample quantized speech feature vector is input into the subsequent processing module in the speech processing model to obtain the sample speech processing result corresponding to the sample speech data to be processed. The speech processing model is trained based on the results of sample speech processing; The quantized speech feature vectors of the samples during the training process are statistically analyzed, and the specified quantized speech feature vectors are determined based on the statistical results.

[0055] In this implementation, the feature processing module is an attention mechanism vector extraction module, and there can be one or more specified quantized speech feature vectors. Determining the specified quantized speech feature vector based on statistical results means using the quantized speech feature vector of the sample with the highest hit rate during training as the specified quantized speech feature vector. Since quantization maps the target speech feature vector to a finite space, in this implementation, one or more sample quantized speech feature vectors with the highest hit rate are selected and fixed through training. Taking the target speech feature vector as a key-value pair as an example, the key-value pair or multiple key-value pairs with the highest hit rate during training are fixed as fixed global features of the speech data. This implementation can effectively reduce the computational load during inference (no specific quantization calculation of the target speech feature vector is needed during inference) and reduce the latency of speech processing tasks.

[0056] Furthermore, in another possible implementation, quantizing the target speech feature vector to obtain the quantized speech feature vector may also include: The target speech feature vector is input into the codec to obtain the reconstructed speech feature vector output by the codec; The reconstructed speech feature vector is quantized to obtain an intermediate quantized speech feature vector; Obtain the specified quantized speech feature vector, and concatenate the intermediate quantized speech feature vector with the specified quantized speech feature vector to obtain the quantized speech feature vector.

[0057] In this implementation, one or more samples with high hit rates during training are quantized speech feature vectors as designated quantized speech feature vectors. The feature vectors obtained by quantizing the data to be processed in the inference stage are used as intermediate quantized speech feature vectors. The intermediate quantized speech feature vectors are concatenated with the designated quantized speech feature vectors to obtain the quantized speech feature vectors. This allows the global features learned during training to provide guidance in the inference stage, improving the ability of the quantized speech feature vectors to represent global information features.

[0058] The speech processing apparatus provided in this application is described below. The speech processing apparatus described below can be referred to in correspondence with the speech processing method described above. For example... Figure 7 As shown, the speech processing apparatus provided in this application includes: Feature extraction module 710 is used to acquire speech data to be processed and extract the target speech feature vector of the speech data to be processed. The quantization module 720 is used to quantize the target speech feature vector to obtain the quantized speech feature vector; The processing result output module 730 is used to obtain the processing result of the speech data to be processed based on the quantized speech feature vector.

[0059] The speech processing apparatus provided in this application extracts features from the speech data to be processed to obtain a target speech feature vector, and then quantizes the target speech feature vector to obtain a quantized speech feature vector. The quantization process maps the feature vector to a finite space. Therefore, the quantized speech feature vector is not a vector in an infinite continuous space, but a feature vector in a finite dictionary. In this way, the speech processing result of the speech data to be processed is determined based on the quantized speech feature vector mapped to the finite space. The global essential features of the speech signal can be learned, thereby improving the quality of the speech data processing result.

[0060] In one possible implementation of the speech processing apparatus provided in this application, the target speech feature vector is an attention mechanism vector; the extraction of the target speech feature vector from the speech data to be processed includes: The speech data to be processed is input into the speech processing model, and the target speech feature vector is obtained through the attention mechanism vector extraction module in the speech processing model.

[0061] In one possible implementation of the speech processing device provided in this application, the target speech feature vector includes key-value vector pairs in the attention mechanism vector.

[0062] In one possible implementation of the speech processing apparatus provided in this application, the step of obtaining the processing result of the speech data to be processed based on the quantized speech feature vector includes: Based on the target speech feature vector and the attention mechanism, attention features are obtained; Based on the attention features, the processing result of the speech data to be processed is obtained.

[0063] In one possible implementation of the speech processing apparatus provided in this application, the step of quantizing the target speech feature vector to obtain a quantized speech feature vector includes: The target speech feature vector is input into the codec to obtain the reconstructed speech feature vector output by the codec; The reconstructed speech feature vector is quantized to obtain the quantized speech feature vector.

[0064] In one possible implementation of the speech processing apparatus provided in this application, the step of quantizing the target speech feature vector to obtain a quantized speech feature vector includes: Replace the target speech feature vector with a preset, specified quantized speech feature vector; The specified quantized speech feature vector is obtained by training multiple times based on multiple samples of speech data to be processed. Each training process includes: The sample speech data to be processed is input into the feature extraction module in the speech processing model to obtain the sample target speech feature vector of the sample speech data to be processed; The sample target speech feature vector is quantized to obtain the sample quantized speech feature vector; The quantized speech feature vector of the sample is input into the subsequent processing module in the speech processing model to obtain the sample speech processing result corresponding to the sample speech data to be processed. The speech processing model is trained based on the sample speech processing results; The quantized speech feature vectors of the samples during the training process are statistically analyzed, and the specified quantized speech feature vectors are determined based on the statistical results.

[0065] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840. The processor 810, communications interface 820, and memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a speech processing method. This speech processing method includes: acquiring speech data to be processed; extracting a target speech feature vector from the speech data to be processed; quantizing the target speech feature vector to obtain a quantized speech feature vector; and obtaining the processing result of the speech data to be processed based on the quantized speech feature vector.

[0066] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0067] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the speech processing methods provided by the above methods. The speech processing method includes: acquiring speech data to be processed; extracting a target speech feature vector from the speech data to be processed; quantizing the target speech feature vector to obtain a quantized speech feature vector; and obtaining a processing result of the speech data to be processed based on the quantized speech feature vector.

[0068] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the speech processing methods provided by the above methods. The speech processing method includes: acquiring speech data to be processed; extracting a target speech feature vector from the speech data to be processed; quantizing the target speech feature vector to obtain a quantized speech feature vector; and obtaining a processing result of the speech data to be processed based on the quantized speech feature vector.

[0069] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0070] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A speech processing method, characterized in that, include: Acquire the speech data to be processed, and extract the target speech feature vector from the speech data to be processed; The target speech feature vector is quantized to obtain a quantized speech feature vector; The processing result of the speech data to be processed is obtained based on the quantized speech feature vector.

2. The speech processing method according to claim 1, characterized in that, The target speech feature vector is an attention mechanism vector; the extraction of the target speech feature vector from the speech data to be processed includes: The speech data to be processed is input into the speech processing model, and the target speech feature vector is obtained through the attention mechanism vector extraction module in the speech processing model.

3. The speech processing method according to claim 2, characterized in that, The target speech feature vector includes key-value vector pairs from the attention mechanism vector.

4. The speech processing method according to claim 2, characterized in that, The processing result of obtaining the speech data to be processed based on the quantized speech feature vector includes: Based on the target speech feature vector and the attention mechanism, attention features are obtained; Based on the attention features, the processing result of the speech data to be processed is obtained.

5. The speech processing method according to claim 1, characterized in that, The step of quantizing the target speech feature vector to obtain a quantized speech feature vector includes: The target speech feature vector is input into the codec to obtain the reconstructed speech feature vector output by the codec; The reconstructed speech feature vector is quantized to obtain the quantized speech feature vector.

6. The speech processing method according to claim 1, characterized in that, The step of quantizing the target speech feature vector to obtain a quantized speech feature vector includes: Replace the target speech feature vector with a preset, specified quantized speech feature vector; The specified quantized speech feature vector is obtained by training multiple times based on multiple samples of speech data to be processed. Each training process includes: The sample speech data to be processed is input into the feature extraction module in the speech processing model to obtain the sample target speech feature vector of the sample speech data to be processed; The sample target speech feature vector is quantized to obtain the sample quantized speech feature vector; The quantized speech feature vector of the sample is input into the subsequent processing module in the speech processing model to obtain the sample speech processing result corresponding to the sample speech data to be processed. The speech processing model is trained based on the sample speech processing results; The quantized speech feature vectors of the samples during the training process are statistically analyzed, and the specified quantized speech feature vectors are determined based on the statistical results.

7. A voice processing device, characterized in that, include: The feature extraction module is used to acquire the speech data to be processed and extract the target speech feature vector of the speech data to be processed. The quantization module is used to quantize the target speech feature vector to obtain a quantized speech feature vector; The processing result output module is used to obtain the processing result of the speech data to be processed based on the quantized speech feature vector.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the speech processing method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the speech processing method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the speech processing method as described in any one of claims 1 to 6.

Images