Method and apparatus for voice enhancement, device, storage medium, program product

Abstract

The application relates to the technical field of audio processing, and discloses a speech enhancement method, which is applied to an electronic device. When the electronic device performs speech enhancement on a noisy speech signal input by a user by using a deployed speech enhancement model, an acoustic feature vector and a semantic feature vector are extracted, and context prompt information is constructed according to speech tokens output by the speech enhancement model at historical times. During the execution of a speech enhancement task by the speech enhancement model, the noisy speech signal can be enhanced by referring to the semantic feature vector under the guidance of the context prompt information, instead of only relying on the acoustic feature vector extracted based on the noisy speech signal to perform speech enhancement, so that the situation that weak speech in a low signal-to-noise ratio speech signal is misjudged as noise is reduced, the accuracy of speech enhancement is improved, and the effect of speech enhancement is improved. The application also discloses a speech enhancement device and equipment, a storage medium and a program product.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to a method, apparatus, device, storage medium, and program product for voice enhancement. Background Technology

[0002] Speech enhancement is a technique that extracts the purest possible speech from noisy speech signals after they have been interfered with or even submerged by noise, thereby suppressing or reducing noise interference. Currently, it is mainly based on artificial intelligence models to enhance noisy speech signals.

[0003] In related technologies, end-to-end models can be used to enhance noisy speech signals. Specifically, the end-to-end model adopts a network architecture of "encoder-transformer-decoder". The encoder converts the noisy speech signal into a noisy token sequence, which is then input into the Transformer to predict a clean token sequence. Finally, the decoder restores the clean token sequence to a clean speech signal, thereby enhancing the noisy speech signal.

[0004] However, in related technologies, when the noisy speech signal is a low signal-to-noise ratio (SNR) speech signal, the end-to-end model is used to enhance the low SNR speech signal. In such cases, weak speech in the low SNR speech signal is easily misjudged as noise, thereby reducing the accuracy of speech enhancement and thus reducing the effect of speech enhancement. Summary of the Invention

[0005] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0006] This application provides a method, apparatus, device, storage medium, and program product for speech enhancement, which can improve the accuracy of speech enhancement of noisy speech signals, thereby improving the effect of speech enhancement.

[0007] In a first aspect, embodiments of this application provide a speech enhancement method applied to an electronic device, the electronic device having a speech enhancement model deployed thereon; the method includes:

[0008] Based on the noisy speech signal input by the user, acoustic feature vectors and semantic feature vectors are extracted;

[0009] Decode the speech tokens output by the speech enhancement model in the historical time period into speech feature vectors;

[0010] The acoustic feature vector is concatenated with the speech feature vector to obtain the noisy speech feature vector;

[0011] The noisy speech feature vector is quantized frame by frame into discrete tokens to obtain the target token sequence;

[0012] The target token sequence and semantic feature vector are input into the speech enhancement model, so that the speech enhancement model performs a cross-attention operation based on the target token sequence and semantic feature vector, and outputs the target speech token according to the result of the cross-attention operation;

[0013] Restore the target voice token to the target voice signal.

[0014] Optionally, the noisy speech feature vector is quantized frame by frame into discrete tokens to obtain the target token sequence, including: performing RVQ quantization on the noisy speech feature vector frame by frame to obtain the index sequence of each audio frame; adjusting the index sequence of each audio frame so that each index in the index sequence is delayed by D1 frames to obtain the target token sequence; where D1 = (N-1) × C + S1 - N, N is the number of audio frames corresponding to the index sequence, C is the number of codebooks used when performing RVQ quantization, and S1 is the position of the index in the index sequence.

[0015] Optionally, the noisy speech feature vector is quantized frame by frame into discrete tokens to obtain the target token sequence, including: performing RVQ quantization on the noisy speech feature vector frame by frame to obtain the index sequence of each audio frame; adjusting the index sequence of each audio frame so that each index in the index sequence is delayed by D2 frames to obtain the target token sequence; where D2=S2-1, and S2 is the position of the index in the index sequence.

[0016] Optionally, based on the noisy speech signal input by the user, an acoustic feature vector is extracted, including: preprocessing the noisy speech signal; performing a short-time Fourier transform on the preprocessed noisy speech signal to obtain the acoustic feature vector of each audio frame in the preprocessed noisy speech signal.

[0017] Optionally, after extracting the acoustic feature vector, the method further includes: converting the dimension of the acoustic feature vector into a target dimension; wherein the target dimension is the input dimension of the speech enhancement model.

[0018] Optionally, after extracting the semantic feature vector, the method further includes: converting the dimension of the semantic feature vector into the target dimension; wherein the target dimension is the input dimension of the speech enhancement model.

[0019] Secondly, embodiments of this application provide a speech enhancement device integrated into an electronic device, the electronic device having a speech enhancement model deployed thereon; the device includes:

[0020] The extraction module is configured to extract acoustic feature vectors and semantic feature vectors based on noisy speech signals input by the user;

[0021] The decoding module is configured to decode the speech tokens output by the speech enhancement model in the historical time into speech feature vectors.

[0022] The integration module is configured to concatenate the acoustic feature vector with the speech feature vector to obtain a noisy speech feature vector;

[0023] The quantization module is configured to quantize the noisy speech feature vector frame by frame into discrete tokens to obtain the target token sequence.

[0024] The enhancement module is configured to input the target token sequence and semantic feature vector into the speech enhancement model, so that the speech enhancement model performs a cross-attention operation based on the target token sequence and semantic feature vector, and outputs the target speech token based on the result of the cross-attention operation;

[0025] The restoration module is configured to restore the target voice token to the target voice signal.

[0026] Thirdly, embodiments of this application provide a voice enhancement device, including a processor and a memory storing program instructions, wherein the processor is configured to execute the voice enhancement method of the first aspect when running the program instructions.

[0027] Fourthly, embodiments of this application provide a storage medium storing program instructions, wherein the program instructions, when executed, perform the speech enhancement method of the first aspect.

[0028] Fifthly, embodiments of this application provide a computer program product, including a computer program, which, when executed by a processor, implements the speech enhancement method of the first aspect.

[0029] The speech enhancement methods, apparatus, devices, storage media, and program products provided in this application can achieve the following technical effects:

[0030] When an electronic device enhances noisy speech signals input by a user based on a deployed speech enhancement model, acoustic and semantic feature vectors are extracted from the noisy speech signal. Speech tokens output from the historical time period of the speech enhancement model can serve as "contextual cues" to guide the model to perform the speech enhancement task more accurately. Specifically, the speech tokens output from the historical time period of the speech enhancement model are decoded into speech feature vectors, and the noisy speech feature vector obtained by concatenating the speech feature vector with the acoustic feature vector is used as input to the speech enhancement model. Before being input into the speech enhancement model, the noisy speech feature vector needs to be quantized frame by frame into discrete tokens to obtain a target token sequence. The target token sequence and semantic feature vector are input into the speech enhancement model, enabling the speech augmentation model to perform a cross-attention operation based on the target token sequence and semantic feature vector, and output the target speech token based on the result of the cross-attention operation. The target speech token is the clean token sequence, which can be restored to a clean target speech signal, thereby achieving speech enhancement of the noisy speech signal.

[0031] In this embodiment, when an electronic device enhances a noisy speech signal input by a user based on a deployed speech enhancement model, the speech enhancement model can, during the execution of the speech enhancement task, refer to the semantic information of the noisy speech signal under the guidance of contextual prompts to enhance the speech signal, rather than relying solely on the acoustic feature vector extracted from the noisy speech signal. In this way, even if the noisy speech signal is a low signal-to-noise ratio (SNR) speech signal, weak speech within the low SNR speech signal can be more accurately identified, reducing the possibility of misclassifying weak speech within the low SNR speech signal as noise, thereby improving the accuracy of speech enhancement and ultimately enhancing its effectiveness.

[0032] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0033] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are considered similar elements. The drawings do not constitute a limitation of scale, and wherein:

[0034] Figure 1 This is a schematic diagram of an electronic device provided in an embodiment of this application;

[0035] Figure 2 This is a flowchart of a speech enhancement method provided in an embodiment of this application;

[0036] Figure 3 This is a schematic diagram of a token sequence provided in an embodiment of this application;

[0037] Figure 4 This is a schematic diagram of another token sequence provided in an embodiment of this application;

[0038] Figure 5 This is a schematic diagram of yet another token sequence provided in an embodiment of this application;

[0039] Figure 6 This is a schematic diagram of a speech enhancement device provided in an embodiment of this application;

[0040] Figure 7 This is a schematic diagram of a voice enhancement device provided in an embodiment of this application. Detailed Implementation

[0041] The terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0042] Unless otherwise stated, the term "multiple" means two or more.

[0043] In this embodiment, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0044] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0045] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0046] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0047] Discriminative models are artificial intelligence models built upon the relationship between unknown and known data. In speech enhancement, discriminative models learn the mapping relationship between noisy and clean speech, thereby acquiring the ability to enhance speech.

[0048] Generative models refer to large-scale artificial intelligence models with a huge number of parameters, based on deep neural network architectures such as Transformer, and learning global statistical laws through massive multimodal data, thereby possessing the ability to generate open content. In speech enhancement, large generative models learn the distribution and latent features of clean speech, thereby possessing the ability to reconstruct or generate high-fidelity, highly intelligible speech based on noisy speech.

[0049] Speech fidelity refers to the degree of similarity between the enhanced speech signal and the original clean speech, and is used to measure the degree of speech reproduction.

[0050] Speech intelligibility: also known as speech clarity, refers to the degree to which a listener can correctly understand the content of speech, and also to the degree to which a speech signal can be correctly recognized after transmission or processing.

[0051] End-to-end models are artificial intelligence models that directly learn the mapping relationship from the raw input to the desired output. They do not rely on manually designed intermediate steps or processes. In speech enhancement, an end-to-end model takes a noisy speech signal as input and can directly output a clean speech signal, thus eliminating the need for manually designed signal processing steps.

[0052] Vector Quantization (VQ) is a digital compression technique used to map high-dimensional continuous vectors to a finite discrete codebook. Here, a vector refers to a data unit to be processed, typically a multi-dimensional feature vector. Quantization refers to discretizing the vector into multiple tokens. The codebook is a pre-trained discrete set containing a finite number of codewords. Each codeword represents a standard vector of the same dimension as the original vector, and each standard vector has a unique ID or index number. The set formed by the index numbers and the standard vectors is called the codebook. A token is the index number of the discretized vector, indicating the position identifier of the codeword in the codebook that is closest to the vector. For example, the vector X quantization process is as follows: calculate the distance between vector X and all codewords in the codebook, and select the codeword C that is closest to vector X. i As the quantization result of vector X, codeword C i The index i in the vector X is the token after discretization.

[0053] Residual Vector Quantization (RVQ) is a process that uses a codebook with multiple levels to progressively approximate a vector. For example, it uses a codebook with K levels {C1, C2, ..., C...} KIn the residual vector quantization process of vector X, the quantization process of vector X is divided into K levels of quantization. The first level uses codebook C1 to quantize vector X, obtaining approximate codeword X1 and residual r1 = X - X1. The second level uses codebook C2 to quantize residual r1, obtaining approximate codeword X2 and residual r2 = r1 - X2, and so on. The Kth level uses codebook C... K Quantize the upper layer residual r K-1 The approximate codeword X is obtained. K and residual r K =r K-1 -X K Finally, the K-layer approximate codewords {X1, X2, ..., X} are obtained. k The K index numbers of the vector X are arranged in the hierarchical order of the K-level codebook to form the index sequence of the vector X. For speech signals, the index sequence of each audio frame in the speech feature vector is arranged in frame order to form a token sequence.

[0054] Cross-attention operation: is a mechanism for information exchange between different modalities or sequences. It can establish attention relationships between different modalities (such as semantic features and acoustic features), so that semantic features can guide the attention of acoustic features.

[0055] The dimension of a feature vector refers to the number of features that make up the feature vector, that is, the number of numerical elements contained in the feature vector, or the number of features describing an object (such as a piece of audio). For example, if the dimension of an audio frame feature vector is 256, it means that 256 different metrics are used to describe the audio frame, and the audio frame feature vector is actually a sequence with 256 features. A feature refers to a representative quantitative attribute extracted from the raw data. For example, in speech, a feature could be the energy, spectral center, or zero-crossing rate of an audio frame in a certain frequency band.

[0056] To provide a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this application. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0057] Currently, the enhancement of noisy speech signals is mainly achieved based on artificial intelligence models.

[0058] In related technologies, discriminative and generative models can be used to enhance noisy speech signals. Specifically, a discriminative model is used to perform coarse noise reduction on the noisy speech signal to distinguish between speech and noise components, while a generative model is used to reconstruct or restore the details of the speech components to obtain a clean speech signal.

[0059] However, in the process of enhancing noisy speech signals using discriminative and generative models, discriminative models may introduce errors when performing coarse noise reduction. They can easily misclassify speech components as noise components, and suppressing these misclassified noise components will cause speech distortion. Thus, even if generative models are used to restore or reconstruct the distorted speech components in detail, the lost speech components cannot be recovered or reconstructed, thereby affecting the speech enhancement effect.

[0060] In response to this, related technologies can utilize end-to-end models to enhance noisy speech signals. Specifically, the end-to-end model employs a network architecture of "encoder-transformer-decoder." The encoder converts the noisy speech signal into a noisy token sequence, which is then input into the Transformer to predict a clean token sequence. Finally, the decoder restores the clean token sequence to a clean speech signal, thereby enhancing the noisy speech signal.

[0061] Although using an end-to-end model to enhance noisy speech signals eliminates the need for a discriminative model to perform coarse noise reduction, thus reducing the likelihood of speech components being mistakenly identified as noise and improving speech enhancement, it can also lead to a decrease in the accuracy of speech enhancement when the noisy speech signal has a low signal-to-noise ratio (SNR). This is because the speech components in low SNR signals have weak energy, making it easy to misclassify weak speech as noise when using an end-to-end model for enhancement, thereby reducing the accuracy of speech enhancement.

[0062] Therefore, embodiments of this application provide methods, apparatuses, devices, storage media, and program products for speech enhancement. When an electronic device enhances a noisy speech signal input by a user based on a deployed speech enhancement model, it can extract acoustic feature vectors and semantic feature vectors, and construct contextual prompts based on speech tokens output by the speech enhancement model over historical time periods. During the speech enhancement task performed by the speech enhancement model, speech enhancement can be performed on the noisy speech signal with reference to the semantic information of the noisy speech signal under the guidance of the contextual prompts, rather than relying solely on the acoustic feature vectors extracted from the noisy speech signal. In this way, even if the noisy speech signal is a low signal-to-noise ratio (SNR) speech signal, weak speech in the low SNR speech signal can be more accurately identified, reducing the possibility of misjudging weak speech in the low SNR speech signal as noise, thereby improving the accuracy of speech enhancement and ultimately enhancing the effect of speech enhancement.

[0063] Combination Figure 1 As shown in the illustration, this application provides an electronic device to which the speech enhancement method is applicable. This electronic device has speech signal processing capabilities and can specifically utilize a deployed end-to-end network model to enhance noisy speech signals. For example, the electronic device can be a wearable device, smartphone, tablet computer, laptop computer, desktop computer, in-vehicle terminal, interactive smart board, smart blackboard, or other terminal device, or a cloud server, edge server, or other server.

[0064] The end-to-end network model architecture includes a semantic encoder, a speech enhancement model, and a decoder. To assist the end-to-end network model in enhancing noisy speech signals, the electronic device also deploys a feature extraction unit to extract acoustic feature vectors. The semantic encoder extracts semantic feature vectors based on the noisy speech signal input to the electronic device; the semantic encoder can be a linear layer within the end-to-end network model architecture. The speech enhancement model predicts clean target speech tokens based on the acoustic and semantic feature vectors; the speech enhancement model is a generative model that supports delayed inference prediction. The decoder is used to restore the clean target speech tokens to clean target speech signals.

[0065] It should be noted that the speech enhancement method provided in this application is applicable to various application scenarios. For example, in a remote conferencing scenario, enhancing the conference speech improves its fidelity, ensuring that participants can better understand the meeting content and thus enhancing the meeting's effectiveness. Similarly, in a smart teaching scenario, enhancing the teacher's speech improves its intelligibility, making it easier for students to understand the content and thus improving teaching effectiveness. It is understood that the above application scenarios are merely illustrative, and the speech enhancement method provided in this application can also be applied to other scenarios requiring speech enhancement; this application does not specifically limit its application in these cases.

[0066] The following describes the method embodiments provided in this application.

[0067] Combination Figure 2 As shown, this application provides a method for voice enhancement, which is applied to the electronic device described above. The method includes the following steps:

[0068] S21, based on the noisy speech signal input by the user, extract acoustic feature vectors and semantic feature vectors.

[0069] In step S21, the noisy speech signal refers to a mixed signal that is clean speech with noise superimposed on it. The noisy speech signal can be pre-uploaded to the electronic device by the user through other devices that have established a network connection with the electronic device, so that the electronic device can perform a noisy speech signal enhancement task. The noisy speech signal can also be collected in real time by an integrated voice acquisition device when the user uses the voice functions of the electronic device (such as call and conference functions). After enhancing the noisy speech signal collected in real time, the effectiveness of voice processing-related functions can be improved, for example, improving the call quality of voice calls.

[0070] In step S21, both the acoustic feature vector and the semantic feature vector are formed from frame-level feature vectors. Specifically, the acoustic feature vector refers to the acoustic feature vector of each audio frame in the noisy speech signal, and the semantic feature vector refers to the semantic feature vector of each audio frame in the noisy speech signal. The acoustic and semantic feature vectors primarily reflect the timbre, emotion, and semantics of the noisy speech signal. This allows the subsequent speech enhancement model to perform speech enhancement based on the acoustic and semantic feature vectors, using more comprehensive information such as the timbre, emotion, and semantics of the noisy speech signal, thereby improving the accuracy of speech enhancement.

[0071] S22 decodes the speech tokens output by the speech enhancement model in the historical time period into speech feature vectors.

[0072] In step S22, the voice token output in historical time refers to the voice token output by the voice enhancement model after enhancing the historical noisy voice signal before processing the current noisy voice signal input by the user. In other words, it's the voice token output by the voice enhancement model after historical time step inference when enhancing the noisy voice signal. Of course, the voice token output in historical time can also be understood as the voice token output by the voice enhancement model when enhancing historical audio frames of the noisy voice signal.

[0073] In step S22, the speech tokens output by the speech enhancement model in the historical time can serve as contextual cues, guiding the speech enhancement model to perform the speech enhancement task more accurately. Specifically, the speech tokens can be input into the embedding lookup layer of the end-to-end network model. The embedding lookup layer decodes the speech tokens into speech feature vectors by looking up the standard vectors corresponding to each index in the speech tokens.

[0074] S23. The acoustic feature vector and the speech feature vector are concatenated to obtain the noisy speech feature vector.

[0075] In step S23, since the speech enhancement model is usually modeled sequentially according to the temporal order of features (i.e., from left to right), when concatenating the acoustic feature vector with the speech feature vector, the speech feature vector, which serves as contextual cue information, needs to be concatenated before the acoustic feature vector. This allows the speech enhancement model to first identify the speech feature vector to obtain contextual cue information, and then identify the acoustic feature vector under the guidance of the contextual cue information.

[0076] S24, quantize the noisy speech feature vector frame by frame into discrete tokens to obtain the target token sequence.

[0077] In step S24, since the speech enhancement model is a generative model, and to adapt to the autoregressive generative architecture of the speech enhancement model, before the noisy speech features are input into the speech enhancement model, the noisy speech feature vector needs to be quantized frame by frame into discrete tokens, resulting in discrete tokens for each audio frame. The discrete tokens of each audio frame are then arranged in time to form the target token sequence. Because the target token sequence is obtained by quantizing the noisy speech feature vector frame by frame, and no enhancement operation is performed during the quantization process, the target token sequence is still a discrete representation of the noisy features.

[0078] In step S24, if a single codebook is used to quantize the noisy speech feature vector frame by frame, then each audio frame vector is quantized into a token. If multiple codebooks are used to perform residual quantization on the noisy speech feature vector frame by frame, then each audio frame vector is quantized into multiple tokens, that is, each audio frame is quantized into an index sequence. The number of indices in the index sequence is the same as the number of codebooks used in residual quantization, and the order of the indices in the index sequence is consistent with the hierarchical order of the multiple codebooks.

[0079] For example, in combination Figure 3 As shown in the diagram, this application provides a schematic diagram of a token sequence. This token sequence is obtained by residual quantization of the noisy speech feature vector frame by frame using a 4-layer codebook. The vertical axis represents the sequentially increasing codebook levels, and the horizontal axis represents the time step of the token sequence. The time step is synchronized with the audio frame, meaning each time step represents one audio frame. B and E are the start and end markers of the token sequence, respectively, and F... nk This represents the index value obtained after quantizing the nth audio frame using the k-th level codebook. The index values ​​in the index sequence of each audio frame are vertically distributed according to the codebook hierarchy and correspond to the same time step or the same audio frame. For example, F in the index sequence of the 1st audio frame... 11 F 12 F 13 F 14 This corresponds to the first time step or the first audio frame.

[0080] S25, the target token sequence and semantic feature vector are input into the speech enhancement model so that the speech enhancement model performs a cross-attention operation based on the target token sequence and semantic feature vector, and outputs the target speech token according to the result of the cross-attention operation.

[0081] In step S25, the target speech token refers to the clean token sequence corresponding to the noisy speech signal. At each time step in the speech enhancement model, when inferring and generating the target speech token, a cross-attention operation is performed between the index sequence in the target token sequence corresponding to that time step and the semantic feature vector. Based on the result of the cross-attention operation, the token for that time step is output. This allows the speech enhancement model to perform inference and prediction with reference to semantic information at each time step, thereby improving the accuracy of speech enhancement.

[0082] S26, restore the target voice token to the target voice signal.

[0083] In step S26, inverse quantization is performed on the target speech token to obtain the target speech feature vector, and the target speech signal is obtained by restoring the target speech feature vector. The process of inverse quantization of the target speech token is similar to the process in step S22 above, where the speech token output from the speech enhancement model's historical time is decoded into a speech feature vector; please refer to the implementation method in step S22 above for details. For token sequences quantized using multiple codebooks, during inverse quantization, the vectors corresponding to the same audio frame are accumulated according to the index value to obtain continuous vectors for each audio frame. The vectors of each audio frame are then arranged in time sequence to obtain the target speech feature vector. The target speech feature vector is then restored to a time-domain waveform using a decoder to obtain the target speech signal.

[0084] The speech enhancement method provided in this application allows the electronic device to enhance noisy speech signals input by the user based on a deployed speech enhancement model. During the speech enhancement task, the model can refer to the semantic information of the noisy speech signal under the guidance of contextual prompts, and perform speech enhancement on the acoustic feature vector extracted from the noisy speech signal, rather than relying solely on the acoustic feature vector extracted from the noisy speech signal. This way, even if the noisy speech signal has a low signal-to-noise ratio (SNR), weak speech within the low SNR speech signal can be more accurately identified, reducing the likelihood of misclassifying weak speech as noise, thereby improving the accuracy and effectiveness of speech enhancement.

[0085] The following describes optional implementation methods of the speech enhancement method provided in the embodiments of this application.

[0086] Optionally, in step S21 above, extracting acoustic feature vectors based on noisy speech signals includes: preprocessing the noisy speech signal and performing a short-time Fourier transform on the preprocessed noisy speech signal to obtain the acoustic feature vectors of each audio frame in the preprocessed noisy speech signal.

[0087] In this embodiment, the preprocessing of the noisy speech signal includes framing and windowing. Framing divides the noisy speech signal into multiple audio segments with stable noise, while windowing improves the local signal-to-noise ratio of the audio segments. Thus, performing a short-time Fourier transform on the preprocessed noisy speech signal improves the quality of the acquired acoustic feature vectors. This allows the subsequent speech enhancement model to provide a clean speech representation based on the high-quality acoustic feature vectors, thereby improving the clarity and naturalness of the denoised speech and ultimately enhancing the speech enhancement effect.

[0088] Optionally, in step S21 above, after extracting the acoustic feature vector based on the noisy speech signal, the method further includes: converting the dimension of the acoustic feature vector into a target dimension. The target dimension is the input dimension of the speech enhancement model.

[0089] In this implementation, since the dimension of the acoustic feature vector differs from the input dimension of the speech enhancement model, the speech enhancement model may be unable to process the acoustic feature vector correctly. Therefore, after extracting the acoustic feature vector, its dimension can be transformed to match the input dimension of the speech enhancement model. Specifically, when the dimension of the acoustic feature vector is lower than the input dimension of the speech enhancement model, its dimension is increased. When the dimension of the acoustic feature vector is higher than the input dimension of the speech enhancement model, its dimension is decreased.

[0090] By adopting this implementation method, the dimension of the acoustic feature vector can be transformed to meet the input dimension condition of the speech enhancement model. This ensures that the acoustic feature vector can be processed normally by the speech enhancement model even when the dimension of the acoustic feature vector is inconsistent with the input dimension of the speech enhancement model, thereby ensuring the normal execution of the speech enhancement task and avoiding abnormalities in the speech enhancement task.

[0091] Optionally, in step S21 above, after extracting the semantic feature vector based on the noisy speech signal, the method further includes: converting the dimension of the semantic feature vector into the target dimension. The target dimension is the input dimension of the speech enhancement model.

[0092] In this implementation, since the semantic feature vector's dimension differs from the input dimension of the speech enhancement model, the speech enhancement model may be unable to process the semantic feature vector correctly. Therefore, after extracting the semantic feature vector, its dimension can be transformed to match the input dimension of the speech enhancement model. Specifically, when the semantic feature vector's dimension is lower than the input dimension of the speech enhancement model, its dimension is increased. When the semantic feature vector's dimension is higher than the input dimension of the speech enhancement model, its dimension is decreased.

[0093] By adopting this implementation method, the dimension of the semantic feature vector can be transformed to meet the input dimension condition of the speech enhancement model. This ensures that the semantic feature vector can be processed normally by the speech enhancement model even when the dimension of the semantic feature vector is inconsistent with the input dimension of the speech enhancement model, thereby ensuring the normal execution of the speech enhancement task and avoiding abnormalities in the speech enhancement task.

[0094] In step S24 of this application embodiment, since the residual quantization of the noisy speech feature vector frame by frame is performed using multiple codebooks, the index values ​​in the index sequence of each audio frame in the quantized target token sequence will correspond to the same time step or the same audio frame. Therefore, when the speech enhancement model processes the target token sequence according to the time step and predicts the enhancement result of the current audio frame, it can only refer to the information of historical audio frames. The index with a later position in the same index sequence cannot refer to the index with a earlier position for prediction. For example, combined with Figure 3 As shown, for the index sequence of the second audio frame, F 21 F 22 F 23 F 24 When the speech enhancement model performs inference at the second time step, it can only refer to the F in the index sequence of the first time step. 11 F 12 F 13 F 14 Due to F in the index sequence 21 F 22 F 23 F 24 Corresponding to the same time step, during the inference process of the second time step, the speech enhancement model adjusts the F... 23 F cannot be referenced when making predictions. 21 and F 22 It is evident that speech enhancement models are effective for... Figure 3 When predicting the target token sequence shown, only different sequences between frames can be referenced; reference between different indices within the same sequence within a frame is not possible. Therefore, this application provides the following optional implementation methods:

[0095] Optionally, in step S24 above, quantizing the noisy speech feature vector frame by frame into discrete tokens to obtain the target token sequence includes: performing RVQ quantization on the noisy speech feature vector frame by frame to obtain the index sequence of each audio frame, adjusting the index sequence of each audio frame respectively, so that each index in the index sequence is delayed by D1 frames to obtain the target token sequence.

[0096] Where D1 = (N-1) × C + S1 - N, N is the number of audio frames corresponding to the index sequence, C is the number of codebooks used when performing RVQ quantization, and S1 is the index position in the index sequence.

[0097] In this embodiment, after obtaining the index sequence of each audio frame through quantization, the index sequence of each audio frame can be adjusted sequentially to completely "stagger" the index sequences corresponding to adjacent audio frames in time steps, so that multiple indices in the index sequence of each audio frame correspond to multiple consecutive time steps.

[0098] For example, in combination Figure 3 and Figure 4 As shown, the index sequence of each audio frame obtained by quantization can be as follows: Figure 3 As shown, the adjustment is carried out according to this embodiment. Figure 3 After obtaining the index sequence of each audio frame, we can get the following: Figure 4 The token sequence shown. For example, F in the index sequence of the first audio frame. 11 F 12 F 13 F 14 F can be adjusted separately. 11 Delay 0 frames, F 12 Delay 1 frame, F 13 Delay 2 frames, F 14 Delay 3 frames. For F in the index sequence of the 2nd audio frame. 21 F 22 F 23 F 24 F can be adjusted separately. 21 Delay 3 frames, F 22 Delay 4 frames, F 23 Delay 5 frames, F 24 Delayed by 6 frames.

[0099] By adopting this implementation method, by adjusting the index sequence of each audio frame, multiple indices in the index sequence of the audio frame can correspond to multiple consecutive time steps, instead of multiple indices in the index sequence corresponding to a single time step. In this way, when the speech enhancement model performs the speech enhancement task based on the index sequence of the audio frames, it can not only refer to the information of historical audio frames, but also refer to the indexes in the same index sequence when predicting the later indexes, thereby improving the prediction accuracy of the speech enhancement model, thus improving the accuracy of speech enhancement and ultimately improving the effect of speech enhancement.

[0100] It should be noted that, in the above implementation method, although adjusting the index sequence of each audio frame can improve the prediction accuracy of the speech enhancement model, it is not feasible when combined with... Figure 4 As shown, the adjusted target token sequence increases the number of time steps, which increases the inference time when the speech enhancement model performs inference based on the target token. Therefore, this application provides another implementation method for constructing the target token sequence, as follows:

[0101] Furthermore, in step S24 above, the noisy speech feature vector is quantized frame by frame into discrete tokens to construct a target token sequence, including: performing RVQ quantization on the noisy speech feature vector frame by frame to obtain the index sequence of each audio frame, adjusting the index sequence of each audio frame respectively, so that each index in the index sequence is delayed by D2 frames to obtain the target token sequence.

[0102] Where D2 = S2-1, and S2 is the index position in the index sequence.

[0103] In this embodiment, after obtaining the index sequence of each audio frame through quantization, the index sequence of each audio frame can be adjusted sequentially to completely "stagger" the index sequences corresponding to each audio frame in terms of time steps, and there is partial "overlap" between the index sequences of adjacent audio frames in terms of time steps, so that multiple indices in the index sequence of each audio frame correspond to multiple consecutive time steps respectively.

[0104] For example, in combination Figure 3 and Figure 5 As shown, the index sequence of each audio frame obtained by quantization can be as follows: Figure 5 As shown, the adjustment is carried out according to this embodiment. Figure 3 After obtaining the index sequence of each audio frame, we can get the following: Figure 5 The token sequence shown. For example, F in the index sequence of the first audio frame. 11 F 12 F 13 F 14 F can be adjusted separately. 11 Delay 0 frames, F 12 Delay 1 frame, F 13 Delay 2 frames, F 14 Delay 3 frames. For F in the index sequence of the 2nd audio frame. 21 F 22 F 23 F 24 F can be adjusted separately. 21 Delay 0 frames, F 22 Delay 1 frame, F 23 Delay 2 frames, F 24 Delayed by 3 frames.

[0105] By adopting this implementation method, by adjusting the index sequence of each audio frame, multiple indices in the index sequence of the audio frame can correspond to multiple consecutive time steps, instead of multiple indices corresponding to one time step. Furthermore, in the adjusted target token sequence, the index sequences of adjacent audio frames are not completely "staggered" but partially overlap. In this way, when the speech enhancement model performs speech enhancement tasks based on the index sequence of audio frames, it can not only refer to the information of historical audio frames, but also refer to the indexes that are earlier in the same index sequence when predicting the later indexes, thereby improving the prediction accuracy of the speech enhancement model and thus improving the accuracy of speech enhancement, thereby enhancing the effect of speech enhancement.

[0106] Furthermore, since the index sequences of adjacent audio frames in the adjusted target token sequence are not completely "staggered" but partially overlap, the adjusted target token sequence does not occupy too many time steps. This improves the prediction accuracy of the speech enhancement model and reduces the inference time steps when the speech enhancement model performs inference based on the target token, thereby reducing the inference duration and improving the efficiency of speech enhancement. Figures 3 to 5 For example, for a noisy speech feature vector containing N audio frames, after performing RVQ quantization frame by frame using a 4-layer codebook, the result is as follows: Figure 3 The target token sequence is shown. For adjustment... Figure 3 After the index sequences in the target token sequence shown, the result is as follows: Figure 4 For the target token sequence shown, the time step increases from N steps to N×4 steps. And for adjustment... Figure 3 After the index sequences in the target token sequence shown, the result is as follows: Figure 5 For the target token sequence shown, the time step increases from N steps to (N+4-1) steps. It can be seen that... Figure 5 The target token sequence shown is compared to Figure 4 The target token sequence shown can significantly reduce the number of time steps.

[0107] To further illustrate the beneficial effects of the speech enhancement method provided in the embodiments of this application, a testing scheme in a conference scenario is provided below:

[0108] In the testing scheme, the speech enhancement method and the STOA-level discriminative model provided in the above embodiments are compared. The P.808 MOS test metric is used under different test sets. The STOA-level discriminative model refers to the discriminative model that has been verified as the best performing model on a specific task or dataset to date. The P.808 MOS test metric is a subjective speech quality assessment method defined in the International Telecommunication Union (ITU-T) standard. It involves human evaluators comprehensively assessing the auditory experience of speech, including clarity, naturalness, and intelligibility, and subjectively scoring the speech. The average of all scores is taken as the MOS (mean opinion score), which measures the audio quality output by the speech enhancement model or system. A higher MOS value indicates higher audio quality after noise reduction. The test sets include a meeting scene test set recorded in a conference setting, the VCTK-DEMAND test set, the public test set of the DNS Challenge, and the URGENT 2025 test set. The VCTK-DEMAND test set is composed of clean VCTK speech and the DEMAND noise library, specifically designed to evaluate the performance of speech denoising models. The DNS Challenge (Deep Noise Suppression Challenge) is an international speech enhancement competition initiated and continuously held by Microsoft, aimed at advancing speech denoising technology in real-world complex scenarios, particularly real-time speech enhancement capabilities under conditions of low latency, real noise, reverberation, and speaker interference. The URGENT 2025 test set is a multilingual, multi-scenario blind test dataset officially released by the URGENT 2025 Challenge (an international competition for speech enhancement and noise suppression), used for subjective listening evaluation. The test results obtained after performing tests based on the above datasets are shown in Table 1 below:

[0109] Table 1: Test Results

[0110] In Table 1 above, the test metrics of the speech enhancement method provided in this application embodiment on each test set are all higher than those of the STOA-level discriminant model on each test set, indicating that the speech enhancement effect of the speech enhancement method provided in this application embodiment is higher than that of the discriminant model.

[0111] The following describes the product embodiments provided in this application.

[0112] Combination Figure 6As shown, this application embodiment provides a speech enhancement device 600, which is integrated into the electronic device of the aforementioned embodiment. The device 600 includes an extraction module 601, a decoding module 602, an integration module 603, a quantization module 604, an enhancement module 605, and a restoration module 606. Wherein:

[0113] Extraction module 601 is configured to extract acoustic feature vectors and semantic feature vectors based on noisy speech signals input by the user.

[0114] The decoding module 602 is configured to decode the speech tokens output by the speech enhancement model in the historical time period into speech feature vectors.

[0115] The integration module 603 is configured to concatenate the acoustic feature vector with the speech feature vector to obtain a noisy speech feature vector.

[0116] The quantization module 604 is configured to quantize the noisy speech feature vector frame by frame into discrete tokens to obtain the target token sequence.

[0117] The enhancement module 605 is configured to input the target token sequence and semantic feature vector into the speech enhancement model, so that the speech enhancement model performs a cross-attention operation based on the target token sequence and semantic feature vector, and outputs the target speech token according to the result of the cross-attention operation.

[0118] The restoration module 606 is configured to restore the target voice token to the target voice signal.

[0119] Optionally, the quantization module 604 quantizes the noisy speech feature vector frame by frame into discrete tokens to obtain the target token sequence. This includes: performing RVQ quantization on the noisy speech feature vector frame by frame to obtain the index sequence of each audio frame; adjusting the index sequence of each audio frame so that each index in the index sequence is delayed by D1 frames to obtain the target token sequence. Where D1 = (N-1) × C + S1 - N, N is the number of audio frames corresponding to the index sequence, C is the number of codebooks used when performing RVQ quantization, and S1 is the position of the index in the index sequence.

[0120] Optionally, the quantization module 604 quantizes the noisy speech feature vector frame by frame into discrete tokens to obtain the target token sequence. This includes: performing RVQ quantization on the noisy speech feature vector frame by frame to obtain the index sequence of each audio frame; adjusting the index sequence of each audio frame so that each index in the index sequence is delayed by D2 frames to obtain the target token sequence. Here, D2 = S2-1, and S2 is the position of the index in the index sequence.

[0121] Optionally, the extraction module 601 extracts acoustic feature vectors based on the noisy speech signal input by the user, including: preprocessing the noisy speech signal and performing a short-time Fourier transform on the preprocessed noisy speech signal to obtain the acoustic feature vectors of each audio frame in the preprocessed noisy speech signal.

[0122] Optionally, after extracting the acoustic feature vectors, the extraction module 601 converts the dimensions of the acoustic feature vectors into a target dimension. The target dimension is the input dimension of the speech enhancement model.

[0123] Optionally, after extracting the semantic feature vector, the extraction module 601 converts the dimension of the semantic feature vector into the target dimension. The target dimension is the input dimension of the speech enhancement model.

[0124] The speech enhancement apparatus provided in this application embodiment can execute the speech enhancement method in the above embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0125] Combination Figure 7 As shown, this application embodiment provides a voice enhancement device 700, including a processor 701 and a memory 702. Optionally, the voice enhancement device 700 may further include a communication interface 703 and a bus 704. The processor 701, memory 702, and communication interface 703 can communicate with each other via the bus 704. The communication interface 703 can be used for information transmission. The processor 701 can call logical instructions in the memory 702 to execute the voice enhancement method described in the above embodiment.

[0126] Furthermore, the logic instructions in the aforementioned memory 702 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0127] The memory 702, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this application. The processor 701 executes functional applications and data processing by running the program instructions / modules stored in the memory 702, that is, it implements the speech enhancement method in the above embodiments.

[0128] The memory 702 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 702 may include high-speed random access memory and may also include non-volatile memory.

[0129] This application provides a storage medium storing computer-executable instructions configured to perform the speech enhancement method described in the above embodiments.

[0130] The aforementioned storage medium can be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0131] The technical solutions of this application embodiment can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this application embodiment. The aforementioned storage medium can be a non-transitory storage medium, including: USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, and other media capable of storing program code; it can also be a transient storage medium.

[0132] This application provides a computer program product, including a computer program that, when executed by a processor, implements the speech enhancement method described in the above embodiments.

[0133] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0134] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0135] The methods and products (including but not limited to devices and equipment) disclosed in the embodiments herein can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to implement this embodiment according to actual needs. In addition, the functional units in the embodiments of this application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0136] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description; sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A method for speech enhancement, characterized in that, Applied to an electronic device, the electronic device having a voice enhancement model deployed thereon; the method includes: Based on the noisy speech signal input by the user, acoustic feature vectors and semantic feature vectors are extracted; The speech tokens output by the speech enhancement model in the historical time are decoded into speech feature vectors; The acoustic feature vector is concatenated with the speech feature vector to obtain a noisy speech feature vector; The noisy speech feature vector is quantized frame by frame into discrete tokens to obtain the target token sequence; The target token sequence and the semantic feature vector are input into the speech enhancement model, so that the speech enhancement model performs a cross-attention operation based on the target token sequence and the semantic feature vector, and outputs the target speech token according to the result of the cross-attention operation; The target voice token is restored to the target voice signal.

2. The method according to claim 1, characterized in that, The step of quantizing the noisy speech feature vector frame by frame into discrete tokens to obtain the target token sequence includes: The noisy speech feature vector is subjected to RVQ quantization frame by frame to obtain the index sequence of each audio frame; Adjust the index sequence of each audio frame separately, delaying each index in the index sequence by D1 frames, to obtain the target token sequence; Wherein, D1 = (N-1) × C + S1 - N, N is the number of audio frames corresponding to the index sequence, C is the number of codebooks used when performing RVQ quantization, and S1 is the index position in the index sequence.

3. The method according to claim 1, characterized in that, The step of quantizing the noisy speech feature vector frame by frame into discrete tokens to obtain the target token sequence includes: The noisy speech feature vector is subjected to RVQ quantization frame by frame to obtain the index sequence of each audio frame; Adjust the index sequence of each audio frame separately, delaying each index in the index sequence by D2 frames, to obtain the target token sequence; Where D2 = S2-1, and S2 is the position of the index in the index sequence.

4. The method according to claim 1, characterized in that, The extraction of acoustic feature vectors from noisy speech signals based on user input includes: The noisy speech signal is preprocessed; A short-time Fourier transform is performed on the preprocessed noisy speech signal to obtain the acoustic feature vectors of each audio frame in the preprocessed noisy speech signal.

5. The method according to claim 1, characterized in that, After extracting the acoustic feature vector, the process also includes: Convert the dimensions of the acoustic feature vectors to the target dimensions; The target dimension is the input dimension of the speech enhancement model.

6. The method according to claim 1, characterized in that, After extracting the semantic feature vector, the process also includes: Convert the dimension of the semantic feature vector to the target dimension; The target dimension is the input dimension of the speech enhancement model.

7. A speech enhancement device, characterized in that, Integrated into an electronic device, the electronic device having a deployed speech enhancement model; the device includes: The extraction module is configured to extract acoustic feature vectors and semantic feature vectors based on noisy speech signals input by the user; The decoding module is configured to decode the speech tokens output by the speech enhancement model in the historical time period into speech feature vectors; The integration module is configured to concatenate the acoustic feature vector with the speech feature vector to obtain a noisy speech feature vector; The quantization module is configured to quantize the noisy speech feature vector frame by frame into discrete tokens to obtain a target token sequence; An enhancement module is configured to input the target token sequence and the semantic feature vector into the speech enhancement model, so that the speech enhancement model performs a cross-attention operation based on the target token sequence and the semantic feature vector, and outputs the target speech token according to the result of the cross-attention operation; The restoration module is configured to restore the target voice token to the target voice signal.

8. A voice enhancement device, comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to perform the speech enhancement method as described in any one of claims 1 to 6 when executing the program instructions.

9. A storage medium storing program instructions, characterized in that, When the program instructions are executed, they perform the speech enhancement method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the speech enhancement method as described in any one of claims 1 to 6.

Images