Speech enhancement method and device, training method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of speech processing, can be applied to the fields of finance and medical treatment, and discloses a speech enhancement method, a training method, a device, computer equipment and a storage medium, and the speech enhancement method comprises the steps: receiving noisy speech input, and carrying out the coding processing of the noisy speech through a pre-trained decomposition codec, and obtaining a hidden representation; performing step-by-step denoising processing on the hidden representation by using a pre-trained semantic diffusion model to obtain a semantic token sequence corresponding to the clear voice; combining the semantic token sequence with the hidden representation of the noisy voice as a condition, and carrying out step-by-step denoising processing through a pre-trained acoustic diffusion model to obtain an acoustic token sequence corresponding to the clear voice; and inputting the semantic token sequence and the acoustic token sequence into a decoder of the decomposition codec, and reconstructing to generate a clear voice signal. According to the method, the robustness of speech enhancement in a complex noise environment is remarkably improved, and the interference of noise on speech key attributes is reduced.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and can be applied to the financial and medical fields. More specifically, it relates to speech enhancement methods, training methods, devices, computer equipment, and storage media. Background Technology

[0002] In the field of speech signal processing, speech enhancement technology, as a key means of extracting clear speech from noisy environments, has significant application value in finance, healthcare, and intelligent communication interaction. In financial scenarios, voice interaction is widely used in remote customer service (such as bank customer service voice consultation), financial transaction instruction confirmation (such as voice verification of transfer operations), and intelligent investment advisor voice interaction. In healthcare scenarios, speech enhancement technology is a crucial support for ensuring diagnostic efficiency and patient safety, and has been deeply integrated into core business processes such as clinical diagnosis, telemedicine, electronic medical record entry, and rehabilitation assistance. For example, in outpatient consultations, doctors need to record symptom descriptions through voice interaction with patients and generate electronic medical records in real time using a voice input system. This information directly determines the diagnostic direction and medication plan. These scenarios have extremely high requirements for the clarity, semantic accuracy, and acoustic feature fidelity of the speech signal.

[0003] Currently, most traditional speech enhancement methods recover clear speech signals from noisy inputs by directly estimating time-frequency masks or spectral features. However, these methods have significant technical limitations in practical applications, especially in financial scenarios where they struggle to meet core requirements.

[0004] First, traditional methods generally overlook the inherent hierarchical nature of speech signals. Speech signals contain multiple attributes, including semantic content (such as key linguistic information like customer account information and transaction instructions), prosody (such as affirmative or questioning emotions in tone), timbre (such as acoustic features related to the speaker's identity), and acoustic details. In financial scenarios, the accuracy of semantic information directly affects transaction security (e.g., misinterpretation of instructions may lead to operational errors), while the fidelity of acoustic features affects the reliability of identity verification. Existing technologies often treat speech signals as a whole, failing to effectively decouple semantic and acoustic attributes. This results in enhanced speech that, while suppressing noise, easily obscures key semantic information (such as numbers and instruction terms) or loses speaker timbre characteristics, affecting the accuracy of identity verification.

[0005] Secondly, performance degrades significantly in complex acoustic environments. For example, in financial service scenarios, customer service centers may experience background noise, equipment hum, and other interference, while remote clients may face issues such as far-field sound pickup and network transmission noise. Under low signal-to-noise ratio or far-field conditions, the difficulty of directly estimating the speech spectrum or mask using traditional methods increases dramatically, leading to distortions in command terminology (e.g., "transfer" mistakenly enhanced as "transfer inflation") and blurred numbers in the enhanced speech, severely impacting the accuracy and security of financial transactions.

[0006] Furthermore, traditional methods are insufficient in supporting downstream tasks in the financial sector. Voice interaction in the financial field often needs to interface with downstream systems such as identity verification and command parsing. These systems are highly dependent on the semantic coherence and acoustic feature integrity of the speech. Although the speech enhanced by traditional methods may improve the subjective listening experience, semantic logical breaks or acoustic feature distortions lead to a decrease in the recognition accuracy of downstream systems (such as voice command recognition models), increase manual verification costs, and even trigger business risks.

[0007] Therefore, how to effectively decouple and preserve the semantic and acoustic properties of speech signals, improve the robustness of speech enhancement in complex noise environments, and ensure the accuracy and security of speech interaction have become key issues that need to be addressed in current speech enhancement technologies. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the prior art and provide a speech enhancement method, training method, device, computer equipment and storage medium, aiming to improve the robustness of speech enhancement in complex noise environments.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] In a first aspect, the present invention provides a speech enhancement method, comprising:

[0011] The system receives noisy speech input and encodes the noisy speech using a pre-trained decomposition codec to obtain a hidden representation.

[0012] By using a pre-trained semantic diffusion model to progressively denoise the hidden representation, a semantic token sequence corresponding to the clear speech is obtained.

[0013] By combining the semantic token sequence with the hidden representation of noisy speech as a condition, and performing stepwise denoising through a pre-trained acoustic diffusion model, the acoustic token sequence corresponding to clear speech is obtained.

[0014] The semantic token sequence and the acoustic token sequence are input into the decoder of the decomposer codec to reconstruct and generate a clear speech signal.

[0015] Further, the receiving of noisy speech input and the encoding of the noisy speech using a pre-trained decomposition codec to obtain a hidden representation includes:

[0016] Noisy speech data is downsampled to extract key speech features;

[0017] By performing feature mapping on the downsampled key speech features through the neural network layer of the encoder, the time-domain speech signal is converted into a hidden representation in a high-dimensional feature space.

[0018] Furthermore, the stepwise denoising process performed on the hidden representation using a pre-trained semantic diffusion model to obtain the semantic token sequence corresponding to the clear speech includes:

[0019] The total number of time steps in the diffusion process is set. At each time step, the semantic diffusion model predicts the semantic tokens masked by the mask tokens based on the current semantic token sequence and the hidden representation of noisy speech.

[0020] Update the current semantic token sequence based on the prediction results, replacing the mask tokens with the predicted clear semantic tokens;

[0021] The process of repeatedly performing the semantic diffusion model, based on the current semantic token sequence and the hidden representation of noisy speech, predicting the semantic tokens masked by the mask tokens, and updating the current semantic token sequence based on the prediction results, replacing the mask tokens with the predicted clear semantic tokens, is repeated at each time step until the denoising process of all time steps is completed and the semantic token sequence corresponding to the clear speech is output.

[0022] Furthermore, the step of combining the semantic token sequence with the hidden representation of noisy speech as a condition, and performing progressive denoising through a pre-trained acoustic diffusion model to obtain the acoustic token sequence corresponding to the clear speech, includes:

[0023] The total time steps of the diffusion process are set. At each time step, the acoustic diffusion model predicts the acoustic tokens that are masked by the masked tokens based on the current acoustic token sequence, semantic tokens, and the hidden representation of noisy speech.

[0024] The current acoustic token sequence is updated based on the prediction results, replacing the masked tokens with the predicted clear acoustic tokens, while retaining the unmasked acoustic tokens as the context for the next time step.

[0025] The process of repeatedly performing the following steps at each time step, whereby the acoustic diffusion model predicts the acoustic tokens masked by the mask tokens based on the current acoustic token sequence, semantic tokens, and the hidden representation of noisy speech, and updates the current acoustic token sequence based on the prediction results, replacing the mask tokens with the predicted clear acoustic tokens, while retaining the unmasked acoustic tokens as the context for the next time step, continues until the denoising process for all time steps is completed, and the acoustic token sequence corresponding to the recovered clear speech is output.

[0026] Further, the step of feeding the semantic token sequence and acoustic token input sequence into the decoder of the decomposition codec to reconstruct and generate a clear speech signal includes:

[0027] The decoder performs feature mapping on the input semantic token sequence, extracts the linguistic content information contained therein, and converts it into a semantic feature vector;

[0028] The decoder performs feature mapping on the input acoustic token sequence, extracts the acoustic details contained therein, and converts them into acoustic feature vectors;

[0029] The semantic feature vector and the acoustic feature vector are fused across modally through the fusion layer of the decoder to generate a joint feature representation containing complete speech attributes;

[0030] The decoder's output layer performs waveform transformation on the joint feature representation, mapping the joint representation in the high-dimensional feature space to a time-domain speech signal, thereby generating a clear speech signal.

[0031] Secondly, the present invention also provides a method for training a speech model, applied to the aforementioned speech enhancement method. The method for training the speech model is used to train a decomposition codec, a semantic diffusion model, and an acoustic diffusion model. The method includes:

[0032] Construct a decomposed codec to train a semantic vector quantizer and an acoustic vector quantizer using clean speech, so that the semantic vector quantizer outputs a semantic token corresponding to the language content and the acoustic vector quantizer outputs an acoustic token corresponding to the acoustic details.

[0033] Using semantic tokens from noisy and clean speech as training data, the forward noise addition and backward noise reduction capabilities of the semantic diffusion model are trained, enabling the semantic diffusion model to generate clear semantic tokens from noisy speech.

[0034] Using semantic tokens and acoustic tokens from noisy speech and clean speech as training data, the forward noise addition and backward noise reduction capabilities of the acoustic diffusion model are trained, enabling the acoustic diffusion model to generate clear acoustic tokens from noisy speech based on semantic tokens.

[0035] The parameters of the decomposer, semantic diffusion model, and acoustic diffusion model are jointly optimized to minimize the difference loss between generated speech and clean speech.

[0036] Thirdly, the present invention also provides a voice enhancement device, comprising:

[0037] The receiving unit is used to receive noisy speech input, encode the noisy speech using a pre-trained decomposition codec, and obtain a hidden representation.

[0038] The first estimation unit is used to perform stepwise denoising on the hidden representation using a pre-trained semantic diffusion model to obtain the semantic token sequence corresponding to the clear speech.

[0039] The second estimation unit is used to combine the semantic token sequence with the hidden representation of noisy speech as a condition, and perform stepwise denoising through a pre-trained acoustic diffusion model to obtain the acoustic token sequence corresponding to the clear speech.

[0040] The reconstruction unit is used to input the semantic token sequence and the acoustic token sequence into the decoder of the decomposition codec to reconstruct and generate a clear speech signal.

[0041] Fourthly, the present invention also provides a training apparatus for a speech model, comprising:

[0042] The building unit is used to build the decomposed codec to train the semantic vector quantizer and acoustic vector quantizer with clean speech, so that the semantic vector quantizer outputs semantic tokens corresponding to the language content and the acoustic vector quantizer outputs acoustic tokens corresponding to the acoustic details.

[0043] The first training unit is used to train the forward noise addition and backward noise reduction capabilities of the semantic diffusion model using semantic tokens of noisy speech and clean speech as training data, so that the semantic diffusion model can generate clear semantic tokens from noisy speech.

[0044] The second training unit is used to train the forward noise addition and backward noise reduction capabilities of the acoustic diffusion model using semantic tokens and acoustic tokens of noisy speech and clean speech as training data, so that the acoustic diffusion model can generate clear acoustic tokens from noisy speech based on semantic tokens.

[0045] The optimization unit is used to jointly optimize the parameters of the decomposition codec, semantic diffusion model, and acoustic diffusion model to minimize the difference loss between generated speech and clean speech.

[0046] Fifthly, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the speech enhancement method or the speech model training method described above.

[0047] In a sixth aspect, the present invention also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the speech enhancement method or the speech model training method described above.

[0048] The beneficial effects of this invention compared to existing technologies are as follows: The speech enhancement method includes: receiving noisy speech input; encoding the noisy speech using a pre-trained decomposition codec to obtain a hidden representation; progressively denoising the hidden representation using a pre-trained semantic diffusion model to obtain a semantic token sequence corresponding to clear speech; combining the semantic token sequence with the hidden representation of the noisy speech as a condition, and progressively denoising the acoustic token sequence using a pre-trained acoustic diffusion model to obtain an acoustic token sequence corresponding to clear speech; inputting the semantic token sequence and acoustic token sequence into the decoder of the decomposition codec to reconstruct and generate a clear speech signal. This invention prioritizes estimating the semantic token sequence from the noisy speech, and then estimates the acoustic token sequence using semantic information as a condition. This effectively decouples the semantic content and acoustic details of the speech, avoids the problem of mutual interference of attributes in traditional methods, ensures that the enhanced speech retains accurate linguistic content, significantly improves the robustness of speech enhancement in complex noisy environments, and reduces the interference of noise on key speech attributes.

[0049] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other objectives, features and advantages of the present invention more obvious and understandable, preferred embodiments are described in detail below. Attached Figure Description

[0050] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a schematic diagram illustrating the application environment of the speech enhancement method provided in a specific embodiment of the present invention;

[0052] Figure 2 The flow chart of the speech enhancement method provided in a specific embodiment of the present invention Figure 1 ;

[0053] Figure 3 The flow chart of the speech enhancement method provided in a specific embodiment of the present invention Figure 2 ;

[0054] Figure 4 Schematic diagram of a speech enhancement device provided in a specific embodiment of the present invention Figure 1 ;

[0055] Figure 5 Schematic diagram of a speech enhancement device provided in a specific embodiment of the present invention Figure 2 ;

[0056] Figure 6 This is a schematic block diagram of a computer device provided for a specific embodiment of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0059] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0060] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0061] The speech enhancement method provided in this embodiment of the invention can be applied to, for example... Figure 1 In the application environment shown, specifically, this speech enhancement method is applied in a speech enhancement system, which includes, for example, Figure 1The diagram shows a client and a server, with the client communicating with the server over a network. The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The client, also known as the user terminal, can be installed on, but is not limited to, various computers, laptops, smartphones, tablets, and portable wearable devices.

[0062] This voice enhancement system can be applied to medical and financial technology. Specifically, it can be used for intelligent customer service in business scenarios such as e-commerce, electronic payment, securities, e-banking, tax transactions, credit cards, online shopping, and insurance to improve the fidelity of voice.

[0063] like Figure 2 As shown, this embodiment of the invention provides a speech enhancement method, including the following steps: S10-S40.

[0064] S10. Receive noisy speech input, encode the noisy speech using a pre-trained decomposition codec, and obtain a hidden representation.

[0065] This step aims to convert the raw noisy speech signal into a high-dimensional feature representation that can be used for subsequent semantic and acoustic attribute estimation.

[0066] In one embodiment, step S10 specifically includes the following steps: S101-S102.

[0067] S101. Downsample the noisy speech data to extract key speech features.

[0068] In financial scenarios, the received noisy voice input may come from remote bank customer service calls, voice transaction command input, etc. For example, when a customer makes a voice transfer through a mobile APP, there may be background human voices, equipment electrical noise or network transmission noise in the environment. The sampling rate of the original voice data is usually 16kHz.

[0069] In this step, the input 16kHz noisy speech data is first downsampled. Specifically, decimation or interpolation algorithms can be used to reduce the sampling rate to 4kHz–8kHz (preferably 4kHz). The purpose of downsampling is to reduce data redundancy and computational load while preserving key speech features (such as fundamental frequency, formants, and other core information related to semantic and acoustic properties).

[0070] S102. The key speech features after downsampling are mapped through the neural network layer of the encoder, and the time-domain speech signal is converted into a hidden representation in a high-dimensional feature space.

[0071] The speech data after downsampling in step S101 is still a time-domain signal, and needs to be feature-mapped by decomposing the encoder neural network layer of the codec.

[0072] In financial scenarios, this process can convert noisy speech signals in the time domain (such as noisy speech containing "confirm transaction password") into high-dimensional hidden representations, which include features related to semantic content (such as linguistic information of "transaction password") as well as features related to acoustic details (such as the customer's timbre and tone features), providing structured input for subsequent denoising processing of semantic diffusion models and acoustic diffusion models.

[0073] For step 10, downsampling effectively filters high-frequency noise and reduces the impact of complex financial environments (such as interference from multi-line calls in customer service centers and noise from remote equipment) on key speech features, making subsequent feature mapping more focused on the core attributes of speech.

[0074] S20. The hidden representation is gradually denoised using a pre-trained semantic diffusion model to obtain the semantic token sequence corresponding to the clear speech.

[0075] Step S20 aims to extract accurate semantic information from the high-dimensional hidden representation of noisy speech, providing a reliable prior for subsequent acoustic attribute estimation:

[0076] In one embodiment, step S20 specifically includes the following steps: S201-S203.

[0077] S201. Set the total time steps of the diffusion process. In each time step, the semantic diffusion model predicts the semantic tokens masked by the masked tokens based on the current semantic token sequence and the hidden representation of noisy speech.

[0078] In financial scenarios, the hidden representation of noisy speech to be processed may come from noisy speech containing key instructions. For example, when a customer asks "check credit card bill" via voice command in a noisy environment, the hidden representation obtained after processing in step S10 still contains noise interference.

[0079] In this step, the total time step T of the diffusion process is first set. The value of T can be dynamically adjusted according to the noise intensity. For example, in a financial customer service scenario with a low signal-to-noise ratio (e.g., SNR = 5dB), T can be set to 100-200 steps (preferably 150 steps). In each time step t (t from 1 to T), the semantic diffusion model uses the current semantic token sequence and the noisy speech hiding representation obtained in step S10 as input to predict the semantic tokens masked by the mask tokens. Here, the mask token is a special marker representing noise interference (e.g., "[MASK]"). The proportion of the mask token in the initial semantic token sequence is controlled by a monotonically increasing function σ(t) = sin(0.5πt / T). For example, when t = 1, σ(t) ≈ 0.016, and the mask proportion is about 1.6%; when t = 75, σ(t) = 0.5, and the mask proportion reaches 50%, ensuring that the model gradually learns the denoising rules.

[0080] Taking the noisy voice message "transfer amount of 10,000 yuan" in a financial scenario as an example, the current semantic token sequence received by the model at t=50 may contain "transfer [MASK] amount [MASK] 10,000 yuan". By analyzing the semantic features in the hidden representation, it is predicted that the tokens at the masked position should be "gold" and "one".

[0081] S202. Update the current semantic token sequence based on the prediction results, and replace the mask tokens in it with the predicted clear semantic tokens.

[0082] Based on the prediction results of step S201, the current semantic token sequence is updated: all "[MASK]" tokens in the sequence are replaced with clear semantic tokens predicted by the model. For example, for a token sequence containing "credit card [MASK] period [MASK] inquiry", it is updated to "credit card to expected inquiry" after prediction, thus initially restoring semantic coherence. During the update process, the model calculates the difference between the predicted tokens and the clean speech semantic tokens through the cross-entropy loss function, and dynamically adjusts the parameters to improve prediction accuracy, especially focusing on optimizing financial terminology (such as "repayment date" and "credit limit").

[0083] S203. Repeat the process described in each time step, whereby the semantic diffusion model predicts the semantic tokens masked by the mask tokens based on the current semantic token sequence and the hidden representation of noisy speech, and updates the current semantic token sequence based on the prediction results, replacing the mask tokens with the predicted clear semantic tokens, until the denoising process of all time steps is completed and the semantic token sequence corresponding to the clear speech is output.

[0084] Repeat steps S201 and S202 until denoising processing is completed for all time steps T. During the iteration process, as the time steps increase, the proportion of mask tokens gradually decreases, and the model's prediction confidence for semantic tokens gradually increases. For example, when processing noisy speech containing "confirm transaction password", the initial sequence may contain a large number of masks. After 50 iterations, the sequence is updated to "confirm [MASK] easy [MASK]"; after continuing to iterate for 150 steps, the final output is a complete and clear semantic token sequence "confirm transaction password". The output semantic token sequence will serve as a key condition for subsequent acoustic diffusion models, supporting the accurate estimation of acoustic properties.

[0085] For step S20, by setting a reasonable diffusion time step and masking mechanism, the model can still accurately extract semantic information in noisy environments in financial scenarios (such as noisy backgrounds in business halls and weak remote call signals), such as accurately identifying key instructions like "redeeming fund shares," thus avoiding business risks caused by semantic ambiguity. Through multiple rounds of iterative prediction and update processes, the model can gradually correct semantic biases and reduce the accumulation of errors caused by one-time denoising, making it particularly suitable for scenarios in the financial field where semantic accuracy requirements are extremely high (such as voice transaction instruction confirmation and identity verification Q&A).

[0086] S30. Combining the semantic token sequence with the hidden representation of noisy speech as a condition, and performing stepwise denoising processing through a pre-trained acoustic diffusion model to obtain the acoustic token sequence corresponding to clear speech.

[0087] In one embodiment, step S30 specifically includes the following steps: S301-S303.

[0088] Step S30 aims to recover the acoustic details of speech (such as prosody, timbre, etc.) from noise by utilizing semantic prior constraints.

[0089] S301. Set the total time steps for the diffusion process. In each time step, the acoustic diffusion model predicts the acoustic tokens that are masked by the masked tokens based on the current acoustic token sequence, semantic tokens, and the hidden representation of noisy speech.

[0090] In financial scenarios, the initial state of the acoustic token sequence to be processed may contain a lot of noise interference. For example, when a customer performs "password verification" by voice, the original voice is distorted due to the device's current noise. After preprocessing, there are multiple "[MASK]" masks in the acoustic token sequence.

[0091] In this step, first set the total number of time steps T for the diffusion process. The value of T can be adjusted according to the complexity of the acoustic noise. For example, in a far-field financial service scenario (such as voice interaction at a self-service terminal), T can be set to 200 steps to ensure sufficient denoising. In each time step t (where t ranges from 1 to T), the acoustic diffusion model takes three conditions as input: the current acoustic token sequence, the semantic token sequence output by step S20 (such as "password verification passed"), and the noisy speech hidden representation obtained from step S10. The model predicts the acoustic tokens masked by "[MASK]" by analyzing the language logic in the semantic tokens (such as the stress position of the word "verification") and the acoustic profile in the hidden representation. For example, it predicts acoustic feature tokens such as the pitch and speaking speed corresponding to the word "password".

[0092] Taking the noisy speech of "transfer amount confirmation" in a financial scenario as an example, the current acoustic token sequence received by the model at t = 100 may contain "transfer [MASK]mount [MASK]onfirmation". Combining the semantic token "transfer amount confirmation" and the spectral features in the hidden representation, the predicted acoustic tokens for the masked positions should be the mid-low pitch token corresponding to the word "gold" and the stress token corresponding to the word "confirm".

[0093] S302. Update the current acoustic token sequence based on the prediction result, replacing the masked tokens with the predicted clear acoustic tokens, and at the same time retaining the unmasked acoustic tokens as the context for the next time step.

[0094] Based on the prediction result of step S301, update the current acoustic token sequence: replace the "[MASK]" tokens in the sequence with the predicted clear acoustic tokens, and at the same time retain the unmasked acoustic tokens as the context for the next time step. For example, for an acoustic token sequence containing "credit card [MASK]ayment date [MASK]uery", after update, it becomes "credit card payment [MASK]uery", where the acoustic tokens corresponding to "payment" (such as the feature of slowed speaking speed) are retained as the context reference for subsequent predictions. During the update process, the model optimizes the parameters by comparing the feature loss (such as the Mel spectrum distance) between the predicted acoustic tokens and the clean speech acoustic tokens, and focuses on retaining the voice characteristics related to identity in the financial scenario (such as the unique voice intonation of the customer).

[0095] S303. Repeat the process where in each time step, the acoustic diffusion model predicts the acoustic tokens masked by the masked tokens based on the current acoustic token sequence, semantic tokens, and the hidden representation of the noisy speech, and updates the current acoustic token sequence based on the prediction result, replacing the masked tokens with the predicted clear acoustic tokens, and at the same time retaining the unmasked acoustic tokens as the context for the next time step, until the denoising process for all time steps is completed, and output the acoustic token sequence corresponding to the restored clear speech.

[0096] Repeat steps S301 and S302 until denoising processing is completed for all time steps T. During the iteration process, the semantic token sequence continuously provides constraints for acoustic prediction. For example, the semantic information of "large transfer" guides the model to prioritize the recovery of the emphatic acoustic features corresponding to the word "large." For instance, when processing noisy speech containing "confirm transaction limit," the initial acoustic token sequence mask accounts for 60%; after 100 iterations, the mask percentage drops to 20%, and the acoustic features are initially recovered; after continuing iterations to 200 steps, a complete and clear acoustic token sequence is output, containing accurate intonation variations, speech rate changes, and other acoustic details. The output acoustic token sequence, along with the semantic token sequence, serves as input to the decoder, supporting clear speech reconstruction.

[0097] In step S30, through the dual conditional constraints of semantic tokens and hidden representations, the model can accurately recover key acoustic features in financial scenarios. For example, it can clearly restore the speech pauses during "password input" and the affirmative tone during "transaction confirmation," improving the naturalness and recognizability of the speech. The conditional guidance of semantic tokens ensures a high degree of matching between acoustic features and language content, avoiding the problem of "semantic correctness but acoustic distortion" in traditional methods. For example, in the speech enhancement of "redeeming funds," it ensures that the acoustic features of the stressed word "redemption" match the semantic meaning. Figure 1 This reduces the unnaturalness of downstream speech synthesis.

[0098] S40. Input the semantic token sequence and the acoustic token sequence into the decoder of the decomposition codec to reconstruct and generate a clear speech signal.

[0099] Step 40 aims to reconstruct complete, clear speech from the token sequence by fusing semantic and acoustic features.

[0100] In one embodiment, step S40 specifically includes the following steps: S401-S404.

[0101] S401. The decoder performs feature mapping on the input semantic token sequence, extracts the linguistic content information contained therein, and converts it into a semantic feature vector.

[0102] In financial scenarios, the semantic token sequence input to the decoder contains key linguistic information, such as the semantic token sequence corresponding to the customer's voice command "Query this month's credit card statement". In this step, the decoder first performs feature mapping on the semantic token sequence: the discrete semantic tokens are converted into continuous vectors through a pre-trained embedding layer, and then the language content features (such as the semantic logic of keywords like "credit card" and "statement") are extracted by a multilayer perceptron (MLP). For example, for the semantic token sequence "Query [this] [month] [credit] [card] [statement]", the embedding layer maps each token to a 32-dimensional vector, and the MLP further aggregates them into a 128-dimensional semantic feature vector through nonlinear transformation. This vector encodes complete linguistic content information, providing semantic logical constraints for subsequent speech reconstruction.

[0103] S402. The decoder performs feature mapping on the input acoustic token sequence, extracts the acoustic detail information contained therein, and converts it into an acoustic feature vector.

[0104] The input acoustic token sequence contains features related to the acoustic details of speech, such as intonation, speech rate, and timbre in the customer's voice. In financial scenarios, such as when a customer performs "identity verification" via voice, the acoustic token sequence carries its unique timbre features. In this step, the decoder performs feature mapping on the acoustic token sequence: using an embedding layer similar to semantic mapping, discrete acoustic tokens are converted into continuous vectors, and then a convolutional neural network (CNN) is used to extract acoustic detail features (such as fundamental frequency, formants, energy changes, etc.). For example, for a sequence containing the acoustic token "identity verification", after the embedding layer maps it to a 32-dimensional vector, the CNN extracts temporal acoustic features through 3 layers of convolutional operations (3×3 kernel size, stride 1), finally converting it into a 128-dimensional acoustic feature vector, fully preserving the customer's timbre and intonation features.

[0105] S403. The semantic feature vector and the acoustic feature vector are fused across modally through the fusion layer of the decoder to generate a joint feature representation containing complete speech attributes.

[0106] The decoder's fusion layer employs an attention mechanism to achieve cross-modal fusion of semantic and acoustic features. In financial scenarios, such as processing the speech reconstruction of "confirm transfer amount of 10,000 yuan," the fusion layer guides the weight allocation of the acoustic feature vector through the keyword "transfer amount" in the semantic feature vector, emphasizing the stressed acoustic features corresponding to "10,000 yuan." Specifically, the attention mechanism calculates the similarity matrix between semantic and acoustic features, assigns higher weights to acoustic segments corresponding to semantic keywords, and then generates a 256-dimensional joint feature representation through weighted summation. This representation simultaneously contains complete linguistic content and matching acoustic details, ensuring semantic and acoustic consistency.

[0107] S404, the output layer of the decoder performs waveform transformation on the joint feature representation, mapping the joint representation in the high-dimensional feature space to a time-domain speech signal, generating a clear speech signal.

[0108] The decoder's output layer uses a deconvolutional neural network (DeCNN) to map the joint feature representation into a time-domain speech signal. For speech reconstruction in financial scenarios, the output layer first uses three layers of deconvolution to progressively convert the 256-dimensional joint features into Mel-spectral features (e.g., 80-dimensional Mel coefficients). Then, it uses the Griffin-Lim algorithm or a pre-trained vocoder (e.g., WaveNet) to convert the Mel-spectrum into a 16kHz time-domain speech waveform. For example, after waveform conversion of the joint feature representation of "transaction password verification passed," the generated clear speech signal accurately reproduces the text of the instruction while preserving the customer's original timbre and intonation, achieving natural and intelligible speech output.

[0109] This invention prioritizes estimating semantic token sequences from noisy speech, and then estimates acoustic token sequences based on semantic information. This effectively decouples the semantic content and acoustic details of speech, avoids the problem of mutual interference between attributes in traditional methods, ensures that the enhanced speech retains accurate language content, significantly improves the robustness of speech enhancement in complex noisy environments, and reduces the interference of noise on key speech attributes.

[0110] like Figure 3 As shown, this embodiment of the invention also provides a method for training a speech model, applied to the aforementioned speech enhancement method, to optimize the decomposition codec, semantic diffusion model, and acoustic diffusion model involved in the speech enhancement method, ensuring that the model achieves high-precision speech enhancement in complex noisy environments such as financial scenarios. The training method includes the following steps: S11-S14.

[0111] S11. Construct a decomposed codec to train a semantic vector quantizer and an acoustic vector quantizer using clean speech, so that the semantic vector quantizer outputs a semantic token corresponding to the language content, and the acoustic vector quantizer outputs an acoustic token corresponding to the acoustic details.

[0112] A decomposed codec framework is constructed, comprising an encoder, a semantic vector quantizer (SemanticVQ), an acoustic vector quantizer (AcousticVQ), and a decoder. In the financial scenario, a clean speech dataset containing clear customer voice commands (such as "check account balance" and "confirm transaction") is used as training data.

[0113] During training, the encoder downsamples and encodes 16kHz clean speech into a hidden representation h. Semantic VQ and acoustic VQ quantize the hidden representation respectively: Semantic VQ projects the encoder output into an 8-dimensional low-dimensional space using an information bottleneck technique and quantizes it to generate semantic tokens corresponding to the language content (such as discrete tokens corresponding to keywords like "account" and "transaction"); Acoustic VQ generates acoustic tokens corresponding to acoustic details (such as speaker timbre and intonation). Simultaneously, a supervised learning strategy is introduced, inputting the semantic tokens into a phoneme predictor. By minimizing the frame-level phoneme prediction loss (phoneme labels are obtained using the MontrealForcedAligner toolkit), the correlation between semantic tokens and language content in the financial domain is strengthened, for example, ensuring accurate mapping of semantic tokens corresponding to terms such as "transfer amount".

[0114] S12. Using semantic tokens from noisy and clean speech as training data, train the forward noise addition and backward noise reduction capabilities of the semantic diffusion model so that the semantic diffusion model can generate clear semantic tokens from noisy speech.

[0115] Using noisy speech collected in financial scenarios (such as the "redeem fund" instruction containing background noise from a customer service center) and semantic tokens of corresponding clean speech as training data, the forward noise addition and backward noise removal capabilities of the semantic diffusion model are trained.

[0116] During the forward noise addition process, the masking probability is controlled by a monotonically increasing function σ(t) = sin(0.5πt / T) (t∈(0,T], where T is the total time step), gradually replacing the tokens in the clean semantic token sequence with the masked tokens. During the reverse denoising process, the model learns to predict the masked semantic tokens conditioned on the hidden representation of the noisy speech. By minimizing the cross-entropy loss between the predicted tokens and the clean semantic tokens, the model's ability to recover semantics from noise is optimized. For example, for noisy "credit card repayment" speech, the model is trained to accurately predict core semantic tokens such as "credit card" and "repayment," ensuring that key information about financial instructions is not lost.

[0117] S13. Using semantic tokens and acoustic tokens of noisy speech and clean speech as training data, train the forward noise addition and backward noise reduction capabilities of the acoustic diffusion model, so that the acoustic diffusion model can generate clear acoustic tokens from noisy speech based on semantic tokens.

[0118] An acoustic diffusion model is trained using semantic and acoustic tokens representing noisy and clean speech as training data. In financial scenarios, speech data containing personalized speaker features (such as the unique voice of VIP customers) is emphasized.

[0119] During training, the forward noise addition process is similar to that of the semantic diffusion model, progressively adding masked noise to the clean acoustic token sequence. In the reverse denoising process, the model predicts the masked acoustic tokens based on the noisy speech hidden representation, the estimated semantic tokens, and the unmasked acoustic tokens. For example, in the "authentication" scenario, the model needs to accurately recover the acoustic token corresponding to the speaker's timbre based on the "verification" semantic token and noisy speech features, ensuring the fidelity of the acoustic features for identity verification. By minimizing the acoustic token prediction loss, the model can utilize semantic constraints to generate matching acoustic details.

[0120] S14. Jointly optimize the parameters of the decomposer, semantic diffusion model, and acoustic diffusion model to minimize the difference loss between generated speech and clean speech.

[0121] The parameters of the decomposer, semantic diffusion model, and acoustic diffusion model are jointly optimized. The optimization objective is to minimize the difference loss between the generated speech and the clean speech, including semantic token prediction loss, acoustic token prediction loss, and waveform loss (such as Mel-spectral distortion loss) between the decoder-reconstructed speech and the clean speech. In financial scenarios, parameters are adjusted through multiple rounds of iteration. For example, for "large transfer confirmation" speech, the semantic token generated by the model accurately corresponds to the instruction content, the acoustic token faithfully reproduces the speaker's tone features, and the waveform difference between the reconstructed speech and the clean speech is ultimately controlled within a preset threshold (such as Mel-spectral error less than 0.05).

[0122] By training the quantizer through information bottlenecks and supervised learning, semantic tokens focus on the linguistic content of financial instructions (such as "transaction password" and "repayment date"), while acoustic tokens retain personalized features such as speaker timbre, avoiding attribute interference problems in traditional models. After step-by-step training of the semantic and acoustic diffusion models, the prediction accuracy of semantic tokens is improved in far-field, low signal-to-noise ratio environments in financial scenarios (such as voice instructions in noisy environments like bank branches), and the recovery quality of acoustic tokens is significantly improved, laying the foundation for downstream augmentation tasks.

[0123] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0124] This invention also provides a speech enhancement apparatus for performing the steps in any of the embodiments of the foregoing speech enhancement method. Specifically, please refer to... Figure 4 , Figure 4 A schematic block diagram of a speech enhancement device 100 provided in an embodiment of this application is shown. The speech enhancement device 100 specifically includes:

[0125] The receiving unit 110 is used to receive noisy speech input and encode the noisy speech using a pre-trained decomposition codec to obtain a hidden representation; the first estimation unit 120 is used to perform progressive denoising on the hidden representation using a pre-trained semantic diffusion model to obtain a semantic token sequence corresponding to clear speech; the second estimation unit 130 is used to combine the semantic token sequence with the hidden representation of the noisy speech as a condition and perform progressive denoising on the acoustic diffusion model using a pre-trained acoustic diffusion model to obtain an acoustic token sequence corresponding to clear speech; the reconstruction unit 140 is used to input the semantic token sequence and the acoustic token sequence into the decoder of the decomposition codec to reconstruct and generate a clear speech signal.

[0126] In one embodiment, the receiving unit 110 is specifically configured to: perform downsampling processing on noisy speech data to extract key speech features; and perform feature mapping on the downsampled key speech features through the neural network layer of the encoder to convert the time-domain speech signal into a hidden representation in a high-dimensional feature space.

[0127] In one embodiment, the first estimation unit 120 is specifically configured to: set a total number of time steps in the diffusion process; in each time step, the semantic diffusion model predicts the semantic tokens masked by the mask tokens based on the current semantic token sequence and the hidden representation of the noisy speech; update the current semantic token sequence based on the prediction result, replacing the mask tokens therein with the predicted clear semantic tokens; repeat the process of the semantic diffusion model predicting the semantic tokens masked by the mask tokens based on the current semantic token sequence and the hidden representation of the noisy speech in each time step, and updating the current semantic token sequence based on the prediction result, replacing the mask tokens therein with the predicted clear semantic tokens, until the denoising processing of all time steps is completed, and the semantic token sequence corresponding to the clear speech is output.

[0128] In one embodiment, the second estimation unit 130 is specifically configured to: set a total number of time steps for the diffusion process; in each time step, the acoustic diffusion model predicts the acoustic tokens masked by the masked tokens based on the current acoustic token sequence, semantic tokens, and the hidden representation of noisy speech; update the current acoustic token sequence based on the prediction results, replacing the masked tokens with the predicted clear acoustic tokens, while retaining the unmasked acoustic tokens as the context for the next time step; repeat the process of the acoustic diffusion model predicting the acoustic tokens masked by the masked tokens based on the current acoustic token sequence, semantic tokens, and the hidden representation of noisy speech in each time step, and updating the current acoustic token sequence based on the prediction results, replacing the masked tokens with the predicted clear acoustic tokens, while retaining the unmasked acoustic tokens as the context for the next time step, until the denoising processing of all time steps is completed, and output the acoustic token sequence corresponding to the recovered clear speech.

[0129] In one embodiment, the reconstruction unit 140 is specifically configured to: perform feature mapping on the input semantic token sequence, extract the linguistic content information contained therein, and convert it into a semantic feature vector; perform feature mapping on the input acoustic token sequence, extract the acoustic detail information contained therein, and convert it into an acoustic feature vector; perform cross-modal fusion of the semantic feature vector and the acoustic feature vector through the fusion layer of the decoder to generate a joint feature representation containing complete speech attributes; and perform waveform transformation on the joint feature representation of the decoder to map the joint representation of the high-dimensional feature space into a time-domain speech signal to generate a clear speech signal.

[0130] The present invention also provides a speech model training apparatus, which is used to perform the steps of any embodiment of the aforementioned speech model training method. Specifically, please refer to... Figure 5 , Figure 5 A schematic block diagram of a speech model training device 1 provided in an embodiment of this application is shown. The speech model training device 1 specifically includes:

[0131] Construction unit 11 is used to construct a decomposition codec to train a semantic vector quantizer and an acoustic vector quantizer using clean speech, so that the semantic vector quantizer outputs a semantic token corresponding to the language content and the acoustic vector quantizer outputs an acoustic token corresponding to the acoustic details; the first training unit 12 is used to train the forward noise addition and backward denoising capabilities of the semantic diffusion model using semantic tokens from noisy speech and clean speech as training data, so that the semantic diffusion model can generate clear semantic tokens from noisy speech; the second training unit 13 is used to train the forward noise addition and backward denoising capabilities of the acoustic diffusion model using semantic tokens and acoustic tokens from noisy speech and clean speech as training data, so that the acoustic diffusion model can generate clear acoustic tokens from noisy speech based on semantic tokens; the optimization unit 14 is used to jointly optimize the parameters of the decomposition codec, the semantic diffusion model, and the acoustic diffusion model to minimize the difference loss between the generated speech and the clean speech.

[0132] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned speech enhancement device 100, speech model training device 1 and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0133] The aforementioned speech enhancement device 100 or speech model training device 1 can be implemented as a computer program, which can be used in, for example... Figure 6 It runs on the computer device shown.

[0134] Please see Figure 6 , Figure 6This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 700 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0135] like Figure 6 As shown, the computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the speech enhancement method or speech model training method as described above.

[0136] The computer device 700 includes a processor 720, a memory, and a network interface 750 connected via a system bus 710, wherein the memory may include a non-volatile storage medium 730 and internal memory 740.

[0137] The non-volatile storage medium 730 may store an operating system 731 and a computer program 732. When the computer program 732 is executed, it enables the processor 720 to perform a speech enhancement method or a speech model training method.

[0138] The processor 720 provides computing and control capabilities to support the operation of the entire computer device 700.

[0139] The internal memory 740 provides an environment for the operation of the computer program 732 in the non-volatile storage medium 730. When the computer program 732 is executed by the processor 720, the processor 720 can execute a speech enhancement method or a speech model training method.

[0140] This network interface 750 is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 700 to which the present application is applied. The specific computer device 700 may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements. The processor 720 is used to run program code stored in memory to implement a speech enhancement method or a speech model training method.

[0141] Those skilled in the art will understand that Figure 6 The embodiments of the computer device shown do not constitute a limitation on the specific configuration of the computer device. In other embodiments, the computer device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. For example, in some embodiments, the computer device may include only memory and a processor. In such embodiments, the structure and function of the memory and processor are different from those shown. Figure 6The embodiments shown are consistent and will not be repeated here.

[0142] It should be understood that in the embodiments of this application, the processor 720 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0143] In another embodiment of the present invention, a computer-readable storage medium is provided. This computer-readable storage medium may be a non-volatile computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the speech enhancement method or speech model training method disclosed in the embodiments of the present invention.

[0144] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0145] In the embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Units with the same function may be grouped into one unit. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, or may be electrical, mechanical, or other forms of connection.

[0146] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0147] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0148] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks.

[0149] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A speech enhancement method, characterized in that, include: The system receives noisy speech input and encodes the noisy speech using a pre-trained decomposition codec to obtain a hidden representation. By using a pre-trained semantic diffusion model to progressively denoise the hidden representation, a semantic token sequence corresponding to the clear speech is obtained. By combining the semantic token sequence with the hidden representation of noisy speech as a condition, and performing stepwise denoising through a pre-trained acoustic diffusion model, the acoustic token sequence corresponding to clear speech is obtained. The semantic token sequence and the acoustic token sequence are input into the decoder of the decomposer codec to reconstruct and generate a clear speech signal.

2. The speech enhancement method according to claim 1, characterized in that, The process of receiving noisy speech input and encoding the noisy speech using a pre-trained decomposition codec to obtain a hidden representation includes: Noisy speech data is downsampled to extract key speech features; By performing feature mapping on the downsampled key speech features through the neural network layer of the encoder, the time-domain speech signal is converted into a hidden representation in a high-dimensional feature space.

3. The speech enhancement method according to claim 1, characterized in that, The stepwise denoising process using a pre-trained semantic diffusion model to obtain a semantic token sequence corresponding to clear speech includes: The total number of time steps in the diffusion process is set. At each time step, the semantic diffusion model predicts the semantic tokens masked by the mask tokens based on the current semantic token sequence and the hidden representation of noisy speech. Update the current semantic token sequence based on the prediction results, replacing the mask tokens with the predicted clear semantic tokens; The process of repeatedly performing the semantic diffusion model, based on the current semantic token sequence and the hidden representation of noisy speech, predicting the semantic tokens masked by the mask tokens, and updating the current semantic token sequence based on the prediction results, replacing the mask tokens with the predicted clear semantic tokens, is repeated at each time step until the denoising process of all time steps is completed and the semantic token sequence corresponding to the clear speech is output.

4. The speech enhancement method according to claim 1, characterized in that, The step of combining the semantic token sequence with the hidden representation of noisy speech as a condition, and performing progressive denoising through a pre-trained acoustic diffusion model to obtain the acoustic token sequence corresponding to the clear speech, includes: The total time steps of the diffusion process are set. At each time step, the acoustic diffusion model predicts the acoustic tokens that are masked by the masked tokens based on the current acoustic token sequence, semantic tokens, and the hidden representation of noisy speech. The current acoustic token sequence is updated based on the prediction results, replacing the masked tokens with the predicted clear acoustic tokens, while retaining the unmasked acoustic tokens as the context for the next time step. The process of repeatedly performing the following steps at each time step, whereby the acoustic diffusion model predicts the acoustic tokens masked by the mask tokens based on the current acoustic token sequence, semantic tokens, and the hidden representation of noisy speech, and updates the current acoustic token sequence based on the prediction results, replacing the mask tokens with the predicted clear acoustic tokens, while retaining the unmasked acoustic tokens as the context for the next time step, continues until the denoising process for all time steps is completed, and the acoustic token sequence corresponding to the recovered clear speech is output.

5. The speech enhancement method according to claim 1, characterized in that, The step of feeding the semantic token sequence and acoustic token input sequence into the decoder of the decomposition codec to reconstruct and generate a clear speech signal includes: The decoder performs feature mapping on the input semantic token sequence, extracts the linguistic content information contained therein, and converts it into a semantic feature vector; The decoder performs feature mapping on the input acoustic token sequence, extracts the acoustic details contained therein, and converts them into acoustic feature vectors; The semantic feature vector and the acoustic feature vector are fused across modally through the fusion layer of the decoder to generate a joint feature representation containing complete speech attributes; The decoder's output layer performs waveform transformation on the joint feature representation, mapping the joint representation in the high-dimensional feature space to a time-domain speech signal, thus generating a clear speech signal.

6. A method for training a speech model, applied in the speech enhancement method according to any one of claims 1-5, wherein the method for training the speech model is used to train a decomposition codec, a semantic diffusion model, and an acoustic diffusion model, characterized in that, The method includes: Construct a decomposed codec to train a semantic vector quantizer and an acoustic vector quantizer using clean speech, so that the semantic vector quantizer outputs a semantic token corresponding to the language content and the acoustic vector quantizer outputs an acoustic token corresponding to the acoustic details. Using semantic tokens from noisy and clean speech as training data, the forward noise addition and backward noise reduction capabilities of the semantic diffusion model are trained, enabling the semantic diffusion model to generate clear semantic tokens from noisy speech. Using semantic tokens and acoustic tokens from noisy speech and clean speech as training data, the forward noise addition and backward noise reduction capabilities of the acoustic diffusion model are trained, enabling the acoustic diffusion model to generate clear acoustic tokens from noisy speech based on semantic tokens. The parameters of the decomposer, semantic diffusion model, and acoustic diffusion model are jointly optimized to minimize the difference loss between generated speech and clean speech.

7. A voice enhancement device, characterized in that, include: The receiving unit is used to receive noisy speech input, encode the noisy speech using a pre-trained decomposition codec, and obtain a hidden representation. The first estimation unit is used to perform stepwise denoising on the hidden representation using a pre-trained semantic diffusion model to obtain the semantic token sequence corresponding to the clear speech. The second estimation unit is used to combine the semantic token sequence with the hidden representation of noisy speech as a condition, and perform stepwise denoising through a pre-trained acoustic diffusion model to obtain the acoustic token sequence corresponding to clear speech. The reconstruction unit is used to input the semantic token sequence and the acoustic token sequence into the decoder of the decomposition codec to reconstruct and generate a clear speech signal.

8. A training device for a speech model, characterized in that, include: The building unit is used to build the decomposed codec to train the semantic vector quantizer and acoustic vector quantizer with clean speech, so that the semantic vector quantizer outputs semantic tokens corresponding to the language content and the acoustic vector quantizer outputs acoustic tokens corresponding to the acoustic details. The first training unit is used to train the forward noise addition and backward noise reduction capabilities of the semantic diffusion model using semantic tokens of noisy speech and clean speech as training data, so that the semantic diffusion model can generate clear semantic tokens from noisy speech. The second training unit is used to train the forward noise addition and backward noise reduction capabilities of the acoustic diffusion model using semantic tokens and acoustic tokens of noisy speech and clean speech as training data, so that the acoustic diffusion model can generate clear acoustic tokens from noisy speech based on semantic tokens. The optimization unit is used to jointly optimize the parameters of the decomposition codec, semantic diffusion model, and acoustic diffusion model to minimize the difference loss between generated speech and clean speech.

9. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the speech enhancement method as described in any one of claims 1 to 5 or the speech model training method as described in claim 6.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the processor performs the speech enhancement method as described in any one of claims 1 to 5 or the speech model training method as described in claim 6.

Images