Background-noise-aware speech editing method and non-transitory computer-readable medium

Abstract

A background noise-aware speech editing method and a non-transitory computer-readable medium are proposed. The method is suitable for execution on a computing device and includes: decomposing an input speech sequence into separate speech and background noise; adjusting the weights of at least one of multiple frequency bands of the separate speech to remove residual noise in the separate speech and generate a filtered speech; inputting the filtered speech and a mask to a first text-to-speech model to generate edited speech; inputting the edited speech and the separate speech to a second text-to-speech model having a multi-head attention module to update the edited speech; and after the edited speech is updated, synthesizing the edited speech with the background noise and outputting the result.

Claims

1. A background noise-aware speech editing method, suitable for execution by a computing device, and comprising: The input speech sequence is decomposed into separate speech and background noise using a diffusion-based stochastic reconstruction model. The weights of at least one of multiple frequency bands of the separated speech are adjusted using sparse Bayesian learning techniques to remove residual noise in the separated speech and generate a filtered speech; the filtered speech and a mask are input to a first text-to-speech model to generate edited speech, wherein the filtered speech includes multiple signals, and the first text-to-speech model is used to modify a portion of these signals according to the mask to generate the edited speech; the edited speech and the separated speech are input to a second text-to-speech model having a multi-head attention module to enhance the human voice features in the edited speech, wherein the multi-head attention module includes multiple attention heads, each of which includes a query vector, a key vector, and a value vector, and the second text-to-speech model is used to use the separated speech as input to the query vector, and the edited speech as input to the key vector and the value vector, and to update the edited speech according to the query vector, the key vector, and the value vector using the multi-head attention module; and after the edited speech is updated, the edited speech is synthesized with the background noise and output.

2. A non-transitory computer-readable medium for storing a plurality of instructions, which, when executed by a computing device, cause a plurality of operations, the operations including: The input speech sequence is decomposed into separate speech and background noise using a diffusion-based stochastic reconstruction model. The weights of at least one of multiple frequency bands of the separated speech are adjusted using sparse Bayesian learning techniques to remove residual noise in the separated speech and generate a filtered speech; the filtered speech and a mask are input to a first text-to-speech model to generate edited speech, wherein the filtered speech includes multiple signals, and the first text-to-speech model is used to modify a portion of these signals according to the mask to generate the edited speech; the edited speech and the separated speech are input to a second text-to-speech model having a multi-head attention module to enhance the human voice features in the edited speech, wherein the multi-head attention module includes multiple attention heads, each of which includes a query vector, a key vector, and a value vector, and the second text-to-speech model is used to use the separated speech as input to the query vector, and the edited speech as input to the key vector and the value vector, and to update the edited speech according to the query vector, the key vector, and the value vector using the multi-head attention module; and after the edited speech is updated, the edited speech is synthesized with the background noise and output.

Images