Voice signal denoising method and device, electronic equipment and program product

Abstract

The invention discloses a voice signal denoising method and device, electronic equipment and a program product, and relates to the technical field of artificial intelligence, and the denoising method comprises the steps: collecting voice data of a target user based on a preset collection strategy, and carrying out the denoising of the voice data through employing a preset back diffusion model, and obtaining initial voice data; inputting the initial voice data into a preset dual-path recursive network, and outputting a voice feature sequence; generating a time frequency mask for the speech feature sequence, and determining a target speech spectrum based on the time frequency mask; and based on the target voice spectrum, performing voice waveform reconstruction by adopting inverse short-time Fourier transform to obtain a target voice signal. According to the invention, the technical problem of low speech recognition accuracy under multi-noise and multi-speaker interference in the prior art is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a speech signal denoising method and device, electronic equipment and program product. BACKGROUND

[0002] With the continuous growth of the number of air passengers, manual check-in services face great pressure during peak hours, and there are problems such as low efficiency and high cost. In order to improve the check-in efficiency and reduce the operation cost of manual services, the airport introduces a self-service check-in system. This kind of system can complete the process of boarding pass printing and seat selection through passenger self-operation, and improve the efficiency of passing. In the self-service check-in mode, voice interaction as a natural and intuitive human-computer interaction mode has good aging characteristics. Especially for elderly passengers or users who are not familiar with the operation process, voice interaction can provide a more convenient and friendly operation experience.

[0003] However, the complex environmental noise of the airport, such as continuous background white noise, system broadcast sound, mutual interference between multiple check-in counters, and frequent human voice interference, affects the accurate capture of the target voice by the system and reduces the voice recognition accuracy, thereby reducing the check-in efficiency and affecting the system usability.

[0004] At present, there is no effective solution to the above problems. SUMMARY

[0005] The embodiments of the present application provide a speech signal denoising method and device, electronic equipment and program product to at least solve the technical problem of low voice recognition accuracy in related technologies under the interference of multiple noises and multiple speakers.

[0006] According to an aspect of an embodiment of the present application, a speech signal denoising method is provided, comprising: collecting speech data of a target user based on a preset collection strategy, and denoising the speech data using a preset reverse diffusion model to obtain initial speech data; inputting the initial speech data into a preset double-path recurrent network to output a speech feature sequence; generating a time-frequency mask for the speech feature sequence, and determining a target voice spectrum based on the time-frequency mask; and performing voice waveform reconstruction using inverse short-time Fourier transform based on the target voice spectrum to obtain a target voice signal.

[0007] Further, before collecting the speech data of the target user based on the preset collection strategy, it further comprises: collecting a plurality of environmental noise data of a preset area, and collecting a plurality of preset speech data; mixing the environmental noise data and the preset speech data to generate a plurality of synthesized speech samples; labeling each synthesized speech sample to obtain labeled synthesized speech samples, and constructing a training data set based on all labeled synthesized speech samples.

[0008] Further, after constructing the training data set based on all the labeled synthesized speech samples, further comprising: constructing an initial reverse diffusion model, wherein the structure of the initial reverse diffusion model at least includes: a forward noise model and a reverse denoising model, the forward noise model is used to add noise to the human voice to obtain noisy speech, and the reverse denoising model is used to denoise the noisy speech to obtain human voice data; training the initial reverse diffusion model based on the training data set to obtain a preset reverse diffusion model.

[0009] Further, before inputting the initial speech data into the preset double-path recurrent network to output the speech feature sequence, further comprising: normalizing the initial speech data to obtain processed audio data; performing sliding window processing on the processed audio data to obtain a plurality of frame data; encoding each frame data to obtain a feature code, and constructing an encoding sequence based on all the feature codes.

[0010] Further, the structure of the preset double-path recurrent network at least includes: a first preset bidirectional recurrent network and a second preset bidirectional recurrent network, and the step of inputting the initial speech data into the preset double-path recurrent network to output the speech feature sequence includes: dividing the encoding sequence into a plurality of sub-encoding sequences according to a preset length; inputting each sub-encoding sequence into the first preset bidirectional recurrent network for feature extraction to obtain a first feature vector, and fusing the sub-encoding sequence with the first feature vector based on a residual structure to obtain a target feature vector; splicing all the target feature vectors to obtain a feature sequence; inputting the feature sequence into the second preset bidirectional recurrent network for feature extraction to obtain a second feature vector, and fusing the feature sequence with the second feature vector based on a residual structure to output the speech feature sequence.

[0011] Further, the step of generating a time-frequency mask for the speech feature sequence includes: obtaining a preset reference speech data, and performing feature extraction on the preset reference speech data to generate a semantic query vector; calculating the similarity between each speech feature in the speech feature sequence and the semantic query vector based on a preset attention mechanism, and assigning a preset weight to the speech feature based on the similarity; generating a time-frequency mask based on the speech feature sequence and all the weights.

[0012] Further, based on the time-frequency mask, the step of determining the target speech spectrum includes: performing short-time Fourier transform on the speech data to obtain a frequency domain speech signal; multiplying the frequency domain speech signal and the time-frequency mask to obtain the target speech spectrum.

[0013] According to another aspect of the embodiments of the present application, a device for de-noising a voice signal is also provided, which comprises: a de-noising unit configured to collect voice data of a target user based on a preset collection strategy, and to de-noise the voice data using a preset reverse diffusion model to obtain initial voice data; a processing unit configured to input the initial voice data into a preset double-path recurrent network to output a voice feature sequence; a determination unit configured to generate a time-frequency mask for the voice feature sequence, and to determine a target voice spectrum based on the time-frequency mask; and a reconstruction unit configured to perform voice waveform reconstruction using an inverse short-time Fourier transform based on the target voice spectrum to obtain a target voice signal.

[0014] Further, the device for de-noising a voice signal comprises: a first collection module configured to collect a plurality of environmental noise data of a preset area and a plurality of preset voice data before collecting voice data of a target user based on a preset collection strategy; a first generation module configured to mix the environmental noise data and the preset voice data to generate a plurality of synthesized voice samples; and a first construction module configured to label each synthesized voice sample to obtain labeled synthesized voice samples, and to construct a training data set based on all the labeled synthesized voice samples.

[0015] Further, the device for de-noising a voice signal further comprises: a second construction module configured to construct an initial reverse diffusion model based on all the labeled synthesized voice samples after constructing the training data set, wherein the structure of the initial reverse diffusion model at least comprises: a forward noise model configured to add noise to a human voice to obtain a noisy voice, and a reverse de-noising model configured to de-noise the noisy voice to obtain human voice data; and a first training module configured to train the initial reverse diffusion model based on the training data set to obtain a preset reverse diffusion model.

[0016] Further, the device for de-noising a voice signal further comprises: a first processing module configured to perform normalization processing on the initial voice data to obtain processed audio data before inputting the initial voice data into the preset double-path recurrent network to output the voice feature sequence; a second processing module configured to perform sliding window processing on the processed audio data to obtain a plurality of frame data; a first encoding module configured to encode each frame data to obtain a feature code, and to construct an encoding sequence based on all the feature codes.

[0017] Further, the structure of the preset bidirectional recurrent network comprises at least: a first preset bidirectional recurrent network and a second preset bidirectional recurrent network, and the processing unit comprises: a first division module configured to divide the coding sequence according to a preset length to obtain a plurality of sub-coding sequences; a first fusion module configured to input each sub-coding sequence into the first preset bidirectional recurrent network to extract features to obtain a first feature vector, and fuse the sub-coding sequence and the first feature vector based on a residual structure to obtain a target feature vector; a first splicing module configured to splice all target feature vectors to obtain a feature sequence; and a second fusion module configured to input the feature sequence into the second preset bidirectional recurrent network to extract features to obtain a second feature vector, and fuse the feature sequence and the second feature vector based on the residual structure to output a speech feature sequence.

[0018] Further, the determining unit comprises: a first extraction module configured to obtain preset reference speech data and perform feature extraction on the preset reference speech data to generate a semantic query vector; a first calculation module configured to calculate the similarity between each speech feature in the speech feature sequence and the semantic query vector based on a preset attention mechanism, and assign a preset weight to the speech feature based on the similarity; and a second generation module configured to generate a time-frequency mask based on the speech feature sequence and all weights.

[0019] Further, the determining unit further comprises: a first transformation module configured to perform short-time Fourier transform on the speech data to obtain a frequency domain speech signal; and a second calculation module configured to perform product calculation on the frequency domain speech signal and the time-frequency mask to obtain a target speech spectrum.

[0020] According to another aspect of the embodiment of the present application, a computer program product is also provided, comprising a non-volatile computer readable storage medium, the non-volatile computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the denoising method of the speech signal of any one of the above.

[0021] According to another aspect of the embodiment of the present application, an electronic device is also provided, comprising one or more processors and a memory, the memory being configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the denoising method of the speech signal of any one of the above.

[0022] In the present application, based on the preset acquisition strategy, the voice data of the target user is collected, and the preset reverse diffusion model is used to denoise the voice data to obtain initial voice data; the initial voice data is input into the preset double-path recurrent network, and the voice feature sequence is output; the time-frequency mask is generated for the voice feature sequence, and the target voice spectrum is determined based on the time-frequency mask; based on the target voice spectrum, the inverse short-time Fourier transform is used for voice waveform reconstruction to obtain the target voice signal, thereby solving the technical problem of low voice recognition accuracy in the related art under the interference of multiple noises and multiple speakers.

[0023] In the present application, according to the preset acquisition strategy, the voice data of the target user in the complex environment is first captured in real time, then the pre-trained reverse diffusion model is used to denoise the collected voice data, which can effectively eliminate background noise and obtain relatively pure initial voice data, then the initial voice data is input as input into the preset double-path recurrent neural network for feature extraction and modeling, the network outputs the feature sequence closely related to the target voice through local and global time sequence modeling, the sequence includes the voice structure information of the target user, based on the above feature sequence, the time-frequency mask is generated through the attention mechanism, the mask can accurately distinguish the target voice from the non-target voice, thereby positioning and screening the target voice in the spectral domain, then the generated time-frequency mask is used to perform point-by-point multiplication operation on the spectral representation of the initial voice data to determine the clear target voice spectrum, then the inverse short-time Fourier transform is used to reconstruct the time-domain waveform of the target voice spectrum, outputting the high-quality and pure target voice signal, realizing the voice denoising and enhancement in the complex environment of the airport, and improving the recognition accuracy of the target voice under the interference of multiple speakers and background noise. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings, which are included to provide a further understanding of the present application, form a part of the present application and illustrate the illustrative embodiments of the present application and together with the description, serve to explain the present application. In the drawings: Figure 1 is a flowchart of an optional voice signal denoising method according to an embodiment of the present application; Figure 2 is a flowchart of an optional voice signal reconstruction according to an embodiment of the present application; Figure 3 is a flowchart of an optional noise modeling and reverse diffusion according to an embodiment of the present application; Figure 4 is a schematic diagram of an optional local bidirectional recurrent-global bidirectional recurrent model structure according to an embodiment of the present application; Figure 5is a flow chart of an optional voice de-noising of a check-in instruction according to an embodiment of the present application; Figure 6 is a schematic diagram of an optional voice signal de-noising device according to an embodiment of the present application; Figure 7 is a hardware structure block diagram of an electronic device (or mobile device) for a voice signal de-noising method according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the personnel in the technical field better understand the present application scheme, the technical scheme in the embodiment of the present application will be described clearly and completely below in combination with the drawings in the embodiment of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by the person skilled in the art without creative labor should belong to the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0027] It should be noted that the relevant information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) collected and involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with relevant laws, regulations and standards in relevant regions, necessary security measures are taken, do not violate public order and good customs, and provide corresponding operation portal for user to choose authorization or refusal. For example, the system and the interface between the relevant users or institutions are provided, and before obtaining the relevant information, the interface needs to send an acquisition request to the aforementioned user or institution, and after receiving the consent information feedback from the aforementioned user or institution, the relevant information is acquired.

[0028] In the present application, the reverse diffusion model is combined with the double-path recurrent network to realize efficient separation and enhancement of the complex noise environment and multi-speaker mixed speech in the airport. Through modeling of the data set of multiple types of background noise and speaker interference, the noise characteristics in the airport environment can be accurately captured, the environmental interference can be effectively suppressed, and the intelligibility and signal-to-noise ratio of the speech signal can be greatly improved. The local-global timing structure modeling and target speech focusing mechanism can also be used to enhance the recognition and separation ability of the target speaker's speech, ensure the high fidelity and natural fluency of the separated speech, effectively improve the accuracy of speech recognition and user interaction experience, and promote the application of intelligent speech interaction technology in self-service check-in, intelligent terminals and human-computer cooperation fields.

[0029] The present application will be described in detail below in conjunction with various embodiments.

[0030] Embodiment one According to the embodiments of the present application, an embodiment of a speech signal denoising method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in a different order.

[0031] Figure 1 is a flowchart of an optional speech signal denoising method according to an embodiment of the present application, as shown in Figure 1 The method comprises the following steps: Step S101, based on a preset acquisition strategy, acquiring speech data of a target user, and using a preset reverse diffusion model to denoise the speech data to obtain initial speech data.

[0032] Optionally, based on a preset acquisition strategy (i.e. a trigger mechanism, including a wake-up word trigger, a physical button trigger, etc.), the speech data of the target user in the airport environment can be collected in real time by the sound collecting equipment arranged at the check-in terminal, self-service counter, etc. to ensure that the speech collection is accurately started at the moment when the target user initiates the interaction, thereby effectively avoiding the interference of non-target speech and improving the effectiveness and accuracy of the collected data.

[0033] In this embodiment, a preset reverse diffusion model (a deep learning model for removing background noise and non-target speech interference in a speech signal, which gradually restores the speech signal through a reverse process to remove noise components) is used to denoise the speech data, improve the intelligibility of the signal, and obtain initial speech data (the speech signal after denoising).

[0034] Step S102, inputting the initial speech data into a preset double-path recurrent network to output a speech feature sequence.

[0035] In the embodiment, the initial speech data is input into a preset dual-path recurrent network for capturing local and global dependencies of the speech signal in time, which at least includes a local path network (i.e., a first preset bidirectional recurrent network for modeling feature dependencies in a short time) and a global path network (i.e., a second preset bidirectional recurrent network for modeling long-time context information), and a speech feature sequence can be output.

[0036] In step S103, a time-frequency mask is generated for the speech feature sequence, and a target speech spectrum is determined based on the time-frequency mask.

[0037] In the embodiment, an attention mechanism can be used to assign a weight feature to each speech feature in the speech feature sequence. The weighted feature representation generated by the attention mechanism for the speech feature sequence can generate a set of time-frequency masks corresponding to the input mixed speech (i.e., the speech data of the target user), which represents the degree of signal preservation. A speech feature with a higher weight indicates a greater degree of preservation, while a speech feature with a lower weight indicates a smaller degree of preservation. The time-frequency mask can be used to weight and filter the spectral representation of the mixed speech point by point to preserve the target speech frequency domain component while suppressing the interfering speech and noise components, thereby extracting the spectral information of the target speech from the noise and interference. This can improve the clarity of the target speech and reduce background noise and interference from other speakers.

[0038] In step S104, based on the target speech spectrum, an inverse short-time Fourier transform is used to reconstruct the speech waveform to obtain the target speech signal.

[0039] In the embodiment, the inverse short-time Fourier transform is the inverse process of the short-time Fourier transform, which is used to convert the signal in the frequency domain back to the time domain to recover the waveform of the speech. The inverse short-time Fourier transform can be used on the target speech spectrum to perform phase recovery and time-domain waveform reconstruction to obtain a target speech signal with the same length as the original mixed speech. To improve the audibility and naturalness of the reconstructed audio, a phase consistency correction mechanism can also be introduced to use the original phase information of the mixed speech as a reference to ensure the coherence of the target speech waveform in terms of speech boundaries and prosodic structure.

[0040] Optionally, a loss function including a spectral loss and a waveform reconstruction error can also be used to improve the reconstruction quality of the target speech through joint optimization (i.e., considering and minimizing both the spectral loss and the waveform reconstruction error at the same time).

[0041] Figure 2 is a flowchart of an optional speech signal reconstruction according to an embodiment of the present application, as Figure 2As shown, the target speech vector can be subjected to speech vector decoding (i.e., a time-frequency mask is generated for the speech feature sequence, and based on the time-frequency mask, a target speech spectrum is determined, and based on the target speech spectrum, speech waveform reconstruction is performed using an inverse short-time Fourier transform), and reconstructed into a continuous speech sequence (i.e., a target speech signal, N representing the feature dimension thereof, and L representing the speech sequence length thereof), ensuring smooth transition and seamless connection of the speech signal at the boundary.

[0042] Optionally, the above scheme is not only applicable to the complex environment of the airport, but also applicable to other complex scenes with high noise and multiple speaker voices.

[0043] In summary, by combining speech data collection under a preset collection strategy, accurate denoising processing of a preset reverse diffusion model, generation of an encoding sequence of target speech features using a preset double-path recurrent network, and accurate focusing and separation of the target speech based on a time-frequency mask generated based on an attention mechanism, high-quality denoising and enhancement of the check-in instruction speech signal in the complex environment of the airport are achieved, the accuracy of target speech extraction is improved, then based on the time-frequency mask, the target speech spectrum is determined, and based on the target speech spectrum, high-fidelity reconstruction of the speech waveform is realized through the inverse short-time Fourier transform, to obtain the target speech signal, thereby solving the technical problem of low speech recognition accuracy in the related art under the interference of multiple noises and multiple speakers.

[0044] In order to accurately construct the training data set, in the denoising method of the speech signal provided in Embodiment One of the present application, a plurality of environmental noise data of a preset area are collected, and a plurality of preset speech data are collected; the environmental noise data and the preset speech data are mixed to generate a plurality of synthesized speech samples; each synthesized speech sample is labeled to obtain a labeled synthesized speech sample, and based on all the labeled synthesized speech samples, a training data set is constructed.

[0045] In this embodiment, in order to enhance the adaptability of the speech processing model to the real airport noise environment, the airport environment speech data set can be constructed by a noise synthesis method. First, environmental noise samples (i.e., environmental noise data) containing multiple sources of interference such as broadcasts, crowd conversations, and equipment operation can be collected in the check-in area, the boarding area, and other airport areas. Then, different amounts of clear human voice corpus (i.e., preset speech data) are mixed with the collected environmental noise samples in different proportions to generate synthesized speech samples covering multiple signal-to-noise ratio conditions, to truly restore the airport speech interaction scene. Then, the synthesized speech samples are structurally labeled to record key information such as noise type and signal-to-noise ratio level, forming a standardized airport environment speech data set (i.e., constructing a training data set) for subsequent model training and evaluation.

[0046] In order to accurately obtain the preset back diffusion model, in the speech signal denoising method provided in Embodiment 1 of this application, an initial back diffusion model is constructed. The structure of the initial back diffusion model includes at least a forward noise model and a reverse denoising model. The forward noise model is used to add noise to human speech to obtain noisy speech, and the reverse denoising model is used to denoise the noisy speech to obtain human speech data. The initial back diffusion model is trained based on the training data set to obtain the preset back diffusion model.

[0047] In this embodiment, the initial backdiffusion model includes at least a forward noise model and a reverse denoising model. The airport noise fusion model (i.e., the forward noise model) simulates the perturbation process of speech signals by complex multi-source noise in an airport environment. This process describes the dynamic trajectory of a clean speech signal gradually degenerating into noisy speech in a real airport context by constructing a noise evolution mechanism under time conditions, enabling a controllable reverse restoration process. The reverse denoising model, based on the constructed forward noise modeling mechanism, learns a conditional probability mapping for gradually restoring the original clean speech signal from any noisy speech state. This effectively removes perturbations such as reverberation and background noise from the speech signal to denoise the noisy speech, obtaining human voice data. Then, based on the training dataset, the initial backdiffusion model can be trained to obtain a preset backdiffusion model (a pre-constructed model).

[0048] Figure 3 This is a flowchart of an optional noise modeling and backdiffusion method according to an embodiment of the present invention, such as... Figure 3 As shown, through forward diffusion in clean speech Adding noise in multiple steps Obtain noisy speech Then, backpropagation can be used to utilize the trainable parameters. The neural network model constituted ( (i.e., the pre-defined backdiffusion model) for noisy speech Denoising (i.e., using) Iterative denoising yields clean speech. .

[0049] In order to accurately construct the encoding sequence, in the speech signal denoising method provided in Embodiment 1 of this application, the initial speech data is normalized to obtain processed audio data; the processed audio data is subjected to sliding window processing to obtain multiple frame data; each frame data is encoded to obtain feature codes, and an encoding sequence is constructed based on all feature codes.

[0050] Optionally, the mixed speech signal (i.e., the initial speech data) can be segmented and feature encoded by an encoder to realize the mapping of the original time-domain signal to a high-dimensional feature space, providing a more expressive input representation for subsequent separation modeling.

[0051] In this embodiment, normalization processing can be performed on the initial speech data to limit its amplitude range to a predefined interval, thereby eliminating amplitude differences between different audios. In addition, according to application requirements, the silent segments in the audio can be optionally removed to improve processing efficiency, and an overlapping sliding window mechanism can be used to perform frame-level segmentation on the audio signal (i.e., the processed audio data). The window length L and the step size S can be set, and the sliding window with a length of L is used to slide on the audio signal with a step size of S to generate a plurality of overlapping frame segments (i.e., frame data). The length T of the speech signal is divided into N frame segments , each frame segment includes consecutive L sample points for subsequent encoding operations. Each frame segment is input into an encoder module, and the encoder includes at least one one-dimensional convolutional neural network with a convolution kernel size equal to the frame segment length and a step size set to the frame shift value S. The convolution operation is performed on each frame segment, and a nonlinear activation function is used to extract local speech features. The encoding results of all frame segments constitute a feature vector representation , where represents the feature vector of the Nth frame segment, N = 1, …, N, and D represents the encoding output dimension, with a specific value depending on parameter settings. represents a feature sequence composed of N speech frames, each frame being encoded as a D-dimensional feature vector that maintains the time sequence order of the speech signal, so that the semantic information of each frame is continuously distributed in the time dimension.

[0052] The structure of the preset double-path recurrent network at least includes: a first preset bidirectional recurrent network and a second preset bidirectional recurrent network. In the speech signal denoising method provided in Embodiment One of the present application, the encoding sequence is divided into a plurality of sub-encoding sequences according to a preset length; each sub-encoding sequence is input into the first preset bidirectional recurrent network for feature extraction to obtain a first feature vector, and based on a residual structure, the sub-encoding sequence and the first feature vector are fused to obtain a target feature vector; all target feature vectors are spliced to obtain a feature sequence; the feature sequence is input into the second preset bidirectional recurrent network for feature extraction to obtain a second feature vector, and based on a residual structure, the feature sequence and the second feature vector are fused to output a speech feature sequence.

[0053] In the embodiment, the structure of the preset bidirectional recurrent network at least includes: a first preset bidirectional recurrent network and a second preset bidirectional recurrent network. The encoding sequence can be divided into a plurality of local segments (i.e., sub-encoding sequences) with a preset length (e.g., a time length T) and a window sliding step s, and can be represented as wherein, s is the window index, and S is the total number of segments after division. Each local segment is then input into a local bidirectional recurrent network (i.e., the first preset bidirectional recurrent network) to extract local context dependencies, capture dynamic information and feature dependencies in a short time scale, and enhance the time consistency and discrimination ability of the speech representation. A first feature vector can be obtained, and the sub-encoding sequence and the first feature vector can be fused based on a residual structure to obtain a target feature vector All target feature vectors are spliced to obtain a feature sequence . Then, the feature sequence is input into a global bidirectional recurrent network (i.e., the second preset bidirectional recurrent network) to capture semantics and speaker continuity in a long time scale, and a second feature vector can be obtained, and the feature sequence and the second feature vector can be fused based on a residual structure to output a speech feature sequence , thereby realizing modeling of the speech signal in different time scales.

[0054] Figure 4 is a schematic diagram of an optional local bidirectional recurrent-global bidirectional recurrent model structure according to an embodiment of the present application, as shown in Figure 4 The model includes a local processing module and a global processing module. The local processing module can perform feature extraction on the input sequence and fuse the input sequence and the extracted features through a residual link to obtain a target feature vector. The global processing module can perform further feature extraction on the target feature vector and fuse the target feature vector and the extracted features through a residual link to output a sequence (i.e., a feature sequence). The network structure of the local processing module includes a Bi-RNN (Bidirectional Recurrent Neural Network), an FC (Fully Connected Layer), and a layerNorm (Layer Normalization). The network structure of the global processing module includes a (Bi)-RNN, an FC, and a layerNorm.

[0055] In order to accurately generate the time-frequency mask, in the speech signal denoising method provided in Embodiment One of the present application, preset reference speech data is obtained, and feature extraction is performed on the preset reference speech data to generate a semantic query vector; based on a preset attention mechanism, the similarity of each speech feature in the speech feature sequence to the semantic query vector is calculated, and based on the similarity, a preset weight is assigned to the speech feature; and based on the speech feature sequence and all the weights, the time-frequency mask is generated.

[0056] In the present embodiment, the preset reference speech data (i.e. data including the voice of the target user, used to guide the model to focus on the features of the target speaker) can be preprocessed in the same way as the mixed speech (i.e. the speech data of the target user) to obtain a feature sequence of the preset reference speech, the feature sequence is aggregated by a voiceprint encoder to extract a fixed-dimension vector (i.e. a semantic query vector q) including speaker personalized features (such as tone, pronunciation habits, etc.), and based on a preset attention mechanism (i.e. an additive attention mechanism), the similarity of each speech feature in the speech feature sequence to the semantic query vector is calculated, and based on the similarity, a preset weight is assigned to the speech feature wherein, , , is a learnable parameter matrix, and tanh() is a hyperbolic tangent activation function, represents the speech feature of the i-th frame segment, represents the speech feature of the n-th frame segment, the model automatically tilts attention to the frame related to the target speech when processing the mixed speech feature, avoiding investing too many modeling resources in the interfering speech or noise frames, and then, based on the speech feature sequence and all the weights, the time-frequency mask is generated.

[0057] In order to accurately obtain the target speech spectrum, in the speech signal denoising method provided in Embodiment One of the present application, short-time Fourier transform is performed on the speech data to obtain a frequency domain speech signal; the frequency domain speech signal is multiplied by the time-frequency mask to obtain the target speech spectrum.

[0058] In the present embodiment, the time-frequency mask can be in the form of an amplitude mask rather than a binary mask, and the element-by-element multiplication operation in the frequency domain with the mixed speech (i.e. the frequency domain speech signal X is multiplied by the time-frequency mask M to obtain the target speech spectrum (i.e. )) is performed to realize spectrum-level speech enhancement, wherein the frequency domain speech signal can be obtained by performing short-time Fourier transform on the speech data.

[0059] Figure 5 is a flowchart of an optional value machine instruction speech denoising according to an embodiment of the present application, as Figure 5As shown, the voice instruction of the user is collected through the microphone device to obtain voice data, in the voice denoising stage, the denoised audio data can be obtained through the steps of airport environment data construction, noise process modeling construction and reverse diffusion voice denoising, then in the voice separation stage, the high-fidelity target voice signal can be obtained through the steps of voice segmentation and coding, voice structure modeling (that is, constructing a preset double-path recurrent network), voice target focusing (that is, a preset attention mechanism, calculating the similarity of each voice feature in the voice feature sequence and the semantic query vector, and based on the similarity, assigning a preset weight to the voice feature) and target voice signal reconstruction.

[0060] In the embodiment of the application, first, the voice signal is normalized and divided by a sliding window to obtain a plurality of frame data, then a deep learning model is used to fuse and model the noise, and a high-precision denoising is realized based on a reverse diffusion process to restore the intelligibility of the target voice, then a double-path network, that is, a local and global bidirectional recurrent network, is used to extract features from the denoised voice features to obtain an enhanced voice feature sequence, at the same time, a semantic query vector and an attention mechanism are used to identify and highlight the voice signal of the target user and generate a time-frequency mask to further filter out interference, then frequency domain signal processing and mask product calculation are used to effectively separate the target voice spectrum, and inverse short-time Fourier transform can be used to reconstruct the voice waveform to obtain the target voice signal, thereby improving the accuracy of voice recognition in a complex noise environment in an airport.

[0061] The following will be described in detail in conjunction with another embodiment.

[0062] Embodiment Two The voice signal denoising device provided in the embodiment comprises a plurality of implementation units, each of which corresponds to each implementation step in the above embodiment one.

[0063] Figure 6 is a schematic diagram of an optional voice signal denoising device according to the embodiment of the application, as Figure 6 shown, the voice signal denoising device can comprise a denoising unit 60, a processing unit 61, a determination unit 62 and a reconstruction unit 63.

[0064] The denoising unit 60 is configured to collect voice data of a target user based on a preset collection strategy, and denoise the voice data using a preset reverse diffusion model to obtain initial voice data. The processing unit 61 is configured to input the initial voice data into a preset double-path recurrent network to output a voice feature sequence. The determination unit 62 is configured to generate a time-frequency mask for the voice feature sequence, and determine a target voice spectrum based on the time-frequency mask. The reconstruction unit 63 is configured to perform speech waveform reconstruction on the target speech spectrum based on the inverse short-time Fourier transform to obtain a target speech signal.

[0065] The speech signal denoising device can collect speech data of the target user based on the preset collection strategy through the denoising unit 60, and perform denoising on the speech data based on the preset reverse diffusion model to obtain initial speech data. The initial speech data can be input into the preset double-path recurrent network through the processing unit 61 to output a speech feature sequence. The determination unit 62 can generate a time-frequency mask for the speech feature sequence, and determine the target speech spectrum based on the time-frequency mask. The reconstruction unit 63 can perform speech waveform reconstruction on the target speech spectrum based on the inverse short-time Fourier transform to obtain a target speech signal.

[0066] Optionally, the speech signal denoising device comprises: a first collection module, configured to collect a plurality of environmental noise data of a preset area and a plurality of preset speech data before collecting speech data of a target user based on a preset collection strategy; a first generation module, configured to mix the environmental noise data and the preset speech data to generate a plurality of synthesized speech samples; and a first construction module, configured to label each synthesized speech sample to obtain a labeled synthesized speech sample, and construct a training data set based on all the labeled synthesized speech samples.

[0067] Optionally, the speech signal denoising device further comprises: a second construction module, configured to construct an initial reverse diffusion model based on all the labeled synthesized speech samples after constructing the training data set, wherein the structure of the initial reverse diffusion model at least comprises: a forward noise model and a reverse denoising model, the forward noise model is configured to add noise to a human voice to obtain a noisy speech, and the reverse denoising model is configured to denoise the noisy speech to obtain human voice data; and a first training module, configured to train the initial reverse diffusion model based on the training data set to obtain a preset reverse diffusion model.

[0068] Optionally, the speech signal denoising device further comprises: a first processing module, configured to perform normalization processing on the initial speech data to obtain processed audio data before inputting the initial speech data into the preset double-path recurrent network to output a speech feature sequence; a second processing module, configured to perform sliding window processing on the processed audio data to obtain a plurality of frame data; and a first encoding module, configured to encode each frame data to obtain a feature code, and construct an encoding sequence based on all the feature codes.

[0069] Optionally, the structure of the preset bidirectional recurrent network comprises at least: a first preset bidirectional recurrent network and a second preset bidirectional recurrent network, the processing unit 61 comprises: a first division module configured to divide the coding sequence into a plurality of sub-coding sequences according to a preset length; a first fusion module configured to input each sub-coding sequence into the first preset bidirectional recurrent network to extract features, obtain a first feature vector, and fuse the sub-coding sequence and the first feature vector based on a residual structure to obtain a target feature vector; a first splicing module configured to splice all target feature vectors to obtain a feature sequence; and a second fusion module configured to input the feature sequence into the second preset bidirectional recurrent network to extract features, obtain a second feature vector, and fuse the feature sequence and the second feature vector based on the residual structure to output a speech feature sequence.

[0070] Optionally, the determination unit 62 comprises: a first extraction module configured to obtain preset reference speech data and extract features of the preset reference speech data to generate a semantic query vector; a first calculation module configured to calculate a similarity between each speech feature in the speech feature sequence and the semantic query vector based on a preset attention mechanism, and assign a preset weight to the speech feature based on the similarity; and a second generation module configured to generate a time-frequency mask based on the speech feature sequence and all weights.

[0071] Optionally, the determination unit 62 further comprises: a first transformation module configured to perform a short-time Fourier transform on the speech data to obtain a frequency domain speech signal; and a second calculation module configured to perform a product calculation on the frequency domain speech signal and the time-frequency mask to obtain a target speech spectrum.

[0072] The above-mentioned speech signal denoising device can further comprise a processor and a memory, and the above-mentioned denoising unit 60, the processing unit 61, the determination unit 62, the reconstruction unit 63 and the like are stored in the memory as program units, and the above-mentioned program units stored in the memory are executed by the processor to realize corresponding functions.

[0073] The above-mentioned processor comprises a core, and the core retrieves corresponding program units from the memory. The core can be set to one or more, and the target speech spectrum is reconstructed by performing an inverse short-time Fourier transform on the target speech spectrum by adjusting the core parameters.

[0074] The above-mentioned memory can comprise a non-permanent memory in a computer readable medium, a random access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM), and the memory comprises at least one memory chip.

[0075] According to another aspect of the embodiments of the present application, there is also provided a computer program product comprising a non-transitory computer readable medium storing a computer program, the computer program, when executed by a processor, implementing the denoising method of a speech signal of any of the above.

[0076] When the computer program product is executed on the data processing device, the program initialized with the following method steps is adapted to be executed: based on a preset acquisition strategy, acquiring speech data of a target user, and using a preset back diffusion model to denoise the speech data to obtain initial speech data; inputting the initial speech data into a preset double-path recurrent network to output a speech feature sequence; generating a time-frequency mask for the speech feature sequence, and determining a target speech spectrum based on the time-frequency mask; based on the target speech spectrum, using inverse short-time Fourier transform to reconstruct a speech waveform to obtain a target speech signal.

[0077] According to another aspect of the embodiments of the present application, there is also provided an electronic device comprising one or more processors and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the denoising method of a speech signal described above.

[0078] Figure 7 is a hardware structure block diagram of an electronic device (or a mobile device) for a denoising method of a speech signal according to an embodiment of the present application. As shown in Figure 7 , the electronic device can include one or more processors (for example, processors 702a, 702b, …, 702n, etc. in Figure 7 , which can include but are not limited to processing devices such as microprocessors MCU or programmable logic devices FPGA, etc.), a memory 704 for storing data. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply and / or a camera. Those skilled in the art can understand that Figure 7 the structure shown is only schematic, and does not limit the structure of the above-mentioned electronic device. For example, the electronic device can also include more or fewer components than those shown in Figure 7 , or have a different configuration from Figure 7 .

[0079] The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0080] The embodiments or examples of the present disclosure are not exhaustive, and are only a part of the embodiments or examples, and are not specific limitations on the protection scope of the present disclosure. Each step in a certain embodiment or example can be implemented as an independent example, and the steps can be combined arbitrarily, for example, a scheme after removing some steps in a certain embodiment or example can also be implemented as an independent example, and the order of the steps in a certain embodiment or example can be exchanged arbitrarily, in addition, the optional ways or optional examples in a certain embodiment or example can be combined arbitrarily; in addition, the embodiments or examples can be combined arbitrarily, for example, the steps of different embodiments or examples can be combined arbitrarily, a certain embodiment or example can be combined with the optional ways or optional examples of other embodiments or examples.

[0081] In the above-described embodiments of the present disclosure, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0082] In the several embodiments of the present disclosure, it should be understood that the disclosed technology can be implemented in other ways. The above-described device embodiments are only illustrative, for example, the division of the units can be a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, units or modules, and can be electrical or other forms.

[0083] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment scheme.

[0084] In addition, each functional unit in each embodiment of the present disclosure can be integrated in a processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0085] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0086] The above is only the preferred embodiment of the present application, it should be pointed out that, for those skilled in the art, without departing from the principles of the present application, can make a number of improvements and refinements, these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A method of de-noising a speech signal, characterized by, The method comprises the following steps: Based on a preset acquisition strategy, the voice data of the target user is collected, and a preset reverse diffusion model is used to denoise the voice data to obtain initial voice data; The initial voice data is input into a preset double-path recurrent network to output a voice feature sequence; A time-frequency mask is generated for the voice feature sequence, and the target voice spectrum is determined based on the time-frequency mask; Based on the target voice spectrum, inverse short-time Fourier transform is used for voice waveform reconstruction to obtain a target voice signal.

2. The method of de-noising a speech signal according to claim 1, wherein, Before collecting the voice data of the target user based on the preset acquisition strategy, the method further comprises the following steps: Collecting a plurality of environmental noise data in a preset area and a plurality of preset voice data; Mixing the environmental noise data and the preset voice data to generate a plurality of synthesized voice samples; Each of the synthesized voice samples is labeled to obtain labeled synthesized voice samples, and a training data set is constructed based on all the labeled synthesized voice samples.

3. The method of de-noising a speech signal according to claim 2, wherein, After constructing the training data set based on all the labeled synthesized voice samples, the method further comprises the following steps: An initial reverse diffusion model is constructed, wherein the structure of the initial reverse diffusion model at least includes a forward noise model and a reverse denoising model, the forward noise model is used to add noise to the human voice to obtain noisy voice, and the reverse denoising model is used to denoise the noisy voice to obtain human voice data; The initial reverse diffusion model is trained based on the training data set to obtain the preset reverse diffusion model.

4. The method of de-noising a speech signal according to claim 1, wherein, Before inputting the initial voice data into the preset double-path recurrent network to output the voice feature sequence, the method further comprises the following steps: The initial voice data is normalized to obtain processed audio data; The processed audio data is subjected to sliding window processing to obtain a plurality of frame data; Each of the frame data is encoded to obtain a feature code, and an encoding sequence is constructed based on all the feature codes.

5. The method of de-noising a speech signal according to claim 4, wherein, The structure of the preset double-path recurrent network at least includes a first preset bidirectional recurrent network and a second preset bidirectional recurrent network, and the step of inputting the initial voice data into the preset double-path recurrent network to output the voice feature sequence comprises the following steps: The encoding sequence is divided into a plurality of sub-encoding sequences according to a preset length; Each of the sub-encoding sequences is input into the first preset bidirectional recurrent network for feature extraction to obtain a first feature vector, and the sub-encoding sequence and the first feature vector are fused based on a residual structure to obtain a target feature vector; All the target feature vectors are spliced to obtain a feature sequence; The feature sequence is input into the second preset bidirectional recurrent network for feature extraction to obtain a second feature vector, and the feature sequence and the second feature vector are fused based on the residual structure to output the voice feature sequence.

6. The method of de-noising a speech signal according to claim 1, wherein, The step of generating a time-frequency mask for the voice feature sequence comprises the following steps: A preset reference voice data is obtained, and a semantic query vector is generated by feature extraction on the preset reference voice data; The similarity between each speech feature in the speech feature sequence and the semantic query vector is calculated based on a preset attention mechanism, and a preset weight is assigned to the speech feature based on the similarity; The time-frequency mask is generated based on the speech feature sequence and all the weights.

7. The method of de-noising a speech signal according to claim 1, wherein, Based on the time-frequency mask, the step of determining the target speech spectrum includes: Performing short-time Fourier transform on the speech data to obtain a frequency domain speech signal; Performing product calculation on the frequency domain speech signal and the time-frequency mask to obtain the target speech spectrum.

8. A speech signal de-noising apparatus characterized by comprising: It comprises: The denoising unit is configured to collect speech data of a target user based on a preset collection strategy, and to denoise the speech data using a preset inverse diffusion model to obtain initial speech data; The processing unit is configured to input the initial speech data into a preset double-path recurrent network to output a speech feature sequence; The determination unit is configured to generate a time-frequency mask for the speech feature sequence, and to determine a target speech spectrum based on the time-frequency mask; The reconstruction unit is configured to perform speech waveform reconstruction using inverse short-time Fourier transform based on the target speech spectrum to obtain a target speech signal.

9. A computer program product, characterised in that, It comprises a non-volatile computer readable storage medium storing a computer program, which is executed by a processor to implement the denoising method of the speech signal according to any one of claims 1 to 7.

10. An electronic device, comprising: It comprises one or more processors and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the denoising method of the speech signal according to any one of claims 1 to 7.

Images