Voice denoising method based on cross Transform, controller and storage medium

Abstract

The invention discloses a voice denoising method based on cross Transform, a controller and a storage medium. The voice denoising method comprises the steps of performing format conversion processing on a noise data set to obtain a layered training data set; constructing a cross Transform neural network model, and inputting the training data set into a cross Transform network to carry out feature extraction processing so as to obtain a feature data set; inputting the training data set into a short-time Fourier transform rapid calculation module for time-frequency feature extraction processing to obtain a time-frequency domain voice signal set; inputting the feature data set and the time-frequency domain voice signal set into a double-domain superposition module for smooth splicing processing to obtain a de-noised voice set; performing training by taking the voice data set as input and the de-noised voice set as output to obtain a trained cross Transform neural network model; and inputting a voice signal to be processed into the trained cross Transform neural network model to obtain a de-noised voice signal. The accuracy of the de-noised voice signal is improved, and the quality of the de-noised voice signal is optimized.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech denoising method, a controller and a storage medium based on a cross Transformer. Background Art

[0002] Speech is one of the most commonly used communication tools for humans. However, the real world is full of various noise sources, such as environmental noise, microphone noise, and electronic equipment noise, which have a negative impact on speech quality and intelligibility. Speech denoising has been widely studied in the fields of automatic speech recognition, hearing aids, telecommunications, etc. Therefore, speech denoising is an important research task.

[0003] Traditional speech denoising methods are usually based on signal processing technology, mainly reducing the impact of noise by reducing noise energy or performing spectral repair on speech signals. However, the robustness and denoising effects of these methods in complex noise environments are limited, and they may not achieve ideal denoising effects in practical applications. For example, the filtering method can selectively retain the frequency components of the target signal and suppress the noise components. However, in the face of dynamically changing noise environments, the use of filtering methods may cause signal distortion. At the same time, the filtering method cannot effectively process nonlinear noise, and the coherence of the generated denoised speech signal is poor, which makes the accuracy of the denoised speech signal low, resulting in poor quality of the denoised speech signal. With the rapid development of artificial intelligence technology, speech denoising methods based on deep neural networks have made significant progress and can better solve the problems faced by traditional denoising methods. The Transformer model shows potential in the field of speech denoising. At present, the speech denoising method of the Transformer model may ignore the local correlation between pixels in pixel-level tasks. Secondly, compared with the traditional neural network model, the Transformer model is more complex, has more parameters and calculations, and the large calculation burden makes the speech denoising method inefficient. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application embodiment provides a speech denoising method, controller and storage medium based on a cross Transformer, which is conducive to improving the accuracy of the denoised speech signal, optimizing the quality of the denoised speech signal, and improving the efficiency of the speech denoising method.

[0005] In a first aspect, an embodiment of the present application provides a speech denoising method based on a cross Transformer, comprising:

[0006] Acquire a speech data set, and add noise to the speech data set to obtain a noise data set;

[0007] Performing format conversion on the noise data set to obtain a hierarchical training data set;

[0008] Constructing a cross-Transformer neural network model, wherein the cross-Transformer neural network model includes a cross-Transformer network, a short-time Fourier transform fast calculation module, and a dual-domain superposition module;

[0009] Inputting the training data set into the cross Transformer network for feature extraction to obtain a feature data set;

[0010] Inputting the training data set into the short-time Fourier transform quick calculation module to perform time-frequency feature extraction processing to obtain a time-frequency domain speech signal set;

[0011] Inputting the feature data set and the time-frequency domain speech signal set into the dual-domain superposition module for smooth splicing processing to obtain a denoised speech set;

[0012] Using the speech data set as the input of the cross Transformer neural network model and the denoised speech set as the output of the cross Transformer neural network model, training the cross Transformer neural network model to obtain a trained cross Transformer neural network model;

[0013] A speech signal to be processed is obtained, and the speech signal to be processed is input into the trained cross-Transformer neural network model to obtain a denoised speech signal.

[0014] According to some embodiments of the present application, the cross Transformer network includes a convolution layer, a normalization layer and a six-layer Transformer encoder layer, and the six-layer Transformer encoder layer is a cross structure; the training data set is input into the cross Transformer network for feature extraction processing to obtain a feature data set, including:

[0015] The training data set is input into a convolution layer with a 1x3 convolution kernel and a stride of 1x2, so that the input sequence length L of the training data set is 512, the number of feature channels width is 64, and the data sample batch is 2;

[0016] The training data set is normalized using a normalization layer with 512 dimensional features to obtain a normalized data set;

[0017] The normalized data set is input into the six-layer Transformer encoder layer, and feature extraction and encoding processing are performed on the normalized data set through the six-layer Transformer encoder layer with a cross structure of different dimensions to obtain a feature data set.

[0018] According to some embodiments of the present application, the cross Transformer network further includes a main component analysis module, and the main component analysis module is embedded in the first Transformer encoder layer, and the method further includes:

[0019] Inputting the feature data set into the main component analysis module,

[0020] The feature data set is subjected to dimensionality reduction processing by the main component analysis module to compress the feature dimension of the feature data set.

[0021] According to some embodiments of the present application, the Transformer encoder layer includes a self-attention module, a feedforward module, a normalization and residual connection module, the self-attention module is used to determine the dependency of features of the input sequence of the normalized data set in different subspaces, the feedforward module is used to extract and transform features of the normalized data set, and the normalization and residual connection module is used to normalize the normalized data set to obtain a feature data set.

[0022] According to some embodiments of the present application, the short-time Fourier transform quick calculation module includes a short-time Fourier transform layer and a two-dimensional convolution layer, and the feature data set is input into the short-time Fourier transform quick calculation module for time-frequency feature extraction processing to obtain a time-frequency domain speech signal set, including:

[0023] Inputting the feature data set into the short-time Fourier transform layer, so as to identify the analysis window size of the feature data set and the degree of overlap between adjacent frames through the short-time Fourier transform layer, and classifying adjacent frames whose degree of overlap is greater than a preset threshold as overlapping frames, thereby obtaining time-frequency information of the feature data set;

[0024] Decomposing the time-frequency information of the feature data set into a real branch and an imaginary branch, wherein the real branch includes and concatenating the real branch and the imaginary branch to form a feature vector, wherein the real part includes the amplitude information of the feature data set, and the imaginary part includes the phase information of the feature data set;

[0025] The feature vector is input into the two-dimensional convolution layer to compress the dimension of the feature vector through the two-dimensional convolution layer, and the channel dimension of the feature vector is linearly transformed through the 1x1 convolution layer to obtain a time-frequency domain speech signal set.

[0026] According to some embodiments of the present application, the dual-domain superposition module includes a residual connection layer and an Overlap Add layer, and the feature data set and the time-frequency domain speech signal set are input into the dual-domain superposition module for smooth splicing processing to obtain a denoised speech set, including:

[0027] Inputting the feature data set and the time-frequency domain speech signal set into the residual connection layer to alleviate the gradient disappearance of the cross Transformer neural network model;

[0028] The feature data set and the time-frequency domain speech signal set are input into the Overlap Add layer, so that the Overlap Add layer performs overlap-addition processing on the overlapping frames of the feature data set and the time-frequency domain speech signal set, smoothly splices multi-frame features, and obtains a denoised speech set.

[0029] According to some embodiments of the present application, the Transformer encoder layer also includes a fully connected layer, which is connected to the output of the self-attention module, and the fully connected layer includes a first linear transformation module and a first activation function, and the first linear transformation module and the first activation function are used to introduce nonlinear characteristics to enhance and learn the complex features output by the self-attention module.

[0030] According to some embodiments of the present application, the cross Transformer neural network model further includes a second linear transformation module and a second activation function, and the second linear transformation module and the second activation function are connected to the output end of the cross Transformer network and the output end of the short-time Fourier transform fast calculation module, and the method further includes:

[0031] The feature data set output by the cross Transformer network and the time-frequency domain speech signal set output by the short-time Fourier transform calculation module are combined with the time-frequency features and context information to be integrated;

[0032] The fused features are input into the second linear transformation module and the second activation function, so as to optimize the fused features through the second linear transformation module and the second activation function.

[0033] In a second aspect, an embodiment of the present application provides a controller comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the method described in the technical solution of the first aspect when running the computer program.

[0034] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method described in the technical solution of the first aspect.

[0035] The cross-Transformer-based speech denoising method, controller and storage medium provided in the embodiments of the present application have at least one of the following advantages or beneficial effects: obtaining a clean speech data set, and manually adding noise to the speech data set to obtain a noise data set, and converting the noise data set into a format and using it as a training data set. Construct a neural network model based on a cross-Transformer. The neural network model based on the cross-Transformer includes three parts: a cross-Transformer network, a short-time Fourier transform quick calculation module and a dual-domain superposition module. Input the training data set into the cross-Transformer network. The cross-Transformer network alternately uses encoder layers in different dimensions to perform feature extraction processing to obtain a feature data set, capture the global dependencies of the training data, and reduce the number of model parameters and computational overhead. Input the training data set into the short-time Fourier transform quick calculation module. The short-time Fourier transform quick calculation module pre-calculates the training data set. The speech data of the training data set is repeated and stored in the GPU to improve the computing efficiency, thereby improving the efficiency of the speech denoising method, and quickly calculating the time-frequency features of the speech data of the training data set to obtain the time-frequency domain speech signal set, providing basic support for signal processing, and faster and better analyzing and understanding the spectral features of the time-frequency domain speech signal set; the feature data set and the time-frequency domain speech signal set are input into the dual-domain superposition module, and the overlap-addition operation is performed to smoothly splice multi-frame features, eliminate boundary discontinuities, and produce a more natural and coherent output to obtain a denoised speech set, enhance the quality of the output denoised speech set, improve the accuracy of the denoised speech signal, and optimize the quality of the denoised speech signal. The cross-Transformer neural network model is trained using the speech data set as input and the denoised speech set as output. After the training is completed, the speech signal to be processed is input into the trained cross-Transformer neural network model to obtain the denoised speech signal.

[0036] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flowchart of a speech denoising method based on a cross Transformer provided in an embodiment of the present application;

[0038] Figure 2 It is a flowchart of a method provided in an embodiment of the present application for inputting a speech signal to be processed into a trained cross-Transformer neural network model to obtain a denoised speech signal;

[0039] Figure 3 It is a flowchart of a method provided in an embodiment of the present application for inputting a training data set into a cross Transformer network for feature extraction processing to obtain a feature data set;

[0040] Figure 4 It is a structural diagram of a cross Transformer network provided in an embodiment of the present application;

[0041] Figure 5 is a flow chart of a method for compressing feature dimensions of a feature data set through a main component analysis module provided by an embodiment of the present application;

[0042] Figure 6 It is a flow chart of a method provided by an embodiment of the present application for inputting a feature data set into a short-time Fourier transform quick calculation module for time-frequency feature extraction processing to obtain a time-frequency domain speech signal set;

[0043] Figure 7 It is a flowchart of a method provided by an embodiment of the present application for inputting a feature data set and a time-frequency domain speech signal set into a dual-domain superposition module for smooth splicing processing to obtain a denoised speech set;

[0044] Figure 8 is a flowchart of a speech denoising method based on a cross Transformer provided in another embodiment of the present application;

[0045] Fig. 9 It is a structural schematic diagram of a controller provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] This section will describe in detail the specific embodiments of the present application. The preferred embodiments of the present application are shown in the accompanying drawings. The purpose of the accompanying drawings is to supplement the description of the text part of the specification with graphics, so that people can intuitively and vividly understand each technical feature and the overall technical solution of the present application, but it cannot be understood as a limitation on the scope of protection of the present application.

[0047] In the description of this application, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed", etc. are understood to exclude the number itself, "above", "below", "within", etc. are understood to include the number itself, "any one" means one or more, "at least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. If there is a description of the first and the second, it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0048] It should be noted that the terms such as setting, installing, and connecting in the embodiments of the present application should be interpreted in a broad sense, and the technical personnel in the relevant technical field can reasonably determine the specific meanings of the above terms in the embodiments of the present application in combination with the specific content of the technical solution. For example, the term "connection" can be a mechanical connection, an electrical connection, or mutual communication; it can be directly connected or indirectly connected through an intermediate medium.

[0049] It should be noted that the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0050] Speech is one of the most commonly used communication tools for humans. However, the real world is full of various noise sources, such as environmental noise, microphone noise, and electronic equipment noise, which have a negative impact on speech quality and intelligibility. Speech denoising has been widely studied in the fields of automatic speech recognition, hearing aids, telecommunications, etc. Therefore, speech denoising is an important research task.

[0051] Traditional speech denoising methods are usually based on signal processing technology, mainly reducing the impact of noise by reducing noise energy or performing spectral repair on speech signals. However, the robustness and denoising effects of these methods in complex noise environments are limited, and they may not achieve ideal denoising effects in practical applications. For example, filtering methods (such as low-pass filters, high-pass filters, or band-pass filters) can selectively retain the frequency components of the target signal and suppress the noise components. However, in the face of dynamically changing noise environments, the use of filtering methods may cause signal distortion. At the same time, filtering methods cannot effectively process nonlinear noise, and the coherence of the generated denoised speech signal is poor, resulting in low accuracy of the denoised speech signal, which leads to poor quality of the denoised speech signal. With the rapid development of artificial intelligence technology, speech denoising methods based on deep neural networks have made significant progress and can better solve the problems faced by traditional denoising methods. The Transformer model shows potential in the field of speech denoising. At present, the speech denoising method of the Transformer model may ignore the local correlation between pixels in pixel-level tasks. Secondly, compared with the traditional neural network model, the Transformer model is more complex, has more parameters and calculations, and the large computational burden makes the speech denoising method inefficient.

[0052] Based on this, the embodiments of the present application provide a speech denoising method, controller and storage medium based on a cross Transformer, which is conducive to improving the accuracy of the denoised speech signal, optimizing the quality of the denoised speech signal, and improving the efficiency of the speech denoising method.

[0053] Please refer to the accompanying drawings below for further explanation of the cross-Transformer-based speech denoising method, controller and storage medium provided in the embodiments of the present application.

[0054] Reference Figure 1 As shown, Figure 1 is a flowchart of a speech denoising method based on a cross Transformer provided in an embodiment of the present application. The speech denoising method based on a cross Transformer includes but is not limited to steps S100 to S800. Specifically,

[0055] Step S100: obtaining a speech data set, adding noise to the speech data set, and obtaining a noise data set;

[0056] Step S200: converting the noise data set into a format to obtain a hierarchical training data set;

[0057] Step S300: constructing a cross Transformer neural network model, wherein the cross Transformer neural network model includes a cross Transformer network, a short-time Fourier transform calculation module, and a dual-domain superposition module;

[0058] Step S400: input the training data set into the cross Transformer network for feature extraction to obtain a feature data set;

[0059] Step S500: input the training data set into the short-time Fourier transform calculation module to perform time-frequency feature extraction processing to obtain a time-frequency domain speech signal set;

[0060] Step S600: inputting the feature data set and the time-frequency domain speech signal set into the dual-domain superposition module for smooth splicing processing to obtain a denoised speech set;

[0061] Step S700: using the speech data set as the input of the cross Transformer neural network model and the denoised speech set as the output of the cross Transformer neural network model, training the cross Transformer neural network model to obtain a trained cross Transformer neural network model;

[0062] Step S800: Obtain a speech signal to be processed, and input the speech signal to be processed into a trained cross-Transformer neural network model to obtain a denoised speech signal.

[0063] In some embodiments of the present application, a speech denoising method based on a cross Transformer includes: obtaining a clean speech data set, manually adding noise to the speech data set to obtain a noise data set, and performing format conversion processing on the noise data set to convert it into hdf5 layered data as a training data set. Construct a neural network model based on a cross Transformer. The neural network model based on the cross Transformer includes three parts: a cross Transformer network, a short-time Fourier transform quick calculation module, and a dual-domain superposition module. Input the training data set into the cross Transformer network. The cross Transformer network alternately uses encoder layers in different dimensions to perform feature extraction processing on the training data set to obtain a feature data set, capture the global dependency of the training data, and reduce the number of model parameters and computational overhead; input the training data set into the short-time Fourier transform quick calculation module. The short-time Fourier transform quick calculation module pre- The repeated intermediate amount of the speech data of the training data set is calculated and stored in the GPU to improve the computing efficiency, thereby improving the efficiency of the speech denoising method, and quickly calculating the time-frequency features of the speech data of the training data set to obtain the time-frequency domain speech signal set, providing basic support for signal processing, and faster and better analyzing and understanding the spectral features of the time-frequency domain speech signal set; the feature data set and the time-frequency domain speech signal set are input into the dual-domain superposition module, and the overlap-addition operation is performed to smoothly splice multi-frame features, eliminate boundary discontinuities, and produce a more natural and coherent output to obtain a denoised speech set, enhance the quality of the output denoised speech set, improve the accuracy of the denoised speech signal, and optimize the quality of the denoised speech signal. The cross-Transformer neural network model is trained using the speech data set as input and the denoised speech set as output. After the training is completed, the speech signal to be processed is input into the trained cross-Transformer neural network model to obtain the denoised speech signal.

[0064] Reference Figure 2 As shown, Figure 2The invention discloses a flow chart of a method for obtaining a denoised speech signal by inputting a speech signal to be processed into a trained cross-Transformer neural network model provided by an embodiment of the present application. The cross-Transformer neural network model comprises a cross-Transformer network, a short-time Fourier transform quick calculation module and a dual-domain superposition module. The speech signal to be processed is obtained and input into the trained cross-Transformer neural network model. The cross-Transformer network alternately uses encoder layers in different dimensions to perform feature extraction processing on the speech signal to be processed. The short-time Fourier transform quick calculation module calculates the time-frequency features of the speech data of the speech signal to be processed. The signal output by the cross-Transformer network and the signal output by the short-time Fourier transform quick calculation module are input into the dual-domain superposition module together. The dual-domain superposition module performs an overlap-addition operation, smoothly splices multi-frame features, eliminates boundary discontinuities, generates a more natural and coherent output, obtains a denoised speech signal, improves the accuracy of the denoised speech signal, and optimizes the quality of the denoised speech signal.

[0065] Reference Figure 3 As shown, Figure 3 4 is a flowchart of a method for inputting a training data set into a cross Transformer network for feature extraction processing to obtain a feature data set provided by an embodiment of the present application. The method for inputting a training data set into a cross Transformer network for feature extraction processing to obtain a feature data set includes but is not limited to steps S410 to S430. Specifically,

[0066] Step S410: input the training data set into a convolution layer with a 1x3 convolution kernel and a stride of 1x2, so that the input sequence length L of the training data set is 512, the number of feature channels width is 64, and the data sample batch is 2;

[0067] Step S420: normalizing the training data set using a normalization layer of 512-dimensional features to obtain a normalized data set;

[0068] Step S430: input the normalized data set into the six-layer Transformer encoder layer, perform feature extraction and encoding processing on the normalized data set through the six-layer Transformer encoder layer with a cross structure of different dimensions to obtain a feature data set.

[0069] In some embodiments of the present application, a cross Transformer network is responsible for feature extraction and encoding of an input signal. The Transformer structure can be used to capture long-distance dependencies in an input sequence. The cross Transformer network includes a convolution layer, a normalization layer, and a six-layer Transformer encoder layer. The six-layer Transformer encoder layer is a cross structure. A method for inputting a training data set into a cross Transformer network for feature extraction to obtain a feature data set includes: first, inputting the training data set into a convolution layer with a 1x3 convolution kernel and a stride of 1x2, setting the input sequence length L of the training data set to 512, the number of feature channels width to 64, and the data sample batch size to 2; then performing normalization processing using a normalization layer suitable for 512-dimensional features to obtain a normalized data set. The cross Transformer network includes 6 layers of Transformer encoder layers, which are used alternately in different dimensions. The 6-layer Transformer encoder layer performs feature extraction and encoding processing on the input normalized data set to form a feature data set.

[0070] In the present application, the cross Transformer network alternately uses encoder layers in different dimensions to perform feature extraction processing to obtain a feature data set, capture the global dependencies of the training data, reduce the number of model parameters and computational overhead, improve computational efficiency, and thus improve the efficiency of the speech denoising method.

[0071] Reference Figure 4 As shown, Figure 4 It is a structural diagram of a cross Transformer network provided in an embodiment of the present application. The cross Transformer network includes 6 layers of Transformer encoder layers. The 6 layers of Transformer encoder layers are cross-connected, and different layers can share information. The output of each encoder layer not only affects the subsequent processing of the same layer, but also affects the calculation of other layers, thereby enhancing the ability of feature learning. Through the interaction between layers, the stacking of multiple encoder layers allows the cross Transformer neural network model to extract multi-level features from the input signal. The cross Transformer neural network model can better understand the structure and pattern in the signal, and combine the contextual information from different layers to improve the understanding of the speech signal of the training data set, especially in a noisy environment, to enhance the clarity and comprehensibility of the signal. Cross-connection can improve the robustness of the cross Transformer neural network model to input changes, so that even in the case of input noise or distortion, the cross Transformer neural network model can still maintain good performance, reduce the risk of overfitting and thus improve performance.

[0072] Reference Figure 5 As shown, Figure 5 1 is a flowchart of a method for compressing the feature dimensions of a feature data set by a main component analysis module provided by an embodiment of the present application. The method for compressing the feature dimensions of a feature data set by a main component analysis module includes but is not limited to steps S900 to S910. Specifically,

[0073] Step S900: Input the feature data set into the main component analysis module,

[0074] Step S910: Perform dimensionality reduction processing on the feature data set through the main component analysis module to compress the feature dimension of the feature data set.

[0075] In some embodiments of the present application, the cross-Transformer network also includes a main component analysis module, which is embedded in the first layer of the Transformer encoder layer. The main component analysis module is used to compress the feature dimension of the feature data set, thereby reducing the complexity and computational burden of the cross-Transformer neural network model. The method of compressing the feature dimension of the feature data set through the main component analysis module includes: inputting the feature data set into the main component analysis module, performing dimensionality reduction processing on the feature data set through the main component analysis module to compress the feature dimension of the feature data set, and realizing that the main component analysis module reduces the burden of the input feature data set, effectively captures the global dependency of the input feature data set, reduces the model parameter amount and computational overhead of the cross-Transformer neural network model, and retains key feature information. Therefore, unlike the traditional Transformer module, in the embodiment of the present application, the main component analysis module is embedded in the Transformer encoder layer, which can reduce the model parameter amount and computational overhead, reduce the model complexity, and can remove redundant features, thereby improving the performance of the cross-Transformer neural network model.

[0076] In some embodiments of the present application, the Transformer encoder layer includes a self-attention module, a feedforward module, a normalization and residual connection module, the self-attention module is used to determine the dependency of features of the input sequence of the normalized data set in different subspaces, the feedforward module is used to extract and transform features of the normalized data set, and the normalization and residual connection module is used to normalize the normalized data set to obtain a feature data set.

[0077] It can be understood that each layer of the Transformer encoder layer includes three modules, namely, the self-attention module, the feedforward module, the normalization and residual connection module. The self-attention module uses a multi-head self-attention mechanism to process the input tensor of the normalized data set to determine the dependency of the features of the input sequence of the normalized data set in different subspaces, and can learn the rich features of speech data from different representation subspaces. The feedforward module uses a double-layer feedforward neural network to extract and transform features of the normalized data set. The nonlinear activation function PReLU is applied between the double-layer feedforward neural network to enhance the expression ability of the cross-Transformer neural network model, so that the cross-Transformer neural network model learns more valuable feature representations, which is conducive to improving the accuracy of the denoised speech signal and optimizing the quality of the denoised speech signal. The normalization and residual connection module is used to normalize the normalized data set to obtain a feature data set; the use of the normalization and residual connection module to normalize the data can stabilize the input distribution, accelerate the training convergence of the cross-Transformer neural network model, and improve the training rate of the cross-Transformer neural network model.

[0078] It can be understood that by setting each Transformer encoder layer to include a multi-head self-attention mechanism, important features are highlighted by calculating the weighted sum of the input data set, while suppressing unimportant information, allowing the cross Transformer neural network model to learn the global dependencies of the input sequence in parallel in different subspaces. Through the multi-head mechanism, the cross Transformer neural network model can focus on different aspects of the speech signal of the input data set in different attention heads, thereby capturing richer contextual information. A residual connection is added between the output of the multi-head attention layer and its input to help the cross Transformer neural network model converge stably during training. At the same time, the output is normalized by layer normalization to improve the training efficiency of the cross Transformer neural network model. This structure can alleviate the gradient vanishing problem in deep networks, allowing information to be transmitted more effectively. Each Transformer encoder layer also contains a normal convolution layer, and the 1x 1 convolution is used to adjust the dimension of the feature, helping the cross Transformer neural network model to maintain spatial information during feature extraction, and can change its number of channels without changing the dimension of the feature space, so that the cross Transformer neural network model can flexibly process features of different dimensions.

[0079] Reference Figure 6 As shown, Figure 6The present invention provides a flowchart of a method for inputting a feature data set into a short-time Fourier transform quick calculation module for time-frequency feature extraction processing to obtain a time-frequency domain speech signal set. The method for inputting a feature data set into a short-time Fourier transform quick calculation module for time-frequency feature extraction processing to obtain a time-frequency domain speech signal set includes but is not limited to steps S510 to S530. Specifically,

[0080] Step S510: inputting the feature data set into the short-time Fourier transform layer, so as to identify the analysis window size of the feature data set and the degree of overlap between adjacent frames through the short-time Fourier transform layer, and classifying adjacent frames whose degree of overlap is greater than a preset threshold as overlapping frames, thereby obtaining the time-frequency information of the feature data set;

[0081] Step S520: decomposing the time-frequency information of the feature data set into a real branch and an imaginary branch, and concatenating the real branch and the imaginary branch to form a feature vector, wherein the real part includes the amplitude information of the feature data set, and the imaginary part includes the phase information of the feature data set;

[0082] Step S530: Input the feature vector into a two-dimensional convolutional layer to compress the dimension of the feature vector through the two-dimensional convolutional layer, and linearly transform the channel dimension of the feature vector through a 1x1 convolutional layer to obtain a time-frequency domain speech signal set.

[0083] In some embodiments of the present application, the short-time Fourier transform calculation module includes a short-time Fourier transform layer and a two-dimensional convolution layer. The short-time Fourier transform layer is responsible for calculating the repeated intermediate amount of speech data in the feature data set in advance, improving computing efficiency, and analyzing and understanding the spectral characteristics of the time domain signal faster and better, providing basic support for various signal processing and analysis tasks.

[0084] The feature data set is input into a short-time Fourier transform quick calculation module for time-frequency feature extraction processing to obtain a time-frequency domain speech signal set, including: first, the feature data set is input into a short-time Fourier transform layer, the short-time Fourier transform layer identifies the analysis window size of the feature data set and the degree of overlap between adjacent frames, and divides adjacent frames with an overlap degree greater than a preset threshold into overlapping frames, and obtains the time-frequency information of the feature data set; in order to optimize performance, the short-time Fourier transform quick calculation module pre-calculates the real part DR and the imaginary part DI of the discrete Fourier transform matrix and the Hamming window W, and stores them in the GPU, thereby obtaining faster calculation speed and practicality and reducing memory usage. Afterwards, the short-time Fourier transform layer outputs the time-frequency information of the feature data set, decomposes the time-frequency information of the feature data set into a real branch and an imaginary branch, the real part includes the amplitude information of the feature data set, and the imaginary part includes the phase information of the feature data set, and then the real branch and the imaginary branch are spliced ​​to form a richer feature vector. The feature vector is input into the two-dimensional convolution layer, which further processes these feature vectors and compresses the 128-dimensional feature vector to 64 dimensions. The dimension of the feature vector is compressed through the two-dimensional convolution layer, effectively reducing the complexity and parameter quantity of the cross-Transformer neural network model. The channel dimension of the feature vector is linearly transformed through 1x1 convolution to more accurately capture the key features of the speech data and obtain the time-frequency domain speech signal set. The short-time Fourier transform module completely retains the amplitude and phase information of the feature vector signal by processing the real and imaginary parts of the feature vector, so that the cross-Transformer neural network model can learn feature representations suitable for specific tasks, thereby enhancing the denoising performance and achieving faster and better analysis and understanding of the spectral characteristics of the time-frequency domain speech signal set, which is conducive to improving the accuracy of the denoised speech signal and optimizing the quality of the denoised speech signal.

[0085] Reference Figure 7 As shown, Figure 7 The present invention provides a flowchart of a method for inputting a feature data set and a time-frequency domain speech signal set into a dual-domain superposition module for smooth splicing to obtain a denoised speech set. The method for inputting a feature data set and a time-frequency domain speech signal set into a dual-domain superposition module for smooth splicing to obtain a denoised speech set includes but is not limited to steps S610 to S620. Specifically,

[0086] Step S610: inputting the feature data set and the time-frequency domain speech signal set into the residual connection layer to alleviate the gradient disappearance of the cross-Transformer neural network model;

[0087] Step S620: Input the feature data set and the time-frequency domain speech signal set to the Overlap Add layer, so as to perform overlap-addition processing on the overlapping frames of the feature data set and the time-frequency domain speech signal set through the Overlap Add layer, smoothly splice multi-frame features, and obtain a denoised speech set.

[0088] In some embodiments of the present application, the dual-domain overlay module includes a residual connection layer and an Overlap Add layer. The residual connection layer can effectively retain the original information and alleviate the gradient vanishing problem of the cross Transformer neural network model, making the training more stable and improving the model performance. The feature data set and the time-frequency domain speech signal set are input into the dual-domain superposition module. The residual connection layer effectively retains the original information of the feature data set and the time-frequency domain speech signal set, and alleviates the gradient disappearance of the cross-Transformer neural network model. After that, the feature data set and the time-frequency domain speech signal set are input into the Overlap Add layer. The Overlap Add layer adjusts the feature dimension of the data set through the 1x1 convolution layer, and performs overlap-addition processing on the overlapping frames of the feature data set and the time-frequency domain speech signal set, dividing the feature data set and the time-frequency domain speech signal set into multiple overlapping frames, each frame containing a certain overlapping part, which helps the cross-Transformer neural network model to capture more continuous information in the time domain, and sums the overlapping parts, smoothly splices the multi-frame features, effectively combines the information between different frames, reduces the discontinuity caused by the boundary effect, and makes the output denoised speech set natural and coherent, and obtains a stable and high-quality denoised speech set that is more in line with human auditory or visual perception. Through residual connection and overlap-addition operations, the signal quality of the denoised speech is effectively enhanced, a more natural and coherent output is generated, and the accuracy of the denoised speech signal is improved.

[0089] In some embodiments of the present application, the Transformer encoder layer also includes a fully connected layer, the fully connected layer and the output connection of the self-attention module, the fully connected layer includes a first linear transformation module and a first activation function, and the first linear transformation module and the first activation function are used to introduce nonlinear characteristics to enhance and learn the complex features output by the self-attention module.

[0090] The Transformer encoder layer contains a fully connected layer, which is connected to the output of the self-attention module for further processing of the output of the multi-head attention layer. The fully connected layer includes a first linear transformation module and a first activation function. The first linear transformation module includes two linear transformations to introduce nonlinear characteristics, which helps to enhance the expressiveness of the cross-Transformer neural network model and allows learning of complex feature maps. Similar to the multi-head attention module, the output of the fully connected layer is also added to the input through a residual connection, and then layer normalization is performed to ensure that the Transformer encoder layer can effectively utilize the information of the previous layer at each layer, maintaining the flow of information and the stability of the model.

[0091] Reference Figure 8 As shown, Figure 8 is a flowchart of a speech denoising method based on a cross Transformer provided by another embodiment of the present application. The speech denoising method based on a cross Transformer also includes steps S920 to S930. Specifically,

[0092] Step S920: fusing the feature data set output by the cross Transformer network and the time-frequency domain speech signal set output by the short-time Fourier transform calculation module with the time-frequency features and context information;

[0093] Step S930: input the fused features into the second linear transformation module and the second activation function, so as to optimize the fused features through the second linear transformation module and the second activation function.

[0094] In some embodiments of the present application, the cross Transformer neural network model also includes a second linear transformation module and a second activation function, and the second linear transformation module and the second activation function are connected to the output end of the cross Transformer network and the output end of the short-time Fourier transform quick calculation module. The cross Transformer speech denoising method also includes fusing the feature data set output by the cross Transformer network and the time-frequency domain speech signal set output by the short-time Fourier transform quick calculation module in combination with the time-frequency features and context information, and the fused features are nonlinearly transformed by the second linear transformation module and the second activation function, and are activated by layer normalization and parameterized ReLU to optimize the fused features and further optimize the feature representation. Finally, the optimized feature data set and time-frequency domain speech signal set are divided into multiple overlapping frames through convolution layers and dual-domain superposition modules. Each frame contains a certain overlapping part, which helps the cross Transformer neural network model to capture more continuous information in the time domain and sum the overlapping parts, effectively combining the information between different frames, reducing the discontinuity caused by boundary effects, making the output denoised speech signal more coherent in time, improving the accuracy of the denoised speech signal, and optimizing the quality of the denoised speech signal.

[0095] Reference Fig. 9 , Fig. 9It is a structural diagram of a controller 1000 provided in an embodiment of the present application, including a processor 1001, which can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the method provided in the embodiment of the present application; a memory 1002, which can be implemented in the form of a read-only memory 1002 (Read Only Memory, ROM), a static storage device, a dynamic storage device or a random access memory 1002 (Random Access Memory, RAM), etc. The memory 1002 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1002 and are called by the processor 1001 to execute the embodiments of this application; the input / output interface 1003 is used to implement information input and output; the communication interface 1004 is used to implement communication interaction between this device and other devices, and communication can be achieved through wired methods (such as USB, network cables, etc.) or wireless methods (such as mobile networks, WIFI, Bluetooth, etc.); a bus transmits information between various components of the device (such as the processor 1001, the memory 1002, the input / output interface 1003 and the communication interface 1004); wherein the processor 1001, the memory 1002, the input / output interface 1003 and the communication interface 1004 are connected to each other within the device through the bus.

[0096] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer-readable storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer-readable storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0097] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings.

Claims

1. A speech denoising method based on a cross Transformer, characterized in that: include: Acquire a speech data set, and add noise to the speech data set to obtain a noise data set; Performing format conversion on the noise data set to obtain a hierarchical training data set; Constructing a cross-Transformer neural network model, wherein the cross-Transformer neural network model includes a cross-Transformer network, a short-time Fourier transform fast calculation module, and a dual-domain superposition module; Inputting the training data set into the cross Transformer network for feature extraction to obtain a feature data set; Inputting the training data set into the short-time Fourier transform quick calculation module to perform time-frequency feature extraction processing to obtain a time-frequency domain speech signal set; Inputting the feature data set and the time-frequency domain speech signal set into the dual-domain superposition module for smooth splicing processing to obtain a denoised speech set; Using the speech data set as the input of the cross Transformer neural network model and the denoised speech set as the output of the cross Transformer neural network model, training the cross Transformer neural network model to obtain a trained cross Transformer neural network model; A speech signal to be processed is obtained, and the speech signal to be processed is input into the trained cross-Transformer neural network model to obtain a denoised speech signal.

2. The speech denoising method based on cross Transformer according to claim 1, characterized in that: The cross Transformer network includes a convolution layer, a normalization layer and a six-layer Transformer encoder layer, and the six-layer Transformer encoder layer is a cross structure; the training data set is input into the cross Transformer network for feature extraction processing, Get the feature data set, including: The training data set is input into a convolution layer with a 1x3 convolution kernel and a stride of 1x2, so that the input sequence length L of the training data set is 512, the number of feature channels width is 64, and the data sample batch is 2; The training data set is normalized using a normalization layer with 512 dimensional features to obtain a normalized data set; The normalized data set is input into the six-layer Transformer encoder layer, and feature extraction and encoding processing are performed on the normalized data set through the six-layer Transformer encoder layer with a cross structure of different dimensions to obtain a feature data set.

3. The speech denoising method based on cross Transformer according to claim 2, characterized in that: The cross Transformer network further includes a main component analysis module, and the main component analysis module is embedded in the first Transformer encoder layer. The method further includes: Inputting the feature data set into the main component analysis module, The feature data set is subjected to dimensionality reduction processing by the main component analysis module to compress the feature dimension of the feature data set.

4. The speech denoising method based on cross Transformer according to claim 2, characterized in that: The Transformer encoder layer includes a self-attention module, a feedforward module, a normalization and residual connection module, the self-attention module is used to determine the dependency relationship between the features of the input sequence of the normalized data set in different subspaces, the feedforward module is used to extract and transform the features of the normalized data set, and the normalization and residual connection module is used to normalize the normalized data set to obtain a feature data set.

5. The speech denoising method based on cross Transformer according to claim 1, characterized in that: The short-time Fourier transform quick calculation module includes a short-time Fourier transform layer and a two-dimensional convolution layer. The feature data set is input into the short-time Fourier transform quick calculation module for time-frequency feature extraction processing to obtain a time-frequency domain speech signal set, including: Inputting the feature data set into the short-time Fourier transform layer, so as to identify the analysis window size of the feature data set and the degree of overlap between adjacent frames through the short-time Fourier transform layer, and classifying adjacent frames whose degree of overlap is greater than a preset threshold as overlapping frames, thereby obtaining time-frequency information of the feature data set; Decomposing the time-frequency information of the feature data set into a real branch and an imaginary branch, wherein the real branch includes and concatenating the real branch and the imaginary branch to form a feature vector, wherein the real part includes the amplitude information of the feature data set, and the imaginary part includes the phase information of the feature data set; The feature vector is input into the two-dimensional convolution layer to compress the dimension of the feature vector through the two-dimensional convolution layer, and the channel dimension of the feature vector is linearly transformed through the 1x1 convolution layer to obtain a time-frequency domain speech signal set.

6. The speech denoising method based on cross Transformer according to claim 5, characterized in that: The dual-domain superposition module includes a residual connection layer and an Overlap Add layer, and the feature data set and the time-frequency domain speech signal set are input into the dual-domain superposition module for smooth splicing processing to obtain a denoised speech set, including: Inputting the feature data set and the time-frequency domain speech signal set into the residual connection layer to alleviate the gradient disappearance of the cross Transformer neural network model; The feature data set and the time-frequency domain speech signal set are input into the Overlap Add layer, so that the Overlap Add layer performs overlap-addition processing on the overlapping frames of the feature data set and the time-frequency domain speech signal set, smoothly splices multi-frame features, and obtains a denoised speech set.

7. The speech denoising method based on cross Transformer according to claim 4, characterized in that: The Transformer encoder layer also includes a fully connected layer, which is connected to the output of the self-attention module. The fully connected layer includes a first linear transformation module and a first activation function. The first linear transformation module and the first activation function are used to introduce nonlinear characteristics to enhance and learn the complex features output by the self-attention module.

8. The speech denoising method based on cross Transformer according to claim 6, characterized in that: The cross Transformer neural network model further includes a second linear transformation module and a second activation function, wherein the second linear transformation module and the second activation function are connected to the output end of the cross Transformer network and the output end of the short-time Fourier transform fast calculation module, and the method further includes: The feature data set output by the cross Transformer network and the time-frequency domain speech signal set output by the short-time Fourier transform calculation module are combined with the time-frequency features and context information to be integrated; The fused features are input into the second linear transformation module and the second activation function, so as to optimize the fused features through the second linear transformation module and the second activation function.

9. A controller, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method according to any one of claims 1 to 8.

Images