Speech denoising method, controller and storage medium based on cross Transformer

Abstract

This application discloses a speech denoising method, controller, and storage medium based on a cross-Transformer architecture. The speech denoising method includes: converting a noisy dataset to obtain a hierarchical training dataset; constructing a cross-Transformer neural network model; inputting the training dataset into the cross-Transformer network for feature extraction to obtain a feature dataset; inputting the training dataset into a short-time Fourier transform (SFT) module for time-frequency feature extraction to obtain a time-frequency domain speech signal set; inputting the feature dataset and the time-frequency domain speech signal set into a dual-domain overlay module for smooth concatenation to obtain a denoised speech set; training the cross-Transformer neural network model using the speech dataset as input and the denoised speech set as output; and inputting the speech signal to be processed into the trained cross-Transformer neural network model to obtain a denoised speech signal. This application improves the accuracy and optimizes the quality of the denoised speech signal.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to a speech denoising method, controller and storage medium based on cross Transformer. Background Technology

[0002] Speech is one of the most commonly used communication tools for humans. However, the real world is full of various noise sources, such as environmental noise, microphone noise, and electronic device noise. These noises negatively affect speech quality and intelligibility. Speech denoising has been extensively studied in fields such as automatic speech recognition, hearing aids, and telecommunications. Therefore, speech denoising is an important research task.

[0003] Traditional speech denoising methods are typically based on signal processing techniques, primarily reducing noise energy or performing spectral restoration on the speech signal to mitigate its impact. However, these methods are limited in their robustness and denoising effectiveness in complex noisy environments, and may not achieve ideal denoising results in practical applications. For example, filtering methods can selectively preserve the frequency components of the target signal while suppressing noise components. However, in dynamically changing noise environments, filtering methods may lead to signal distortion. Furthermore, filtering methods cannot effectively handle nonlinear noise, resulting in poor coherence of the generated denoised speech signal, leading to low accuracy and ultimately poor quality. With the rapid development of artificial intelligence, speech denoising methods based on deep neural networks have made significant progress, effectively addressing the problems faced by traditional denoising methods. The Transformer model has shown potential in the field of speech denoising. However, current Transformer-based speech denoising methods may neglect local correlations between pixels in pixel-level tasks. Moreover, compared to traditional neural network models, the Transformer model is more complex, with more parameters and computational load, resulting in low efficiency due to the heavy computational burden. Summary of the Invention

[0004] This application aims to address at least one of the technical problems existing in the prior art. To this end, embodiments of this application provide a speech denoising method, controller, and storage medium based on a cross-transformer, which helps to improve the accuracy of denoised speech signals, optimize the quality of denoised speech signals, and simultaneously improve the efficiency of the speech denoising method.

[0005] In a first aspect, embodiments of this application provide a speech denoising method based on cross-Transformer, including:

[0006] Obtain a speech dataset, add noise to the speech dataset to obtain a noise dataset;

[0007] The noise dataset is converted to a hierarchical training dataset.

[0008] A cross-Transformer neural network model is constructed, which includes a cross-Transformer network, a short-time Fourier transform fast calculation module, and a dual-domain superposition module;

[0009] The training dataset is input into the cross-transformer network for feature extraction to obtain the feature dataset.

[0010] The training dataset is input into the short-time Fourier transform fast calculation module for time-frequency feature extraction processing to obtain a time-frequency domain speech signal set;

[0011] The feature dataset and the time-frequency domain speech signal set are input into the dual-domain overlay module for smooth splicing to obtain a denoised speech set.

[0012] Using the speech dataset as the input to the cross-Transformer neural network model and the denoised speech set as the output of the cross-Transformer neural network model, the cross-Transformer neural network model is trained to obtain a trained cross-Transformer neural network model.

[0013] The speech signal to be processed is acquired and input into the trained cross-transformer neural network model to obtain a denoised speech signal.

[0014] According to some embodiments of this application, the cross-Transformer network includes convolutional layers, normalization layers, and a six-layer Transformer encoder layer, wherein the six-layer Transformer encoder layer has a cross structure; the step of inputting the training dataset into the cross-Transformer network for feature extraction processing to obtain a feature dataset includes:

[0015] The training dataset is input into a convolutional layer with a 1x3 kernel and a stride of 1x2, so that the input sequence length L of the training dataset is 512, the number of feature channels width is 64, and the data sample batch is 2.

[0016] The training dataset is normalized using a normalization layer with 512-dimensional features to obtain a normalized dataset.

[0017] The normalized dataset is input into the six-layer Transformer encoder layer. The six-layer Transformer encoder layer with different dimensions and cross structure performs feature extraction and encoding on the normalized dataset to obtain the feature dataset.

[0018] According to some embodiments of this application, the cross-Transformer network further includes a principal component analysis module, which is embedded in the first Transformer encoder layer, and the method further includes:

[0019] The feature dataset is input into the principal component analysis module.

[0020] The principal component analysis module performs dimensionality reduction on the feature dataset to compress the feature dimension of the feature dataset.

[0021] According to some embodiments of this application, the Transformer encoder layer includes a self-attention module, a feedforward module, a normalization and residual connection module, wherein the self-attention module is used to determine the feature dependencies of the input sequence of the normalized dataset in different subspaces, the feedforward module is used to perform feature extraction and transformation on the normalized dataset, and the normalization and residual connection module is used to perform normalization processing on the normalized dataset to obtain a feature dataset.

[0022] According to some embodiments of this application, the short-time Fourier transform fast calculation module includes a short-time Fourier transform layer and a two-dimensional convolutional layer. The step of inputting the feature dataset into the short-time Fourier transform fast calculation module for time-frequency feature extraction processing to obtain a time-frequency domain speech signal set includes:

[0023] The feature dataset is input into the short-time Fourier transform layer to identify the analysis window size of the feature dataset and the degree of overlap between adjacent frames through the short-time Fourier transform layer, and adjacent frames with an overlap degree greater than a preset threshold are divided into overlapping frames to obtain the time-frequency information of the feature dataset.

[0024] The time-frequency information of the feature dataset is decomposed into real and imaginary components. The real component includes the amplitude information of the feature dataset, and the imaginary component includes the phase information of the feature dataset.

[0025] The feature vector is input into the two-dimensional convolutional layer to compress the dimension of the feature vector, and a 1x1 convolutional layer is used to linearly transform the channel dimension of the feature vector to obtain a time-frequency domain speech signal set.

[0026] According to some embodiments of this application, the dual-domain overlay module includes a residual connection layer and an overlap add layer. The step of inputting the feature dataset and the time-frequency domain speech signal set into the dual-domain overlay module for smooth concatenation processing to obtain a denoised speech set includes:

[0027] The feature dataset and the time-frequency domain speech signal set are input into the residual connection layer to alleviate the gradient vanishing of the cross-Transformer neural network model;

[0028] The feature dataset and the time-frequency domain speech signal set are input into the Overlap Add layer, so that the overlapping frames of the feature dataset and the time-frequency domain speech signal set are processed by the Overlap Add layer to smoothly splice the features of multiple frames to obtain the denoised speech set.

[0029] According to some embodiments of this application, the Transformer encoder layer further includes a fully connected layer, which is connected to the output of the self-attention module. The fully connected layer includes a first linear transformation module and a first activation function, which are used to introduce nonlinear characteristics to enhance and learn the complex features output by the self-attention module.

[0030] According to some embodiments of this application, the cross-Transformer neural network model further includes a second linear transformation module and a second activation function, wherein the second linear transformation module and the second activation function are connected to the output of the cross-Transformer network and the output of the short-time Fourier transform quick calculation module, and the method further includes:

[0031] The feature dataset output by the cross-Transformer network and the time-frequency domain speech signal set output by the short-time Fourier transform fast calculation module are fused together with time-frequency features and contextual information;

[0032] The fused features are input into the second linear transformation module and the second activation function to optimize the fused features.

[0033] In a second aspect, embodiments of this application provide a controller, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the method described in the first aspect above when running the computer program.

[0034] Thirdly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the method described in the first aspect.

[0035] The speech denoising method, controller, and storage medium based on cross-Transformer provided in this application embodiment have at least one of the following advantages or beneficial effects: A clean speech dataset is obtained, and noise is manually added to the speech dataset to obtain a noisy dataset. The noisy dataset is then converted into a format and used as a training dataset. A neural network model based on cross-Transformer is constructed, comprising a cross-Transformer network, a short-time Fourier transform fast calculation module, and a dual-domain superposition module. The training dataset is input into the cross-Transformer network, which alternately uses encoder layers in different dimensions for feature extraction to obtain a feature dataset, capturing the global dependencies of the training data and reducing the number of model parameters and computational overhead. The training dataset is then input into the short-time Fourier transform fast calculation module, which pre-calculates the training data. The training dataset's speech data is repeatedly processed and stored in the GPU to improve computational efficiency, thereby enhancing the efficiency of the speech denoising method. It also rapidly calculates the time-frequency features of the training dataset's speech data, obtaining a time-frequency domain speech signal set, providing fundamental support for signal processing and enabling faster and better analysis and understanding of the spectral features of the time-frequency domain speech signal set. The feature dataset and the time-frequency domain speech signal set are input into a dual-domain overlay module, performing an overlap-addition operation to smoothly stitch together features from multiple frames, eliminating boundary discontinuities and producing a more natural and coherent output, resulting in a denoised speech set. This enhances the quality of the output denoised speech set, improves the accuracy of the denoised speech signal, and optimizes its overall quality. Using the speech dataset as input and the denoised speech set as output, a cross-Transformer neural network model is trained. After training, the speech signal to be processed is input into the trained cross-Transformer neural network model to obtain the denoised speech signal.

[0036] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0037] Figure 1 This is a flowchart of a speech denoising method based on cross-Transformer provided in an embodiment of this application;

[0038] Figure 2 This is a flowchart illustrating a method for obtaining a denoised speech signal by inputting a speech signal to be processed into a trained cross-transformer neural network model, as provided in an embodiment of this application.

[0039] Figure 3 This is a flowchart illustrating a method for obtaining a feature dataset by inputting a training dataset into a cross-Transformer network for feature extraction, as provided in an embodiment of this application.

[0040] Figure 4 This is a schematic diagram of the structure of a cross Transformer network provided in an embodiment of this application;

[0041] Figure 5 This is a flowchart illustrating a method for compressing the feature dimensions of a feature dataset using a principal component analysis module, as provided in an embodiment of this application.

[0042] Figure 6 This is a flowchart of a method for obtaining a time-frequency domain speech signal set by inputting a feature dataset into a short-time Fourier transform fast calculation module for time-frequency feature extraction, as provided in an embodiment of this application.

[0043] Figure 7 This is a flowchart of a method provided in this application embodiment for inputting a feature dataset and a time-frequency domain speech signal set into a dual-domain overlay module for smooth splicing processing to obtain a denoised speech set;

[0044] Figure 8 This is a flowchart of a speech denoising method based on cross-Transformer provided in another embodiment of this application;

[0045] Figure 9 This is a schematic diagram of the structure of a controller provided in an embodiment of this application. Detailed Implementation

[0046] This section will describe in detail the specific embodiments of this application. Preferred embodiments of this application are shown in the accompanying drawings. The purpose of the drawings is to supplement the textual description with graphics, so that people can intuitively and vividly understand each technical feature and the overall technical solution of this application, but they should not be construed as limiting the scope of protection of this application.

[0047] In the description of this application, "several" means one or more, "more than" means two or more, "greater than," "less than," "exceeding," etc. are understood to exclude the number itself, while "above," "below," "within," etc. are understood to include the number itself. "Any one" refers to one or more, and "at least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. If the terms "first" and "second" are used only for the purpose of distinguishing technical features, and should not be construed as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the order of the indicated technical features.

[0048] It should be noted that the terms "setting," "installing," and "connecting" in the embodiments of this application should be interpreted broadly. Those skilled in the art can reasonably determine the specific meaning of the above terms in the embodiments of this application based on the specific content of the technical solution. For example, the term "connection" can be a mechanical connection, an electrical connection, or a connection that allows for mutual communication; it can be a direct connection or an indirect connection through an intermediate medium.

[0049] It should be noted that the technical features involved in the various embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0050] Speech is one of the most commonly used communication tools for humans. However, the real world is full of various noise sources, such as environmental noise, microphone noise, and electronic device noise. These noises negatively affect speech quality and intelligibility. Speech denoising has been extensively studied in fields such as automatic speech recognition, hearing aids, and telecommunications. Therefore, speech denoising is an important research task.

[0051] Traditional speech denoising methods are typically based on signal processing techniques, primarily reducing noise energy or performing spectral restoration on the speech signal to mitigate its impact. However, these methods are limited in their robustness and denoising effectiveness in complex noisy environments, and may not achieve ideal denoising results in practical applications. For example, filtering methods (such as low-pass, high-pass, or band-pass filters) can selectively preserve the frequency components of the target signal while suppressing noise components. However, in dynamically changing noise environments, filtering methods may lead to signal distortion. Furthermore, filtering methods cannot effectively handle nonlinear noise, resulting in poor coherence of the denoised speech signal and low accuracy, thus leading to poor quality denoised speech signals. With the rapid development of artificial intelligence, speech denoising methods based on deep neural networks have made significant progress, effectively addressing the problems faced by traditional denoising methods. The Transformer model has shown potential in the field of speech denoising. However, current Transformer-based speech denoising methods may neglect local correlations between pixels in pixel-level tasks. Moreover, compared to traditional neural network models, the Transformer model is more complex, with more parameters and computational load, resulting in low efficiency due to the heavy computational burden.

[0052] Based on this, embodiments of this application provide a speech denoising method, controller, and storage medium based on cross-transformer, which helps to improve the accuracy of denoised speech signals, optimize the quality of denoised speech signals, and improve the efficiency of speech denoising methods.

[0053] The following description, with reference to the accompanying drawings, further illustrates the speech denoising method, controller, and storage medium based on the cross-transformer provided in the embodiments of this application.

[0054] Reference Figure 1 As shown, Figure 1 This is a flowchart of a speech denoising method based on cross-Transformer provided in an embodiment of this application. The speech denoising method based on cross-Transformer includes, but is not limited to, steps S100 to S800. Specifically,

[0055] Step S100: Obtain the speech dataset, add noise to the speech dataset to obtain the noise dataset;

[0056] Step S200: Convert the format of the noise dataset to obtain a hierarchical training dataset;

[0057] Step S300: Construct a cross-Transformer neural network model, which includes a cross-Transformer network, a short-time Fourier transform fast calculation module, and a dual-domain superposition module;

[0058] Step S400: Input the training dataset into the cross-Transformer network for feature extraction to obtain the feature dataset;

[0059] Step S500: Input the training dataset into the short-time Fourier transform fast calculation module for time-frequency feature extraction processing to obtain a time-frequency domain speech signal set;

[0060] Step S600: Input the feature dataset and the time-frequency domain speech signal set into the dual-domain overlay module for smooth splicing processing to obtain the denoised speech set;

[0061] Step S700: Use the speech dataset as the input of the cross-Transformer neural network model and the denoised speech dataset as the output of the cross-Transformer neural network model to train the cross-Transformer neural network model and obtain the trained cross-Transformer neural network model.

[0062] Step S800: Obtain the speech signal to be processed, input the speech signal to be processed into the trained cross-transformer neural network model, and obtain the denoised speech signal.

[0063] In some embodiments of this application, the speech denoising method based on cross-Transformer includes: acquiring a clean speech dataset, manually adding noise to the speech dataset to obtain a noisy dataset, and converting the noisy dataset into HDF5 hierarchical data as a training dataset. A neural network model based on cross-Transformer is constructed, comprising a cross-Transformer network, a short-time Fourier transform (SFT) fast calculation module, and a dual-domain stacking module. The training dataset is input into the cross-Transformer network, which alternately uses encoder layers in different dimensions to extract features from the training dataset, obtaining a feature dataset that captures the global dependencies of the training data, reducing the number of model parameters and computational overhead. The training dataset is then input into the SFT fast calculation module, which pre-calculates the SFT data. The process calculates and stores the intermediate repetitions of the speech data in the training dataset on the GPU, improving computational efficiency and thus enhancing the efficiency of the speech denoising method. It also rapidly calculates the time-frequency features of the speech data in the training dataset, obtaining a time-frequency domain speech signal set, providing fundamental support for signal processing and enabling faster and better analysis and understanding of the spectral characteristics of the time-frequency domain speech signal set. The feature dataset and the time-frequency domain speech signal set are input into a dual-domain overlay module, performing an overlap-addition operation to smoothly stitch together features from multiple frames, eliminating boundary discontinuities and producing a more natural and coherent output, resulting in a denoised speech set. This enhances the quality of the output denoised speech set, improves the accuracy of the denoised speech signal, and optimizes the overall quality of the denoised speech signal. Using the speech dataset as input and the denoised speech set as output, a cross-Transformer neural network model is trained. After training, the speech signal to be processed is input into the trained cross-Transformer neural network model to obtain the denoised speech signal.

[0064] Reference Figure 2 As shown, Figure 2This is a flowchart illustrating a method for obtaining a denoised speech signal by inputting a speech signal to be processed into a trained cross-Transformer neural network model, as provided in this application embodiment. The cross-Transformer neural network model includes a cross-Transformer network, a short-time Fourier transform (SFT) fast calculation module, and a dual-domain superposition module. The speech signal to be processed is acquired and input into the trained cross-Transformer neural network model. The cross-Transformer network alternately uses encoder layers in different dimensions to extract features from the speech signal. The SFT fast calculation module calculates the time-frequency features of the speech data of the speech signal to be processed. The signals output from the cross-Transformer network and the SFT fast calculation module are input together into the dual-domain superposition module. The dual-domain superposition module performs an overlap-addition operation, smoothly splicing features from multiple frames, eliminating boundary discontinuities, and producing a more natural and coherent output, thus obtaining a denoised speech signal, improving the accuracy of the denoised speech signal, and optimizing the quality of the denoised speech signal.

[0065] Reference Figure 3 As shown, Figure 3 This is a flowchart illustrating a method for inputting a training dataset into a cross-Transformer network for feature extraction to obtain a feature dataset, as provided in this application embodiment. The method includes, but is not limited to, steps S410 to S430. Specifically,

[0066] Step S410: Input the training dataset into a convolutional layer with a 1x3 kernel and a stride of 1x2, so that the length L of the input sequence of the training dataset is 512, the width of the feature channels is 64, and the batch size of the data samples is 2.

[0067] Step S420: Normalize the training dataset using a normalization layer with 512-dimensional features to obtain a normalized dataset;

[0068] Step S430: Input the normalized dataset into a six-layer Transformer encoder layer. The normalized dataset is processed by feature extraction and encoding through a six-layer Transformer encoder layer with different cross-structures to obtain the feature dataset.

[0069] In some embodiments of this application, a cross-Transformer network is responsible for feature extraction and encoding of the input signal. The Transformer structure can capture long-range dependencies in the input sequence. The cross-Transformer network includes convolutional layers, normalization layers, and six Transformer encoder layers. The six Transformer encoder layers are a cross structure. The method of inputting the training dataset into the cross-Transformer network for feature extraction to obtain the feature dataset includes: first, inputting the training dataset into a convolutional layer with a 1x3 kernel and a stride of 1x2, setting the input sequence length L of the training dataset to 512, the feature channel width to 64, and the data sample batch size to 2; then performing normalization processing using a normalization layer with 512-dimensional features to obtain a normalized dataset. The cross-Transformer network contains six Transformer encoder layers, which are used alternately in different dimensions. The six Transformer encoder layers perform feature extraction and encoding processing on the input normalized dataset to form the feature dataset.

[0070] In this application, the cross-Transformer network alternates between encoder layers in different dimensions to perform feature extraction processing, obtains a feature dataset, captures the global dependencies of the training data, reduces the number of model parameters and computational overhead, improves computational efficiency, and thus improves the efficiency of the speech denoising method.

[0071] Reference Figure 4 As shown, Figure 4 This is a schematic diagram of a cross-Transformer network structure provided in an embodiment of this application. The cross-Transformer network includes six Transformer encoder layers. These six layers are cross-connected, allowing them to share information. The output of each encoder layer not only affects subsequent processing within the same layer but also influences the computation of other layers, enhancing feature learning capabilities. Through the interaction between layers, the stacking of multiple encoder layers allows the cross-Transformer neural network model to extract multi-level features from the input signal. The cross-Transformer neural network model can better understand the structure and patterns in the signal, and by combining contextual information from different layers, it improves the understanding of speech signals in the training dataset, especially in noisy environments, enhancing signal clarity and intelligibility. Cross-connections improve the robustness of the cross-Transformer neural network model to input variations, enabling it to maintain good performance even under input noise or distortion, reducing the risk of overfitting and thus improving performance.

[0072] Reference Figure 5 As shown, Figure 5 This is a flowchart illustrating a method for compressing the feature dimensions of a feature dataset using a principal component analysis module, as provided in this application embodiment. The method includes, but is not limited to, steps S900 to S910. Specifically,

[0073] Step S900: Input the feature dataset into the principal component analysis module.

[0074] Step S910: Perform dimensionality reduction on the feature dataset using the principal component analysis module to compress the feature dimension of the feature dataset.

[0075] In some embodiments of this application, the cross-Transformer network further includes a principal component analysis (PCA) module embedded in the first Transformer encoder layer. The PCA module compresses the feature dimension of the feature dataset, reducing the complexity and computational burden of the cross-Transformer neural network model. The method for compressing the feature dimension of the feature dataset using the PCA module includes: inputting the feature dataset into the PCA module, and performing dimensionality reduction processing on the feature dataset to compress its feature dimension. This reduces the burden on the input feature dataset, effectively capturing the global dependencies of the input feature dataset, reducing the number of model parameters and computational overhead of the cross-Transformer neural network model, while retaining key feature information. Therefore, unlike traditional Transformer modules, embedding the PCA module into the Transformer encoder layer in this embodiment reduces the number of model parameters and computational overhead, lowers model complexity, removes redundant features, and thus improves the performance of the cross-Transformer neural network model.

[0076] In some embodiments of this application, the Transformer encoder layer includes a self-attention module, a feedforward module, a normalization and residual connection module. The self-attention module is used to determine the feature dependencies of the input sequence of the normalized dataset in different subspaces. The feedforward module is used to extract and transform features from the normalized dataset. The normalization and residual connection module is used to normalize the normalized dataset to obtain a feature dataset.

[0077] Understandably, each Transformer encoder layer comprises three modules: a self-attention module, a feedforward module, and a normalization and residual connection module. The self-attention module employs a multi-head self-attention mechanism to process the input tensor of the normalized dataset, determining the dependencies of features in different subspaces of the input sequence. This allows it to learn rich features from different representation subspaces of the speech data. The feedforward module uses a two-layer feedforward neural network for feature extraction and transformation, applying the non-linear activation function PReLU between the two layers to enhance the expressive power of the cross-Transformer neural network model. This enables the model to learn more valuable feature representations, improving the accuracy and quality of the denoised speech signal. The normalization and residual connection module normalizes the normalized dataset to obtain a feature dataset. Using this module stabilizes the input distribution, accelerates the training convergence of the cross-Transformer neural network model, and increases its training speed.

[0078] Understandably, by including a multi-head self-attention mechanism in each Transformer encoder layer, the important features are highlighted by calculating a weighted sum of the input dataset, while suppressing unimportant information. This allows the cross-Transformer neural network model to learn the global dependencies of the input sequence in parallel across different subspaces. Through the multi-head mechanism, the cross-Transformer neural network model can focus on different aspects of the speech signal in the input dataset across different attention heads, thereby capturing richer contextual information. Adding residual connections between the output and input of the multi-head attention layer helps the cross-Transformer neural network model achieve stable convergence during training. Simultaneously, layer normalization standardizes the output, improving the training efficiency of the cross-Transformer neural network model. This structure alleviates the gradient vanishing problem in deep networks, allowing information to be transferred more effectively. Each Transformer encoder layer also includes a regular convolutional layer; the 1x1 convolution is used to adjust the feature dimension, helping the cross-Transformer neural network model maintain spatial information during feature extraction. It can change the number of channels without altering the feature space dimension, enabling the cross-Transformer neural network model to flexibly handle features of different dimensions.

[0079] Reference Figure 6 As shown, Figure 6This is a flowchart illustrating a method for obtaining a time-frequency domain speech signal set by inputting a feature dataset into a short-time Fourier transform (SFT) fast calculation module for time-frequency feature extraction, as provided in this application embodiment. The method includes, but is not limited to, steps S510 to S530. Specifically,

[0080] Step S510: Input the feature dataset into the short-time Fourier transform layer to identify the analysis window size of the feature dataset and the degree of overlap between adjacent frames through the short-time Fourier transform layer, and divide adjacent frames with an overlap degree greater than a preset threshold into overlapping frames to obtain the time-frequency information of the feature dataset.

[0081] Step S520: Decompose the time-frequency information of the feature dataset into real and imaginary branches, and concatenate the real and imaginary branches to form a feature vector, wherein the real part includes the amplitude information of the feature dataset, and the imaginary part includes the phase information of the feature dataset.

[0082] Step S530: Input the feature vector into a two-dimensional convolutional layer to compress the dimension of the feature vector through the two-dimensional convolutional layer, and perform a linear transformation on the channel dimension of the feature vector through a 1x1 convolutional layer to obtain a time-frequency domain speech signal set.

[0083] In some embodiments of this application, the short-time Fourier transform fast calculation module includes a short-time Fourier transform layer and a two-dimensional convolutional layer. The short-time Fourier transform layer is responsible for pre-calculating the intermediate amount of repeated speech data in the feature dataset, improving computational efficiency, and analyzing and understanding the spectral characteristics of time-domain signals faster and better, providing basic support for various signal processing and analysis tasks.

[0084] By inputting the feature dataset into the Short-Time Fourier Transform (SFT) fast calculation module for time-frequency feature extraction, a time-frequency domain speech signal set is obtained. This process includes: First, the feature dataset is input into the SFT layer. The SFT layer identifies the analysis window size of the feature dataset and the degree of overlap between adjacent frames, classifying adjacent frames with an overlap greater than a preset threshold as overlapping frames, thus obtaining the time-frequency information of the feature dataset. To optimize performance, the SFT fast calculation module pre-calculates the real part (DR), imaginary part (DI), and Hamming window (W) of the Discrete Fourier Transform matrix and stores them in the GPU, thereby achieving faster computation speed and practicality while reducing memory usage. Next, the SFT layer outputs the time-frequency information of the feature dataset, decomposing it into real and imaginary branches. The real part includes the amplitude information of the feature dataset, and the imaginary part includes the phase information. Subsequently, the real and imaginary branches are concatenated to form a richer feature vector. Feature vectors are input into a 2D convolutional layer, which further processes these feature vectors, compressing the 128-dimensional feature vectors to 64 dimensions. This 2D convolutional layer effectively reduces the complexity and number of parameters in the cross-Transformer neural network model by compressing the dimensionality of the feature vectors. A 1x1 convolution linearly transforms the channel dimension of the feature vectors, more accurately capturing the key features of the speech data to obtain a time-frequency domain speech signal set. The short-time Fourier transform module processes the real and imaginary parts of the feature vectors, fully preserving the amplitude and phase information of the feature vector signal. This allows the cross-Transformer neural network model to learn feature representations suitable for specific tasks, thereby enhancing denoising performance and enabling faster and better analysis and understanding of the spectral characteristics of the time-frequency domain speech signal set. This improves the accuracy and quality of the denoised speech signal.

[0085] Reference Figure 7 As shown, Figure 7 This is a flowchart illustrating a method for obtaining a denoised speech set by inputting a feature dataset and a time-frequency domain speech signal set into a dual-domain overlay module for smooth concatenation processing, as provided in this application embodiment. The method includes, but is not limited to, steps S610 to S620. Specifically,

[0086] Step S610: Input the feature dataset and the time-frequency domain speech signal set into the residual connection layer to alleviate gradient vanishing in the cross-Transformer neural network model;

[0087] Step S620: Input the feature dataset and the time-frequency domain speech signal set into the Overlap Add layer, so that the Overlap Add layer performs overlap-add processing on the overlapping frames of the feature dataset and the time-frequency domain speech signal set, smoothly splices the features of multiple frames, and obtains the denoised speech set.

[0088] In some embodiments of this application, the dual-domain overlay module includes a residual connection layer and an Overlap Add layer. The residual connection layer can effectively preserve the original information and alleviate the gradient vanishing problem in the cross-Transformer neural network model, making training more stable and improving model performance. The feature dataset and time-frequency domain speech signal set are input into the dual-domain overlay module. The residual connection layer effectively preserves the original information of the feature dataset and time-frequency domain speech signal set, alleviating the gradient vanishing phenomenon in the cross-Transformer neural network model. Then, the feature dataset and time-frequency domain speech signal set are input into the Overlap Add layer. The Overlap Add layer adjusts the feature dimension of the dataset through a 1x1 convolutional layer and performs overlap-add processing on the overlapping frames of the feature dataset and time-frequency domain speech signal set, dividing them into multiple overlapping frames. Each frame contains a certain overlap, helping the cross-Transformer neural network model capture more continuous information in the time domain. The overlapping parts are summed to smoothly stitch together the features from multiple frames, effectively combining information between different frames and reducing discontinuities caused by boundary effects. This results in a natural and coherent output denoised speech set, yielding a stable and high-quality denoised speech set that better matches human auditory or visual perception. Through residual connections and overlap-add operations, the signal quality of the denoised speech is effectively enhanced, producing a more natural and coherent output and improving the accuracy of the denoised speech signal.

[0089] In some embodiments of this application, the Transformer encoder layer further includes a fully connected layer, and the output connection between the fully connected layer and the self-attention module is provided. The fully connected layer includes a first linear transformation module and a first activation function. The first linear transformation module and the first activation function are used to introduce nonlinear characteristics to enhance and learn the complex features of the output of the self-attention module.

[0090] The Transformer encoder layer contains a fully connected layer, which is connected to the output of the self-attention module for further processing of the output from the multi-head attention layer. The fully connected layer includes a first linear transformation module and a first activation function. The first linear transformation module comprises two linear transformations to introduce non-linear characteristics, which helps enhance the expressive power of the cross-Transformer neural network model and allows for the learning of complex feature maps. Similar to the multi-head attention module, the output of the fully connected layer is added to the input through a residual connection and then normalized to ensure that the Transformer encoder layer effectively utilizes information from the previous layer at each layer, maintaining information flow and model stability.

[0091] Reference Figure 8 As shown, Figure 8 This is a flowchart of a speech denoising method based on cross-Transformer provided in another embodiment of this application. The speech denoising method based on cross-Transformer further includes steps S920 to S930, specifically,

[0092] Step S920: Combine the feature dataset output by the cross-Transformer network and the time-frequency domain speech signal set output by the short-time Fourier transform fast calculation module with time-frequency features and contextual information for fusion;

[0093] Step S930: Input the fused features into the second linear transformation module and the second activation function to optimize the fused features through the second linear transformation module and the second activation function.

[0094] In some embodiments of this application, the cross-Transformer neural network model further includes a second linear transformation module and a second activation function. The second linear transformation module and the second activation function are connected to the output of the cross-Transformer network and the output of the short-time Fourier transform fast calculation module. The speech denoising method of the cross-Transformer further includes fusing the feature dataset output by the cross-Transformer network and the time-frequency domain speech signal set output by the short-time Fourier transform fast calculation module with time-frequency features and context information. The fused features are then subjected to nonlinear transformation by the second linear transformation module and the second activation function, and further optimized by layer normalization and parameterized ReLU activation. Finally, the optimized feature dataset and time-frequency domain speech signal set are divided into multiple overlapping frames through convolutional layers and a dual-domain overlay module. Each frame contains a certain overlap, which helps the cross-Transformer neural network model capture more continuous information in the time domain. The overlapping parts are summed to effectively combine the information between different frames, reduce the discontinuity caused by boundary effects, and make the output denoised speech signal more coherent in time, thereby improving the accuracy and quality of the denoised speech signal.

[0095] Reference Figure 9 , Figure 9This is a schematic diagram of the structure of a controller 1000 provided in an embodiment of this application. It includes a processor 1001, which can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the method provided in the embodiment of this application; and a memory 1002, which can be implemented using a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM), etc. The memory 1002 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001. The input / output interface 1003 is used to implement information input and output. The communication interface 1004 is used to realize communication interaction between this device and other devices. Communication can be realized by wired means (e.g., USB, network cable, etc.) or by wireless means (e.g., mobile network, WIFI, Bluetooth, etc.). The bus transmits information between the various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003 and communication interface 1004). The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device through the bus.

[0096] Those skilled in the art will understand that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer-readable storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer-readable storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0097] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings.

Claims

1. A speech denoising method based on cross-Transformer, characterized in that, include: Obtain a speech dataset, add noise to the speech dataset to obtain a noise dataset; The noise dataset is converted to a hierarchical training dataset. A cross-Transformer neural network model is constructed, which includes a cross-Transformer network, a short-time Fourier transform fast calculation module, and a dual-domain superposition module; The training dataset is input into the cross-transformer network for feature extraction to obtain the feature dataset. The training dataset is input into the short-time Fourier transform fast calculation module for time-frequency feature extraction processing to obtain a time-frequency domain speech signal set; The feature dataset and the time-frequency domain speech signal set are input into the dual-domain overlay module for smooth splicing to obtain a denoised speech set. Using the speech dataset as the input to the cross-Transformer neural network model and the denoised speech set as the output of the cross-Transformer neural network model, the cross-Transformer neural network model is trained to obtain a trained cross-Transformer neural network model. The speech signal to be processed is acquired and input into the trained cross-transformer neural network model to obtain a denoised speech signal.

2. The speech denoising method based on cross-Transformer according to claim 1, characterized in that, The cross-Transformer network includes convolutional layers, normalization layers, and a six-layer Transformer encoder layer, wherein the six-layer Transformer encoder layer has a cross structure; the training dataset is input into the cross-Transformer network for feature extraction processing. The feature dataset is obtained, including: The training dataset is input into a convolutional layer with a 1x3 kernel and a stride of 1x2, so that the input sequence length L of the training dataset is 512, the number of feature channels width is 64, and the data sample batch is 2. The training dataset is normalized using a normalization layer with 512-dimensional features to obtain a normalized dataset. The normalized dataset is input into the six-layer Transformer encoder layer. The six-layer Transformer encoder layer with different dimensions and cross structure performs feature extraction and encoding on the normalized dataset to obtain the feature dataset.

3. The speech denoising method based on cross-Transformer according to claim 2, characterized in that, The cross-Transformer network further includes a principal component analysis module, which is embedded in the first Transformer encoder layer. The method further includes: The feature dataset is input into the principal component analysis module. The principal component analysis module performs dimensionality reduction on the feature dataset to compress the feature dimension of the feature dataset.

4. The speech denoising method based on cross-Transformer according to claim 2, characterized in that, The Transformer encoder layer includes a self-attention module, a feedforward module, a normalization module, and a residual connection module. The self-attention module is used to determine the feature dependencies of the input sequence of the normalized dataset in different subspaces. The feedforward module is used to extract and transform features from the normalized dataset. The normalization module and residual connection module are used to normalize the normalized dataset to obtain a feature dataset.

5. The speech denoising method based on cross-Transformer according to claim 1, characterized in that, The short-time Fourier transform (SFT) fast calculation module includes a SFT layer and a two-dimensional convolutional layer. The feature dataset is input into the SFT fast calculation module for time-frequency feature extraction processing to obtain a time-frequency domain speech signal set, including: The feature dataset is input into the short-time Fourier transform layer to identify the analysis window size of the feature dataset and the degree of overlap between adjacent frames through the short-time Fourier transform layer, and adjacent frames with an overlap degree greater than a preset threshold are divided into overlapping frames to obtain the time-frequency information of the feature dataset. The time-frequency information of the feature dataset is decomposed into real and imaginary components. The real component includes the amplitude information of the feature dataset, and the imaginary component includes the phase information of the feature dataset. The feature vector is input into the two-dimensional convolutional layer to compress the dimension of the feature vector, and a 1x1 convolutional layer is used to linearly transform the channel dimension of the feature vector to obtain a time-frequency domain speech signal set.

6. The speech denoising method based on cross-Transformer according to claim 5, characterized in that, The dual-domain overlay module includes a residual connection layer and an overlap add layer. The feature dataset and the time-frequency domain speech signal set are input into the dual-domain overlay module for smooth concatenation processing to obtain a denoised speech set, including: The feature dataset and the time-frequency domain speech signal set are input into the residual connection layer to alleviate the gradient vanishing of the cross-Transformer neural network model; The feature dataset and the time-frequency domain speech signal set are input into the Overlap Add layer, so that the overlapping frames of the feature dataset and the time-frequency domain speech signal set are processed by the Overlap Add layer to smoothly splice the features of multiple frames to obtain the denoised speech set.

7. The speech denoising method based on cross-Transformer according to claim 4, characterized in that, The Transformer encoder layer further includes a fully connected layer, which is connected to the output of the self-attention module. The fully connected layer includes a first linear transformation module and a first activation function, which are used to introduce nonlinear characteristics to enhance and learn the complex features output by the self-attention module.

8. The speech denoising method based on cross-Transformer according to claim 6, characterized in that, The cross-Transformer neural network model further includes a second linear transformation module and a second activation function, which are connected to the output of the cross-Transformer network and the output of the short-time Fourier transform fast calculation module. The method further includes: The feature dataset output by the cross-Transformer network and the time-frequency domain speech signal set output by the short-time Fourier transform fast calculation module are fused together with time-frequency features and contextual information; The fused features are input into the second linear transformation module and the second activation function to optimize the fused features.

9. A controller, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, performs the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the method as described in any one of claims 1 to 8.

Images