A Deep Learning-Based Speech Enhancement and Emotion Recognition Method for Noisy Environments

Abstract

This application relates to the field of speech signal processing technology and discloses a deep learning-based speech enhancement and emotion recognition method for noisy environments, aiming to solve the speech enhancement problem. The method includes: preprocessing the input noisy speech signal to extract its time-frequency domain feature representation; inputting the time-frequency domain features into a cascaded deep neural network architecture, which includes a speech enhancement module based on attention mechanism and multi-scale convolution and an emotion recognition module incorporating prior knowledge of emotion; in the speech enhancement module, through adaptive noise estimation and feature masking mechanisms, an enhancement masking matrix for the current speech frame is generated and applied to the input features to suppress noise components. By adopting the above technical solution, this application can improve the signal-to-noise ratio while retaining key acoustic cues for emotion discrimination, enhance the model's generalization ability to different types of noise, and improve the robustness and discriminativeness of emotion feature extraction.

Description

Technical Field

[0001] This invention belongs to the field of speech signal processing technology, specifically relating to a deep learning speech enhancement and emotion recognition method for noisy environments. Background Technology

[0002] With the continuous evolution of artificial intelligence and human-computer interaction technologies, speech signal processing plays an increasingly crucial role in scenarios such as intelligent customer service, in-vehicle systems, remote conferencing, and mental health assistance. As a vital carrier of semantic and emotional information, the clarity and integrity of speech directly impact the accuracy of subsequent recognition tasks. However, in real-world applications, speech signals are highly susceptible to various noise interferences, including steady-state background noise, non-steady-state burst noise, and reverberation effects, leading to a significant deterioration in speech quality and consequently affecting speech enhancement and emotion recognition performance.

[0003] Deep learning-driven speech enhancement technology has become a research hotspot in recent years, with its core objective being to recover high-quality, clean speech signals from noisy speech using neural network models. These methods typically rely on large-scale training data and complex network structures to learn the nonlinear mapping relationship between noise and speech, thereby achieving effective suppression of noise components and preservation of speech features.

[0004] Existing technologies still face multiple challenges in noisy environments: First, most speech enhancement models assume that the noise type is known or has a stable distribution during the training phase, making it difficult to generalize to complex and varied real-world noise scenarios. Second, the enhancement process often excessively smooths speech details, resulting in the loss of emotion-related prosodic features and weakening the discriminative ability of downstream emotion recognition modules. Third, speech enhancement and emotion recognition are usually designed as two independent tasks, lacking an end-to-end joint optimization mechanism, leading to improved auditory quality but a lack of synchronous improvement in emotion representation. Finally, under low signal-to-noise ratio conditions, existing methods are prone to artificial distortion or residual noise, further interfering with the stability of emotion feature extraction. These problems make achieving high-quality speech enhancement and high-precision emotion recognition simultaneously in high-noise environments a technical challenge that urgently needs to be overcome. Summary of the Invention

[0005] The purpose of this invention is to provide a deep learning speech enhancement and emotion recognition method for noisy environments. By designing a cascaded, end-to-end jointly optimized deep neural network architecture, speech enhancement and emotion recognition tasks are deeply integrated, avoiding the unintentional destruction of emotional features by the enhancement process in the traditional serial process. This ensures that the enhanced speech retains the key acoustic cues for emotion discrimination to the greatest extent while improving the signal-to-noise ratio.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A deep learning-based speech enhancement and emotion recognition method for noisy environments, comprising the following steps:

[0008] Step S110 involves preprocessing the input noisy speech signal to extract its time-frequency domain feature representation. Specifically, this includes: performing frame segmentation and windowing on the original time-domain speech signal; applying short-time Fourier transform to each frame of the speech signal to calculate its complex spectrum; extracting the amplitude spectrum of the complex spectrum as the basic time-frequency domain feature, and further calculating its Mel-frequency cepstral coefficients and fundamental frequency contour features to form a multidimensional feature vector.

[0009] Step S120: Input the time-frequency domain features into a cascaded deep neural network architecture, which includes a speech enhancement module based on attention mechanism and multi-scale convolution and an emotion recognition module that integrates prior emotional knowledge.

[0010] Step S130: In the speech enhancement module, an enhancement masking matrix for the current speech frame is generated through an adaptive noise estimation and feature masking mechanism, and applied to the input features to suppress noise components.

[0011] Step S140: Perform residual connection between the enhanced features output by the speech enhancement module and the original input features, and input them into the emotion recognition module;

[0012] Step S150: In the emotion recognition module, a multi-head self-attention mechanism is used to capture long-range contextual dependencies related to emotions in the enhanced speech, and combined with pre-trained prior emotional acoustic features, to classify emotional states.

[0013] Step S160: Design a joint loss function that simultaneously constrains the reconstruction accuracy of the speech enhancement task and the classification accuracy of the emotion recognition task, and perform end-to-end joint training and optimization on the cascaded deep neural network architecture.

[0014] Preferably, in step S130, in the speech enhancement module, an enhancement masking matrix for the current speech frame is generated through an adaptive noise estimation and feature masking mechanism, and applied to the input features to suppress noise components; specifically, after the multi-scale convolutional attention block of the speech enhancement module, a gated recurrent unit network is connected to model the temporal dynamics of speech features.

[0015] Preferably, the last hidden state of the gated recurrent unit network is fed into a noise estimation sub-network, which consists of two fully connected layers and outputs a noise estimation vector with the same dimension as the input features. The noise estimation vector is then compared element-wise with the output features of the multi-scale convolutional attention block, and a soft masking matrix between 0 and 1 is generated by a sigmoid activation function. ,in Represents the time frame index. Represents the frequency index; the soft masking matrix is ​​applied to the input feature amplitude spectrum according to the following formula. The enhanced characteristic amplitude spectrum is obtained as follows:

[0016] .

[0017] Preferably, in step S140, the step of performing a residual connection between the enhanced features output by the speech enhancement module and the original input features, and then inputting them into the emotion recognition module, specifically involves: processing the amplitude spectrum of the enhanced features output by the speech enhancement module. The amplitude spectrum of the original input features extracted in step S110 By adding elements one by one, we obtain the fusion feature:

[0018] ,

[0019] in It is a learnable scaling factor.

[0020] Preferably, the emotion recognition module in step S150 is a fusion layer of a multi-head self-attention mechanism and an emotion prior; the input of the multi-head self-attention mechanism is the fused feature. The sequence representation is obtained after a linear projection layer; the emotion prior fusion layer loads an acoustic feature extractor pre-trained on a clean emotion speech dataset, which has fixed parameters for extracting features from the fused features. Deep features highly correlated with emotions are extracted; the self-attention enhanced feature representation is concatenated with the pre-extracted deep emotional features along the channel dimension; finally, a classifier consisting of two fully connected layers and a softmax layer is used to output the probability distribution corresponding to different emotion categories.

[0021] Preferably, the joint loss function It consists of a weighted sum of two parts:

[0022] ,

[0023] in, For speech enhancement loss, the scale-invariant signal-to-distortion ratio is used as a metric, and its calculation formula is as follows:

[0024] ,

[0025] in For the reconstructed time-domain speech signal, This serves as a reference for the corresponding clean speech signal; For emotion recognition loss, a labeled smoothed cross-entropy loss function is used; weight coefficients... and Determined through grid search.

[0026] This invention also discloses a deep learning-based speech enhancement and emotion recognition system for noisy environments, the system comprising the following components:

[0027] Preprocessing unit: used to preprocess the input noisy speech signal and extract its time-frequency domain feature representation. Specifically, it includes: performing frame segmentation and windowing processing on the original time-domain speech signal, applying short-time Fourier transform to each frame of speech signal, and calculating its complex spectrum; extracting the amplitude spectrum of the complex spectrum as the basic time-frequency domain feature, and further calculating its Mel-frequency cepstral coefficients and fundamental frequency contour features to form a multi-dimensional feature vector.

[0028] Cascaded deep neural network architecture: includes a speech enhancement module based on attention mechanism and multi-scale convolution and an emotion recognition module that integrates prior emotional knowledge, for receiving the multi-dimensional feature vector output by the preprocessing unit;

[0029] The speech enhancement module is configured to generate an enhancement masking matrix for the current speech frame through an adaptive noise estimation and feature masking mechanism, and apply it to the input features to suppress noise components; the emotion recognition module is configured to use a multi-head self-attention mechanism to capture long-range contextual dependencies related to emotions in the enhanced speech, and combine it with pre-trained emotional acoustic feature priors to classify emotional states.

[0030] The system also includes a training optimization unit configured to apply a joint loss function to perform end-to-end joint training and optimization of the cascaded deep neural network architecture.

[0031] Preferably, the voice enhancement module includes:

[0032] A feature encoder, consisting of multiple one-dimensional convolutional layers, is used to map the input multi-dimensional feature vector to a high-dimensional feature space.

[0033] Multi-scale convolutional attention blocks contain multiple depthwise separable convolutional branches with different kernel sizes in parallel, and each branch connects to a channel attention submodule;

[0034] A gated recurrent unit network, connected after the multi-scale convolutional attention block, is used to model the temporal dynamics of speech features;

[0035] The noise estimation subnetwork, connected to the output of the gated recurrent unit network, consists of two fully connected layers and is used to output the noise estimation vector.

[0036] The masking generation and application unit is configured to compare the noise estimation vector with the output features of the multi-scale convolutional attention block, and generate a soft masking matrix using a sigmoid activation function. and according to the formula The characteristic amplitude spectrum acting on the input The enhanced characteristic amplitude spectrum is obtained. .

[0037] Preferably, the emotion recognition module includes:

[0038] Linear projection layer, used to fuse the input features. Convert to sequence representation;

[0039] Multi-head self-attention layer, used to compute self-attention-enhanced feature representations;

[0040] An emotion-prior fusion layer is loaded with a pre-trained acoustic feature extractor for extracting features from the fused data. Extracting deep emotional features;

[0041] The feature concatenation layer is used to concatenate the self-attention-enhanced feature representation with the deep emotion feature along the channel dimension.

[0042] The classifier, consisting of two fully connected layers and a softmax layer, is used to output the probability distribution of sentiment categories.

[0043] The technical effects and advantages of the present invention in the above technical solution are as follows:

[0044] 1. In this invention, by designing a cascaded, end-to-end jointly optimized deep neural network architecture, speech enhancement and emotion recognition tasks are deeply integrated, avoiding the unintentional destruction of emotional features by the enhancement process in the traditional serial process, and ensuring that the enhanced speech retains the key acoustic cues for emotion discrimination to the greatest extent while improving the signal-to-noise ratio.

[0045] 2. The proposed speech enhancement module based on attention mechanism and multi-scale convolution can adaptively estimate and suppress complex and variable noise. Its multi-scale convolution structure enhances the model's generalization ability to different types of noise (steady-state and non-steady-state), while the channel attention mechanism strengthens the focus on the dominant frequency band features of speech and reduces the loss of emotional prosodic details caused by excessive smoothing.

[0046] 3. In this invention, the emotion recognition module, which introduces prior knowledge of emotion and combines it with a multi-head self-attention mechanism, can effectively capture long-range contextual dependencies and deep acoustic patterns related to emotion in enhanced speech, thereby improving the robustness and discriminativeness of emotion feature extraction under conditions of residual noise or low signal-to-noise ratio.

[0047] 4. In this invention, a joint loss function is used to optimize the overall model, so that the goal of the speech enhancement sub-network is not only to minimize speech waveform distortion, but also to improve the accuracy of downstream emotion classification. This achieves performance balance and joint optimization of the two sub-tasks at the system level, and solves the performance bottleneck problem caused by independent optimization. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the overall technical solution architecture of the deep learning speech enhancement and emotion recognition method for noisy environments proposed in this invention.

[0049] Figure 2 This is a schematic diagram of the core principle framework of the speech enhancement module based on attention mechanism and multi-scale convolution in this invention;

[0050] Figure 3 This is a schematic diagram of the multi-head self-attention and emotion prior fusion principle framework of the emotion recognition module in this invention;

[0051] Figure 4 This is a flowchart illustrating the logical flow of the end-to-end joint training of speech enhancement and emotion recognition tasks in this invention.

[0052] Figure 5 This is a schematic diagram illustrating the principle of the adaptive noise estimation and feature masking mechanism in this invention. Detailed Implementation

[0053] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the specific embodiments according to the present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments.

[0054] Example 1

[0055] In practical applications such as intelligent cockpit in-vehicle voice interaction systems, remote psychological service hotline centers, and intelligent monitoring in noisy public places, background noise severely interferes with voice communication quality and voice-based emotion analysis. For example, in a moving car, engine noise, wind noise, and road noise can contaminate the driver's or passenger's voice commands and emotional expressions; in customer service hotlines, environmental noise may mask emotional cues in the user's voice, affecting service quality assessment. The method provided by this invention aims to solve the technical challenge of simultaneously achieving high-fidelity voice enhancement and high-precision emotion recognition in such complex noisy environments.

[0056] See Figure 1 The overall technical architecture of this invention includes a cascaded deep neural network. This network takes a noisy speech signal as input and processes it sequentially through a preprocessing unit, a speech enhancement module, and an emotion recognition module, ultimately outputting an enhanced speech signal and its corresponding emotion category probability. The entire architecture is trained and optimized end-to-end using a joint loss function, ensuring that the speech enhancement and emotion recognition subtasks work collaboratively.

[0057] First, step S110 is executed to preprocess the input noisy speech signal and extract its time-frequency domain feature representation. The specific operation is as follows: The system receives the original time-domain speech signal, which is typically a mono PCM format audio stream with a sampling rate of 16kHz. The preprocessing unit first performs frame processing on the continuous speech stream, with the frame length strictly set to 25 milliseconds, corresponding to 400 sampling points; the frame shift is set to 10 milliseconds, i.e., 160 sampling points, thus achieving a balance between time resolution and computational efficiency. A Hanning window function is applied to each frame of the speech signal to reduce spectral leakage. Subsequently, a 256-point short-time Fourier transform is applied to each windowed frame to calculate its complex spectrum. From this complex spectrum, the amplitude spectrum is extracted as the most basic time-frequency domain feature, with a dimension of 129×T, where 129 is the number of frequency points and T is the total number of frames. To further enrich the feature representation, the system calculates the Mel-frequency cepstral coefficients and fundamental frequency contour features of the speech frame in parallel. The calculation process for the Mel frequency cepstral coefficients includes: mapping the amplitude spectrum to a 40-channel Mel filter bank, taking the logarithm, applying a discrete cosine transform, and finally selecting the first 13 dimensions of coefficients and their first and second-order differences to form a 39-dimensional feature vector. The fundamental frequency contour feature is calculated by estimating the fundamental frequency of each frame using the autocorrelation function method or the cepstral method, and then calculating its smooth trajectory over multiple consecutive frames. Finally, the amplitude spectrum, the 39-dimensional Mel frequency cepstral coefficient feature, and the fundamental frequency feature of each frame are concatenated along the feature dimensions to form a high-dimensional multidimensional feature vector, which serves as the input to the subsequent deep neural network.

[0058] Step S120: The multi-dimensional feature vector obtained from the preprocessing is input into a cascaded deep neural network architecture. The core of this architecture consists of two main modules connected sequentially: a speech enhancement module based on attention mechanisms and multi-scale convolutions, and an emotion recognition module that integrates prior knowledge of emotions. The speech enhancement module is responsible for separating and suppressing noise components from noisy features; the emotion recognition module is responsible for parsing the user's emotional state from the enhanced features.

[0059] Step S130 specifically implements the adaptive noise estimation and feature masking mechanism within the speech enhancement module. See [link / reference] Figure 2 The speech enhancement module first processes the input multidimensional feature vector through a feature encoder. This feature encoder consists of three stacked one-dimensional convolutional layers, each followed by a batch normalization layer and a ReLU activation function. The first convolutional layer has a kernel size of 7 and a stride of 1, expanding the number of channels from the input feature dimension to 64; the second and third convolutional layers both have a kernel size of 5, with the number of channels increasing to 128 and 256 respectively. The function of this encoder is to map the input time-frequency features to a higher-dimensional, more expressive feature space.

[0060] The encoder output is fed into a multi-scale convolutional attention block. This block is designed to capture local acoustic patterns at different time scales in parallel. It contains three independent branches, each with a depthwise separable convolutional layer at its core, but with different kernel sizes of 3, 5, and 7 respectively. The depthwise separable convolution performs channel-wise convolution followed by pointwise convolution, significantly reducing the number of parameters while maintaining performance. After processing the input features, each branch generates a set of feature maps. Subsequently, the output of each branch passes through a channel attention submodule. This submodule first performs global average pooling on the feature maps along both time and frequency dimensions to obtain a compressed channel description vector. This vector then passes through two fully connected layers. The first fully connected layer compresses the number of channels to one-sixteenth of the original number and applies ReLU activation, while the second fully connected layer restores the number of channels to the original number and applies Sigmoid activation, thus generating a channel weight vector between 0 and 1. This weight vector is multiplied channel-wise with the original branch output to enhance important feature channels and suppress less important channels. After weighting, the outputs of the three branches are concatenated along the channel dimension to form a feature tensor with a significantly increased number of channels. Finally, a 1×1 convolutional layer is applied to this concatenated feature to achieve cross-channel feature fusion and dimensionality reduction, restoring the number of output channels to 256.

[0061] See Figure 5Following the multi-scale convolutional attention block, the system connects a two-layer gated recurrent unit (GRU) network to model the temporal dynamics of speech features. The GRU network processes the output of the multi-scale convolutional attention block sequentially, with 256 hidden units. The hidden state at the last time step is considered to contain summary information about the noise characteristics of the current speech segment. This hidden state is fed into a noise estimation sub-network, which consists of two fully connected layers: the first layer maps the 256-dimensional input to 128 dimensions and applies ReLU activation; the second layer maps from 128 dimensions to a space with the same time-frequency dimension as the input features, for example, 129 frequency points. The output of this sub-network is a noise estimation vector, where each element corresponds to the estimated noise energy of a frequency point in the current time frame.

[0062] Next, the system implements a feature masking mechanism. The noise estimation vector is compared element-wise with the output features of the multi-scale convolutional attention block along the corresponding dimensions. A soft masking matrix is ​​generated using a sigmoid activation function. ,in Represents the time frame index. This represents the frequency index. The value of this matrix ranges from 0 to 1. The closer the value is to 1, the higher the probability that the time-frequency point is dominated by speech; the closer it is to 0, the higher the probability that it is dominated by noise. This soft masking matrix is ​​applied to the input feature amplitude spectrum according to the following formula. The characteristic amplitude spectrum after preliminary enhancement was obtained. :

[0063] ;

[0064] The essence of this operation is to perform soft attenuation on noisy speech in the time-frequency domain. Regions with high noise estimates experience greater attenuation, while regions dominated by speech are preserved.

[0065] Step S140 performs a residual connection operation. To preserve some details of the original input and stabilize the training process, the enhanced feature amplitude spectrum output by the speech enhancement module is... The amplitude spectrum of the original input features extracted in step S110 By adding elements one by one, the fusion characteristics are obtained. The specific formula is:

[0066] ;

[0067] in This is a learnable scaling factor during model training, initially set to 0.2. This factor allows the model to adaptively determine how much information to retain from the original features. This fused feature... It serves as the final output of the speech enhancement stage and is also the input to the emotion recognition module.

[0068] Step S150 involves classifying the emotional state in the emotion recognition module. (See also...) Figure 3 The core of the emotion recognition module is composed of a multi-head self-attention mechanism and an emotion prior fusion layer. First, the fusion features...

[0069] The data is fed into a linear projection layer, which transforms the two-dimensional time-frequency representation into a sequence representation composed of a series of feature vectors for processing by the self-attention mechanism. Specifically, all frequency features of each time frame are concatenated into a vector, and the entire speech segment forms a sequence.

[0070] Subsequently, the system employs a multi-head self-attention mechanism, with eight attention heads. For each attention head, the input sequence is processed through three independent linear transformation layers to generate query vectors, key vectors, and value vectors, respectively. The computation process for each head is as follows: the similarity score between the query vector and all key vectors is calculated using a scaled dot product attention formula, normalized to weights using Softmax, and then the value vectors are weighted and summed to obtain the context-aware output of that head. The eight attention heads compute in parallel, each focusing on dependencies in different subspaces or aspects of the sequence. The outputs of the eight heads are then concatenated along the feature dimension and fused and dimensionality-reduced through another linear projection layer to obtain a self-attention-enhanced feature representation. This process enables the model to effectively capture long-range contextual dependencies related to emotion in the enhanced speech, such as intonation variations within a sentence or speech features before and after specific emotional words.

[0071] Meanwhile, the emotion prior fusion layer begins its work. This layer loads an acoustic feature extractor pre-trained on a large, clean emotion speech dataset. This extractor is typically a deep convolutional neural network or recurrent neural network, whose parameters are fixed during the training of the model in this invention and do not participate in updates. This pre-trained extractor uses the aforementioned fused features... As input, it is specifically designed to extract deep, abstract acoustic features highly correlated with emotions, such as prosodic features reflecting emotional intensity and timbre features reflecting emotional quality. These pre-extracted deep emotional features are considered valuable prior knowledge.

[0072] Finally, the self-attention-enhanced feature representation output by the multi-head self-attention mechanism is concatenated with the pre-extracted deep emotion features output by the emotion prior fusion layer along the channel dimension. The concatenated comprehensive feature vector is then fed into a classifier. This classifier consists of two fully connected layers and a softmax layer: the first fully connected layer reduces the feature dimension to 128 and applies ReLU activation and Dropout regularization; the second fully connected layer further reduces the dimension to the number of emotion categories; the softmax layer ultimately outputs a probability distribution vector, where each element corresponds to the probability of a basic emotion category, such as happy, sad, angry, neutral, etc.

[0073] Step S160 involves designing and applying a joint loss function to perform end-to-end joint training and optimization of the entire cascaded deep neural network architecture described above. See [link / reference] Figure 4 The training process requires clean speech signals and their corresponding real emotion labels as supervision information. Joint loss function.

[0074] ;

[0075] in, For speech enhancement loss, a scale-invariant signal-to-distortion ratio is used as a metric. Its calculation requires analyzing the enhanced feature amplitude spectrum of the network output. Combined with the original phase, the time-domain speech signal is reconstructed through inverse short-time Fourier transform. The formula for calculating the loss is:

[0076] ;

[0077] in This serves as a reference for the corresponding clean speech signal. The loss function is insensitive to the overall energy scale of the signal, focusing instead on waveform shape differences.

[0078] For emotion recognition loss, a labeled smoothed cross-entropy loss function is used. The label smoothing factor is set to 0.1, which means that when calculating the loss, the 1 in the one-hot vector of the true label is replaced with 0.9, and the 0 is evenly distributed as 0.1 / (number of categories - 1), thereby alleviating the problems of model overfitting and overconfidence.

[0079] Weighting coefficient and This is used to balance the importance of the two tasks. The preferred setting was determined through a grid search on a validation set. , This indicates that joint optimization places greater emphasis on ensuring the quality of speech enhancement, as it is fundamental to the effective performance of emotion recognition. During the training phase, the backpropagation algorithm is used to simultaneously calculate the gradient of the joint loss function with respect to all trainable parameters in the network, and the Adam optimizer is employed to update the parameters. This drives the speech enhancement module and the emotion recognition module to learn collaboratively, enabling the enhancement process to not only remove noise but also consciously preserve acoustic features crucial for emotion discrimination.

[0080] Example 2

[0081] In remote fault diagnosis scenarios for industrial equipment, operators make voice reports via handheld terminals in noisy workshop environments, with strong non-steady-state noise such as machine roaring and metallic clanging in the background. This embodiment applies the method of the present invention to clearly recover the operator's voice from strong interference and accurately determine the urgency or anxiety level of the report, so as to prioritize high-risk reports.

[0082] The system in this embodiment is deployed on a cloud server and receives audio streams from terminal devices. The preprocessing parameters in step S110 are fine-tuned based on the characteristics of industrial noise. Since industrial noise may have a wider spectrum, the number of points in the short-time Fourier transform is increased to 512 to provide higher frequency resolution, facilitating the differentiation of certain high-frequency mechanical noise from speech. The number of filter bank channels for the Mel-frequency cepstral coefficients is increased to 64 to more precisely characterize the speech formant structure that may become blurred under noise masking.

[0083] The model architecture in steps S120 to S150 is basically the same as in Example 1, but the multi-scale convolutional attention block in the speech enhancement module is strengthened to address the non-steady-state characteristics of industrial noise. The kernel sizes of the three depthwise separable convolutional branches are adjusted to 5, 9, and 13 to expand the receptive field and better capture impactful noise patterns that last for a long time or change rapidly. Simultaneously, in the channel attention submodule, in addition to global average pooling, a global max pooling path is introduced. The description vectors generated by the two pooling methods are concatenated before being fed into the fully connected layer, enabling the model to simultaneously focus on the general activity and significant peaks of features, which is more effective for identifying sudden, strong noise.

[0084] In the adaptive noise estimation in step S130, the number of layers in the gated recurrent unit network is increased to 3 to enhance the ability to model complex temporal dynamics of noise. The output dimension of the first fully connected layer of the noise estimation subnetwork is increased to 256 to accommodate more complex noise feature representations.

[0085] In the emotion recognition module of step S150, the emotion categories are set to five categories: "normal report," "emergency report," "anxiety," "anger," and "neutral," which better meets the needs of industrial scenarios. The pre-trained model loaded in the emotion prior fusion layer is trained on a dataset containing speech under various stress situations, so that the deep features extracted by it can better reflect states such as tension and anxiety.

[0086] In the joint training phase of step S160, the training data uses a speech database recorded under various industrial noise environments. The noise types include steady-state motor noise, periodic impact noise, and random broadcast noise. The weights of the joint loss function are re-searched and adjusted under this scenario. , This slightly raises the requirements for the accuracy of emotion recognition, as accurately judging the urgency of emotions is crucial for scheduling maintenance resources in this scenario. The trained system can effectively suppress various types of industrial noise and maintain an emotion recognition accuracy of over 85% even with a signal-to-noise ratio as low as 0 dB, significantly improving the efficiency and reliability of remote fault diagnosis.

[0087] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention. The basic principles, main features, and advantages of the present invention have been shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely the principles of the present invention. Various changes and modifications can be made to the present invention without departing from the spirit and scope of the present invention, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A deep learning-based speech enhancement and emotion recognition method for noisy environments, characterized in that, The method includes the following steps: Step S110 involves preprocessing the input noisy speech signal to extract its time-frequency domain feature representation. Specifically, this includes: performing frame segmentation and windowing on the original time-domain speech signal; applying short-time Fourier transform to each frame of the speech signal to calculate its complex spectrum; extracting the amplitude spectrum of the complex spectrum as the basic time-frequency domain feature, and further calculating its Mel-frequency cepstral coefficients and fundamental frequency contour features to form a multidimensional feature vector. Step S120: Input the time-frequency domain features into a cascaded deep neural network architecture, which includes a speech enhancement module based on attention mechanism and multi-scale convolution and an emotion recognition module that integrates prior emotional knowledge. Step S130: In the speech enhancement module, an enhancement masking matrix for the current speech frame is generated through an adaptive noise estimation and feature masking mechanism, and applied to the input features to suppress noise components. Step S140: Perform residual connection between the enhanced features output by the speech enhancement module and the original input features, and input them into the emotion recognition module; Step S150: In the emotion recognition module, a multi-head self-attention mechanism is used to capture long-range contextual dependencies related to emotions in the enhanced speech, and combined with pre-trained prior emotional acoustic features, to classify emotional states. Step S160: Design a joint loss function that simultaneously constrains the reconstruction accuracy of the speech enhancement task and the classification accuracy of the emotion recognition task, and perform end-to-end joint training and optimization on the cascaded deep neural network architecture. The emotion recognition module in step S150 is a fusion layer of a multi-head self-attention mechanism and an emotion prior; the input of the multi-head self-attention mechanism is the fused features. The sequence representation is obtained after a linear projection layer; the emotion prior fusion layer loads an acoustic feature extractor pre-trained on a clean emotion speech dataset, which has fixed parameters for extracting features from the fused features. Extract deep features that are highly correlated with emotions; concatenate the self-attention enhanced feature representation with the pre-extracted deep emotion features along the channel dimension; finally, output the probability distribution corresponding to different emotion categories through a classifier consisting of two fully connected layers and a softmax layer. The joint loss function It consists of a weighted sum of two parts: ； in, For speech enhancement loss, the scale-invariant signal-to-distortion ratio is used as a metric, and its calculation formula is as follows: ； in For the reconstructed time-domain speech signal, This serves as a reference for the corresponding clean speech signal; For emotion recognition loss, a labeled smoothed cross-entropy loss function is used; weight coefficients... and Determined through grid search.

2. The deep learning speech enhancement and emotion recognition method for noisy environments according to claim 1, characterized in that, In step S130, in the speech enhancement module, an enhancement masking matrix for the current speech frame is generated through an adaptive noise estimation and feature masking mechanism. Specifically, after the multi-scale convolutional attention block of the speech enhancement module, a gated recurrent unit network is connected to model the temporal dynamics of speech features.

3. The deep learning speech enhancement and emotion recognition method for noisy environments according to claim 2, characterized in that, The last hidden state of the gated recurrent unit network is fed into a noise estimation subnetwork, which consists of two fully connected layers and outputs a noise estimation vector with the same dimension as the input features. The noise estimation vector is then compared element-wise with the output features of the multi-scale convolutional attention block, and a soft masking matrix between 0 and 1 is generated by a sigmoid activation function. ,in Represents the time frame index. Represents the frequency index; the soft masking matrix is ​​applied to the input feature amplitude spectrum according to the following formula. The enhanced characteristic amplitude spectrum is obtained as follows: 。 4. The deep learning speech enhancement and emotion recognition method for noisy environments according to claim 1, characterized in that, In step S140, the step of performing a residual connection between the enhanced features output by the speech enhancement module and the original input features, and then inputting them into the emotion recognition module, specifically involves: processing the amplitude spectrum of the enhanced features output by the speech enhancement module... The characteristic amplitude spectrum acting on the input in step S110 By adding elements one by one, we obtain the fusion feature: ； in It is a learnable scaling factor.

5. A deep learning speech enhancement and emotion recognition system for noisy environments, applied to the deep learning speech enhancement and emotion recognition method for noisy environments as described in any one of claims 1-4, characterized in that, The system includes the following components: Preprocessing unit: used to preprocess the input noisy speech signal and extract its time-frequency domain feature representation. Specifically, it includes: performing frame segmentation and windowing processing on the original time-domain speech signal, applying short-time Fourier transform to each frame of speech signal, and calculating its complex spectrum; extracting the amplitude spectrum of the complex spectrum as the basic time-frequency domain feature, and further calculating its Mel-frequency cepstral coefficients and fundamental frequency contour features to form a multi-dimensional feature vector. Cascaded deep neural network architecture: includes a speech enhancement module based on attention mechanism and multi-scale convolution and an emotion recognition module that integrates prior emotional knowledge, for receiving the multi-dimensional feature vector output by the preprocessing unit; The speech enhancement module is configured to generate an enhancement masking matrix for the current speech frame through an adaptive noise estimation and feature masking mechanism, and apply it to the input features to suppress noise components; the emotion recognition module is configured to use a multi-head self-attention mechanism to capture long-range contextual dependencies related to emotions in the enhanced speech, and combine it with pre-trained emotional acoustic feature priors to classify emotional states. The system also includes a training optimization unit configured to apply a joint loss function to perform end-to-end joint training and optimization of the cascaded deep neural network architecture.

6. The deep learning speech enhancement and emotion recognition system for noisy environments according to claim 5, characterized in that, The speech enhancement module includes: A feature encoder, consisting of multiple one-dimensional convolutional layers, is used to map the input multi-dimensional feature vector to a high-dimensional feature space. Multi-scale convolutional attention blocks contain multiple depthwise separable convolutional branches with different kernel sizes in parallel, and each branch connects to a channel attention submodule; A gated recurrent unit network, connected after the multi-scale convolutional attention block, is used to model the temporal dynamics of speech features; The noise estimation subnetwork, connected to the output of the gated recurrent unit network, consists of two fully connected layers and is used to output the noise estimation vector. The masking generation and application unit is configured to compare the noise estimation vector with the output features of the multi-scale convolutional attention block, and generate a soft masking matrix using a sigmoid activation function. and according to the formula The characteristic amplitude spectrum acting on the input The enhanced characteristic amplitude spectrum is obtained. .

7. The deep learning speech enhancement and emotion recognition system for noisy environments according to claim 5, characterized in that, The emotion recognition module includes: Linear projection layer, used to fuse the input features. Convert to sequence representation; Multi-head self-attention layer, used to compute self-attention-enhanced feature representations; An emotion-prior fusion layer is loaded with a pre-trained acoustic feature extractor for extracting features from the fused data. Extracting deep emotional features; The feature concatenation layer is used to concatenate the self-attention-enhanced feature representation with the deep emotion feature along the channel dimension. The classifier, consisting of two fully connected layers and a softmax layer, is used to output the probability distribution of sentiment categories.

Images