Mine scene speech enhancement method and device based on multi-modal noise feature learning

Abstract

The invention relates to the technical field of voice signal processing and industrial communication, and discloses a mine scene voice enhancement method and device based on multi-modal noise feature learning, and the method comprises the following steps: synchronously collecting acoustics, vibration and acceleration signals of a mine scene, and extracting multi-dimensional features after trend term removal and standardization preprocessing; dynamically fusing the vibration and acceleration characteristics representing the physical working condition of the pure noise source with the acoustic characteristics by using a self-attention mechanism; inputting the fusion features into a CNN-LSTM deep network, and reasoning to obtain an ideal ratio masking vector; and using the masking vector to correct the amplitude spectrum, and combining original phase reconstruction to generate enhanced speech. According to the method, the non-acoustic physical mode is introduced to serve as auxiliary information, causal association between deep network mining mechanical vibration and acoustic noise is utilized, the problem that the non-stationary impact noise is difficult to restrain under the low signal-to-noise ratio is effectively solved, and the definition and intelligibility of voice communication in the complex mine environment are improved.

Description

Technical Field

[0001] This invention relates to the fields of speech signal processing and industrial communication technology, specifically to a method and device for speech enhancement in mining scenarios based on multimodal noise feature learning. Background Technology

[0002] Mining operations typically take place in confined underground spaces or complex open-pit pits, characterized by challenging terrain and high-intensity operation of large machinery such as drilling rigs, tunneling machines, local ventilation fans, and transport vehicles. These machines generate extremely high background noise levels during operation, with a wide range of noise types, including broadband steady-state noise from fans, non-steady-state impact noise from drilling into rock, and electromagnetic noise from motors. In such high-risk, high-noise environments, clear and continuous voice communication between workers and between workers and the ground command center is crucial for ensuring production safety, scheduling work processes, and emergency response command.

[0003] To extract target speech signals from noisy speech, existing technologies typically employ single-channel speech enhancement algorithms based on statistical models, such as spectral subtraction, Wiener filtering, and minimum mean square error estimators. These traditional algorithms largely rely on the assumption that noise is short-term stationary, meaning that the statistical characteristics of noise change slowly over a short period. However, in actual mining operations, noise generated by drill pipe impacts and falling ore exhibits strong non-stationarity and abruptness, with its energy fluctuating dramatically in the time and frequency domains. This makes it difficult for traditional algorithms to accurately track changes in the noise spectrum, resulting in processed speech often retaining significant musical noise or causing auditory distortion, failing to meet practical communication needs.

[0004] With the development of deep learning technology, speech enhancement methods based on deep neural networks (DNNs), convolutional neural networks (CNNs), or recurrent neural networks (RNNs) have gradually become mainstream. These methods learn a nonlinear mapping relationship from noisy spectrum to clean spectrum through supervised training on a large number of noisy and clean speech pairs, which improves the enhancement effect under non-stationary noise to a certain extent. However, most existing deep learning speech enhancement schemes rely solely on a single acoustic modality input (i.e., signals collected only by a microphone). In the extremely low signal-to-noise ratio (SNR) environment of mines (such as -5dB or even lower), high-intensity mechanical noise often completely masks the formant structure of the speech signal in the frequency domain, and the spectral texture of mechanical noise is sometimes extremely similar to that of speech. In the absence of external auxiliary information, relying solely on the characteristics of the acoustic signal itself, deep neural networks struggle to accurately distinguish between speech components and noise components from severe spectral aliasing, easily leading to problems such as over-suppression resulting in damage to the target speech or insufficient suppression resulting in strong noise residue.

[0005] Furthermore, while some technologies attempt to suppress noise using microphone arrays through beamforming, the narrow tunnels and rough walls of underground mines cause severe sound wave reflection and reverberation, significantly compromising the spatial phase information needed for sound source localization and limiting the performance of the array processing. More critically, all the aforementioned methods based on acoustic sensors essentially process the resulting signal already mixed with noise, failing to utilize the physical information of the noise source. Mine machinery noise is primarily caused by the mechanical vibration of the equipment itself. This vibration signal has a high causal correlation with acoustic noise and contains no target speech components, making it an ideal noise reference source. However, current technologies have not yet established an effective mechanism to deeply fuse non-acoustic physical modes such as vibration and acceleration from the equipment surface with acoustic signals, failing to fully explore the complementary potential of multimodal information in speech enhancement tasks in complex industrial scenarios. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a method and device for speech enhancement in mining scenarios based on multimodal noise feature learning. This solves the problem that existing single-channel speech enhancement technologies, in complex and noisy mining environments, cannot distinguish between non-stationary mechanical impact noise and speech signals due to the inability to utilize the physical characteristics of noise sources, resulting in low intelligibility and severe residual noise after speech enhancement.

[0007] To achieve the above objectives, the present invention provides the following technical solution: In a first aspect, the present invention provides a method for speech enhancement in mining scenes based on multimodal noise feature learning, comprising the following steps: Step S1: Acquire air conduction signals, solid conduction vibration signals, and acceleration signals from the mining site through a synchronous acquisition module, and perform time synchronization calibration on the above signals to generate an original discrete signal sequence; Step S2: Perform detrending and standardization processing on the original discrete signal sequence, and perform frame windowing operation to generate a preprocessed discrete frame sequence; Step S3: Extract acoustic feature vectors representing the mixed state of speech and noise, and vibration feature vectors and acceleration feature vectors representing the physical conditions of pure noise sources from the discrete frame sequence, respectively. Step S4: The acoustic feature vector, the vibration feature vector, and the acceleration feature vector are structurally concatenated, and the concatenated features are weighted and fused using a self-attention mechanism to generate a multimodal fusion feature vector; Step S5: Input the multimodal fusion feature vector into the pre-constructed CNN-LSTM deep augmentation network for inference, and output the spectral masking vector for the current time frame through the network; Step S6: The frequency domain amplitude spectrum of the air conduction signal generated in step S2 is corrected using the spectral masking vector, and the enhanced speech signal is reconstructed by combining it with the original phase spectrum.

[0008] Furthermore, in step S1, the acquisition process uses a high dynamic range microphone to collect mixed sound waves, a piezoelectric vibration sensor rigidly connected to the surface of the noise source device to collect mechanical vibration waves, and a triaxial accelerometer installed on the surface of the noise source device to collect spatial impact characteristics; sampling is triggered by a multi-channel parallel analog-to-digital converter based on a unified clock source to ensure the alignment of each modal data in the time domain.

[0009] Further, in step S2, the detrending term processing is performed by calculating the arithmetic mean of the signal within the current processing window and subtracting the value point by point to eliminate DC bias; the normalization processing uses the Z-Score method to map each modal signal to the standard normal distribution space; the frame windowing operation uses the Hamming window function to divide the signal into a short-time frame sequence with overlapping regions.

[0010] Further, in step S3, the extraction of the acoustic feature vector includes performing a short-time Fourier transform on the air conduction signal and calculating the short-time energy, short-time zero-crossing rate, spectral centroid, and Mel frequency cepstral coefficients; the extraction of the vibration feature vector and acceleration feature vector includes calculating the mean, variance, peak value, and kurtosis, and performing a discrete wavelet transform to calculate the multi-scale wavelet coefficient energy. The kurtosis is used to characterize the tail thickness of the signal probability density function to identify non-Gaussian impulsive noise components.

[0011] Further, in step S4, the execution process of the self-attention mechanism includes: firstly, concatenating the feature vectors of each modality into a joint feature vector; then, projecting the joint feature vector into a query vector, a key vector, and a value vector through three independent learnable linear transformation matrices; calculating the dot product of the query vector and the transpose of the key vector, and scaling and Softmax normalizing the dot product result to obtain the attention weight matrix; finally, using the weight matrix to perform a weighted summation of the value vector, thereby dynamically adjusting the contribution of physical features to the acoustic enhancement task.

[0012] Further, in step S5, the CNN-LSTM deep augmentation network includes a local feature encoder, a temporal correlation modeler, and a spectral masking estimator. The local feature encoder is configured as a one-dimensional convolutional neural network for extracting local feature representations; the temporal correlation modeler is configured as a bidirectional long short-term memory network, which uses a gating mechanism to capture contextual temporal dependencies; and the spectral masking estimator is configured as a fully connected layer for mapping temporal features to an ideal ratio masking vector.

[0013] Further, in step S6, the reconstruction process includes calculating the Hadamard product of the spectral masking vector and the original noisy speech amplitude spectrum to obtain the enhanced amplitude spectrum, using Euler's formula to combine the original noisy speech phase spectrum to synthesize a complex spectrum, and outputting a time-domain waveform through inverse short-time Fourier transform and overlapping addition method.

[0014] A second aspect of the present invention provides a speech enhancement device based on multimodal noise feature learning, comprising: The multimodal data acquisition module is configured to connect to an external acquisition unit and synchronously receive air conduction signals, solid conduction vibration signals, and acceleration signals from the mining site. The signal preprocessing module is configured to perform detrending, standardization, and frame windowing processing on the signal received by the multimodal data acquisition module; The multidimensional feature extraction module is configured to extract Mel frequency cepstral coefficients from acoustic modes and kurtosis and wavelet energy features from vibration and acceleration modes, respectively. The attention feature fusion module is configured to calculate the attention weights of physical features to acoustic features through a self-attention mechanism, and generate a weighted multimodal fusion feature vector. The deep network inference module is configured to load a pre-trained CNN-LSTM network and infer an ideal ratio mask vector based on the multimodal fusion feature vector. The signal reconstruction output module is configured to correct the original noisy amplitude spectrum using the ideal ratio masking vector and reuse the original phase spectrum to synthesize an enhanced speech signal.

[0015] This invention provides a method and device for speech enhancement in mining scenes based on multimodal noise feature learning. It has the following beneficial effects: 1. This invention solves the technical challenge of distinguishing speech from noise in low signal-to-noise ratio environments using a single microphone by simultaneously acquiring air conduction signals, solid conduction vibration signals, and acceleration signals, and by introducing non-acoustic modes as auxiliary information. Since the noise of mining machinery is essentially caused by equipment vibration, vibration and acceleration signals can directly characterize the physical excitation of the noise source and are not affected by speech signals. By utilizing these pure physical reference features (such as kurtosis and wavelet energy), the model can pinpoint the noise components at the source, thereby improving the ability to suppress non-stationary mechanical impact noise.

[0016] 2. This invention employs a self-attention mechanism in the feature fusion stage. Through matrix operations of query vectors, key vectors, and value vectors, it achieves dynamic weighted fusion of multimodal features. It can adaptively adjust the weight of physical features in the fusion vector according to the changes in the contribution of vibration to noise under different working conditions of mining equipment (such as idling, loading, and drilling). This allows the model to automatically focus on highly correlated physical cues, avoiding feature mutual exclusion or redundancy caused by simple linear splicing, and improving the accuracy of spectral masking estimation.

[0017] 3. This invention constructs a deep augmentation network comprising a one-dimensional convolutional neural network (1D-CNN) and a bidirectional long short-term memory network (Bi-LSTM), achieving deep spatiotemporal modeling of multimodal features. By utilizing CNN to extract local short-term patterns of features and combining it with Bi-LSTM to capture long-term temporal contextual dependencies of speech signals, this network structure can effectively track rapidly changing transient noise while maintaining the continuity and integrity of speech signals in the temporal domain. This significantly reduces residual background noise while minimizing spectral distortion of the target speech, thereby improving the intelligibility of the reconstructed speech. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the overall process of the method of the present invention. Figure 2 This is a schematic diagram of the feature fusion and network architecture of the present invention; Figure 3 This is a hardware connection diagram of the multimodal speech enhancement device of the present invention. Detailed Implementation

[0019] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] See attached document Figure 1 This invention provides a voice enhancement device for mining scenarios based on multimodal noise feature learning. The device mainly consists of a multimodal perception module, a synchronous acquisition module, and a computing and processing module connected by a signal transmission link.

[0021] The multimodal sensing module is used to capture signal data of different dimensions in the physical space of a mining operation site. The multimodal sensing module includes an acoustic acquisition unit, a vibration acquisition unit, and an acceleration acquisition unit.

[0022] The acoustic acquisition unit is configured as a high dynamic range explosion-proof microphone. The acoustic acquisition unit is worn on the worker's safety helmet or fixedly deployed on the surface of the communication terminal. The acoustic acquisition unit is used to acquire mixed sound wave signals propagating through the air medium. We define this mixed sound wave signal as an air-conducted signal, denoted as [insert denoting ] ,in This represents a continuous-time variable. It includes the target speech component and the environmental background noise component.

[0023] The vibration acquisition unit is an industrial-grade piezoelectric vibration sensor. It is rigidly connected to the outer casing of the main noise source equipment in the mine via a magnetic base or threaded connection. The noise source equipment includes, but is not limited to, rock drilling rigs, local ventilation fans, or mining locomotives. The vibration acquisition unit is used to directly acquire the mechanical vibration waves generated during equipment operation. We define this signal as a solid-conducted vibration signal, denoted as [symbol missing]. Because the vibration signal is directly transmitted through the solid medium, the It does not include speech components in the air, but only includes the physical characteristics of noise sources related to the operating status of the equipment.

[0024] The acceleration acquisition unit is configured as a triaxial MEMS accelerometer. It is installed in the area of ​​the noise source device experiencing severe vibration, and is physically adjacent to the vibration acquisition unit. The acceleration acquisition unit is used to acquire the instantaneous acceleration changes of the device in three-dimensional space. This signal is defined as an acceleration signal, denoted as [missing information]. The aforementioned It includes impact characteristic data of the equipment in the X, Y and Z axis directions.

[0025] The synchronous acquisition module is physically connected to the acoustic acquisition unit, the vibration acquisition unit, and the acceleration acquisition unit via a wired shielded cable. The synchronous acquisition module is equipped with a multi-channel parallel analog-to-digital converter. The synchronous acquisition module is used to receive analog signals from the acoustic acquisition unit, vibration acquisition unit, and acceleration acquisition unit. The above and the aforementioned The above signals are synchronously discretized and sampled based on a unified clock source. The sampling frequency is set to... The sampling precision is 16 bits. After processing by the synchronous acquisition module, discrete time sequence signals are generated, denoted as follows: , and ,in This serves as an index for discrete sampling points. The synchronous acquisition module ensures that the phase deviation between different channels is less than a preset microsecond threshold.

[0026] The computing processing module is connected to the synchronous acquisition module via a high-speed data bus. The computing processing module includes a central processing unit (CPU), a graphics processing unit (GPU), and a memory. The memory stores computer-executable instructions; when these instructions are executed by the CPU or the GPU, they enable the acquisition of the data. , and The data processing logic.

[0027] The computational processing module is configured to perform computational tasks including data preprocessing, feature extraction, feature fusion, model inference, and signal reconstruction. The module first performs detrending and Z-score normalization on the received multimodal discrete signal. For discrete signals from any input channel... The computing module calculates the normalized signal according to the following formula.

[0028] ; in, This represents the arithmetic mean of the signal channel within the current time window. This represents the standard deviation of the signal channel within the current time window. The normalization process eliminates the differences in physical dimensions between sound pressure, vibration velocity, and acceleration, ensuring that data from different modes fall within the same numerical distribution range. The computational processing module further divides the normalized signal into short-time frames with overlapping regions. The frame length is set to... Frame shift is For the first Frame signal, the computing processing module utilizes Hamming window function Windowing is applied to suppress spectral leakage. The windowed... The frame signal is represented as

[0029] ; Based on the aforementioned hardware architecture and preprocessing logic, the computational processing module constructs a multimodal data frame sequence for subsequent deep learning model input. The multimodal perception module provides a direct physical reference for noise sources, the synchronous acquisition module ensures strict alignment of different modal features in the temporal domain, and the computational processing module provides the computing power environment for algorithm execution.

[0030] See attached document Figure 2 The specific execution logic of step S1 in the method of the present invention is as follows: First, start the multi-channel synchronous data acquisition task. The computing and processing module sends an initialization command to the synchronous acquisition module to set the global sampling clock frequency. The frequency is 16kHz. The synchronous acquisition module locks the phase of each physical channel to ensure that the acoustic acquisition unit, vibration acquisition unit, and acceleration acquisition unit are triggered for sampling on the same clock edge. The acquisition system converts the continuous analog electrical signal into a discrete digital signal sequence, generating the original acoustic sequence. Original vibration sequence and the original acceleration sequence .

[0031] Secondly, detrending processing is performed on the original discrete sequence. Because piezoelectric sensors and capacitive microphones are susceptible to DC bias due to temperature drift or power supply voltage fluctuations in the low-frequency range, this bias affects the accuracy of subsequent energy characteristic calculations. The arithmetic mean of the signal calculated by the processing module within a preset time window is calculated, and this mean is subtracted from the original sequence to eliminate the DC component. For a length of... The processing window, the detrended signal The calculation logic is as follows: ; Next, amplitude standardization is performed on the detrended signal. This is because the dynamic range of the signal fluctuates significantly during drilling or blasting operations, and the physical dimensions of different modal signals, such as sound pressure (Pa) and acceleration (m / s²), differ. 2 The differences lead to inconsistent numerical magnitudes, and directly inputting these values ​​into the neural network can cause unstable gradient updates. The computational processing module uses the Z-Score normalization method based on statistical characteristics to map each modal signal to a standard normal distribution space with a mean of 0 and a standard deviation of 1.

[0032] Subsequently, framing and windowing operations are performed. To analyze the short-time stationary characteristics of the signal, the one-dimensional long-sequence signal is divided into a short-time frame sequence. The frame length is set. Corresponding to a duration of 25ms (i.e.) (sampling points), frame shift Corresponding to a duration of 10ms (i.e.) (Sampling points). The computational processing module multiplies each frame of signal by a Hamming window function. This window function smooths the discontinuity at the signal truncation point by attenuating the amplitude value at the frame edge, thereby suppressing sidelobe leakage during the frequency domain transformation process.

[0033] Finally, a time-frequency domain transformation is performed on the framed acoustic and physical vibration signals. Although some features are extracted in the time domain, frequency domain features are crucial for describing noise distribution. The computational processing module uses a short-time Fourier transform to convert the time-domain frame signal into a complex spectrum. For the first... Frame input signal Its frequency domain representation The calculation is as follows: ; in, This represents the frequency index, with values ​​ranging from 0 to... This indicates the number of points for the Fast Fourier Transform (FFT), set to 512 to cover the effective frequency band; The imaginary unit is used. This step maps a one-dimensional time-domain waveform to a two-dimensional time-frequency spectrum, generating a spectrum including amplitude. Phase spectrum The data structure is as follows. The amplitude spectrum is used for subsequent feature extraction and masking value estimation, and the phase spectrum is cached for subsequent signal reconstruction steps. The acceleration signal also undergoes the above-described framing process, but in this embodiment, its time-domain and wavelet transform features are mainly retained, and whether to perform STFT transformation depends on the specific configuration.

[0034] In step S2 of the method of the present invention, the computational processing module extracts feature vectors from the acoustic mode and the non-acoustic physical mode for each frame of preprocessed signal sequence. The specific execution content is as follows: First, regarding acoustic signal frames Temporal and frequency domain features are extracted to characterize the mixed acoustic state of speech and environmental noise. In the temporal domain, the short-time energy and short-time zero-crossing rate of the current frame are calculated. Short-time energy... Reflects the amplitude intensity of the current frame, serving as an indicator for distinguishing between high-noise and silent segments; short-time zero-crossing rate It reflects the number of times the signal crosses the zero-level axis, and is used to distinguish between low-frequency mechanical noise (low zero-crossing rate) and high-frequency airflow noise or voiceless speech (high zero-crossing rate).

[0035] In the frequency domain, based on the amplitude spectrum generated in step S1 Extract the spectral centroid and spectral bandwidth. Spectral centroid The centroid location of the spectral energy distribution is described, and its calculation formula is as follows: ; in, Frequency Index The corresponding actual physical frequency value. The spectral centroid can effectively characterize the frequency shift characteristics of mine noise (e.g., the centroid shifts to higher frequencies when the drilling rig is running at high speed). The spectral bandwidth describes the degree of diffusion of spectral energy around the centroid and is used to characterize the bandwidth of the noise.

[0036] Furthermore, Mel-frequency cepstral coefficients (MFCCs) are extracted to simulate the characteristics of human hearing. The computational processing module passes the amplitude spectrum through a set of Mel-scaled triangular filters, takes the logarithmic energy of the filter output, and performs a discrete cosine transform (DCT) to extract the first 13 dimensions of coefficients as the MFCC feature vector. This set of features can compress redundant spectral information and retain the envelope features that contribute most to speech intelligibility.

[0037] Secondly, regarding the vibration signal frame With acceleration signal frame Statistical and time-frequency features are extracted to characterize the physical conditions of the pure background noise source. The computational processing module extracts the mean, variance, and peak value of the aforementioned physical signal. The mean reflects the DC offset of the signal; the variance (i.e., the second moment) directly corresponds to the average power of the mechanical vibration; and the peak value reflects the maximum instantaneous impact intensity within the current frame. To more accurately capture the impact noise unique to mining (such as the moment of blasting or drill pipe impacting rock), the computational processing module further calculates the kurtosis of the signal. Kurtosis is the fourth central moment, used to measure the tail thickness of the signal's probability density function. For non-Gaussian vibration signals containing pulse impacts, the kurtosis value will be significantly greater than 3. kurtosis of the physical signal in a frame The calculation is as follows: ; in and These are the mean and standard deviation of the current frame, respectively.

[0038] Finally, wavelet transform is performed on the vibration and acceleration signals to extract time-frequency domain energy features. Since the short-time Fourier transform is limited by a fixed window length and cannot simultaneously account for high-frequency and low-frequency resolution, the computational processing module employs discrete wavelet transform to decompose the physical signal frames into multi-scale components. The signal is decomposed into approximate coefficients (low-frequency components) and detail coefficients (high-frequency components), recursively layer by layer. The computational processing module calculates the energy sum of the wavelet coefficients at each scale, which serves as the time-frequency feature of that physical mode. This feature effectively separates the steady-state low-frequency vibrations from sudden high-frequency impact vibrations during mining machinery operation, providing multi-resolution noise reference information for subsequent models.

[0039] After the above steps, for each frame time window, the computational processing module outputs an acoustic feature vector containing the aforementioned acoustic features. Vibration feature vectors containing the above-mentioned statistical and wavelet features and acceleration eigenvectors .

[0040] The specific execution logic of step S3 in the method of this invention is as follows: The computational processing module first performs a structured splicing operation of multimodal features. In order to preserve the physical hierarchical structure of the features, the computational processing module strictly follows a preset splicing sequence strategy: for the current frame, first arrange the temporal features (including short-time energy, zero-crossing rate, mean, variance, kurtosis, etc.), then arrange the frequency domain features (including spectral centroid, bandwidth, MFCC coefficients, etc.), and finally arrange the time-frequency domain features (wavelet coefficient energy). This strategy will combine the feature vectors from the acoustic acquisition unit... , characteristic vector of vibration acquisition unit and the feature vector of the acceleration acquisition unit Combined into a single high-dimensional joint feature vector, denoted as This vector numerically contains the full-dimensional information required for speech enhancement at the current moment, covering the complete link from acoustic performance to the physical excitation of the noise source.

[0041] Subsequently, the computational processing module introduced a self-attention mechanism to... Deep feature interaction and weighting are performed. Due to the highly non-stationary nature of the noise environment in mines, the contribution of vibration features to acoustic noise varies dynamically under different operating conditions (e.g., idling and drilling). Simple linear concatenation cannot reflect this complex non-linear dependence. Therefore, this embodiment uses an attention mechanism to allow the model to adaptively learn the correlations within features.

[0042] Specifically, the computational processing module first processes the joint feature vector. Through three independent learnable linear transformation matrices respectively , and Perform projection to generate query vector Key vector Sum value vector

[0043] ; Next, the computational processing module calculates the attention weight distribution. This is done by calculating the query vector. With key vector The dot product of the transpose of measures the correlation strength between different feature elements. To prevent the dot product result from becoming too large and causing the gradient vanishing in the subsequent Softmax function, the dot product result is divided by a scaling factor. ( (This refers to the dimension of the key vector). Then, the result is normalized using the Softmax function to obtain the attention weight matrix. Finally, this weight matrix is ​​used to adjust the value vector. Perform a weighted summation to obtain the fused feature vector. The core calculation formula for this process is described as follows: ; Through this mechanism, when the vibration sensor detects a high-intensity impact signal (high kurtosis feature), the attention mechanism automatically increases the weight of this physical feature in the fusion vector, enabling the model to accurately distinguish and suppress the corresponding acoustic impact noise component in subsequent processing. Finally, the computational processing module outputs a weighted fused feature vector. This serves as the input stimulus for subsequent deep learning models.

[0044] In step S4 of the method of the present invention, the computational processing module constructs and initializes a cascaded deep neural network, which mainly consists of a local feature encoder (CNN part), a temporal correlation modeler (LSTM part), and a spectral masking estimator (output layer).

[0045] The computational processing module first processes the fused feature vector sequence output from step S3. The input is fed into a local feature encoder. This encoder is configured as a one-dimensional convolutional neural network (1D-CNN). Since the input feature vectors are continuous along the time axis, and the phoneme features of speech or the impact features of mechanical noise often exhibit short-term local patterns, the 1D-CNN utilizes sliding convolutional kernels to extract deep abstract features at each time step. For the first... Each convolutional layer performs convolution operations and nonlinear activations in its computational processing module. Its computational logic is described below: ; in, The output of the previous layer (the input of the first layer is...) ), * indicates a one-dimensional convolution operation. and These are the kernel weight matrix and bias vector of this layer, respectively. To correct the linear unit activation function, nonlinearity is introduced to alleviate the gradient vanishing problem. By stacking multiple layers of convolutional operations, the computational processing module maps the original feature space into local feature representations with a higher semantic level. .

[0046] Subsequently, the computational processing module represents the local features. The input is fed into a temporal correlation modeler. This modeler is configured as a bidirectional long short-term memory (LSTM) network. Because speech signals have long-term correlations, and the enhancement effect of the current frame depends not only on past information but also on the signal change trend at future moments, the bidirectional structure can utilize contextual information simultaneously. The LSTM units control the flow of information through a gating mechanism. For each time step... The LSTM unit is based on the current input (i.e., the corresponding) (components) and the hidden state of the previous time step Update the internal state. Its core gating calculation logic is as follows: ; ; ; in, , , These represent the input gate, forget gate, and output gate, respectively. The sigmoid activation function limits the gate value to between 0 and 1, determining the proportion of information retained or discarded. Cell state. The old state is preserved by the forget gate, and the new information is injected into the input gate to update the output, ultimately outputting the hidden state. Bidirectional LSTM processes the forward and reverse sequences separately. The computation module concatenates the hidden states from both directions to form a contextual feature vector that includes global temporal dependencies. .

[0047] Finally, the computational processing module will process the context feature vector. The input is fed into the spectral masking estimator, which consists of fully connected layers. The computational processing module maps the high-dimensional LSTM output back to the frequency dimension of the target speech through the fully connected layers, and outputs a real-valued masking vector with values ​​ranging from [0,1] through the Sigmoid activation function, denoted as the ideal ratio mask. Each element of this masking vector corresponds to a frequency point; the closer the value is to 1, the more likely the frequency point is mainly speech, and the closer the value is to 0, the more likely the frequency point is mainly noise. This output is the network's prediction of the spectral gain of the speech in the current frame.

[0048] The specific execution logic of step S5 in this invention is as follows: First, the computational processing module constructs a dedicated dataset for supervised learning. Based on the aforementioned 100 sets of raw data, each 10 minutes long, the computational processing module divides them into a training set and a test set. 80 sets of data are randomly selected as the training set, and the remaining 20 sets are used as the test set to verify the model's generalization ability. For each training sample in the dataset, it contains paired input data and target labels. The input data is a temporally aligned multimodal fusion feature sequence containing acoustic, vibration, and acceleration information, obtained through steps S1 to S3. The target label is the pure speech amplitude spectrum corresponding to that data segment. To obtain high-quality label data, the pure speech amplitude spectrum can be obtained by collecting data in a relatively silent environment during mine equipment downtime, or by linearly mixing a standard pure speech library with independently collected mine background noise data at different signal-to-noise ratios and strictly aligning them, thereby ensuring the temporal consistency between the input multimodal features and the output target speech.

[0049] Secondly, the computational processing module defines a loss function to measure the model's prediction quality. To ensure that the enhanced speech signal approximates real, clean speech as closely as possible in terms of spectral structure, this embodiment uses mean squared error as the objective function. During the forward propagation phase of training, the model outputs a predicted spectral masking vector. The computational processing module then multiplies this masking vector element-wise with the input noisy speech amplitude spectrum to obtain the predicted enhanced speech amplitude spectrum. Subsequently, the predicted spectrum was calculated and compared with the amplitude spectrum of the true clean speech used as a label. The differences between them. Loss function The specific calculation logic is as follows: ; in, This indicates the total number of time frames within the current training batch. This represents the total number of frequency points (corresponding to half the number of FFT points). This formula quantifies the energy distribution deviation between the predicted signal and the real signal in the frequency domain, forcing the model to learn how to accurately identify and remove noise components in the spectrum using vibration and acceleration features while preserving the formant structure of the speech, in the process of minimizing this deviation.

[0050] Finally, the computational processing module performs iterative parameter optimization based on backpropagation. This embodiment uses the Adam optimizer, which combines the advantages of the momentum method and the RMSProp algorithm, and can adaptively adjust the learning rate for different network parameters based on the first and second moment estimates of the gradient. The computational processing module sets the initial learning rate to 0.001 and the total number of training epochs to 50. In each training epoch, the computational processing module feeds the training set data into the network in batches and calculates the aforementioned loss function. The gradient is calculated and the convolutional kernel parameters of the CNN layer, the gate weights of the LSTM layer, and the bias parameters of the fully connected layer are updated in reverse. As the training progresses, when the loss function value converges to a stable range or reaches the preset 50 epochs, the network parameters of the processing module are frozen, and the final speech enhancement model file is generated.

[0051] In step S6 of the method of this invention, the computational processing module performs an inverse transformation from the frequency domain feature space to the time domain waveform space to generate the final enhanced speech signal. The specific execution logic is as follows: First, the computational processing module performs amplitude spectrum correction and restoration. In step S4 mentioned above, the deep neural network outputs a spectral masking vector for the current frame, which represents the signal-to-noise ratio gain weight at each frequency point. The computational processing module then retrieves the original noisy speech amplitude spectrum cached in memory. (i.e., the magnitude of the STFT transform in step S1), and compare it with the masking vector predicted by the network. Element-wise Hadamard product operations are performed. Through this operation, the computational processing module suppresses the energy of frequency points with low weights in the masking vector (i.e., those determined by the multimodal model to be dominated by drilling rig vibration or fan noise), while retaining the energy of frequency points with high weights (i.e., those dominated by speech formants), thus obtaining the enhanced speech amplitude spectrum. .

[0052] Secondly, the computational processing module performs complex spectrum synthesis. Since the human auditory system is insensitive to the short-time phase spectrum of speech signals, and accurately estimating the phase of clean speech in a noisy environment is extremely difficult, this embodiment employs a noisy phase multiplexing strategy. The computational processing module reads the original noisy speech phase spectrum generated in step S1 from the cache. and compared it with the corrected amplitude spectrum Combined, construct the enhanced complex spectrum The synthesis process follows Euler's formula, and its mathematical expression is as follows: ; in The unit is the imaginary unit. Although noisy phase is used, the perceptual clarity of the reconstructed speech is mainly determined by the amplitude spectrum structure because the amplitude spectrum has been accurately corrected through a deep model that incorporates physical features of vibration and acceleration. Therefore, this strategy can effectively ensure the intelligibility of the speech. Next, the computational processing module performs an inverse short-time Fourier transform on the synthesized complex spectrum. This step maps the frequency domain data of each frame back to the time domain, generating an enhanced time-domain frame signal. .

[0053] Finally, the computational processing module uses an overlap-addition method to stitch the discrete time-domain frames into a continuous waveform. Because an overlap interval (frame shift of 10ms is less than frame length of 25ms) is introduced during the framing process in step S1, direct stitching would cause the signal amplitude of the overlapped portion to double and the waveform to become discontinuous. The computational processing module aligns and superimposes adjacent frames on the time axis according to the frame shift step size. Simultaneously, to eliminate the periodic amplitude modulation caused by the Hamming window in step S1, the computational processing module normalizes the superimposed signal, i.e., divides it by the accumulated value of the square of the window function. After this de-windowing and smoothing process, the Gibbs effect and discontinuous transitions at frame boundaries are eliminated, ultimately outputting a continuous, smooth, and background mechanically noise-removed time-domain enhanced speech signal. It is then output to an audio playback device or communication transmission interface via a digital-to-analog converter.

[0054] See attached document Figure 3 The present invention provides a speech enhancement device based on multimodal noise feature learning, which includes: a multimodal data acquisition module, a signal preprocessing module, a multidimensional feature extraction module, an attention feature fusion module, a deep network inference module, and a signal reconstruction output module.

[0055] The multimodal data acquisition module is configured to connect to an external synchronous acquisition hardware communication interface. This module receives three parallel discrete signal streams from the mining site in real time, specifically including: raw acoustic signals characterizing air-conducted sound waves, raw vibration signals characterizing mechanical vibrations on equipment surfaces, and raw acceleration signals characterizing triaxial impacts on the equipment. The multimodal data acquisition module incorporates a clock synchronization verification unit to detect and correct timestamp discrepancies between the three signal streams, ensuring that all input data maintains strict alignment at the microsecond level of time accuracy.

[0056] The signal preprocessing module is connected to the multimodal data acquisition module. The signal preprocessing module performs standardization cleaning on the received raw signal. Specifically, the signal preprocessing module first calculates and subtracts the statistical mean of each channel signal within a sliding window to eliminate DC bias caused by sensor hardware temperature drift. Subsequently, the module performs Z-score normalization to uniformly map data with different physical dimensions, such as sound pressure, vibration velocity, and acceleration, to a standard normal distribution range. Furthermore, the signal preprocessing module is also equipped with a frame-segmentation windowing unit, which uses a Hamming window function to divide a continuous long-time-series signal into a short-time frame sequence with overlapping regions to satisfy the short-time stationarity assumption of subsequent frequency domain transformation.

[0057] The multidimensional feature extraction module is connected to the signal preprocessing module. The multidimensional feature extraction module is configured to perform acoustic feature calculations and physical feature calculations in parallel. For acoustic signal frames, this module generates spectral data through short-time Fourier transform and calculates short-time energy, zero-crossing rate, spectral centroid, and Mel-frequency cepstral coefficients accordingly to construct an acoustic feature vector characterizing the mixed state of speech and noise. For vibration and acceleration signal frames, this module focuses on calculating the kurtosis index characterizing non-Gaussian impact characteristics and the multi-scale energy distribution extracted through discrete wavelet transform to construct a physical feature vector characterizing the physical condition of a pure noise source.

[0058] The attention feature fusion module is connected to the multi-dimensional feature extraction module. The attention feature fusion module addresses the contribution distribution of different modal features in noise suppression tasks. First, the module performs a high-dimensional concatenation of the acoustic feature vector and the physical feature vector. Then, it executes a self-attention mechanism, dynamically evaluating the correlation strength between physical features (especially high-kurtosis impact vibration features) and the current acoustic noise through matrix operations on the query vector, key vector, and value vector, and generates a weighted fused feature vector accordingly. This process enables the device to automatically focus on acoustic noise components directly caused by mechanical vibration.

[0059] The deep network inference module is connected to the attention feature fusion module. The deep network inference module is loaded with a pre-trained CNN-LSTM composite neural network model. This module takes the fused feature vector as input, first extracting local deep features using a one-dimensional convolutional layer, and then capturing long-term temporal contextual dependencies of the speech signal using a bidirectional LSTM layer. The output of the deep network inference module is mapped through a fully connected layer to generate a target spectral masking vector for the current frame. This vector quantifies the predicted proportion of speech components at each frequency point.

[0060] The signal reconstruction output module is connected to the deep network inference module and the signal preprocessing module. The signal reconstruction output module is used to synthesize the final enhanced speech. This module uses the spectral masking vector output by the deep network inference module to perform weighted filtering on the buffered original noisy speech amplitude spectrum to suppress noise bands. Simultaneously, this module reuses the phase spectrum of the original noisy speech and combines it with the corrected amplitude spectrum to synthesize a complex spectrum. Finally, the signal reconstruction output module performs inverse short-time Fourier transform and overlap-add operation to eliminate inter-frame discontinuities and output clean speech waveform data in the time domain.

Claims

1. A speech enhancement method for mining scenes based on multimodal noise feature learning, characterized in that, Includes the following steps: S1. Acquire air conduction signals, solid conduction vibration signals, and acceleration signals at the mine operation site through a synchronous acquisition module, and perform time synchronization calibration on multiple of the signals to generate an original discrete signal sequence; S2. Perform detrending and standardization processing on the original discrete signal sequence, and perform frame windowing operation to generate a preprocessed discrete frame sequence. S3. Extract acoustic feature vectors representing the mixed state of speech and noise, and vibration feature vectors and acceleration feature vectors representing the physical conditions of pure noise sources from the discrete frame sequence, respectively. S4. The acoustic feature vector, the vibration feature vector, and the acceleration feature vector are structurally spliced ​​together, and the spliced ​​features are weighted and fused using a self-attention mechanism to generate a multimodal fusion feature vector. S5. Input the multimodal fusion feature vector into the pre-constructed CNN-LSTM deep augmentation network for inference, and output the spectral masking vector for the current time frame through the network. S6. The frequency domain amplitude spectrum of the air conduction signal generated in step S2 is corrected using the spectral masking vector, and the enhanced speech signal is reconstructed by combining it with the original phase spectrum.

2. The method for speech enhancement in mining scenes based on multimodal noise feature learning according to claim 1, characterized in that, Step S1 specifically includes: A high dynamic range microphone is used to acquire a mixed sound wave containing the target speech and background noise as the air conduction signal; A piezoelectric vibration sensor rigidly connected to the surface of the noise source device is used to collect mechanical vibration waves as the air conduction signal; The spatial impact characteristics are collected using a triaxial accelerometer mounted on the surface of the noise source device as the acceleration signal; By using a multi-channel parallel analog-to-digital converter to trigger the sampling of the air conduction signal, the air conduction signal, and the acceleration signal based on a unified clock source, it is ensured that the data of each channel are strictly aligned in the time domain.

3. The method for speech enhancement in mining scenes based on multimodal noise feature learning according to claim 1, characterized in that, Step S2 specifically includes: Calculate the arithmetic mean of the signals within the current processing window, and subtract the arithmetic mean point by point from the original discrete signal sequence to eliminate DC bias; The Z-Score normalization method is used to map each modal signal to the standard normal distribution space; The standardized signal is divided into short-time frame sequences with overlapping regions using the Hamming window function.

4. The method for speech enhancement in mining scenes based on multimodal noise feature learning according to claim 1, characterized in that, In step S3, the specific process of feature extraction includes: A short-time Fourier transform is performed on the air conduction signal to generate an amplitude spectrum. Based on the amplitude spectrum, the short-time energy, short-time zero-crossing rate, spectral centroid, and Mel frequency cepstral coefficients are calculated to construct the acoustic feature vector. The mean, variance, peak value, and kurtosis of the solid-conducted vibration signal and the acceleration signal are calculated respectively, and the discrete wavelet transform is performed to calculate the energy of the multi-scale wavelet coefficients, thereby constructing the vibration feature vector and the acceleration feature vector. The kurtosis is used to characterize the tail thickness of the signal probability density function and to identify impulsive noise components.

5. The method for speech enhancement in mining scenes based on multimodal noise feature learning according to claim 1, characterized in that, In step S4, the specific process of weighted fusion using the self-attention mechanism includes: The acoustic feature vector, the vibration feature vector, and the acceleration feature vector are concatenated into a joint feature vector according to a preset sequence strategy; The joint feature vector is projected through three independent learnable linear transformation matrices to generate a query vector, a key vector, and a value vector. Calculate the dot product of the query vector and the transpose of the key vector, divide the dot product result by the scaling factor, and then normalize it using the Softmax function to obtain the attention weight matrix; The value vector is weighted and summed using the attention weight matrix to output the multimodal fusion feature vector.

6. The method for speech enhancement in mining scenes based on multimodal noise feature learning according to claim 1, characterized in that, The CNN-LSTM deep augmentation network includes a local feature encoder, a temporal correlation modeler, and a spectral masking estimator connected in sequence. Step S5 specifically includes: Using the local feature encoder configured as a one-dimensional convolutional neural network, convolution operations and modified linear unit activations are performed on the input multimodal fusion feature vector to extract local feature representations; The local feature representation is processed using the temporal association modeler configured as a bidirectional long short-term memory network. The cell state is updated through a gating mechanism of input gate, forget gate and output gate, and the output is a temporal feature vector containing contextual information. The spectral masking estimator, configured as a fully connected layer, maps the temporal feature vector to an ideal ratio masking vector corresponding to the number of frequency points.

7. The method for speech enhancement in mining scenes based on multimodal noise feature learning according to claim 6, characterized in that, The CNN-LSTM deep augmentation network is pre-trained in the following manner: Construct a training set containing pairs of data, each pair of data including the multimodal fusion feature vector sequence as input and the pure speech amplitude spectrum as label; Define the mean squared error loss function and calculate the difference between the predicted enhanced speech amplitude spectrum and the pure speech amplitude spectrum output by the network. The Adam optimizer is used to update the convolution kernel parameters of the one-dimensional convolutional neural network, the gating weights of the bidirectional long short-term memory network, and the parameters of the fully connected layer in reverse according to the gradient of the loss function.

8. The method for speech enhancement in mining scenes based on multimodal noise feature learning according to claim 1, characterized in that, Step S6 specifically includes: Obtain the original noisy speech amplitude spectrum and the original noisy speech phase spectrum by performing a short-time Fourier transform on the air conduction signal; The enhanced amplitude spectrum is obtained by calculating the Hadamard product of the spectral masking vector and the original noisy speech amplitude spectrum; The enhanced amplitude spectrum and the original noisy speech phase spectrum are synthesized into a complex spectrum according to Euler's formula; The inverse short-time Fourier transform is performed on the complex spectrum to obtain the time-domain frame signal, and the overlapping addition method is used to eliminate the inter-frame discontinuity, outputting the final enhanced speech signal.

9. A method for speech enhancement in mining scenes based on multimodal noise feature learning according to claim 8, characterized in that, The overlapping addition method includes: According to the frame shift step size set during framing, adjacent time-domain frame signals are aligned and superimposed on the time axis; The sum of the superimposed signals divided by the square of the window function is normalized to eliminate periodic amplitude modulation and smooth frame boundaries.

10. A speech enhancement device for mining scenes based on multimodal noise feature learning, characterized in that, A method for speech enhancement in a mining scene based on multimodal noise feature learning, applicable to any one of claims 1-9, includes: The multimodal data acquisition module is configured to connect to an external acquisition unit and synchronously receive air conduction signals, solid conduction vibration signals, and acceleration signals from the mining site. The signal preprocessing module is configured to perform detrending, standardization, and frame windowing processing on the signal received by the multimodal data acquisition module; The multidimensional feature extraction module is configured to extract Mel frequency cepstral coefficients from acoustic modes and kurtosis and wavelet energy features from vibration and acceleration modes, respectively. The attention feature fusion module is configured to calculate the attention weights of physical features to acoustic features through a self-attention mechanism, and generate a weighted multimodal fusion feature vector. The deep network inference module is configured to load a pre-trained CNN-LSTM network and infer an ideal ratio mask vector based on the multimodal fusion feature vector. The signal reconstruction output module is configured to correct the original noisy amplitude spectrum using the ideal ratio masking vector and reuse the original phase spectrum to synthesize an enhanced speech signal.

Images