Dual-path feature fusion and harmonic reconstruction method and system for coal mine over-inhibited speech

Abstract

The application discloses a double-path feature fusion and harmonic reconstruction method and system for over-inhibited speech in coal mines, and the method comprises the following steps: acquiring original noisy speech signals collected from a coal mine site and obtaining corresponding over-inhibited speech signals; constructing a damage indication matrix according to the original noisy speech signals and the over-inhibited speech signals; obtaining an optimal fundamental frequency estimation value by performing low-pass filtering, autocorrelation function operation, searching for a maximum peak position in a fundamental frequency range of the speech signal and eliminating abnormal values on the over-inhibited speech signals; determining a harmonic center frequency point index according to the optimal fundamental frequency estimation value, and converting the harmonic center frequency point index into continuous time-frequency weights to construct a harmonic prior masking matrix; concatenating and splicing the amplitude spectrum of the over-inhibited speech signals, the damage indication matrix and the harmonic prior masking matrix to construct a composite feature tensor; and generating an enhanced speech signal according to the composite feature tensor. The application can solve the problems of over-inhibited speech features and semantic distortion.

Description

Technical Field

[0001] This invention relates to the field of speech signal processing technology, and in particular to a dual-path feature fusion and harmonic reconstruction method and system for oversuppressed speech in coal mines. Background Technology

[0002] In recent years, deep learning-based speech enhancement technology has become a mainstream solution for handling complex industrial noise due to its powerful nonlinear mapping capabilities. Existing technologies typically employ architectures such as deep neural networks, convolutional recurrent networks, or Conformer to perform supervised learning on massive noisy datasets.

[0003] Existing solutions typically use the short-time Fourier transform to extract the linear amplitude spectrum of the signal as the global input, use the backbone network to estimate the time-frequency mask or directly fit the pure spectrum, and minimize the numerical difference through symmetric loss functions such as mean square error. The aim is to forcibly suppress all non-speech components through end-to-end pure data mapping.

[0004] However, the acoustic environment of coal washing plants is extremely complex. The broadband noise generated by large washing machinery and conveyor belt equipment is not only enormous in energy but also exhibits significant non-Gaussian, non-stationary, and transient impact characteristics. Existing general solutions are prone to severe over-suppression when dealing with such high-energy and complex conditions, causing key harmonic components of speech to be crudely truncated by the algorithm as noise. Because purely data-driven networks lack prior knowledge of the physical laws of human vocalization, existing enhancement models cannot adaptively repair the damaged spectral skeleton after filtering out noise, resulting in enhanced speech with severe mechanical sounds and a significant loss of naturalness, greatly affecting the communication experience in actual industrial environments.

[0005] Therefore, existing deep learning enhancement schemes often adopt overly aggressive strategies in pursuit of extreme background noise suppression when dealing with extremely noisy conditions such as coal washing plant workshops, resulting in the following shortcomings: (1) The limitation of single-path repair capability leads to permanent information loss: Once the existing model over-suppresses the vulnerable speech during the noise reduction process, the speech features that are mistakenly deleted will be permanently lost in the single forward propagation path. The existing technology lacks an effective interaction and dual-path fusion mechanism between deep network features and over-suppressed residual information, which makes it impossible for the algorithm to find and repair the lost acoustic details from multi-dimensional hidden features; (2) Lack of physical priors leads to the inability to reconstruct the harmonic skeleton: Existing pure data-driven mapping, which lacks acoustic physical constraints, lacks understanding of the human speech harmonic generation mechanism when faced with strong noise coverage. When the harmonics of speech break under forced suppression, the network cannot adaptively reconstruct a continuous spectral skeleton, resulting in a severe mechanical sound in the final output enhanced speech waveform and a significant loss of naturalness.

[0006] The disclosure of the above background information is only for the purpose of assisting in understanding the concept and technical solution of this application, and does not necessarily provide technical instruction. Summary of the Invention

[0007] The purpose of this invention is to provide a dual-path feature fusion and harmonic reconstruction method and system for oversuppressed speech in coal mines, which can solve the problems of oversuppression of speech features and semantic distortion.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A dual-path feature fusion and harmonic reconstruction method for oversuppressed speech in coal mines includes the following steps: The original noisy speech signal collected at the coal mine site was acquired, and an oversuppressed speech signal was obtained based on the original noisy speech signal. The damage indication matrix is ​​constructed based on the original noisy speech signal and the oversuppressed speech signal, including: calculating the difference between the amplitude spectrum of the original noisy speech signal and the amplitude spectrum of the oversuppressed speech signal to obtain the damage energy residual, extracting the damage energy residuals that are greater than zero and setting the remaining damage energy residuals to zero to obtain the damage indication matrix. The optimal fundamental frequency estimate is obtained by performing low-pass filtering, autocorrelation function calculation, searching for the maximum peak position within the fundamental frequency range of the speech signal, and removing outliers on the oversuppressed speech signal. The harmonic center frequency index is determined based on the optimal fundamental frequency estimate, and the harmonic center frequency index is converted into continuous time-frequency weights to construct the harmonic prior masking matrix. The amplitude spectrum, damage indication matrix, and harmonic prior masking matrix of the oversuppressed speech signal are concatenated and spliced ​​to construct a composite feature tensor; Enhanced speech signals are generated based on composite feature tensors.

[0009] Furthermore, following any one or a combination of the aforementioned technical solutions, generating an enhanced speech signal based on a composite feature tensor includes the following steps: A composite feature tensor is fed into a pre-defined deep inpainting network to output a speech inpainting spectrum increment. The deep inpainting network includes an encoder, an attention module, and a decoder connected in sequence. The encoder is configured to perform two-dimensional convolution and downsampling operations on the input to extract deep acoustic semantic features layer by layer. The attention module is configured to assign corresponding gating weights to the acoustic semantic features based on a spatial attention weight mask and a priori masking matrix. The decoder is configured to perform an upsampling and transpose convolution operation on the acoustic semantic features after superimposing the gating weights to output the speech inpainting spectrum increment. The spatial attention weight mask is a mask for suppressing severe energy breaks caused by oversuppression in the speech signal, annotated by a damage indicator matrix. The high-fidelity speech amplitude spectrum is reconstructed by adding the speech restoration spectrum increment to the oversuppressed speech signal amplitude spectrum point by point in the frequency domain, and an enhanced speech signal is generated based on the high-fidelity speech amplitude spectrum.

[0010] Furthermore, in accordance with any one or a combination of the aforementioned technical solutions, generating enhanced speech signals based on composite feature tensors further includes the following steps: The optimal reconstructed phase is determined based on the speech restoration spectrum increment, the phase of the original noisy speech signal, and the phase of the oversuppressed speech signal. This includes: if the phase of the original noisy speech signal is greater than a preset threshold, then the phase of the original noisy speech signal is determined to be the optimal phase; otherwise, the phase of the oversuppressed speech signal is determined to be the optimal phase. The high-fidelity speech amplitude spectrum is combined with the optimal reconstructed phase, and the complete complex spectrum matrix is ​​constructed using Euler's formula. The enhanced speech signal is obtained by performing an inverse short-time Fourier transform on the complex spectral matrix.

[0011] Furthermore, following any one or a combination of the aforementioned technical solutions, the attention module is configured to assign corresponding gating weights to the acoustic semantic features based on a spatial attention weight mask and a priori masking matrix, thereby creating composite attention gating weights. The expression is as follows: ;in, This represents the self-attention mapping operator in the bottleneck layer. This represents the spatial attention weight mask. Denotes the prior masking matrix. For time frame indexing, This is a frequency point index.

[0012] Furthermore, following any one or a combination of the aforementioned technical solutions, the speech restoration spectrum increment is reconstructed by adding the amplitude spectrum of the oversuppressed speech signal point by point in the frequency domain, resulting in a high-fidelity speech amplitude spectrum. The expression is: ;in, For time frame indexing, For frequency point index, To suppress the amplitude spectrum of the speech signal, Incremental spectrum for speech restoration.

[0013] Furthermore, based on any one or a combination of the aforementioned technical solutions, the complex spectrum matrix... The expression is: ;in, For time frame indexing, For frequency point index, The imaginary unit, For high-fidelity speech amplitude spectrum, For optimal reconstruction phase, This represents an exponential function with the natural constant e as its base.

[0014] Furthermore, following any one or a combination of the aforementioned technical solutions, a spatial attention weight mask is generated based on the damage indication matrix, including the following steps: Extract mid-frequency slices within a preset mid-frequency range from the damage indication matrix to serve as the mid-frequency residual distribution matrix; Calculate the mean residual energy of each frame corresponding to the mid-frequency residual distribution matrix; and compare the coordinate point values ​​in the mid-frequency residual distribution matrix with the mean residual energy to filter out the high coordinate point values ​​of the target to determine the high-probability damage area; The high-probability damage region is processed using a normalized activation function to generate a spatial attention weight mask, which is configured to assign corresponding weight values ​​to the feature stream during the generation of the enhanced speech signal.

[0015] Furthermore, following any one or a combination of the aforementioned technical solutions, the average residual energy within the intermediate frequency slice is calculated using the following method. : ; This is the index corresponding to the upper limit frequency of the preset mid-frequency range. This is the index corresponding to the lower limit frequency of the preset mid-frequency range; And / or, High-probability damage areas are identified using the following method: If If so, it is determined to be a high-probability damage area. This is the residual distribution matrix for the mid-frequency band. For time frame indexing, Frequency point index; For threshold coefficient, , The mean residual energy; And / or, Spatial attention weight mask The expression is: ,in, For time frame indexing, For frequency point index, For threshold coefficient, , This represents the mean residual energy.

[0016] Furthermore, following any one or a combination of the aforementioned technical solutions, a damage indication matrix is ​​constructed based on the original noisy speech signal and the oversuppressed speech signal, including the following steps: Perform a Fourier transform on the original noisy speech signal to obtain the amplitude spectrum of the original noisy speech signal. Perform a Fourier transform on the oversuppressed speech signal to obtain the amplitude spectrum of the oversuppressed speech signal. ,in, For time frame indexing, Frequency point index; Calculate the amplitude spectrum of the original noisy speech signal amplitude spectrum of oversuppressed speech signal The difference; The damage indication matrix is ​​constructed using the ReLU activation function and based on the following formula. : .

[0017] Furthermore, following any one or a combination of the aforementioned technical solutions, the optimal fundamental frequency estimate is obtained through the following steps: Oversuppressed speech signals Low-pass filtering and frame segmentation are performed to obtain low-frequency signals. , n For sampling points, For frame index, Intra-frame sampling point index; The following formula is used to... Perform autocorrelation function calculation, the first Frame autocorrelation function The expression is: ,in, For frame length, This is the amount of time delay; Based on the following formula, within the preset time delay range The maximum peak position of the internal autocorrelation function is searched to extract the optimal delay period. : , Indicates in Calculation within the interval And determine the maximum value corresponding to The optimal delay period is determined based on the following formula. Convert to frequency to obtain fundamental frequency detection candidate results : ,in, Sampling rate; For the fundamental frequency detection candidate results Median smoothing filtering is performed to remove outliers, and the processed fundamental frequency detection candidate results are then analyzed. Sort and take the median to obtain the optimal fundamental frequency estimate. .

[0018] Furthermore, following any one or a combination of the aforementioned technical solutions, the harmonic center frequency index is determined based on the optimal fundamental frequency estimate, and the harmonic center frequency index is converted into continuous time-frequency weights to construct the harmonic prior masking matrix. This includes the following steps: Calculate the first based on the following formula. Frame, number Index of center frequencies of first harmonics : ;in, For time frame indexing, For harmonic series, The maximum harmonic order under the Nyquist frequency limit; The number of points in the Fourier transform. The system sampling rate, This is the rounding function; The discrete harmonic center frequency indexes are transformed into continuous time-frequency weights, and the prior masking matrix is ​​constructed using the following formula. : ; in, For time frame indexing, For frequency point index, To control the standard deviation parameter of the harmonic physical bandwidth, This represents an exponential function with the natural constant e as its base.

[0019] Furthermore, following any one or a combination of the aforementioned technical solutions, the method further includes having the original noisy speech signal and the oversuppressed speech signal have the same sampling rate, and performing frame-by-frame, frequency-by-frequency spatial physical alignment on the original noisy speech signal and the oversuppressed speech signal.

[0020] According to another aspect of the present invention, the present invention provides a dual-path feature fusion and harmonic reconstruction system for oversuppressed speech in coal mines, comprising a processor configured to process noisy speech signals based on the dual-path feature fusion and harmonic reconstruction method for oversuppressed speech in coal mines as described in any one or a combination of the above technical solutions.

[0021] The beneficial effects of the technical solution provided by this invention are as follows: a. This invention constructs an impairment indicator matrix by isomorphically aligning the original noisy speech signal and the oversuppressed speech signal and calculating the amplitude difference between the two, which accurately extracts the energy residual caused by oversuppression, and can effectively overcome the speech deletion impairment caused by blind noise reduction in traditional pure data-driven models; b. This invention solves the technical bottleneck of the inability to adaptively repair broken harmonic skeletons by introducing a harmonic physical prior masking matrix based on optimal fundamental frequency tracking and Gaussian mapping as an acoustic constraint; c. This invention employs a composite feature tensor and attention mechanism to directionally compensate for effective speech energy from the residuals, and combines it with optimal phase alignment technology to completely eliminate mechanical sounds during the enhancement process, thereby significantly improving the naturalness and intelligibility of speech under extreme conditions. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A flowchart of a dual-path feature fusion and harmonic reconstruction method for oversuppressed speech in coal mines, provided as an exemplary embodiment of the present invention; Figure 2 A block diagram of a dual-path feature fusion and harmonic reconstruction system for oversuppressed speech in coal mines, provided as an exemplary embodiment of the present invention; Figure 3 A flowchart for constructing a damage indication matrix and a spatial attention weight mask is provided as an exemplary embodiment of the present invention; Figure 4 A flowchart for reconstructing the amplitude spectrum of high-fidelity speech based on oversuppressed speech signals is provided as an exemplary embodiment of the present invention; Figure 5 A flowchart for obtaining an enhanced speech signal based on reconstructing the high-fidelity speech amplitude spectrum is provided as an exemplary embodiment of the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0026] To address the shortcomings of existing technologies, this invention proposes a dual-path feature fusion and harmonic reconstruction method for oversuppressed speech in coal mines. By constructing a damage indication matrix and performing dual-path deep fusion with the intermediate layer features of the network, and introducing a harmonic prior masking matrix as a physical constraint, the model is guided to accurately locate the damaged speech region and actively reconstruct the physical harmonic skeleton. This aims to fundamentally solve the speech distortion problem caused by oversuppression in complex working conditions such as coal washing plant workshops, ensuring that the final output enhanced speech waveform has both high communication intelligibility and naturalness.

[0027] In one embodiment of the present invention, a dual-path feature fusion and harmonic reconstruction method for oversuppressed speech in coal mines is provided, see [link to relevant documentation]. Figures 1 to 3 This includes the following steps: The original noisy speech signal was collected from a coal mine site, such as a coal washing plant, and an oversuppressed speech signal was obtained based on the original noisy speech signal. The damage indication matrix is ​​constructed based on the original noisy speech signal and the oversuppressed speech signal, including: calculating the difference between the amplitude spectrum of the original noisy speech signal and the amplitude spectrum of the oversuppressed speech signal to obtain the damage energy residual, extracting the damage energy residuals that are greater than zero and setting the remaining damage energy residuals to zero to obtain the damage indication matrix. The optimal fundamental frequency estimate is obtained by performing low-pass filtering, autocorrelation function calculation, searching for the maximum peak position within the fundamental frequency range of the speech signal, and removing outliers on the oversuppressed speech signal. The harmonic center frequency index is determined based on the optimal fundamental frequency estimate, and the harmonic center frequency index is converted into continuous time-frequency weights to construct the harmonic prior masking matrix. The amplitude spectrum, damage indication matrix, and harmonic prior masking matrix of the oversuppressed speech signal are concatenated and spliced ​​to construct a composite feature tensor; Enhanced speech signals are generated based on composite feature tensors.

[0028] The following detailed description of the dual-path feature fusion and harmonic reconstruction method for oversuppressed speech in coal mines, provided by this invention, is illustrated through a specific embodiment. It should be noted that the numbers in steps one through ten are for ease of understanding of the technical solution and do not represent the execution order of the dual-path feature fusion and harmonic reconstruction method for oversuppressed speech in coal mines. For example, steps four and five are processing steps for the frequency domain branch, steps six, seven, and eight are processing steps for the time domain branch, and steps nine and ten are processing steps for the combining path, wherein the frequency domain branch and the time domain branch are processed in parallel.

[0029] Step 1: Time-domain signal acquisition and preprocessing.

[0030] The raw, noisy speech signal x(n) in the time domain, collected from the coal washing plant workshop, is acquired by the acquisition module and then input into the existing noise reduction system to obtain the oversuppressed speech signal in the time domain, which is damaged by excessive suppression of intermediate frequency mechanical noise. Existing noise reduction systems can be traditional systems based on spectral subtraction, or intelligent noise reduction systems based on deep learning architectures (such as BSRNN, Conformer, or MossFormer). To ensure that subsequent processing fully covers the core frequency bands of human semantics, the two signals, namely x(n) and... The sampling rate is uniformly set to 16kHz. It should be noted that x(n) and A sampling rate of 16kHz is preferred, but other sampling rates can be selected as needed.

[0031] Step 2: Short-time Fourier transform and two-dimensional spectrum extraction.

[0032] The two one-dimensional time-domain signals are respectively fed into a preset feature extraction module to perform short-time Fourier transform (STFT). The feature extraction module is configured to perform short-time Fourier transform. Specifically, the feature extraction module includes a framing unit, a windowing unit, and a transform unit connected in sequence. The framing unit is responsible for dividing the continuous speech signal into short segments of fixed length; the windowing unit is responsible for adding Hamming windows to these short segments to avoid signal distortion; the transform unit performs discrete Fourier transform calculations on the processed segments to directly obtain the complex spectrum.

[0033] The same overlapping Hamming window was used in the processing, with a frame length of 32ms, a frame shift of 16ms, and a Fast Fourier Transform (FFT) point count of 512. The corresponding two-dimensional amplitude spectrum matrix was calculated by taking the absolute value of the complex spectrum. ; ; in, For time frame indexing, For frequency point index, The amplitude spectrum of the original noisy speech signal. This is to suppress the amplitude spectrum of the speech signal.

[0034] Simultaneously, the complex spectrum phase information generated by the short-time Fourier transform is retained, and the original noisy signal phase matrix is ​​extracted. Phase matrix of oversuppressed signal The complex spectral phase matrix is ​​then obtained and stored in a preset cache for use in the waveform reconstruction of the subsequent step ten.

[0035] Step 3: Sampling point alignment and time-frequency isomorphism processing.

[0036] Since the two time-domain signals have the same sampling rate and are absolutely synchronized on the physical time axis, and employ strictly consistent framing and windowing functions in the feature extraction module, their output two-dimensional time-frequency matrices (i.e., the aforementioned two-dimensional amplitude spectrum matrix and complex spectral phase matrix) are naturally isomorphic in the spatial dimension. The two-dimensional coordinate mapping module directly achieves frame-by-frame, frequency-by-frequency spatial physical alignment on the coordinate dimension of the two-dimensional time-frequency matrix, ensuring that the two signals are aligned at any identical coordinate point on the time-frequency matrix. They all point to the exact same physical time and frequency band.

[0037] Step 4: Construction of the damage indication matrix.

[0038] The amplitude spectra of the aligned original noisy speech signal and the amplitude spectra of the oversuppressed speech signal are synchronously input to the matrix arithmetic operation module to extract the energy residuals lost during the noise reduction process. The matrix arithmetic operation module includes a subtractor unit and a ReLU activation unit connected in sequence.

[0039] The subtractor unit calculates the amplitude difference between the two in the coordinate system, that is, it calculates the difference between the original noisy amplitude spectrum and the oversuppressed amplitude spectrum point by point. Then, the ReLU activation unit is used to filter out the values ​​less than or equal to zero in the difference to construct the damage indication matrix. .

[0040] The expression is as follows: ; This step introduces The function aims to avoid the situation where the oversuppressed signal is actually higher than the original signal at some pure noise points due to algorithm artifacts. It calculates the amplitude spectrum of the original noisy speech signal and the amplitude spectrum of the oversuppressed speech signal as the energy residual. The portion of the energy residual greater than 0 is retained, while the portion not greater than 0 is set to zero, thus obtaining the damage indication matrix. By constructing a damage indication matrix, undamaged or abnormally enhanced regions are strictly set to zero, retaining only the amplitude spectrum of the original noisy speech signal in the cut-off energy region.

[0041] Step 5: Generation of intermediate frequency residual slices and spatial attention weight masks.

[0042] This step extracts the mid-frequency residuals by using a preset spatial attention weight generation module and compares them with the mean to locate high-probability speech impairment points, thereby dynamically generating a spatial attention weight mask to guide the network's focus.

[0043] Considering the physical characteristics of mechanical noise (such as that from crushing and screening equipment) in coal washing plant workshops, where energy is highly concentrated in the mid-frequency range of 500Hz to 3000Hz, a damage indication matrix is ​​used. Perform the following frequency domain masking slicing process.

[0044] First, based on the 16kHz sampling rate and the frequency resolution of the 512-point Fast Fourier Transform, the bandwidth pass-through filter is set. Extract the mid-frequency residual distribution matrix using the upper and lower bound indices. : ; ; in, The index corresponding to the lower limit frequency set according to the frequency band characteristics of the coal washing plant. This is the index corresponding to the upper limit frequency set according to the frequency band characteristics of the coal washing plant.

[0045] Then, calculate the current number. The intermediate frequency band residual distribution matrix is ​​the mean residual energy within the intermediate frequency slice. : ; To accurately locate high-probability speech impairment points, it is necessary to determine whether the coordinate values ​​of the points are significantly higher than the average residual level of the frame.

[0046] like If the region is identified as a high-probability damage area, then a continuous spatial attention weight mask is generated using normalized activation logic. : ; in, The threshold coefficient, used to control the stringency of the screening, typically ranges from [value range missing]. ; Generated spatial attention weight mask The value of is between 0 and 1. This matrix accurately marks the severe energy break regions caused by oversuppression, using them as spatial priors to provide strongly guided spatial focusing for feature filtering and repair of the deep network in subsequent step nine.

[0047] Step Six: Fundamental Frequency Tracking and Preliminary Detection under Low-Frequency Constraints.

[0048] This step prioritizes performing autocorrelation calculations in the low-frequency band to extract the fundamental frequency, and uses a median smoothing filter to constrain adjacent frames to filter out burst interference in order to obtain the optimal fundamental frequency estimate.

[0049] An acoustic physical feature extraction module is introduced to provide physical priors for the reconstruction of fractured spectral textures. Considering that in complex industrial environments such as coal washing plants, the mid-to-high frequency features of speech are often completely masked or mistakenly deleted by noise reduction algorithms, the system prioritizes the acquisition of oversuppressed speech signals in the time domain. The signal is preprocessed by a low-pass filter with a cutoff frequency of 500Hz to obtain a low-frequency time-domain signal with less damage. .

[0050] For low-frequency signals after framing (in, For frame index, (For the intra-frame sampling point index), perform short-time autocorrelation function (ACF) calculations to track the vocal cord vibration period of the speech signal. Frame autocorrelation function Defined as: ;in, For frame length, This represents the time delay.

[0051] To avoid interference from low-frequency mechanical vibrations and to pinpoint the true range of human vocalization, the delay amount is limited. The search scope is Based on a system sampling rate of 16kHz and the normal human fundamental frequency range (typically 60Hz~500Hz), the following settings were made: , .

[0052] Search for the location of the maximum peak of the autocorrelation function within a limited range to extract the optimal delay period. : , Indicates in Calculation within the interval And determine the maximum value corresponding to Finally, the optimal delay period is converted into frequency to initially obtain the low-frequency time-domain signal of the current frame. Candidate results for fundamental frequency detection : ;in, The sampling rate.

[0053] The candidate result This provides the most crucial reference parameters for subsequent physical constraints and harmonic generation.

[0054] Step 7: Fundamental frequency time domain constraint and smoothing (removing false data from Step 6).

[0055] Utilizing the statistically stationary nature of human vocal organs over a short period (i.e., the vocal cord vibration frequency does not undergo physical abrupt changes within tens of milliseconds), the system analyzes the preliminary fundamental frequency detection candidate results obtained in step six. Apply time-domain continuity constraints.

[0056] To address fundamental frequency jumps caused by sudden mechanical collisions (such as coal falling or impacts from heavy components) in coal washing plant workshops, a sliding median smoothing filter is introduced to remove outliers. A filter with a length of [value missing] is set. Odd-numbered observation windows for the current frame and before and after The candidate fundamental frequencies of the frames are sorted and the median is taken to output the low-frequency time-domain signal of the current frame. Optimal fundamental frequency estimation Its mathematical expression is: ; in, To smooth the number of frames extended on one side of the window, satisfying Based on a 16ms frame shift parameter, this embodiment sets... That is, the length of the observation window This corresponds to a physical time span of 80ms. This operation effectively filters out sudden extreme values ​​that do not conform to the physiological laws of human vocalization, ensuring the smoothness of the estimation trajectory and outputting the optimal fundamental frequency estimation. As an absolute benchmark for the subsequent generation of harmonic physics a priori.

[0057] Step 8: Generation of harmonic physics prior masking matrix.

[0058] Based on the preset harmonic center frequency calculation unit, a harmonic generation operator is introduced, and the optimal fundamental frequency estimate output in step seven is used. As a physical benchmark for human vocalization, the center positions of each harmonic level are automatically derived on a discrete frequency axis. The harmonic center frequency calculation unit is configured to perform fundamental frequency multiplication and rounding operations.

[0059] For the Frame, number Index of center frequencies of first harmonics The calculation formula is: ; in, For harmonic series, The maximum harmonic order under the Nyquist frequency limit; The number of points in the Fourier transform. The system sampling rate, This is the rounding function.

[0060] Since real speech harmonics are not infinitely fine discrete spectral lines in the frequency domain, but rather possess a certain energy leakage and physical bandwidth, this acoustic characteristic is simulated. The discrete harmonic centers are calculated using the optimal fundamental frequency, and a Gaussian distribution mapping is used to simulate the real physical bandwidth, constructing a continuous time-frequency prior masking matrix as the repair benchmark. The system utilizes the Gaussian distribution mapping module to transform the discrete harmonic center frequency indices into continuous time-frequency weights, constructing a time-frequency prior masking matrix with spatial dimensions consistent with the input characteristics. The Gaussian distribution mapping calculation unit is configured to perform calculations of the squared frequency point distance difference and exponential function operations with the natural constant as the base.

[0061] Time-frequency prior masking matrix The expression is as follows: ; in, To control the standard deviation parameter of the harmonic physical bandwidth, which reflects the attenuation rate of energy diffusion to adjacent frequency points during actual sound generation, it is preferably set to... To adapt to complex operating conditions, this formula generates values ​​that are strictly bounded to the Gaussian mapping of each harmonic level by calculating the maximum value of each level. Continuous prior masking matrices between This matrix provides a complete, lossless harmonic skeleton diagram, which can be directly used to guide the frequency domain repair work of subsequent deep networks.

[0062] Step Nine: Multidimensional Tensor Fusion and Deep Repair Network Using a multidimensional tensor fusion unit to perform standard channel cascading operations, the amplitude spectrum of oversuppressed speech signals is converted. The damage indication matrix generated in step four and the harmonic prior masking matrix generated in step eight By cascading and stitching along the channel dimension, a composite feature tensor with consistent spatial dimensions and a channel depth of 3 is constructed. (in ).

[0063] Composite feature tensor The data is fed into a deep repair network based on an encoder-decoder architecture, and the time-frequency attention module of the deep repair network accurately extracts the compensation energy from the residual.

[0064] The encoder uses consecutive two-dimensional convolutional layers and downsampling operations to extract deep acoustic semantic features layer by layer.

[0065] A time-frequency attention module is introduced in the central region of the network for prior physical constraints. This module utilizes the harmonic prior masking matrix generated in step eight. With the spatial attention weight mask generated in step five Together, they selectively filter the deep acoustic semantic feature map output by the encoder, and set composite attention gating weights. The following physical mapping logic is satisfied: ;in, This represents the self-attention mapping operator in the bottleneck layer.

[0066] When the network detects the original residual energy region, i.e., the damage indicator matrix When the harmonic trajectory highly overlaps with the mid-frequency severe impairment characteristics, high activation weights are assigned to the corresponding time-frequency coordinates to guide the network to accurately extract the effective speech components from the impairment channel of the input tensor and discard pure machine noise.

[0067] Subsequently, the decoder restores the feature stream output by the attention module to the original two-dimensional time-frequency dimension through an upsampling transpose convolution operation symmetric to that of the encoder, outputting the speech restoration spectral increment of the depth prediction. .

[0068] Finally, the system introduces a global residual learning mechanism, which adds the speech restoration spectrum increment output by the deep network to the amplitude spectrum of the input oversuppressed speech signal point by point in the frequency domain to obtain the reconstructed high-fidelity speech amplitude spectrum. : This global residual structure not only significantly reduces the mapping difficulty of deep networks, but also ensures that the speech features of undamaged regions are fully preserved, achieving accurate repair of broken spectra.

[0069] In a specific application example, the deep inpainting network can adopt the classic U-Net architecture. The encoder consists of multiple layers of two-dimensional convolutional layers, and the decoder consists of symmetrical transposed convolutional layers. An attention module is configured between the encoder and decoder. The deep inpainting network is pre-trained using offline supervised training: clean speech samples and real industrial noise are mixed to generate noisy and clean speech data pairs with a signal-to-noise ratio of -5dB to 15dB as training samples. The number of samples, i.e., the size of the training set, is preferably no less than 50 hours to ensure the network's generalization ability. The network parameters use the Adam optimizer with an initial learning rate of 1×10⁻⁶. -4 The loss function is dynamically reduced with the number of training rounds. The mean squared error loss function is adopted, and the error between the predicted speech restoration spectrum increment output by the network and the ideal spectrum increment calculated by the real clean label is used as a penalty term. The network weights are continuously updated through backpropagation until the loss function converges and the training is completed, thus obtaining the trained deep restoration network.

[0070] Step 10: Time-frequency reconstruction and high-fidelity waveform synthesis.

[0071] Upon entering the time-frequency reconstruction and synthesis module, the system needs to reconstruct the amplitude spectrum from the output of step nine using high fidelity. The signal is then restored to a one-dimensional time domain signal. To ensure the time-domain continuity of the waveform, the system extracts the phase of the original noisy signal from step two. Phase with oversuppressed signal And based on the repair increment output in step nine Execute optimal phase alignment logic.

[0072] Set phase selection threshold Based on the amplitude repair increment, the optimal phase component is selected for alignment, and the optimal reconstructed phase at the current frequency is extracted. : .

[0073] That is, for damaged points that have been significantly compensated for by the network, the original physical phase is used to maintain the authenticity of the temporal structure; for finely adjusted or undamaged points, the oversuppressed phase is retained.

[0074] Subsequently, the reconstructed amplitude spectrum is combined with the optimal phase, and the complete complex spectrum matrix is ​​constructed using Euler's formula. : ;in, It is the imaginary unit.

[0075] Finally, the constructed complex spectrum matrix is ​​subjected to inverse short-time Fourier transform (iSTFT). A Hamming synthesis window, completely symmetrical to the analysis end, is used, and the overlapping addition logic strictly follows a 16ms frame shift setting. The time-domain segments reconstructed from each frame are smoothly spliced ​​together, ultimately outputting an enhanced speech signal that retains the complete speech harmonic skeleton and has no physical distortion. .

[0076] The dual-path feature fusion and harmonic reconstruction method for oversuppressed speech in coal mines provided by this invention can solve the problems of oversuppression and structural distortion, as well as harmonic skeleton breakage and physical structure damage caused by the aggressive suppression strategy adopted by existing speech enhancement systems under the extreme high noise conditions in coal mines.

[0077] First, to address the problem that existing algorithms may mistakenly remove overlapping speech features due to forced filtering of broadband noise in high-noise environments like coal mines, this invention introduces a dual-path residual compensation mechanism to recover the mistakenly deleted energy units, thereby eliminating muffled, discontinuous, and semantically missing speech and ensuring the integrity of communication information.

[0078] Secondly, addressing the technical pain point that pure data-driven models lack prior knowledge of the physical laws governing human speech, resulting in irreversible damage to speech harmonic texture and formants, this invention introduces physical constraints based on fundamental frequency trajectories for spectrum reconstruction, enabling adaptive repair of broken harmonics. This solves the problems of harmonic skeleton breakage and physical structural damage, significantly improving the naturalness of the enhanced speech and the intelligibility of on-site communication.

[0079] In one embodiment of the present invention, a dual-path feature fusion and harmonic reconstruction system for oversuppressed speech in coal mines is provided, including a processor configured to process noisy speech signals based on the dual-path feature fusion and harmonic reconstruction method for oversuppressed speech in coal mines as described in any of the above embodiments.

[0080] Specifically, the processor is equipped with the feature extraction module, matrix arithmetic operation module, spatial attention weight generation module, multidimensional tensor fusion unit, acoustic physical feature extraction module, sliding median smoothing filter, harmonic center frequency calculation unit, Gaussian distribution mapping module, deep repair network based on encoder-decoder architecture, and time-frequency reconstruction and synthesis module as described in the above embodiments.

[0081] In this architecture, the feature extraction module, matrix arithmetic operation module, and spatial attention weight generation module constitute the first branch, i.e., the frequency domain branch. The acoustic physical feature extraction module, sliding median smoothing filter, harmonic center frequency calculation unit, and Gaussian distribution mapping module are sequentially connected to form the second branch, i.e., the time domain branch. The outputs of the two branches are configured to be connected to the inputs of the multidimensional tensor fusion unit, the deep inpainting network based on the encoder-decoder architecture, and the time-frequency reconstruction and synthesis module, which are connected sequentially. The frequency domain branch executes steps one, two, four, and five in the above embodiments sequentially; the time domain branch executes steps six, seven, and eight in the above embodiments sequentially. The outputs of the two branches are processed by the multidimensional tensor fusion unit, the deep inpainting network, and the time-frequency reconstruction and synthesis module to obtain a low-damage, high-fidelity enhanced speech signal. The specific details are as described in the above embodiments and will not be repeated here.

[0082] It should be noted that the above-described embodiment of the dual-path feature fusion and harmonic reconstruction system for oversuppressed speech in coal mines and the embodiment of the dual-path feature fusion and harmonic reconstruction method for oversuppressed speech in coal mines share the same inventive concept. All contents of the embodiment of the dual-path feature fusion and harmonic reconstruction method for oversuppressed speech in coal mines are incorporated into the embodiment of the dual-path feature fusion and harmonic reconstruction system for oversuppressed speech in coal mines by way of reference.

[0083] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0084] The above description is only a specific embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A dual-path feature fusion and harmonic reconstruction method for oversuppressed speech in coal mines, characterized in that, Includes the following steps: The original noisy speech signal collected on site was acquired, and the oversuppressed speech signal was obtained based on the original noisy speech signal; The damage indication matrix is ​​constructed based on the original noisy speech signal and the oversuppressed speech signal, including: calculating the difference between the amplitude spectrum of the original noisy speech signal and the amplitude spectrum of the oversuppressed speech signal to obtain the damage energy residual, extracting the damage energy residuals that are greater than zero and setting the remaining damage energy residuals to zero to obtain the damage indication matrix. The optimal fundamental frequency estimate is obtained by performing low-pass filtering, autocorrelation function calculation, searching for the maximum peak position within the fundamental frequency range of the speech signal, and removing outliers on the oversuppressed speech signal. The harmonic center frequency index is determined based on the optimal fundamental frequency estimate, and the harmonic center frequency index is converted into continuous time-frequency weights to construct the harmonic prior masking matrix. The amplitude spectrum of the oversuppressed speech signal, the damage indication matrix, and the harmonic prior masking matrix are concatenated and spliced ​​to construct a composite feature tensor. Enhanced speech signals are generated based on composite feature tensors.

2. The dual-path feature fusion and harmonic reconstruction method for oversuppressed speech in coal mines according to claim 1, characterized in that, Generating enhanced speech signals based on composite feature tensors includes the following steps: A composite feature tensor is fed into a pre-defined deep inpainting network to output a speech inpainting spectrum increment. The deep inpainting network includes an encoder, an attention module, and a decoder connected in sequence. The encoder is configured to perform two-dimensional convolution and downsampling operations on the input to extract deep acoustic semantic features layer by layer. The attention module is configured to assign corresponding gating weights to the acoustic semantic features based on a spatial attention weight mask and a priori masking matrix. The decoder is configured to perform an upsampling and transpose convolution operation on the acoustic semantic features after superimposing the gating weights to output the speech inpainting spectrum increment. The spatial attention weight mask is a suppression of severely energy-breaking regions in the speech signal caused by oversuppression, annotated by a damage indicator matrix. The high-fidelity speech amplitude spectrum is reconstructed by adding the speech restoration spectrum increment to the oversuppressed speech signal amplitude spectrum point by point in the frequency domain, and an enhanced speech signal is generated based on the high-fidelity speech amplitude spectrum.

3. The dual-path feature fusion and harmonic reconstruction method for oversuppressed speech in coal mines according to claim 2, characterized in that, The process of generating enhanced speech signals based on composite feature tensors also includes the following steps: The optimal reconstructed phase is determined based on the speech restoration spectrum increment, the phase of the original noisy speech signal, and the phase of the oversuppressed speech signal. This includes: if the phase of the original noisy speech signal is greater than a preset threshold, then the phase of the original noisy speech signal is determined to be the optimal phase; otherwise, the phase of the oversuppressed speech signal is determined to be the optimal phase. The high-fidelity speech amplitude spectrum is combined with the optimal reconstructed phase, and the complete complex spectrum matrix is ​​constructed using Euler's formula. The enhanced speech signal is obtained by performing an inverse short-time Fourier transform on the complex spectral matrix.

4. The dual-path feature fusion and harmonic reconstruction method for oversuppressed speech in coal mines according to claim 2, characterized in that, The attention module is configured to assign corresponding gating weights to the acoustic semantic features based on a spatial attention weight mask and a priori masking matrix, resulting in composite attention gating weights. The expression is as follows: ;in, This represents the self-attention mapping operator in the bottleneck layer. This represents the spatial attention weight mask. Denotes the prior masking matrix. For time frame indexing, This is a frequency point index.

5. The dual-path feature fusion and harmonic reconstruction method for oversuppressed speech in coal mines according to claim 2, characterized in that, The high-fidelity speech amplitude spectrum is reconstructed by adding the speech restoration spectrum increment to the amplitude spectrum of the oversuppressed speech signal point by point in the frequency domain. The expression is: ;in, For time frame indexing, For frequency point index, To suppress the amplitude spectrum of the speech signal, Incremental speech spectrum for restoration.

6. The dual-path feature fusion and harmonic reconstruction method for oversuppressed speech in coal mines according to claim 2, characterized in that, Complex spectrum matrix The expression is: ;in, For time frame indexing, For frequency point index, The imaginary unit, For high-fidelity speech amplitude spectrum, For optimal reconstruction phase, This represents an exponential function with the natural constant e as its base.

7. The dual-path feature fusion and harmonic reconstruction method for oversuppressed speech in coal mines according to claim 2, characterized in that, Generate a spatial attention weight mask based on the damage indication matrix, including the following steps: Extract mid-frequency slices within a preset mid-frequency range from the damage indication matrix to serve as the mid-frequency residual distribution matrix; Calculate the mean residual energy of each frame corresponding to the mid-frequency residual distribution matrix; and compare the coordinate point values ​​in the mid-frequency residual distribution matrix with the mean residual energy to filter out the high coordinate point values ​​of the target to determine the high-probability damage area; The high-probability damage region is processed using a normalized activation function to generate a spatial attention weight mask, which is configured to assign corresponding weight values ​​to the feature stream during the generation of the enhanced speech signal.

8. The dual-path feature fusion and harmonic reconstruction method for oversuppressed speech in coal mines according to claim 7, characterized in that, The mean residual energy within the intermediate frequency slice is calculated using the following method. : ; This is the index corresponding to the upper limit frequency of the preset mid-frequency range. This is the index corresponding to the lower limit frequency of the preset mid-frequency range; And / or, High-probability damage areas are identified using the following method: If If so, it is determined to be a high-probability damage area. This is the residual distribution matrix in the mid-frequency band. For time frame indexing, Frequency point index; For threshold coefficient, , The mean residual energy; And / or, Spatial attention weight mask The expression is: ,in, For time frame indexing, For frequency point index, For threshold coefficient, , This represents the mean residual energy.

9. The dual-path feature fusion and harmonic reconstruction method for oversuppressed speech in coal mines according to claim 1, characterized in that, Constructing a damage indication matrix based on the original noisy speech signal and the oversuppressed speech signal includes the following steps: performing a Fourier transform on the original noisy speech signal to obtain the amplitude spectrum of the original noisy speech signal. Perform a Fourier transform on the oversuppressed speech signal to obtain the amplitude spectrum of the oversuppressed speech signal. ,in, For time frame indexing, Frequency index; calculate the amplitude spectrum of the original noisy speech signal. amplitude spectrum of oversuppressed speech signal The difference; using the ReLU activation function, the damage indication matrix is ​​constructed based on the following formula. : ; And / or, It also includes having the same sampling rate for the original noisy speech signal and the oversuppressed speech signal, and performing frame-by-frame, frequency-by-frequency spatial physical alignment on the original noisy speech signal and the oversuppressed speech signal.

10. The dual-path feature fusion and harmonic reconstruction method for oversuppressed speech in coal mines according to claim 1, characterized in that, The optimal fundamental frequency estimate is obtained through the following steps: for oversuppressed speech signals... Low-pass filtering and frame segmentation are performed to obtain low-frequency signals. , n For sampling points, For frame index, Intra-frame sampling point index; The following formula is used to... Perform autocorrelation function calculation, the first Frame autocorrelation function The expression is: ,in, For frame length, The time delay amount; based on the following formula, within the preset time delay range The maximum peak position of the internal autocorrelation function is searched to extract the optimal delay period. : , Indicates in Calculation within the interval And determine the maximum value corresponding to The optimal delay period is determined based on the following formula. Convert to frequency to obtain fundamental frequency detection candidate results : ,in, The sampling rate; for the fundamental frequency detection candidate results Median smoothing filtering is performed to remove outliers, and the processed fundamental frequency detection candidate results are then analyzed. Sort and take the median to obtain the optimal fundamental frequency estimate. ; And / or, The harmonic center frequency index is determined based on the optimal fundamental frequency estimate, and the harmonic center frequency index is converted into continuous time-frequency weights to construct the harmonic prior masking matrix. The process includes the following steps: Calculate the first... Frame, number Index of center frequencies of first harmonics : ;in, For time frame indexing, For harmonic series, The maximum harmonic order under the Nyquist frequency limit; The number of points in the Fourier transform. The system sampling rate, This is a rounding function; it transforms discrete harmonic center frequency indices into continuous time-frequency weights, and constructs the aforementioned prior masking matrix using the following formula. : ;in, For time frame indexing, For frequency point index, To control the standard deviation parameter of the harmonic physical bandwidth, This represents an exponential function with the natural constant e as its base.

11. A dual-path feature fusion and harmonic reconstruction system for oversuppressed speech in coal mines, characterized in that, The system includes a processor configured to process noisy speech signals based on a dual-path feature fusion and harmonic reconstruction method for oversuppressed speech in coal mines as described in any one of claims 1 to 10.

Images