Sound signal enhancement processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a sound signal enhancement processing method and device, electronic equipment and a storage medium, and relates to the technical field of audio signal processing or other related fields, and the method comprises the steps: carrying out the preprocessing of a to-be-enhanced noisy time domain sound signal, obtaining the three-channel spectrogram data of a corresponding frequency domain, comprising a real number spectrogram, an imaginary number spectrogram and an energy spectrogram; noise type information and three-channel spectrogram enhancement data are obtained through enhancement recognition analysis of a target detection model, the three-channel spectrogram enhancement data comprise an enhancement real number spectrogram, an enhancement imaginary number spectrogram and an enhancement energy spectrogram, and the target detection model comprises a noise information extraction branch and a sound main signal enhancement branch; generating a noise reduction time domain sound signal based on the noisy time domain sound signal and the three-channel spectrogram enhancement data; and outputting a sound signal enhancement processing result based on the noise type information and the noise reduction time domain sound signal. According to the invention, the technical problem of poor noise reduction effect on sound signals in a complex noise environment in the prior art is solved.

Description

Technical Field

[0001] This invention relates to the field of audio signal processing technology, and more specifically, to a method and apparatus for enhancing sound signals, an electronic device, and a storage medium. Background Technology

[0002] Audio signal denoising aims to extract the purest possible target sound from noisy frequencies, improving the signal-to-noise ratio (SNR) to enhance its practicality and robustness in applications such as communication, recognition, and audio analysis. Traditional audio signal enhancement methods typically rely on specific prior assumptions, such as the stationarity of noise and the statistical independence of noise from the target sound signal. However, noise in complex acoustic environments is often highly non-stationary, making it difficult for traditional denoising methods based on fixed prior assumptions to achieve ideal results. Furthermore, signal processing-based methods (such as spectral subtraction and Wiener filtering) often exhibit artifacts at low SNR conditions, affecting the enhanced sound quality.

[0003] In recent years, deep learning-based audio signal denoising methods have made significant progress. These methods do not require specific assumptions about the target audio signal and noise characteristics; instead, they directly learn the mapping relationship between noisy and clean signals through a data-driven approach. Although deep learning-based methods have demonstrated superior performance in denoising tasks, achieving consistent enhancement across different noise types remains a challenge. Because different noises have varying degrees of impact on the target audio signal, uniform training on datasets containing multiple noise types may lead to model bias, resulting in decreased denoising performance and poor results in complex noise scenarios.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] The present invention provides a method and apparatus for enhancing sound signals, an electronic device, and a storage medium, so as to at least solve the technical problem of poor noise reduction effect of sound signals in complex noise environments in related technologies.

[0006] According to one aspect of the present invention, a method for enhancing a sound signal is provided, comprising: preprocessing a noisy time-domain sound signal to be enhanced to obtain three-channel spectrogram data in the corresponding frequency domain, wherein the three-channel spectrogram data includes a real spectrum, an imaginary spectrum, and an energy spectrum; performing enhancement identification analysis on the three-channel spectrogram data using a target detection model to obtain noise type information and enhanced three-channel spectrogram data, wherein the target detection model includes a noise information extraction branch and a main sound signal enhancement branch, and the enhanced three-channel spectrogram data includes: an enhanced real spectrum, an enhanced imaginary spectrum, and an enhanced energy spectrum; generating an enhanced denoised time-domain sound signal based on the noisy time-domain sound signal and the enhanced three-channel spectrogram data; and outputting a sound signal enhancement processing result based on the noise type information and the denoised time-domain sound signal.

[0007] Further, the step of preprocessing the noisy time-domain audio signal to be enhanced to obtain the corresponding three-channel spectrum data in the frequency domain includes: determining a frequency threshold based on the sampling frequency of the noisy time-domain audio signal; using the frequency threshold as the cutoff frequency of a high-pass filter and filtering the noisy time-domain audio signal using the high-pass filter to obtain a high-frequency enhanced audio signal; processing the high-frequency enhanced audio signal to the frequency domain space using a short-time Fourier transform to obtain frequency domain features including the real-valued spectrum and the imaginary spectrum; calculating the energy value of the noisy time-domain audio signal at each frequency point based on the frequency domain features, and plotting the energy spectrum based on the energy value at each frequency point.

[0008] Furthermore, the noise information extraction branch includes a dynamic convolutional layer, a self-attention layer, and a fully connected layer. The step of performing enhanced identification analysis on the three-channel spectrogram data using a target detection model to obtain noise type information includes: processing the three-channel spectrogram data through the dynamic convolutional layer to obtain time-varying frequency features; modeling the time-varying frequency features in the time dimension through the self-attention layer to obtain a time-frequency dependency model; processing the time-frequency dependency model through the fully connected layer to obtain a noise type probability distribution, and determining the noise type information based on the noise type probability distribution.

[0009] Furthermore, the main sound signal enhancement branch includes an encoding layer, a core processing layer, and a decoding layer. The core processing layer includes an upper branch and a lower branch. The step of performing enhancement recognition analysis on the three-channel spectrogram data using a target detection model to obtain enhanced three-channel spectrogram data includes: extracting features from the three-channel spectrogram data through the encoding layer to obtain local signal features; performing time correlation modeling on the local signal features in the upper branch of the core processing layer to obtain a first feature map; performing frequency correlation modeling on the local signal features in the lower branch of the core processing layer to obtain a second feature map; and fusing and decoding the first feature map and the second feature map through the decoding layer to obtain the enhanced three-channel spectrogram data.

[0010] Furthermore, the step of generating an enhanced denoised time-domain sound signal based on the noisy time-domain sound signal and the three-channel spectrogram enhancement data includes: performing data fusion processing on the noisy time-domain sound signal and the three-channel spectrogram enhancement data to obtain fused feature data; and reconstructing the time-domain signal using inverse short-time Fourier transform based on the fused feature data to obtain the denoised time-domain sound signal.

[0011] Further, the step of performing data fusion processing on the noisy time-domain audio signal and the three-channel spectral enhancement data to obtain fused feature data includes: based on the phase information of the enhanced energy spectrum and the noisy time-domain audio signal, performing spectrum reconstruction according to Euler's formula to obtain a reconstructed real spectrum and a reconstructed imaginary spectrum; superimposing and averaging the enhanced real spectrum and the reconstructed real spectrum to obtain a fused real spectrum; superimposing and averaging the enhanced imaginary spectrum and the reconstructed imaginary spectrum to obtain a fused imaginary spectrum; and outputting the fused feature data containing the fused real spectrum and the fused imaginary spectrum.

[0012] Furthermore, the modeling steps of the target detection model include: acquiring a pre-constructed sample dataset, wherein the sample dataset contains at least: noisy signal samples and corresponding clean signal samples, and noise type labels; training an initial detection model using the sample dataset, and adjusting the model parameters according to the model loss during the training process, wherein the noise information extraction branch in the initial detection model has a first loss function, and the main sound signal enhancement branch has a second loss function; and stopping the training process and adjusting the model parameters when the model loss meets preset conditions to obtain the target detection model.

[0013] According to another aspect of the present invention, an audio signal enhancement processing apparatus is also provided, comprising: a preprocessing unit, configured to preprocess a noisy time-domain audio signal to be enhanced to obtain three-channel spectrogram data in the corresponding frequency domain, wherein the three-channel spectrogram data includes a real spectrum, an imaginary spectrum, and an energy spectrum; an analysis unit, configured to perform enhancement recognition analysis on the three-channel spectrogram data using a target detection model to obtain noise type information and three-channel spectrogram enhancement data, wherein the target detection model includes a noise information extraction branch and a main audio signal enhancement branch, and the three-channel spectrogram enhancement data includes: an enhanced real spectrum, an enhanced imaginary spectrum, and an enhanced energy spectrum; a generation unit, configured to generate an enhanced denoised time-domain audio signal based on the noisy time-domain audio signal and the three-channel spectrogram enhancement data; and an output unit, configured to output the audio signal enhancement processing result based on the noise type information and the denoised time-domain audio signal.

[0014] Further, the preprocessing unit includes: a determination module, used to determine a frequency threshold based on the sampling frequency of the noisy time-domain sound signal; a first processing module, used to use the frequency threshold as the cutoff frequency of a high-pass filter and to filter the noisy time-domain sound signal using the high-pass filter to obtain a high-frequency enhanced sound signal; a second processing module, used to process the high-frequency enhanced sound signal to the frequency domain space using a short-time Fourier transform to obtain frequency domain features including the real-valued spectrum and the imaginary spectrum; and a calculation module, used to calculate the energy value of the noisy time-domain sound signal at each frequency point based on the frequency domain features, and to draw the energy spectrum based on the energy value at each frequency point.

[0015] Furthermore, the noise information extraction branch includes: a dynamic convolutional layer, a self-attention layer, and a fully connected layer. The analysis unit includes: a third processing module, used to process the three-channel spectral data through the dynamic convolutional layer to obtain time-varying frequency features; a first modeling module, used to model the time-varying frequency features in the time dimension through the self-attention layer to obtain a time-frequency dependency model; and a fourth processing module, used to process the time-frequency dependency model through the fully connected layer to obtain a noise type probability distribution, and determine the noise type information based on the noise type probability distribution.

[0016] Furthermore, the main sound signal enhancement branch includes an encoding layer, a core processing layer, and a decoding layer. The core processing layer includes an upper branch and a lower branch. The analysis unit further includes: a feature extraction module, used to extract features from the three-channel spectrogram data through the encoding layer to obtain local signal features; a second modeling module, used to perform time-correlation modeling on the local signal features in the upper branch of the core processing layer to obtain a first feature map; a third modeling module, used to perform frequency-correlation modeling on the local signal features in the lower branch of the core processing layer to obtain a second feature map; and a decoding module, used to fuse and decode the first feature map and the second feature map through the decoding layer to obtain the three-channel spectrogram enhancement data.

[0017] Furthermore, the generation unit includes: a fifth processing module, used to perform data fusion processing on the noisy time-domain sound signal and the three-channel spectrogram enhancement data to obtain fused feature data; and a reconstruction module, used to reconstruct the time-domain signal using inverse short-time Fourier transform based on the fused feature data to obtain the denoised time-domain sound signal.

[0018] Further, the fifth processing module includes: a reconstruction submodule, used to reconstruct the spectrum according to Euler's formula based on the enhanced energy spectrum and the phase information of the noisy time-domain sound signal, to obtain a reconstructed real spectrum and a reconstructed imaginary spectrum; a first superposition averaging submodule, used to superimpose and average the enhanced real spectrum and the reconstructed real spectrum to obtain a fused real spectrum; a second superposition averaging submodule, used to superimpose and average the enhanced imaginary spectrum and the reconstructed imaginary spectrum to obtain a fused imaginary spectrum; and an output submodule, used to output the fused feature data containing the fused real spectrum and the fused imaginary spectrum.

[0019] Furthermore, the sound signal enhancement processing device further includes: a modeling unit for performing the modeling steps of the target detection model, the modeling unit including: an acquisition module for acquiring a pre-constructed sample dataset, wherein the sample dataset includes at least: noisy signal samples and corresponding clean signal samples, and noise type labels; a training module for training an initial detection model using the sample dataset, and adjusting the model parameters according to the model loss during the training process, wherein the noise information extraction branch in the initial detection model has a first loss function, and the main sound signal enhancement branch has a second loss function; and a stopping module for stopping the training process and adjusting the model parameters when the model loss meets preset conditions, thereby obtaining the target detection model.

[0020] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the sound signal enhancement processing method described in any one of the above embodiments.

[0021] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the sound signal enhancement processing method described in any one of the preceding embodiments.

[0022] According to another aspect of the present invention, a computer program product is also provided, including computer instructions, wherein when the computer instructions are executed by a processor, they implement the steps of the sound signal enhancement processing method described in any one of the above embodiments.

[0023] This invention proposes a method for enhancing sound signals. First, the noisy time-domain sound signal to be enhanced is preprocessed to obtain three-channel spectrogram data in the corresponding frequency domain. The three-channel spectrogram data includes a real spectrum, an imaginary spectrum, and an energy spectrum. Then, an enhancement identification analysis is performed on the three-channel spectrogram data using a target detection model to obtain noise type information and enhanced three-channel spectrogram data. The target detection model includes a noise information extraction branch and a main sound signal enhancement branch. The enhanced three-channel spectrogram data includes an enhanced real spectrum, an enhanced imaginary spectrum, and an enhanced energy spectrum. Next, an enhanced denoised time-domain sound signal is generated based on the noisy time-domain sound signal and the enhanced three-channel spectrogram data. Finally, based on the noise type information and the denoised time-domain sound signal, the sound signal enhancement processing result is output.

[0024] In this invention, a deep learning framework guided by integrated prior information is adopted. By designing a noise information extraction branch and a main sound signal enhancement branch, the invention achieves the effect of accurately diagnosing noise type and optimizing signal enhancement in a targeted manner. This results in a significant improvement in noise reduction quality and audio clarity in complex noise environments, thereby solving the technical problem of poor noise reduction effect on sound signals in complex noise environments in related technologies.

[0025] Specifically, this invention's preprocessing of noisy time-domain audio signals goes beyond traditional spectrum analysis. It transforms the signals into three-channel spectrogram data, including real, imaginary, and energy spectra, providing richer and more detailed input information for subsequent deep learning model analysis. This ensures the model can fully understand the signal's spectral characteristics. Subsequently, a detection model is used to conduct in-depth analysis of the spectrogram data, identifying noise types and generating enhanced three-channel spectrogram data for different noise types, more accurately reflecting the signal's purity. Finally, combining the noisy time-domain audio signal and the enhanced three-channel spectrogram data, an inverse transform technique is used to generate a denoised time-domain signal, restoring the signal's temporal continuity and preserving its original rhythm and cadence, greatly improving the user's auditory experience in noisy environments. Based on the comprehensive analysis of noise type information and the denoised time-domain audio signal, this invention outputs high-quality audio signal enhancement results, providing strong technical support for fields such as sound recognition, communication, and audio analysis. Attached Figure Description

[0026] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0027] Figure 1 This is a flowchart of an optional sound signal enhancement processing method according to an embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram of an optional frequency domain feature evolution according to an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram of an optional sound signal enhancement processing device according to an embodiment of the present invention;

[0030] Figure 4 This is a structural block diagram of an electronic device for performing a method for enhancing audio signals according to an embodiment of the present invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] To facilitate understanding of the present invention by those skilled in the art, some terms or nouns involved in the various embodiments of the present invention are explained below:

[0034] A time-domain signal refers to a sound signal represented on the time axis, reflecting the waveform characteristics of the signal as it changes over time. A noisy time-domain sound signal is a complex signal that has noise superimposed on a pure sound signal.

[0035] Frequency-domain signals are signals that have been converted from time-domain signals to frequency-domain signals using techniques such as Fourier transform, in order to reflect the amplitude and phase of different frequency components in the signal.

[0036] Three-channel spectral data, including real, imaginary, and energy spectra, provides a comprehensive spectral representation that fully reflects the spectral characteristics of a signal. The energy spectrum is particularly helpful in identifying the energy distribution within a signal.

[0037] In this invention, the target detection model refers not only to a model for identifying and locating targets in an image, but also to a deep learning model used for analyzing and processing frequency domain signals, capable of identifying noise types and performing signal enhancement analysis.

[0038] The Noise Information Extraction Branch is a sub-model or functional module in the target detection model specifically designed to extract noise type information from the signal, providing prior guidance for subsequent signal enhancement.

[0039] The Main Signal Enhancement Branch is another key part of the target detection model. It is used to enhance the main spectral features of the signal, reduce the impact of noise, and improve the clarity and quality of the signal.

[0040] Multi-task learning is a machine learning strategy that trains a model to perform multiple related tasks simultaneously, improving model performance by sharing learned representations. In this invention, the noise information extraction branch and the main sound signal enhancement branch work collaboratively through multi-task learning to improve the model's performance in noise reduction and classification tasks.

[0041] ISTFT, Inverse Short Time Fourier Transform, is a signal processing technique that, in contrast to Short Time Fourier Transform, is used to convert frequency domain signals back to time domain signals. In this invention, it is used to reconstruct the noise-reduced time domain audio signal.

[0042] Real and imaginary spectra, the spectral representations after short-time Fourier transform, respectively reflect the real and imaginary components of the signal in the frequency domain. Using them in combination can more accurately reconstruct the phase information of the signal.

[0043] An energy spectrum represents the energy distribution of a signal at different frequencies and time points. It is used to analyze the spectral characteristics of a signal in order to identify noise types and evaluate the effectiveness of noise reduction.

[0044] The following embodiments of the present invention can be applied to various systems / applications / devices requiring high-precision audio signal denoising and audio enhancement in complex scenarios. They enable fine-grained enhancement of audio signals using a deep learning model guided by prior information. The present invention uses a target detection model for noise type identification and signal spectrum analysis and enhancement. Then, it combines the noisy time-domain audio signal and the enhanced frequency-domain spectrogram data to generate a denoised time-domain audio signal, which can more effectively and specifically remove noise components, significantly improving the clarity and resolution of the audio signal.

[0045] In its specific implementation, this invention first preprocesses the noisy audio signal, generating three-channel spectral data including real, imaginary, and energy spectra through signal conversion, ensuring that noise characteristics and the main components of the audio signal are fully characterized. Subsequently, the noise information extraction branch in the target detection model performs deep learning analysis on these spectral data to accurately identify the noise type, while the main sound signal enhancement branch performs targeted enhancement processing on the spectral data based on the identified noise type information, generating three-channel enhanced spectral data. Finally, based on the noisy time-domain audio signal and the enhanced spectral information, the inverse short-time Fourier transform technique is used to reconstruct a clean time-domain audio signal, achieving effective noise suppression and signal quality improvement.

[0046] This invention avoids the problem of traditional noise reduction techniques adversely affecting signal quality in complex noise environments by integrating a deep learning framework guided by prior information and a multi-task learning strategy. At the same time, it overcomes the technical obstacle that a single model cannot adapt to multiple noise types, and significantly improves the robustness and flexibility of sound signal enhancement.

[0047] The present invention will now be described in detail with reference to various embodiments.

[0048] Example 1

[0049] According to an embodiment of the present invention, an embodiment of a method for enhancing sound signals is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0050] The implementation subject of this invention can be an intelligent audio signal processing system, or integrated into various communication devices, audio monitoring devices, communication equipment, and field biological research tools, as well as various devices and platforms that need to process sound signals in noisy environments. Combining deep learning and multi-task strategies, it is used for sound signal enhancement processing in complex noise scenarios. In particular, it addresses the problem of poor noise reduction effect of sound signals in complex noise environments in related technologies. By designing a noise prior information extraction auxiliary branch and a sound signal enhancement main branch, the specific steps are to perform preprocessing, use a target detection model to identify noise type and perform signal enhancement analysis, generate a noise-reduced time-domain signal, and output the enhancement processing result, so as to achieve the purpose of accurate noise suppression and signal clarity improvement.

[0051] The embodiments of the present invention will now be described in detail with reference to the specific implementation steps.

[0052] Figure 1 This is a flowchart of an optional sound signal enhancement processing method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0053] Step S101: Preprocess the noisy time-domain sound signal to be enhanced to obtain the corresponding three-channel spectrum data in the frequency domain. The three-channel spectrum data includes a real spectrum, an imaginary spectrum, and an energy spectrum.

[0054] Specifically, noisy time-domain sound signals refer to unprocessed sound signals collected in natural or man-made environments, including target sounds (such as speeches, music, animal calls, etc.) and various background noises (car noises, wind noises, crowd noises, etc.). In the time domain, these signals appear as time-series waveforms with characteristics that change over time. However, due to the superposition of noise, the quality of the original sound signal is affected, leading to unclear information transmission and difficulties in analysis.

[0055] Converting time-domain signals to the frequency domain allows for clearer observation of noise superposition and original sound signal information, reducing the difficulty of analysis. Furthermore, compared to a single spectral representation, three-channel spectral data provides more complete and accurate signal characteristics. Noisy time-domain audio signals can be converted to the frequency domain using methods such as Short-Time Fourier Transform (STFT) to obtain three different types of spectral representations, specifically including:

[0056] Real number spectrum : Reflects the real part of the signal in the frequency domain, that is, the amplitude of the signal at different frequencies;

[0057] Imaginary number spectrum : The imaginary part of the signal in the frequency domain, which reflects the phase information of the signal;

[0058] Energy spectrum Displaying the energy distribution of various frequency components of a signal, i.e., the power or intensity of the signal at different frequencies, is one of the key bases for identifying noise and target signals.

[0059] The three-channel spectral data composed of the aforementioned spectral information provides richer foundational information for noise identification and signal enhancement. For example, the energy spectrum can highlight which frequency bands carry the majority of energy, while the real and imaginary spectra can help the model more accurately understand the amplitude and phase characteristics of the signal, thereby making more informed decisions during noise reduction and signal enhancement.

[0060] The above steps can improve the noise reduction effect of sound signals in complex noise environments. Specifically, by acquiring three-channel spectral data, a more comprehensive noise analysis perspective is provided, enabling the model to identify noise types based on richer spectral information and to perform targeted noise suppression, ultimately achieving the goal of improving signal clarity, enhancing noise reduction robustness and applicability.

[0061] Further, the step of preprocessing the noisy time-domain audio signal to be enhanced to obtain the corresponding three-channel spectrum data in the frequency domain includes: determining a frequency threshold based on the sampling frequency of the noisy time-domain audio signal; using the frequency threshold as the cutoff frequency of a high-pass filter and filtering the noisy time-domain audio signal using the high-pass filter to obtain a high-frequency enhanced audio signal; processing the high-frequency enhanced audio signal to the frequency domain space using short-time Fourier transform to obtain frequency domain features containing real and imaginary spectra; calculating the energy value of the noisy time-domain audio signal at each frequency point based on the frequency domain features, and plotting the energy spectrum based on the energy value at each frequency point.

[0062] It should be noted that in digital signal processing, the sampling frequency refers to the number of times a continuous signal is sampled per second (i.e., converted into a digital signal). It is a key parameter for determining the upper limit of signal spectrum analysis and is the highest frequency (Nyquist frequency) in the frequency domain that the signal can be accurately reproduced during the digitization of sound signals.

[0063] This embodiment determines a frequency threshold based on the sampling frequency of the noisy time-domain audio signal. This threshold serves as the basis for setting the cutoff frequency of the subsequent high-pass filter, aiming to remove low-frequency noise or unnecessary components from the signal and improve its clarity. For example, if the sampling frequency is 44.1 kHz, the frequency threshold is set above 50 Hz to ensure that common frequency components in human voices are preserved while filtering out most low-frequency background noise.

[0064] Another point to note is that a high-pass filter is a signal processing tool that only passes through the high-frequency components of a signal, blocking or attenuating sound signals below a specific cutoff frequency. In this embodiment, based on the physical characteristics of sound, the high-frequency components of human voice and many target audio signals often carry important identification features, while background noise is mostly concentrated in the low-frequency region. Therefore, using a high-pass filter to preprocess the noisy time-domain sound signal, by setting the filter's cutoff frequency to the aforementioned determined frequency threshold, can effectively enhance the high-frequency components in the signal, i.e., achieve a high-frequency enhancement effect for the sound signal. This helps reduce low-frequency background noise in the signal before processing, thereby improving the accuracy of subsequent spectrum analysis and the efficiency of noise identification. In practice, this process can be achieved by designing digital filtering algorithms, such as infinite impulse response (IIR) or finite impulse response (FIR) filters.

[0065] It should be further explained that Short-Time Fourier Transform (STFT) is a commonly used method to convert time-domain audio signals into frequency-domain features. It divides the signal into multiple short time intervals and performs a Fourier transform on each interval to obtain the signal's spectral information as a function of time. In this embodiment, STFT is used to convert the preprocessed audio signal to the frequency domain, obtaining not only the signal's spectral amplitude information but also its phase information, represented as a real spectrum and an imaginary spectrum, respectively. The result of STFT is a complex number; its real and imaginary parts correspond to the real and imaginary spectra of the signal, respectively, providing a complete representation of the signal in the frequency domain.

[0066] Furthermore, an energy spectrum is a graph that describes how signal energy changes with frequency. Calculating the energy spectrum based on frequency domain characteristics involves squaring the amplitude of the signal at each frequency point to reflect the energy distribution of the signal at that frequency. Energy values ​​are used in signal processing to reveal the spectral density and intensity distribution of a signal, helping to identify which frequencies carry the main information and which frequencies are dominated by noise. Plotting the energy spectrum graphically presents the calculated energy values, facilitating a direct and intuitive assessment of the signal's spectral structure and providing a basis for noise type identification and signal enhancement strategy development.

[0067] This embodiment, through a series of preprocessing steps, including frequency threshold determination, application of high-pass filter and short-time Fourier transform, and calculation of energy spectrum, not only helps in the early optimization of signal, but also generates spectral data that comprehensively reflects the spectral characteristics of the signal, making it suitable for signal processing in complex noise environments.

[0068] Step S102: Enhance the three-channel spectrum data by using the target detection model to obtain noise type information and enhanced three-channel spectrum data. The target detection model includes a noise information extraction branch and a main sound signal enhancement branch. The enhanced three-channel spectrum data includes: enhanced real spectrum, enhanced imaginary spectrum, and enhanced energy spectrum.

[0069] In this embodiment, the object detection model is applied to the enhanced recognition and analysis of sound signals, extracting noise type information from complex three-channel spectrogram data and generating enhanced signal features. The base model can be selected from, but is not limited to, convolutional neural networks (CNNs), recurrent neural networks (RNNs), attention mechanisms, and Transformer models.

[0070] If CNN is chosen as the base model, dynamic convolutional layers and self-attention mechanisms can be added to enhance the model's sensitivity in the frequency direction and its ability to perform time series analysis. Dynamic convolutional layers can adaptively adjust to the signal characteristics of different frequency bands, while self-attention mechanisms help capture the long-term dependencies of signals.

[0071] If an RNN is chosen, it can effectively process sequential data. By adding gated recurrent units (GRUs) or long short-term memory units (LSTMs), it can better handle the dynamic changes of time-domain signals, thereby improving the accuracy of noise type identification. At the same time, it can be combined with the structure of CNNs for spectral feature extraction, building a deep learning model that integrates time-frequency information.

[0072] Furthermore, the noise information extraction branch can be based on a CNN structure, including dynamic convolutional layers, self-attention mechanisms, and fully connected layers. Dynamic convolutional layers capture the dynamic changes in frequency domain features, self-attention mechanisms model audio features from a temporal perspective, and fully connected layers output the probability distribution of noise types. This branch identifies and classifies noise types through deep analysis of three-channel spectrogram data, providing prior guidance for the model.

[0073] The main audio signal enhancement branch is based on an encoder and decoder architecture, employing multi-layer dilated convolutions for complex feature extraction and encoding, a self-attention mechanism with positional encoding for modeling long-term temporal correlations, and a decoder for spectrum reconstruction. Its main task is to combine the output of the noise information extraction branch with adjustments to the three-channel spectral data to reduce noise impact, generating enhanced real, imaginary, and energy spectra.

[0074] In the enhanced three-channel spectral data, the enhanced real spectrum This refers to the real-valued spectrum of the signal after noise removal, which more accurately reflects the frequency and amplitude information of the target sound; similarly, the enhanced imaginary spectrum is also processed. This refers to the imaginary spectrum that retains the signal's phase information, used to reconstruct a clean time-domain signal; enhanced energy spectrum. Used to display the energy distribution of a signal at different frequency points, after noise reduction processing, the energy of the target signal is more prominent at key frequency points, which helps to enhance the model signal.

[0075] The above steps can simultaneously handle noise classification and signal enhancement tasks through the object detection model. By utilizing noise type information to guide the noise reduction process, more efficient and accurate signal enhancement can be achieved, improving the noise reduction performance of sound signals in complex noise environments. Further, the modeling steps of the object detection model include: acquiring a pre-constructed sample dataset, which at least contains: noisy signal samples and corresponding clean signal samples, and noise type labels; training an initial detection model using the sample dataset, and adjusting the model parameters based on the model loss during training. The initial detection model has a first loss function for the noise information extraction branch and a second loss function for the main sound signal enhancement branch; stopping the training process and adjusting the model parameters when the model loss meets preset conditions, thus obtaining the object detection model.

[0076] In this embodiment, the sample dataset is constructed by combining actually collected clean signal samples with various types of noise samples, and through a series of preprocessing and mixing operations to simulate different noise reduction scenarios. The sample dataset contains at least noisy signal samples, corresponding clean signal samples, and noise type labels. The noisy signal samples are formed by mixing clean signals and noise at a certain signal-to-noise ratio, and the noise type labels are manually annotated by experts based on noise characteristics to guide the model in learning the characteristics of noise.

[0077] The noisy signal samples are generated by mixing clean audio signals collected in real time with noise data from the outdoor environment. The noise data includes various types, such as wind noise, crowd noise, and mechanical noise, to cover different noise scenarios. The clean signal samples refer to sound signals without any background noise, which are used to compare and evaluate the noise reduction effect of the model. The noise type label means that each piece of noise data is assigned one or more labels to indicate the category to which the noise belongs, which helps the model learn noise classification during training.

[0078] The initial detection model typically consists of a noise information extraction branch and a main sound signal enhancement branch. Model parameters include, but are not limited to, convolutional kernel weights, parameters of the self-attention mechanism, weights and biases of fully connected layers, etc. The model loss is jointly determined by the first loss function (classification loss) of the noise information extraction branch and the second loss function (such as mean squared error loss) of the main sound signal enhancement branch.

[0079] During training, the model predicts noise type information and enhanced frequency domain features based on the input three-channel spectrogram data. The first loss function measures the difference between the noise type predicted by the model and the actual label, while the second loss function evaluates the gap between the generated enhanced data and the spectral features of the clean signal. Through the calculation and feedback of these loss functions, the model parameters are adjusted to optimize noise recognition and signal enhancement performance.

[0080] Specifically, the first loss function (classification loss) calculates the difference between the noise type probability distribution predicted by the model and the actual noise type label, typically using cross-entropy loss to quantify this difference. An example of the first loss function is provided below: ;in, Representing the The noisy signal sample at the th The actual noise type label (1 if it belongs to that type, 0 otherwise). The branch representing the noise prior information extraction is the first... The noisy signal sample at the th Predicted probability of noise and These represent the total number of samples and the number of noise categories, respectively.

[0081] Additionally, the second loss function (noise reduction loss) measures how closely the enhanced signal generated by the model approximates the clean signal in the frequency domain, and can be evaluated using mean squared error (MSE) or spectral match error. An example of the second loss function is provided below: ;in, Representing the The clean sound signal corresponding to each noisy signal sample. This represents the enhanced time-domain audio signal output from the audio signal enhancement branch. Represents the number of samples.

[0082] Furthermore, the formula for calculating the total loss function of the object detection model can be as follows: ;in, and All represent preset coefficients. The loss function characterizing the noise information extraction branch. The loss function characterizing the enhancement branch of the main sound signal.

[0083] In each training iteration, the model makes predictions based on the input noisy spectral data, then compares the predictions with labeled and clean signal samples to calculate the loss function value. The direction and magnitude of the model parameter adjustments are determined by the gradient of the loss function, and the model "learns" how to more accurately identify noise types and more effectively enhance the signal through backpropagation.

[0084] Furthermore, the preset conditions refer to the standards used during the training process to judge whether the model has reached the expected performance level. These conditions may include, but are not limited to: loss function convergence, i.e., the model's loss value no longer decreases significantly over multiple consecutive training cycles, indicating that the model has learned stable and effective parameters; validation set performance meeting the standards, i.e., the noise recognition accuracy and signal enhancement effect on non-training data meet the predetermined standards, proving that the model has good generalization ability; and training cycle limit, i.e., reaching the preset maximum number of training iterations. Even if the model performance does not fully meet the above conditions, training should be stopped to avoid overfitting or excessive resource consumption.

[0085] For example, the preset conditions might be set such that the total loss value of the model does not change by more than 1% in the last ten rounds of training, or that the average accuracy of noise recognition on the validation set reaches 95%, and the average SNR improvement after signal enhancement is higher than 10dB, at which point the training process and parameter adjustments are stopped.

[0086] The above steps developed a deep learning model capable of automatically identifying noise types and performing targeted signal enhancement to improve audio processing performance in complex noise environments. By incorporating prior noise information to enhance the model's adaptability and robustness under different noise scenarios, high-quality noise reduction and signal enhancement of noisy audio signals are ultimately achieved under low signal-to-noise ratio and non-stationary noise conditions.

[0087] Furthermore, the noise information extraction branch includes: dynamic convolutional layers, self-attention layers, and fully connected layers. The step of enhancing the recognition and analysis of the three-channel spectrogram data through the target detection model to obtain noise type information includes: processing the three-channel spectrogram data through dynamic convolutional layers to obtain frequency features that change over time; modeling the frequency features that change over time in the time dimension through self-attention layers to obtain a time-frequency dependence model; processing the time-frequency dependence model through fully connected layers to obtain the probability distribution of noise types, and determining the noise type information based on the probability distribution of noise types.

[0088] It should be noted that dynamic convolutional layers are used to capture the dynamic changes in frequency-direction features in three-channel spectral data (real spectrum, imaginary spectrum, and energy spectrum). Due to the non-stationarity and complexity of noise, dynamic convolutional layers use their unique two-dimensional convolutional kernels and frequency-direction attention mechanisms to perform deep learning on the input spectral information, identifying and distinguishing different types of noise features. During computation, the convolutional kernels of dynamic convolutional layers not only perform sliding and dot product operations in space (such as frequency and time), but also automatically adjust the weights in the convolution process through the frequency-direction attention mechanism, enhancing feature extraction of key frequency bands. This allows the model to more accurately focus on frequency components related to specific noise types, thereby improving the accuracy of noise recognition.

[0089] A self-attention layer models audio features in the time dimension, aiming to capture the dependencies between signals that change over time. The model uses a self-attention mechanism to find and correlate signal features at different times within spectral data. During computation, the self-attention layer forms an attention weight matrix by calculating the dot product of the query, key, and value of the input features, and then integrates the obtained time-series features through a weighted sum. This process allows the model to focus on the most important time segments of the signal while ignoring irrelevant information, thus more accurately extracting the signal's inherent patterns even in complex noisy environments.

[0090] The fully connected layer, serving as the output layer of the noise information extraction branch, further integrates the time-frequency dependent model processed by the dynamic convolutional layer and the self-attention layer, outputting the predicted probability distribution of noise types. Through the processing of the fully connected layer, the model can learn the statistical regularities of noise types from spectral features, thus providing a basis for noise classification. During computation, the fully connected layer receives the feature vector output by the self-attention layer and integrates the features through a weighted sum operation (where the weights and biases are model parameters optimized during training). Each element in the final vector corresponds to a noise type, representing the predicted probability of that noise type. By comparing the magnitudes of each element in the probability distribution vector, the dominant noise type information in the input signal can be determined.

[0091] In practical implementation, the configuration and parameters of dynamic convolutional layers, self-attention layers, and fully connected layers require careful design. For example, the kernel size of the dynamic convolutional layer should consider the average span of the harmonic structure in the signal; the query, key, and value vector dimensions of the self-attention layer need to match the feature dimensions of the input spectral data; and the output dimension of the fully connected layer is equal to the number of noise types. Furthermore, each layer may contain multiple corresponding operational units to deepen the model's hierarchy and improve its feature representation capabilities.

[0092] The core of the above steps lies in automatically identifying and classifying noise types through a deep learning model, and then performing targeted enhancement processing on the audio signal. The noise prior information extraction branch not only improves the model's noise recognition ability but also provides crucial guidance for the main audio signal enhancement branch, ensuring good noise reduction performance and signal clarity in various noise environments.

[0093] Furthermore, the main sound signal enhancement branch includes an encoding layer, a core processing layer, and a decoding layer. The core processing layer includes an upper branch and a lower branch. The steps of enhancing and identifying the three-channel spectrogram data through the target detection model to obtain the enhanced three-channel spectrogram data include: extracting features from the three-channel spectrogram data through the encoding layer to obtain local signal features; modeling the temporal correlation of the local signal features in the upper branch of the core processing layer to obtain a first feature map; modeling the frequency correlation of the local signal features in the lower branch of the core processing layer to obtain a second feature map; and fusing and decoding the first and second feature maps through the decoding layer to obtain the enhanced three-channel spectrogram data.

[0094] It should be noted that the encoding layer is responsible for deep feature extraction from the input three-channel spectral data (real spectrum, imaginary spectrum, and energy spectrum), reducing data dimensionality, enhancing feature expressiveness, and preserving key signal information. In this embodiment, the encoding layer consists of multiple dilated convolutions. Since dilated convolutions can expand the receptive field while maintaining input resolution, they are particularly suitable for capturing complex time-frequency correlations in audio signals, thereby obtaining local signal features. In practice, each dilated convolution layer in the encoding layer performs a convolution operation on the input data, increasing the size of the receptive field to integrate a wider range of time and frequency information. During the calculation, the dilated convolution layer adjusts the filter width according to a preset dilation rate (i.e., the number of zero elements inserted into the convolution kernel), thus considering dependencies at different time scales when extracting features.

[0095] The core processing layer is the key part of the model responsible for in-depth modeling of the local features of the signal output from the coding layer. Through temporal correlation modeling in the upper branch and frequency correlation modeling in the lower branch, it generates the first and second feature maps respectively, providing more comprehensive time-frequency domain information for signal enhancement. Specifically, the upper branch models the local features of the signal using a self-attention mechanism with positional encoding, capturing the long-term dependencies of the signal in the time dimension and generating the first feature map. The introduction of positional encoding enables the self-attention mechanism to distinguish features at different time points, thereby more accurately learning the patterns of the time series. The lower branch focuses on the relationship of the signal in the frequency dimension, generating the second feature map through another self-attention mechanism with positional encoding, strengthening the model's perception of spectral information and ensuring noise reduction and signal enhancement effects in the frequency dimension.

[0096] The decoding layer's task is to fuse the first and second feature maps generated by the core processing layer and decode them into enhanced three-channel spectral data: enhanced real spectrum, enhanced imaginary spectrum, and enhanced energy spectrum. As a crucial step in signal enhancement, it utilizes signal features learned during encoding and core processing to generate cleaner, more focused spectral information for the target sound. Specifically, the decoding layer first performs feature fusion on the first and second feature maps, typically involving weighted combinations of feature maps along channel and spatial dimensions. Then, through multiple dilated convolutions (configured as mirrors of the convolutional layers in the encoding layer to restore the data to its original resolution), the fused feature maps are decoded into enhanced spectral data. Finally, this spectral data is converted back to a time-domain signal, completing the signal enhancement process.

[0097] The above steps enable a deep learning model that can automatically identify noise types and specifically enhance target sound signals in complex noise environments. By improving the clarity and recognizability of sound signals through feature extraction in the encoding layer, time-frequency domain modeling in the core processing layer, and feature fusion and decoding in the decoding layer, it is particularly suitable for signal processing in low signal-to-noise ratio and non-stationary noise environments, such as outdoor animal sound recording and industrial noise monitoring.

[0098] Step S103: Generate an enhanced denoised time-domain audio signal based on the noisy time-domain audio signal and the three-channel spectrogram enhancement data.

[0099] The core objective of the above steps is to convert the enhanced frequency domain signal (i.e., the three-channel spectrum enhancement data) back into a time domain audio signal, thereby achieving noise reduction and signal enhancement. It also transforms the noise identification and signal feature optimization work done in the frequency domain in the previous steps into an audible, higher-quality audio signal.

[0100] The Inverse Short Time Fourier Transform (ISTFT) is the most direct conversion method, which converts the enhanced real, imaginary, and energy spectra back into a time-domain signal. ISTFT restores the time-domain form of the original signal based on the enhanced spectral information, while simultaneously using the energy spectrum to guide the reconstruction of phase information, resulting in a more natural sound signal.

[0101] In practice, the enhanced real spectrum, enhanced imaginary spectrum, and enhanced energy spectrum output by the decoder are first processed to ensure compatibility with the input requirements of ISTFT. Then, the ISTFT algorithm converts these three sets of spectral information into an enhanced time-domain audio signal. The calculation principle of ISTFT involves recombining the spectral data into short time windows, performing an inverse Fourier transform, and finally merging the signals from each short time window to obtain a complete audio signal.

[0102] Furthermore, the step of generating an enhanced denoised time-domain audio signal based on the noisy time-domain audio signal and the three-channel spectrogram enhancement data includes: performing data fusion processing on the noisy time-domain audio signal and the three-channel spectrogram enhancement data to obtain fused feature data; and reconstructing the time-domain signal using inverse short-time Fourier transform based on the fused feature data to obtain the denoised time-domain audio signal.

[0103] In this embodiment, data fusion processing refers to the process of combining the enhanced three-channel spectrogram data (i.e., enhanced real spectrum, enhanced imaginary spectrum, and enhanced energy spectrum) output by the decoder with the original noisy time-domain audio signal. This process remaps the enhanced frequency domain features back to the time domain, generating a high-quality, denoised audio signal. Data fusion processing typically involves combining the enhanced frequency domain features with the phase information of the noisy signal, preserving the natural characteristics of the signal while reducing the impact of noise.

[0104] The fused feature data is a comprehensive audio feature obtained after data fusion processing, containing both enhanced spectral information and original phase information. In this stage, the enhanced energy spectrum is used as a guide and combined with the real and imaginary spectra (i.e., phase information) of the noisy signal to form a new frequency domain representation. This representation includes both the noise-reduced signal features and, as far as possible, preserves the time-domain characteristics of the original signal.

[0105] Time-domain signal reconstruction refers to the process of converting fused feature data into a time-domain signal by applying the inverse short-time Fourier transform. In this step, the frequency-enhanced features are converted into time-domain sound, ensuring that the frequency-domain processing results can be presented in a form that humans can understand.

[0106] The noise-reduced time-domain audio signal is the result of time-domain signal reconstruction. Compared with the original noisy time-domain audio signal, the noise component in the noise-reduced time-domain audio signal is significantly reduced, the clarity of the target sound is improved, and the quality is closer to the pure audio signal without noise.

[0107] The above steps enable efficient noise reduction and sound signal enhancement based on prior information. By designing an auxiliary branch for noise prior information extraction and a main branch for sound signal enhancement, combined with a multi-task learning strategy, the model can adaptively identify noise types in complex noise environments and perform targeted noise elimination, thereby improving the clarity and recognizability of the sound signal after noise reduction.

[0108] Further, the steps of performing data fusion processing on the noisy time-domain audio signal and the three-channel spectrogram enhancement data to obtain fused feature data include: based on the phase information of the enhanced energy spectrum and the noisy time-domain audio signal, performing spectrum reconstruction according to Euler's formula to obtain a reconstructed real spectrum and a reconstructed imaginary spectrum; superimposing and averaging the enhanced real spectrum and the reconstructed real spectrum to obtain a fused real spectrum; superimposing and averaging the enhanced imaginary spectrum and the reconstructed imaginary spectrum to obtain a fused imaginary spectrum; and outputting fused feature data containing the fused real spectrum and the fused imaginary spectrum.

[0109] It should be noted that Euler's formula is a mathematical concept that describes the relationship between the exponential form of complex numbers and trigonometric functions. In this embodiment, Euler's formula is used to describe and process spectral data of noisy audio signals. Specifically, Euler's formula can convert a complex number representation with phase (such as spectral data of noisy audio signals) into a linear combination of its real and imaginary parts.

[0110] Since the spectral representation of an audio signal typically includes both amplitude and phase, while the energy spectrum only reflects the amplitude information, and phase information affects the time-domain shape of the audio signal (such as waveform shape and temporal synchronization), it is crucial for accurate signal recovery. Therefore, this embodiment does not directly use Euler's formula to generate the reconstructed energy spectrum, but instead utilizes principles to reconstruct the real and imaginary spectra of the signal. Specifically, the phase angle is determined based on the phase information of the original noisy signal, and the amplitude is determined based on the enhanced energy spectrum, forming complex-form spectral information, where the real part is the reconstructed real spectrum. The imaginary part is the reconstructed imaginary spectrum. .

[0111] Furthermore, superposition averaging is a processing technique that combines multiple signal sources or multiple signal representations to generate a more stable and accurate output signal. In this embodiment, superposition averaging is used to fuse the enhanced real-valued spectrum and the reconstructed real-valued spectrum, as well as the enhanced imaginary spectrum and the reconstructed imaginary spectrum, to obtain a fused real-valued spectrum and a fused imaginary spectrum. Superposition averaging can smooth signal fluctuations, reduce the impact of noise, and enhance signal strength. By superimposing and averaging the enhanced real-valued spectrum with the real-valued spectrum reconstructed based on Euler's formula, a fused real-valued spectrum that balances noise suppression and signal integrity preservation can be obtained. The same process can be applied to the imaginary spectrum to obtain the fused imaginary spectrum. .

[0112] It should be noted that, Figure 2 This is a schematic diagram of an optional frequency domain feature evolution according to an embodiment of the present invention, such as... Figure 2 As shown, in the implementation evolution of this embodiment: Enhanced real-valued spectra and enhanced imaginary spectrum It is the direct output of the sound signal enhancement model, reflecting the model's removal of noise components from noisy audio signals and enhancement of the target audio signal; reconstructing the real-valued spectrogram. and reconstructing the imaginary spectrum It reconstructs the real and imaginary spectra with phase information based on the enhanced energy spectrum and the original phase information of the noisy signal using Euler's formula. These two spectra preserve the time-frequency characteristics of the signal, especially the phase details; the real spectrum is then fused. And the fusion of imaginary number spectrum The spectrum is generated after superposition and averaging, which is the sum of the results of balancing noise reduction effect and signal fidelity. It has both high signal-to-noise ratio and maintains good sound quality and naturalness.

[0113] The above steps can automatically identify noise types and perform targeted noise suppression and signal enhancement to improve the clarity, fidelity, and perceived quality of sound signals in complex environments. Through the data fusion and spectral reconstruction processes described above, this invention aims to provide a more comprehensive and accurate noise reduction and signal enhancement solution, particularly for non-stationary noise and low signal-to-noise ratio situations, significantly improving the quality and efficiency of signal processing.

[0114] Step S104: Based on the noise type information and the noise reduction time-domain audio signal, output the audio signal enhancement processing result.

[0115] It should be noted that after completing the steps of noise prior information extraction, sound signal enhancement, and time-domain signal reconstruction, all previous processing results are integrated to output the final sound signal enhancement result. First, using the noise category information obtained from the noise prior information extraction branch, the enhanced denoised time-domain sound signal is further optimized. This allows the model to employ specific post-processing strategies based on the identified noise type, such as secondary filtering of specific noise or sound quality adjustment, to further improve signal quality and adaptability. Then, based on the optimized denoised time-domain sound signal, the final sound signal enhancement result is generated, including but not limited to saving the denoised signal as an audio file for playback via an audio interface, or sending the audio signal to other signal processing systems or applications, such as speech recognition, audio communication, and natural sound signal analysis.

[0116] By employing deep learning and multi-task learning strategies, this embodiment utilizes noise reduction and sound signal enhancement guided by prior noise information to output high-quality, high-definition audio signal processing results, maintaining signal recognizability and naturalness, especially in complex noisy environments. This embodiment aims to improve performance in processing natural sound signals (such as bird calls), achieving effective noise suppression and a significant improvement in signal quality.

[0117] Through steps S101 to S104 above, the noisy time-domain audio signal to be enhanced can be preprocessed to obtain the corresponding three-channel spectrum data in the frequency domain. The three-channel spectrum data includes a real spectrum, an imaginary spectrum, and an energy spectrum. Then, the three-channel spectrum data is enhanced and identified by a target detection model to obtain noise type information and enhanced three-channel spectrum data. The target detection model includes a noise information extraction branch and a main audio signal enhancement branch. The enhanced three-channel spectrum data includes: enhanced real spectrum, enhanced imaginary spectrum, and enhanced energy spectrum. Then, the enhanced denoised time-domain audio signal is generated based on the noisy time-domain audio signal and the enhanced three-channel spectrum data. Finally, the audio signal enhancement processing result is output based on the noise type information and the denoised time-domain audio signal.

[0118] In this embodiment, a deep learning framework guided by integrated prior information is adopted. By designing a noise information extraction branch and a main sound signal enhancement branch, the method achieves the effect of accurately diagnosing noise type and optimizing signal enhancement in a targeted manner. This results in a significant improvement in noise reduction quality and audio clarity in complex noise environments, thereby solving the technical problem of poor noise reduction effect of sound signals in complex noise environments in related technologies.

[0119] Specifically, this embodiment preprocesses the noisy time-domain audio signal beyond traditional spectrum analysis, transforming it into three-channel spectrogram data containing real, imaginary, and energy spectra. This provides richer and more detailed input information for subsequent deep learning model analysis, ensuring the model can fully understand the signal's spectral characteristics. Subsequently, a detection model is used to conduct in-depth analysis of the spectrogram data, identifying not only the type of noise but also generating enhanced three-channel spectrogram data for different noise types, more accurately reflecting the signal's purity. Finally, combining the noisy time-domain audio signal and the enhanced three-channel spectrogram data, an inverse transform technique is used to generate a denoised time-domain signal, restoring the signal's temporal continuity and preserving its original rhythm and cadence, greatly improving the user's auditory experience in noisy environments. Based on the comprehensive analysis of noise type information and the denoised time-domain audio signal, this invention outputs high-quality audio signal enhancement results, providing strong technical support for fields such as sound recognition, communication, and audio analysis.

[0120] The invention will now be described in conjunction with another alternative embodiment.

[0121] Example 2

[0122] The sound signal enhancement processing device provided in this embodiment includes multiple implementation units, each of which corresponds to a specific implementation step in Embodiment 1 above.

[0123] Figure 3This is a schematic diagram of an optional audio signal enhancement processing device according to an embodiment of the present invention, such as... Figure 3 As shown, the device may include: a preprocessing unit 31, an analysis unit 32, a generation unit 33, and an output unit 34.

[0124] The preprocessing unit 31 is used to preprocess the noisy time-domain sound signal to be enhanced to obtain three-channel spectrum data in the corresponding frequency domain. The three-channel spectrum data includes a real spectrum, an imaginary spectrum, and an energy spectrum.

[0125] Analysis unit 32 is used to perform enhanced identification analysis on the three-channel spectrum data through a target detection model to obtain noise type information and enhanced three-channel spectrum data. The target detection model includes a noise information extraction branch and a main sound signal enhancement branch. The enhanced three-channel spectrum data includes: enhanced real spectrum, enhanced imaginary spectrum, and enhanced energy spectrum.

[0126] The generation unit 33 is used to generate an enhanced denoised time-domain sound signal based on the noisy time-domain sound signal and the three-channel spectrogram enhancement data.

[0127] The output unit 34 is used to output the audio signal enhancement processing result based on noise type information and the noise-reduced time-domain audio signal.

[0128] The aforementioned sound signal enhancement processing device can first preprocess the noisy time-domain sound signal to be enhanced through the preprocessing unit 31 to obtain three-channel spectrum data in the corresponding frequency domain. The three-channel spectrum data includes a real spectrum, an imaginary spectrum, and an energy spectrum. Then, the analysis unit 32 performs enhancement recognition analysis on the three-channel spectrum data through a target detection model to obtain noise type information and enhanced three-channel spectrum data. The target detection model includes a noise information extraction branch and a main sound signal enhancement branch. The enhanced three-channel spectrum data includes: enhanced real spectrum, enhanced imaginary spectrum, and enhanced energy spectrum. Then, the generation unit 33 generates an enhanced denoised time-domain sound signal based on the noisy time-domain sound signal and the enhanced three-channel spectrum data. Finally, the output unit 34 outputs the sound signal enhancement processing result based on the noise type information and the denoised time-domain sound signal.

[0129] In this embodiment of the invention, a deep learning framework guided by integrated prior information is adopted. By designing a noise information extraction branch and a main sound signal enhancement branch, the method achieves the effect of accurately diagnosing noise type and optimizing signal enhancement in a targeted manner. This results in a significant improvement in noise reduction quality and audio clarity in complex noise environments, thereby solving the technical problem of poor noise reduction effect on sound signals in complex noise environments in related technologies.

[0130] Specifically, this embodiment preprocesses the noisy time-domain audio signal beyond traditional spectrum analysis, transforming it into three-channel spectrogram data containing real, imaginary, and energy spectra. This provides richer and more detailed input information for subsequent deep learning model analysis, ensuring the model can fully understand the signal's spectral characteristics. Subsequently, a detection model is used to conduct in-depth analysis of the spectrogram data, identifying not only the type of noise but also generating enhanced three-channel spectrogram data for different noise types, more accurately reflecting the signal's purity. Finally, combining the noisy time-domain audio signal and the enhanced three-channel spectrogram data, an inverse transform technique is used to generate a denoised time-domain signal, restoring the signal's temporal continuity and preserving its original rhythm and cadence, greatly improving the user's auditory experience in noisy environments. Based on the comprehensive analysis of noise type information and the denoised time-domain audio signal, this invention outputs high-quality audio signal enhancement results, providing strong technical support for fields such as sound recognition, communication, and audio analysis.

[0131] Furthermore, the preprocessing unit includes: a determination module for determining a frequency threshold based on the sampling frequency of the noisy time-domain sound signal; a first processing module for using the frequency threshold as the cutoff frequency of a high-pass filter and filtering the noisy time-domain sound signal using the high-pass filter to obtain a high-frequency enhanced sound signal; a second processing module for processing the high-frequency enhanced sound signal into the frequency domain space using a short-time Fourier transform to obtain frequency domain features including real and imaginary spectra; and a calculation module for calculating the energy value of the noisy time-domain sound signal at each frequency point based on the frequency domain features and drawing an energy spectrum based on the energy value at each frequency point.

[0132] Furthermore, the noise information extraction branch includes: a dynamic convolutional layer, a self-attention layer, and a fully connected layer. The analysis unit includes: a third processing module, used to process the three-channel spectral data through the dynamic convolutional layer to obtain the frequency features that change over time; a first modeling module, used to model the frequency features that change over time in the time dimension through the self-attention layer to obtain a time-frequency dependence model; and a fourth processing module, used to process the time-frequency dependence model through the fully connected layer to obtain the noise type probability distribution, and to determine the noise type information based on the noise type probability distribution.

[0133] Furthermore, the main sound signal enhancement branch includes an encoding layer, a core processing layer, and a decoding layer. The core processing layer includes an upper branch and a lower branch. The analysis unit also includes: a feature extraction module, used to extract features from the three-channel spectrogram data through the encoding layer to obtain local signal features; a second modeling module, used to perform time correlation modeling on the local signal features in the upper branch of the core processing layer to obtain a first feature map; a third modeling module, used to perform frequency correlation modeling on the local signal features in the lower branch of the core processing layer to obtain a second feature map; and a decoding module, used to fuse and decode the first and second feature maps through the decoding layer to obtain the three-channel spectrogram enhancement data.

[0134] Furthermore, the generation unit includes: a fifth processing module, used to perform data fusion processing on the noisy time-domain sound signal and the three-channel spectrogram enhancement data to obtain fused feature data; and a reconstruction module, used to reconstruct the time-domain signal based on the fused feature data using inverse short-time Fourier transform to obtain a denoised time-domain sound signal.

[0135] Furthermore, the fifth processing module includes: a reconstruction submodule, used to reconstruct the spectrum according to Euler's formula based on the phase information of the enhanced energy spectrum and the noisy time-domain sound signal, to obtain a reconstructed real spectrum and a reconstructed imaginary spectrum; a first superposition averaging submodule, used to superimpose and average the enhanced real spectrum and the reconstructed real spectrum to obtain a fused real spectrum; a second superposition averaging submodule, used to superimpose and average the enhanced imaginary spectrum and the reconstructed imaginary spectrum to obtain a fused imaginary spectrum; and an output submodule, used to output fused feature data containing the fused real spectrum and the fused imaginary spectrum.

[0136] Furthermore, the sound signal enhancement processing device also includes: a modeling unit for performing the modeling steps of the target detection model, the modeling unit including: an acquisition module for acquiring a pre-constructed sample dataset, wherein the sample dataset includes at least: noisy signal samples and corresponding clean signal samples, and noise type labels; a training module for training an initial detection model using the sample dataset, and adjusting the model parameters according to the model loss during the training process, wherein the noise information extraction branch in the initial detection model has a first loss function, and the main sound signal enhancement branch has a second loss function; and a stopping module for stopping the training process and adjusting the model parameters when the model loss meets preset conditions, thereby obtaining the target detection model.

[0137] The aforementioned sound signal enhancement processing device may also include a processor and a memory. The aforementioned preprocessing unit 31, analysis unit 32, generation unit 33, output unit 34, etc., are all stored in the memory as program units, and the processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0138] The aforementioned processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured. By adjusting kernel parameters, the noisy time-domain audio signal to be enhanced is preprocessed to obtain three-channel spectrogram data in the corresponding frequency domain. An object detection model is then used to perform enhancement identification analysis on the three-channel spectrogram data to obtain noise type information and enhanced three-channel spectrogram data. Based on the noisy time-domain audio signal and the enhanced three-channel spectrogram data, an enhanced denoised time-domain audio signal is generated. Finally, based on the noise type information and the denoised time-domain audio signal, the audio signal enhancement processing result is output.

[0139] The aforementioned memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0140] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program with the following initialization steps: preprocessing the noisy time-domain sound signal to be enhanced to obtain three-channel spectrogram data in the corresponding frequency domain, wherein the three-channel spectrogram data includes a real spectrum, an imaginary spectrum, and an energy spectrum; performing enhancement recognition analysis on the three-channel spectrogram data through a target detection model to obtain noise type information and enhanced three-channel spectrogram data, wherein the target detection model includes a noise information extraction branch and a main sound signal enhancement branch, and the enhanced three-channel spectrogram data includes: enhanced real spectrum, enhanced imaginary spectrum, and enhanced energy spectrum; generating an enhanced denoised time-domain sound signal based on the noisy time-domain sound signal and the enhanced three-channel spectrogram data; and outputting the sound signal enhancement processing result based on the noise type information and the denoised time-domain sound signal.

[0141] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the sound signal enhancement processing method of any one of the above embodiments.

[0142] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the sound signal enhancement processing method of any one of the above embodiments.

[0143] Figure 4 This is a structural block diagram of an electronic device for performing a sound signal enhancement processing method according to an embodiment of the present invention, such as... Figure 4As shown, the electronic device may include: one or more ( Figure 4 (Only one is shown) Processor 402, memory 404, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0144] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the sound signal enhancement processing method and apparatus in this application embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned sound signal enhancement processing method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0145] Those skilled in the art will understand that Figure 4 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 4 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 4 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 4 The different configurations shown.

[0146] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0147] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0148] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0149] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0150] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0151] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0152] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0153] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method of enhancing processing of a sound signal, characterized by, The method comprises the following steps: preprocessing a noisy time-domain sound signal to be enhanced to obtain three-channel spectrogram data corresponding to a frequency domain, wherein the three-channel spectrogram data comprises a real spectrogram, an imaginary spectrogram and an energy spectrogram; performing enhancement recognition analysis on the three-channel spectrogram data by a target detection model to obtain noise type information and three-channel spectrogram enhancement data, wherein the target detection model comprises a noise information extraction branch and a sound main signal enhancement branch, and the three-channel spectrogram enhancement data comprises an enhanced real spectrogram, an enhanced imaginary spectrogram and an enhanced energy spectrogram; generating an enhanced noise-reduced time-domain sound signal based on the noisy time-domain sound signal and the three-channel spectrogram enhancement data; outputting a sound signal enhancement processing result based on the noise type information and the noise-reduced time-domain sound signal.

2. The method of claim 1, wherein the method is characterized by: The step of preprocessing a noisy time-domain sound signal to be enhanced to obtain three-channel spectrogram data corresponding to a frequency domain comprises the following steps: determining a frequency threshold based on a sampling frequency of the noisy time-domain sound signal; using the frequency threshold as a cutoff frequency of a high-pass filter and performing filtering processing on the noisy time-domain sound signal by using the high-pass filter to obtain a high-frequency enhanced sound signal; processing the high-frequency enhanced sound signal to a frequency domain space by using a short-time Fourier transform to obtain frequency domain features comprising the real spectrogram and the imaginary spectrogram; calculating energy values of the noisy time-domain sound signal at each frequency point based on the frequency domain features, and drawing the energy spectrogram according to the energy values at the each frequency point.

3. The method of claim 1, wherein the method further comprises: The noise information extraction branch comprises a dynamic convolution layer, a self-attention layer and a fully connected layer, wherein The step of performing enhancement recognition analysis on the three-channel spectrogram data by a target detection model to obtain noise type information comprises the following steps: processing the three-channel spectrogram data by the dynamic convolution layer to obtain time-varying frequency features; modeling the time-varying frequency features in a time dimension by the self-attention layer to obtain a time-frequency dependence model; processing the time-frequency dependence model by the fully connected layer to obtain a noise category probability distribution, and determining the noise type information based on the noise category probability distribution.

4. The method of claim 3, wherein the method further comprises: The sound main signal enhancement branch comprises an encoding layer, a core processing layer and a decoding layer, wherein the core processing layer comprises an upper branch and a lower branch, The step of performing enhancement recognition analysis on the three-channel spectrogram data by a target detection model to obtain three-channel spectrogram enhancement data comprises the following steps: extracting signal local features from the three-channel spectrogram data by the encoding layer; modeling the signal local features in a time correlation in the upper branch of the core processing layer to obtain a first feature map; modeling the signal local features in a frequency correlation in the lower branch of the core processing layer to obtain a second feature map; fusing and decoding the first feature map and the second feature map by the decoding layer to obtain the three-channel spectrogram enhancement data.

5. The method of claim 1, wherein The step of generating an enhanced de-noised time-domain sound signal based on the noisy time-domain sound signal and the three-channel spectrogram enhancement data comprises: performing data fusion processing on the noisy time-domain sound signal and the three-channel spectrogram enhancement data to obtain fusion feature data; reconstructing a time-domain signal by using an inverse short-time Fourier transform according to the fusion feature data to obtain the de-noised time-domain sound signal.

6. The method of claim 5, wherein the method further comprises: The step of performing data fusion processing on the noisy time-domain sound signal and the three-channel spectrogram enhancement data to obtain fusion feature data comprises: reconstructing a spectrogram according to an Euler formula based on the enhanced energy spectrogram and phase information of the noisy time-domain sound signal to obtain a reconstructed real spectrogram and a reconstructed imaginary spectrogram; performing superimposed average processing on the enhanced real spectrogram and the reconstructed real spectrogram to obtain a fusion real spectrogram; performing superimposed average processing on the enhanced imaginary spectrogram and the reconstructed imaginary spectrogram to obtain a fusion imaginary spectrogram; outputting the fusion feature data containing the fusion real spectrogram and the fusion imaginary spectrogram.

7. The method of claim 1, wherein the method further comprises: The modeling step of the target detection model comprises: obtaining a pre-constructed sample data set, wherein the sample data set at least contains noisy signal samples and corresponding pure signal samples, noise type labels; training an initial detection model using the sample data set, and adjusting model parameters during the training process according to model loss, wherein the noise information extraction branch in the initial detection model is provided with a first loss function, and the sound main signal enhancement branch is provided with a second loss function; stopping the training process and adjusting the model parameters when the model loss meets a preset condition to obtain the target detection model.

8. An apparatus for enhancing a sound signal, the apparatus comprising: comprises: a preprocessing unit configured to preprocess a noisy time-domain sound signal to be enhanced to obtain three-channel spectrogram data in a frequency domain, wherein the three-channel spectrogram data comprises a real spectrogram, an imaginary spectrogram and an energy spectrogram; an analysis unit configured to perform enhancement recognition analysis on the three-channel spectrogram data by using a target detection model to obtain noise type information and three-channel spectrogram enhancement data, wherein the target detection model comprises a noise information extraction branch and a sound main signal enhancement branch, and the three-channel spectrogram enhancement data comprises an enhanced real spectrogram, an enhanced imaginary spectrogram and an enhanced energy spectrogram; a generation unit configured to generate an enhanced de-noised time-domain sound signal based on the noisy time-domain sound signal and the three-channel spectrogram enhancement data; an output unit configured to output a sound signal enhancement processing result based on the noise type information and the de-noised time-domain sound signal.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium comprises a stored computer program, wherein the computer program controls a device in which the computer-readable storage medium is located to perform the sound signal enhancement processing method of any one of claims 1 to 7 when the computer program is running.

10. An electronic device, comprising: The device comprises one or more processors and a memory configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the sound signal enhancement processing method of any one of claims 1 to 7.

11. A computer program product, characterised in that, Computer instructions, wherein the computer instructions, when executed by a processor, implement the steps of the method for enhancing a sound signal according to any one of claims 1 to 7.

Images