Noise cancellation method and computer program product

Abstract

The application discloses a noise cancellation method and a computer program product, relates to the technical field of large models, and makes nonlinear distortion characteristics that may be originally covered more prominent through energy enhancement processing on nonlinear signal components in a mixed audio frame; further, the nonlinear distortion characteristics of the nonlinear signal components are captured and memorized; a noise suppression mask generated based on the results of distortion characteristic identification can more accurately distinguish nonlinear signals, noise and pure audio frames, and further noise cancellation based on the noise suppression mask can efficiently remove nonlinear echoes and background noise in the mixed audio frame, and obtain a high-fidelity pure audio stream. It can be seen that the application can realize noise suppression without relying on a reference signal, and in the case that there is nonlinear echo with nonlinear distortion characteristics in the mixed audio stream, the noise suppression effect of the mixed audio stream can also be effectively improved, and the audio processing performance of the device in a complex acoustic environment is improved.

Description

Technical Field

[0001] This disclosure generally relates to the field of large model technology, and more particularly to a noise cancellation method and computer program product. Background Technology

[0002] Currently, acoustic echo cancellation technology is a technique for eliminating echo interference in voice communication. Its core task is to identify and remove echo interference from the mixed audio signal captured by the microphone in real time, thereby improving call quality. Moreover, existing acoustic echo cancellation methods are usually based on linear assumptions, and the adaptive filtering signal processing methods and deep neural network learning methods included all heavily rely on the reference signal. Only by utilizing the linear or nonlinear relationship between the reference signal and the microphone signal can the echo interference be estimated and the echo eliminated by subtraction.

[0003] However, due to the difficulty in obtaining or the unreliability of reference signals in practical applications, the echo cancellation effect is reduced. Furthermore, the severe nonlinear transformation generated by the end-side miniature loudspeaker at high volumes causes the echo interference and the reference signal to no longer have a linear superposition relationship, making it impossible for acoustic echo cancellation technology based on the linear assumption to completely eliminate the echo signal. Summary of the Invention

[0004] In view of the above-mentioned defects or deficiencies in the prior art, it is desirable to provide a noise cancellation method and computer program product that can memorize and identify the nonlinear distortion characteristics of nonlinear signal components in mixed audio frames through a noise suppression model, achieve noise suppression without relying on a reference signal, and effectively improve the noise suppression effect of mixed audio streams when there are nonlinear echoes with nonlinear distortion characteristics in the mixed audio stream, thus significantly improving the audio processing performance of the device in complex acoustic environments.

[0005] Firstly, this application provides a noise cancellation method. The method includes: Acquire a mixed audio stream, divide the mixed audio stream into frames, and obtain multiple mixed audio frames; The mixed audio frame is input into a noise suppression model, and energy enhancement processing is performed on the nonlinear signal components in the mixed audio frame. Distortion characteristics are identified based on the energy-enhanced mixed audio frame. The distortion characteristics identification is used to identify and memorize the nonlinear distortion characteristics of the nonlinear signal components in the mixed audio frame. Based on the result of the distortion characteristic identification, a noise suppression mask is generated. The noise suppression mask is used to distinguish the nonlinear signal component, the noise signal component, and the clean audio signal component. The noise suppression mask is used to perform noise cancellation on the mixed audio frames to eliminate nonlinear signal components and noise signal components in the mixed audio frames, thereby obtaining clean audio frames corresponding to each of the mixed audio frames output by the noise suppression model, and obtaining a clean audio stream based on each of the clean audio frames.

[0006] In conjunction with the first aspect, in one embodiment, the energy enhancement processing of the nonlinear signal components in the mixed audio frame includes: The mixed audio frame is converted to a time-frequency signal to obtain a complex spectrum of the mixed audio frame. The complex spectrum is used to characterize the amplitude and phase of the sound signal at each time-frequency position. Energy enhancement is performed on the low-amplitude spectrum in the complex spectrum to enhance the nonlinear signal components in the mixed audio frame.

[0007] In conjunction with the first aspect, in one embodiment, the distortion characteristic identification based on the energy-enhanced hybrid audio frame includes: Feature extraction is performed on the energy-enhanced complex spectrum to obtain spectrum-based acoustic features; The spectrum-based acoustic features and the complex spectrum are fused, and the fused result is encoded to obtain a high-dimensional acoustic feature map. The high-dimensional acoustic feature map includes the nonlinear acoustic features of the nonlinear signal component, the statistical texture features of the noise signal component, and the high-dimensional latent space features of the clean audio signal component. Identify and memorize the nonlinear distortion characteristics of the nonlinear acoustic features in the high-dimensional acoustic feature map as they change over time.

[0008] In conjunction with the first aspect, in one embodiment, generating a noise suppression mask based on the result of the distortion characteristic identification includes: Based on the results of the distortion characteristic identification, the signal components with nonlinear distortion characteristics in the mixed audio frame are marked; The signal components with statistical texture features in the mixed audio frame are labeled, and the noise suppression mask is generated based on the labeling results.

[0009] In conjunction with the first aspect, in one embodiment, the training process of the noise suppression model includes: A training sample set is obtained, wherein each training sample in the training sample set includes a mixed audio signal and a target speech signal, the mixed audio signal being determined based on the target speech signal, a background noise signal, and a nonlinear echo signal; the nonlinear echo signal is determined based on the device acoustic fingerprint features; the device acoustic fingerprint features include the physical nonlinear features of the audio signal transmission link from the device's speaker to the microphone; Using the target speech signal as a label, the pre-constructed initial neural network model is trained based on the training sample set to obtain the noise suppression model corresponding to the model training result satisfying the preset convergence condition.

[0010] In conjunction with the first aspect, in one embodiment, the mixed audio signal is determined based on the target speech signal, the background noise signal, and the nonlinear echo signal, including: The nonlinear echo signal is obtained by convolution processing the acoustic fingerprint features of the device and the target speech signal. The target speech signal, the nonlinear echo signal, and the background noise signal are fused to obtain the mixed audio signal.

[0011] In conjunction with the first aspect, in one embodiment, the mixed audio signal is determined based on the target speech signal, the background noise signal, and the nonlinear echo signal, including: The nonlinear echo signal is subjected to random signal enhancement, and the enhanced nonlinear echo signal, the target speech signal, and the background noise signal are fused to obtain the mixed audio signal; wherein, the random signal enhancement includes random resampling and / or amplitude gain perturbation.

[0012] In conjunction with the first aspect, in one embodiment, obtaining the training sample set includes: Acquire the acoustic fingerprint features of the device when playing a preset audio signal under different audio playback parameters; the audio playback parameters include at least one of the speaker playback volume and the degree of occlusion of the speaker; Multiple nonlinear echo signals are determined based on multiple device acoustic fingerprint features and the target speech signal; The training sample set is obtained based on the multiple nonlinear echo signals, the target speech signal, and the background noise signal.

[0013] In conjunction with the first aspect, in one embodiment, obtaining the device acoustic fingerprint features when the device plays a preset audio signal under different audio playback parameters includes: The preset audio signal and the audio signal recorded synchronously during the playback of the preset audio signal under the corresponding audio playback parameters are aligned, and the total harmonic distortion characteristics and intermodulation distortion characteristics of the speaker are determined based on the two aligned audio signals. The nonlinear harmonic energy envelope of the loudspeaker is constructed based on the total harmonic distortion characteristics and the intermodulation distortion characteristics, and the nonlinear harmonic energy envelope is normalized to obtain the acoustic fingerprint characteristics of the device.

[0014] Secondly, this application also provides a computer program product. The computer program product includes instructions that, when executed, cause the noise cancellation method described in the first aspect to be implemented.

[0015] This application provides a noise cancellation method and computer program product. The noise cancellation method enhances the nonlinear signal components in the mixed audio frames by performing energy enhancement processing, making the nonlinear distortion features that might otherwise be masked more prominent, thus laying the foundation for subsequent identification. By identifying distortion characteristics, the nonlinear distortion characteristics of the nonlinear signal components are directly captured and memorized, which not only eliminates the dependence on external reference signals but also overcomes the limitations of unreliable reference signals. The noise suppression mask generated based on the results of distortion characteristic identification can more accurately distinguish between nonlinear signals, background noise, and clean audio frames. Furthermore, through noise cancellation based on the noise suppression mask, nonlinear echoes and background noise in each mixed audio frame can be efficiently removed, resulting in a high-fidelity clean audio stream. Therefore, without relying on reference signals, various noises in the mixed audio stream can be efficiently and comprehensively eliminated simply by calling a pre-trained noise suppression model, improving the noise suppression efficiency and robustness of the mixed audio stream. In addition, the severe physical clipping and harmonic distortion generated by the end-side miniature loudspeakers at high volumes make the echo signal no longer a linear superposition of the reference signal, but a nonlinear echo signal with complex nonlinear distortion characteristics. For mixed audio streams with nonlinear echo signals, the noise suppression effect of the mixed audio stream can be effectively improved by nonlinear signal component energy enhancement, distortion characteristic identification and noise suppression masking, which significantly improves the audio processing performance of the device in complex acoustic environments. Attached Figure Description

[0016] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This diagram illustrates application scenarios for noise cancellation methods. Figure 2 This is one of the flowcharts illustrating a noise cancellation method in one embodiment; Figure 3 This is a second flowchart illustrating a noise cancellation method in one embodiment; Figure 4 This is the third flowchart of a noise cancellation method in one embodiment; Figure 5 This is a fourth flowchart illustrating a noise cancellation method in one embodiment; Figure 6 This is the fifth flowchart of a noise cancellation method in one embodiment; Figure 7This is a schematic diagram of the model structure of the noise suppression model in one embodiment; Figure 8 This is a flowchart of a noise cancellation method in one embodiment, number six. Figure 9 This is the seventh flowchart of a noise cancellation method in one embodiment; Figure 10 This is a schematic diagram illustrating a scenario for obtaining the acoustic fingerprint features of a device in one embodiment; Figure 11 This is the eighth flowchart of a noise cancellation method in one embodiment; Figure 12 This is a flowchart of a noise cancellation method in one embodiment, number nine. Figure 13 The waveforms and spectrograms before and after model processing are shown. Figure 14 This is a schematic diagram of the device in one embodiment. Detailed Implementation

[0017] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0018] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The application will now be described in detail with reference to the accompanying drawings and embodiments. Furthermore, the term "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The terms "first" and "second," etc., in the specification and claims of the embodiments of this application are used to distinguish different objects, not to describe a specific order of objects.

[0019] First, the terms used in this application will be explained: Nonlinear distortion: During signal transmission or electroacoustic conversion, the output waveform of a signal is distorted relative to the input waveform due to the deviation of the device or acoustic system from the ideal linear operating characteristics, and harmonics and intermodulation components that are not present in the original signal are introduced.

[0020] With the popularization of smart terminal technology, applications such as Voice over Internet Protocol (VoIP), live streaming, and game voice communication have become core functions of mobile devices and Internet of Things (IoT) devices. In full-duplex voice interaction systems, acoustic echo cancellation (AEC) is a key technology to ensure call quality. Its main task is to eliminate far-end voice signals played through speakers and coupled to the microphone through environmental reflections.

[0021] Current mainstream AEC (Augmentation Echo) technologies mainly include traditional signal processing methods based on adaptive filtering and learning methods based on deep neural networks. Both types of technologies typically rely heavily on a reference signal as the system input, utilizing the linear or nonlinear correlation between the reference signal and the microphone signal to estimate and cancel the echo. However, practical end-side applications suffer from the following three drawbacks: First, the reference signal is unreliable or unavailable: Due to limitations of operating system permissions (such as third-party application development) or hardware architecture, the application layer may be unable to obtain the underlying reference signal; even if it is obtained, severe nonlinear delays are often caused by clock drift or transmission jitter, resulting in a decrease in algorithm performance.

[0022] Secondly, nonlinear distortion is difficult to handle: At high volumes, end-side miniature loudspeakers exhibit severe physical clipping and harmonic distortion. Specifically, due to their small size, thin diaphragm, and limited magnetic circuit and suspension travel, under high-volume driving, the input electrical signal causes the diaphragm's vibration amplitude to approach or even exceed its maximum physical displacement range. When the diaphragm reaches its mechanical limit, it can no longer follow the signal's movement, and the top and bottom of the waveform are forcibly "truncated," resulting in significant physical clipping. Furthermore, under high-volume driving, the vibration of the diaphragm, suspension system, and cavity in the miniature loudspeaker no longer exhibits an ideal linear relationship. The originally clean single-frequency signal is excited with a large number of high-order frequency components, manifesting as harsh, rough additional sound—this is harmonic distortion. Therefore, this complex nonlinear transformation of physical clipping and harmonic distortion means that the echo signal is no longer a linear superposition of the reference signal, causing the AEC algorithm based on linear assumptions to fail to eliminate residual echoes.

[0023] Third, the general blind source separation may damage human voice: Although the single-channel blind source separation algorithm does not rely on the reference signal, it is difficult to distinguish between echoes with highly overlapping spectra and near-end human voices when there is a lack of specific prior information about the device, which can easily cause speech damage. The blind source component here is a signal processing technique that separates the original independent signals based solely on the observed mixed signals under the condition that the characteristics of the source signal and the mixing process are unknown.

[0024] To address the aforementioned technical problems, this application provides a noise cancellation method and a computer program product. The application scenarios of the noise cancellation method include, but are not limited to, situations where existing AEC algorithms cannot eliminate nonlinear distortion echoes generated during full-duplex voice interaction; for example, referring to… Figure 1 The illustrated application scenario shows that when a smartwatch is used for hands-free calling at maximum volume, the speaker produces severe distortion (non-linear distortion). Existing AEC algorithms cannot eliminate this distorted echo, and users developing third-party hardware lack system permissions to access the underlying reference signal, resulting in the other party hearing a severe non-linear echo signal. This is understandable. Figure 1 The application scenario shown is a most demanding one, and the method described in this application is applicable to... Figure 1 If the application scenario shown is applicable, then the method of this application can also effectively solve similar technical problems in other ordinary and relaxed scenarios.

[0025] The noise cancellation method provided in this application can be implemented by a processing unit, which can be built into a device. The device can include, but is not limited to, smartwatches, augmented reality (AR) / virtual reality (VR) glasses, low-end / outdated Android phones, IoT smart speakers, and walkie-talkies. Alternatively, when various third-party VoIP calling applications developed based on Web Real-Time Communication (WebRTC) are installed on the device, the third-party VoIP calling application can also implement the noise cancellation method by calling the processing unit on the device.

[0026] Below, in conjunction with Figures 2 to 13 This application describes a noise cancellation method. To facilitate understanding of the noise cancellation method provided, the following exemplary embodiments will be used to illustrate the method in detail. It is understood that these exemplary embodiments can be combined with each other, and similar concepts or processes may not be repeated in some embodiments.

[0027] Reference Figure 2 This is a flowchart illustrating the noise cancellation method provided in an embodiment of this application, as shown below. Figure 2 As shown, the cooking video generation method includes the following steps 101 to 104.

[0028] Step 101: Obtain the mixed audio stream, divide the mixed audio stream into frames, and obtain multiple mixed audio frames.

[0029] Among them, the mixed audio stream is a single-channel mixed audio stream, that is, a mixed audio stream acquired in real time in a single speech signal path.

[0030] The mixed audio stream can be a mixture of a clean audio stream, background noise, and echo signals generated by the audio played by the device's own speakers. Since the echo signal is a linear echo when the speaker volume is normal and there is no distortion, and a non-linear echo when the speaker volume is too high, there is distortion, or the diaphragm is distorted, the echo signal here can be either a linear echo signal or a non-linear echo signal.

[0031] A linear echo signal can be an audio signal played when the speaker volume is normal and there is no distortion, which is then transmitted through the air and reflected in the room, and picked up by a microphone.

[0032] Nonlinear echo signals can be the sound signals picked up by microphones after the audio signal played by the speaker is transmitted through the air and reflected in the room when the speaker volume is too high, the sound is distorted, or the diaphragm is distorted.

[0033] Specifically, the processing unit in the device acquires the mixed audio stream either through real-time acquisition via the device's microphone or by selecting from pre-stored audio files. In practical applications, the mixed audio stream can be acquired in real-time through the device's built-in microphone, for example, the processing unit directly acquires the mixed audio stream captured by the microphone during actual voice interaction; or it can be received through an external audio interface, for example, by directly receiving mixed audio streams transmitted in real-time from other devices.

[0034] The purpose of framing a mixed audio stream is to divide a continuous mixed audio stream into shorter, easier-to-process segments, each of which is a mixed audio frame. For example, a continuous mixed audio stream can be segmented into segments of fixed duration (e.g., 20 milliseconds), with each segment being a mixed audio frame. This framing method facilitates independent signal processing for each mixed audio frame, thereby improving the real-time performance and parallelism of the processing.

[0035] It should be noted that when the device's built-in microphone collects voice signals, the timestamp function can be used to automatically mark the timestamp; alternatively, if the device or application supports active insertion of marking during the sound pickup process, the user can manually mark the timestamp to facilitate rapid frame segmentation later.

[0036] At this point, the mixed audio stream carrying timestamps is framed. First, timing alignment and buffer management are completed based on the timestamps. Then, the continuous mixed audio stream is divided into multiple mixed audio frames according to a fixed time window (such as 25ms) to balance timing accuracy and feature extraction effectiveness.

[0037] For example, since the mixed audio stream originates from a microphone and carries a timestamp, the timestamp can be used for precise time alignment. Then, the time-aligned mixed audio stream can be further processed using standard framing. The framing parameters involved are typically tied to the sampling rate and application scenario: Frame Length: 25ms → corresponding to 400 sampling points (16000 × 0.025); Frame Shift: 10ms → corresponding to 160 sampling points (16000 × 0.01); Overlap Rate: 60% (because frame length > frame shift); Framing Formula: The starting position of the i-th mixed audio frame is i × hop_size, where hop_size is the sampling point step size between two adjacent frames; The mixed audio stream is framed into multiple mixed audio frames in this way.

[0038] Step 102: Input the mixed audio frame into the noise suppression model, perform energy enhancement processing on the nonlinear signal components in the mixed audio frame, and perform distortion characteristic identification based on the mixed audio frame after energy enhancement processing; distortion characteristic identification is used to identify and memorize the nonlinear distortion characteristics of the nonlinear signal components in the mixed audio frame.

[0039] The noise suppression model is a trained network model used to identify and eliminate different types of noise signals in mixed audio frames, and the network model is a neural network model or a large model; the different types of noise here include background noise and echo signals.

[0040] For example, if a network model is only used to eliminate different types of noise in mixed audio frames and output clean audio frames, that is, it only has noise suppression function, then the network model is a neural network model. If the network model simultaneously meets the following conditions: the parameter scale exceeds 1 billion, it adopts a deep encoder / decoder architecture; the training samples reach tens of millions or more, covering multiple noise types, scenes and speech features; it can not only complete the core noise suppression task, but also adapt to at least two speech domain related tasks (such as speech separation, cross-scene noise suppression); and its inference performance meets the standards, supporting batch processing or real-time noise reduction (such as being adaptable to terminal or server deployment after quantization optimization), then the network model is a large model.

[0041] Nonlinear signal components can include signal distortion caused by the physical characteristics or operating state of the speakers in the device (e.g., playing at high volume) during audio signal transmission or processing. This signal distortion makes the output signal no longer a simple linear relationship with the input signal, but exhibits complex nonlinear characteristics.

[0042] Energy enhancement processing can be an operation that boosts the energy of specific signal components (such as nonlinear signal components) in a mixed audio frame. By amplifying the energy of weaker nonlinear harmonic distortion components in the mixed audio frame, the originally weaker nonlinear signal components become more significant in the feature space, thus facilitating subsequent identification and elimination.

[0043] Distortion characteristic identification involves analyzing energy-enhanced mixed audio frames to identify and memorize the nonlinear distortion patterns or characteristics unique to the nonlinear signal components within them. This identification process aims to capture the nonlinear distortion characteristics of nonlinear echoes in order to distinguish them from clean audio frames and background noise.

[0044] Understandably, when the end-side micro loudspeaker is driven at high volume, it enters the nonlinear operating range, producing physical clipping and harmonic distortion. This complex nonlinear transformation causes the echo signal to no longer satisfy the linear relationship, but to become a nonlinear echo signal with nonlinear distortion characteristics. Since this nonlinear echo signal is essentially a nonlinear distortion signal, it is necessary to identify and memorize the nonlinear distortion characteristics.

[0045] Specifically, the mixed audio frames are input into a noise suppression model, which analyzes and processes the input mixed audio frames. Internally, the noise suppression model first performs energy enhancement on the nonlinear signal components in the mixed audio frames. This energy enhancement can be achieved by applying a uniform gain factor to the lower-amplitude signal components in the mixed audio frames to boost their energy; alternatively, an energy threshold can be set to uniformly boost signal components below this threshold to a uniform energy level. Through energy enhancement, nonlinear signal components in the mixed audio frames that might otherwise be masked by other signals become more prominent, facilitating subsequent identification.

[0046] Subsequently, distortion characteristics are identified based on the energy-enhanced hybrid audio frames. The purpose of distortion characteristic identification is to identify and memorize the nonlinear distortion characteristics of the nonlinear signal components in the hybrid audio frames. For example, time-domain statistical analysis can be performed on the energy-enhanced hybrid audio frames to calculate statistics such as kurtosis and skewness to capture nonlinear distortion patterns. Alternatively, some typical nonlinear distortion templates can be preset, and the energy-enhanced hybrid audio frames can be matched with these templates to identify the nonlinear distortion characteristics of the nonlinear signal components in the hybrid audio frames; through this identification, unique fingerprint information of the nonlinear signal components can be established.

[0047] Step 103: Generate a noise suppression mask based on the result of distortion characteristic identification. The noise suppression mask is used to distinguish nonlinear signal components, noise signal components, and clean audio signal components.

[0048] The noise suppression mask can be a weight or label matrix used to indicate the distribution of different signal components (nonlinear signal components, noise signal components, and clean audio signal components) in the mixed audio frame; the noise suppression mask can accurately separate the clean audio frames that need to be retained and suppress unwanted nonlinear signals and noise signals.

[0049] Specifically, a noise suppression mask is generated based on the results of the aforementioned distortion characteristic identification. This mask aims to accurately distinguish between nonlinear signal components, noise signal components, and clean audio signal components in a mixed audio frame. For example, a simple energy threshold can be set based on the identified linear and nonlinear distortion characteristics. Signals above the energy threshold are marked as clean audio frames, while signals below the threshold are marked as noise or nonlinear signals. Alternatively, a mapping table reflecting the relationship between non-distortion and distortion characteristics and mask values ​​can be pre-constructed, directly mapping different identified distortion and non-distortion characteristics to predefined mask values ​​to obtain the noise suppression mask. This noise suppression mask is a crucial intermediate product, providing precise guidance for subsequent noise cancellation and ensuring targeted processing of different signal components.

[0050] Step 104: Perform noise cancellation on the mixed audio frames based on the noise suppression mask to eliminate nonlinear signal components and noise signal components in the mixed audio frames, obtain the clean audio frames corresponding to each mixed audio frame output by the noise suppression model, and obtain the clean audio stream based on each of the clean audio frames.

[0051] Specifically, noise cancellation is performed on the mixed audio frames based on a noise suppression mask. The purpose of noise cancellation is to eliminate nonlinear and noise signal components in the mixed audio frames, thereby obtaining clean audio frames corresponding to each mixed audio frame. For example, nonlinear and noise signal components in the mixed audio frames can be eliminated by multiplying the noise suppression mask point-by-point with the time-frequency graph of the mixed audio frames; alternatively, the noise suppression mask can be used as a gating signal to suppress or attenuate signal components marked as nonlinear and noise signals in the mixed audio frames. In this way, unwanted interference signals can be effectively removed, retaining only the clean audio frames in the mixed audio frames. Finally, a clean audio stream is obtained based on each clean audio frame, that is, all the processed clean audio frames are reassembled according to the timestamp order of their respective mixed audio frames to form a continuous, high-quality clean audio stream.

[0052] The noise cancellation method provided in this application enhances the nonlinear signal components in the mixed audio frames by performing energy enhancement processing, making the nonlinear distortion features that might otherwise be masked more prominent, thus laying the foundation for subsequent identification. By identifying distortion characteristics, the nonlinear distortion characteristics of the nonlinear signal components are directly captured and memorized, which not only eliminates the dependence on external reference signals but also overcomes the limitations of unreliable reference signals. The noise suppression mask generated based on the results of distortion characteristic identification can more accurately distinguish between nonlinear signals, background noise, and clean audio frames. Furthermore, through noise cancellation based on the noise suppression mask, nonlinear echoes and background noise in each mixed audio frame can be efficiently removed, resulting in a high-fidelity clean audio stream. Thus, without relying on reference signals, various noises in the mixed audio stream can be efficiently and comprehensively eliminated simply by calling a pre-trained noise suppression model, improving the noise suppression efficiency and robustness of the mixed audio stream. In addition, the severe physical clipping and harmonic distortion generated by the end-side miniature loudspeakers at high volumes make the echo signal no longer a linear superposition of the reference signal, but a nonlinear echo signal with complex nonlinear distortion characteristics. For mixed audio streams with nonlinear echo signals, the noise suppression effect of the mixed audio stream can be effectively improved by nonlinear signal component energy enhancement, distortion characteristic identification and noise suppression masking, which significantly improves the audio processing performance of the device in complex acoustic environments.

[0053] Based on the above Figure 2 In one example embodiment of the method shown, step 102 involves energy enhancement processing of the nonlinear signal components in the mixed audio frame. The specific process in this embodiment can be achieved through… Figure 3 Steps 201 and 202 shown are implemented.

[0054] Step 201: Perform time-frequency conversion on the mixed audio frame to obtain the complex spectrum of the mixed audio frame. The complex spectrum is used to characterize the amplitude and phase of the sound signal at each time-frequency position.

[0055] Step 202: Perform energy enhancement on the low-amplitude spectrum in the complex spectrum to enhance the nonlinear signal components in the mixed audio frame.

[0056] The complex spectrum is used to characterize the amplitude and phase of the sound signal at various time-frequency positions. This means that each point of the complex spectrum corresponds to a specific time-frequency unit. The magnitude (amplitude) of the complex value represents the energy intensity of the time-frequency unit, and its argument (phase) represents the initial phase of the time-frequency unit. This representation completely preserves all the information of the original mixed audio frame, enabling subsequent signal processing to be based on accurate frequency and phase information, thereby avoiding the introduction of unnecessary distortion.

[0057] Specifically, when the noise suppression model includes a Short-Time Fourier Transform (STFT) module and an energy enhancement module, after the mixed audio frame enters the noise suppression model, it first undergoes time-frequency conversion via the STFT module. That is, by dividing the mixed audio frame into multiple short-time windows and performing a Fourier transform on each window signal, the local spectral characteristics of the mixed audio frame in time and frequency are obtained, resulting in the complex spectrum of the mixed audio frame output by the STFT module. Then, this complex spectrum enters the energy enhancement module for low-amplitude spectrum energy enhancement.

[0058] Considering that the main function of STFT is to map the time-domain signal to the complex representation in the time-frequency domain and obtain the complex spectrum, the STFT module in the noise suppression model can be replaced by the Quadrature Mirror Filterbank (QMF) module, the Complex Gabor Transform (CQT) module, or the Complex Wavelet Transform module to obtain the complex spectrum of the mixed audio frame.

[0059] The purpose of performing time-frequency conversion on the mixed audio frames is to transform the time-domain mixed audio frames into a frequency-domain representation, thereby revealing the energy distribution of the signal at different times and frequencies. The complex spectrum contains not only the amplitude information of each frequency component but also its phase information, which is crucial for accurate signal reconstruction or fine analysis. This time-frequency conversion is implemented using complex wavelet transform. Wavelet transform can provide multi-resolution time-frequency analysis and has advantages for the analysis of non-stationary signals, allowing for more flexible capture of the transient characteristics of the signal.

[0060] Based on this, energy enhancement is performed on the low-amplitude spectrum of the complex spectrum. The purpose is to construct a feature analysis path and extract input representations that are sensitive to nonlinear features, that is, to compress the dynamic range of the input complex spectrum, for example, by using power-law compression. , Alternatively, logarithmic compression can be used. In this embodiment, power-law compression is employed to enhance the energy of low-amplitude spectra in the complex spectrum. The aim is to amplify the weaker nonlinear harmonic distortion components in the mixed audio frame, making their re-characteristic space more significant. For the energy enhancement of low-amplitude spectra, the purpose is to highlight signal components that were originally masked by high-amplitude signals or had low energy due to nonlinear distortion, making them easier for subsequent distortion characteristic identification modules to detect and analyze. Ultimately, through the above processing, energy enhancement of nonlinear signal components in the mixed audio frame is achieved, improving the detectability of nonlinear signal components in the subsequent identification process, thus laying the foundation for accurate identification of their nonlinear distortion characteristics.

[0061] It should be noted that the noise suppression model may also include a signal reconstruction path, which is used to maintain the physical accuracy of waveform synthesis. Its purpose is to treat the original linear amplitude spectrum (i.e., the complex spectrum of the mixed audio frame) as the object to be processed, so as to ensure that the subsequent masking operation conforms to the signal superposition principle.

[0062] The noise cancellation method provided in this application converts the time-domain signal into a complex spectrum by performing time-frequency conversion on the mixed audio frames, thereby obtaining the amplitude and phase information of the mixed audio frames at various time-frequency positions, which provides a foundation for refined processing. Then, the low-amplitude spectrum in the complex spectrum is further enhanced. Since nonlinear signal components are usually low-amplitude and difficult to detect in mixed audio, by selectively enhancing the energy of these low-amplitude spectra, the nonlinear signal components that are originally low-amplitude and poorly distributed in the time domain can be effectively highlighted, making them more significant in the frequency domain. This greatly improves the detectability and identifiability of the nonlinear signal components, ensuring the accuracy of subsequent distortion characteristic identification. Furthermore, by more accurately identifying nonlinear distortion characteristics, a more precise noise suppression mask can be generated, thereby improving the overall noise cancellation effect and ultimately obtaining a higher-quality, clean audio stream. This avoids noise interference or signal distortion that may be introduced during direct time-domain processing, improving the reliability of the overall noise cancellation.

[0063] Based on the above Figure 2 In one example embodiment of the method shown, step 102 involves identifying distortion characteristics based on the hybrid audio frame after energy enhancement processing. The specific process in this embodiment can be achieved through… Figure 4 Steps 301 to 303 shown are implemented.

[0064] Step 301: Extract features from the energy-enhanced complex spectrum to obtain spectrum-based acoustic features.

[0065] Step 302: Fuse the spectrum-based acoustic features and the digital spectrum, encode the fused result to obtain a high-dimensional acoustic feature map. The high-dimensional acoustic feature map includes the nonlinear acoustic features of the nonlinear signal components, the statistical texture features of the noise signal components, and the high-dimensional latent space features of the clean audio signal components.

[0066] Step 303: Identify and memorize the nonlinear distortion characteristics of nonlinear acoustic features over time in the high-dimensional acoustic feature map.

[0067] Specifically, in a noise suppression model that includes an STFT module, an energy enhancement module, a feature extraction module, an encoder module, and a distortion characteristic recognition module, the mixed audio frames are sequentially processed by the time-frequency conversion of the STFT module and the low-amplitude spectral energy enhancement of the energy enhancement module to obtain an energy-enhanced complex spectrum, which is then fed into the feature extraction module for feature extraction, resulting in spectrum-based acoustic features. Feature extraction reduces the dimensionality of the data while retaining key acoustic information, avoiding interference from redundant data in the original complex spectrum on the recognition process. Here, spectrum-based acoustic features include, but are not limited to, Mel-Frequency Cepstral Coefficients (MFCCs), Linear Predictive Coding (LPC) coefficients, or Perceptual Linear Prediction (PLP) coefficients.

[0068] Subsequently, to ensure the comprehensiveness and consistency of feature representation and avoid the loss of details during feature extraction, the identified spectrum-based acoustic features are first fused with the complex spectrum of the mixed audio frame. That is, the complex spectrum and the real and imaginary parts of the spectrum-based acoustic features are concatenated. The fused result is then further processed by the encoder module. The encoding operation maps the fused information to a high-dimensional space to form a high-dimensional acoustic feature map, thereby providing richer contextual information for different signal components.

[0069] The high-dimensional acoustic feature map output by the encoder significantly improves the accuracy and robustness of subsequent recognition by encoding the unique features of nonlinear signal components, noise signal components, and clean audio signal components into different dimensions or regions. Specifically, the nonlinear acoustic features of the nonlinear signal components can be represented by specific patterns of nonlinear effects such as harmonic distortion and intermodulation distortion in the spectrum or time domain; the statistical texture features of the noise signal components can include statistical quantities such as energy distribution, spectral flatness, and spectral entropy, which can effectively describe the randomness and persistence of noise; and the high-dimensional latent space features of the clean audio signal components can be abstract representations automatically extracted by the deep learning model during the learning process, effectively distinguishing clean speech or music. These features typically possess high semantic information and discriminative power.

[0070] For example, the encoder module can be a module including N layers of encoders. Each encoder layer includes a convolutional (conv2d) layer, a batch normalization (BNU) layer, and an activation (ELU) layer. The convolutional layer is used to extract the local structure and high-level acoustic features of the complex spectrum; the batch normalization layer performs normalization processing on the features to improve training stability and model convergence speed; the activation layer introduces nonlinear transformation to enhance the modeling and fitting ability for nonlinear echoes and complex noise interference. In this embodiment, in order to balance computing power and performance, a 2-layer convolutional encoding is used.

[0071] Subsequently, the high-dimensional acoustic feature map output by the encoder enters the distortion characteristic recognition module. This module identifies and memorizes the temporal variation patterns of the nonlinear acoustic features in the high-dimensional acoustic feature map, enabling the model to better adapt to the dynamic nature of nonlinear distortion in practical applications, thereby improving recognition accuracy and long-term stability. The distortion characteristic recognition module can employ sequence models, such as Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), or Transformer models. These models excel at processing time-series data and can capture long-term dependencies between features. Considering that nonlinear echo signals are often accompanied by specific time-decaying Pauli and periodic harmonic structures, this embodiment preferentially selects an N-layer bidirectional Long Short-Term Memory (BI-LSTM) layer for the distortion characteristic recognition module to memorize and recognize these time-varying nonlinear distortion characteristics.

[0072] The noise cancellation method provided in this application extracts features from the energy-enhanced complex spectrum, ensuring that even low-amplitude nonlinear signal components can be effectively captured. Then, the spectrum-based acoustic features are fused with the complex spectrum and encoded into a high-dimensional acoustic feature map. This high-dimensional acoustic feature map can finely distinguish between nonlinear signal components, noise signal components, and clean audio signal components, laying the foundation for accurate identification. By recognizing and memorizing the time-varying characteristics of the nonlinear acoustic features in the high-dimensional acoustic feature map, it can dynamically adapt to changes in nonlinear distortion, significantly improving the accuracy and robustness of distortion characteristic identification. This allows the subsequently generated noise suppression mask to more accurately distinguish and suppress nonlinear signal components and noise signal components, thereby obtaining a higher quality clean audio stream in complex acoustic environments, especially in the presence of dynamic nonlinear distortion, significantly improving the audio processing performance of the device.

[0073] Based on the above Figure 2 In one example embodiment of the method shown, step 103 generates a noise suppression mask based on the result of distortion characteristic identification. The specific process of this step in this embodiment can be achieved through… Figure 5 Steps 401 and 402 shown are implemented.

[0074] Step 401: Based on the results of distortion characteristic identification, mark the signal components with nonlinear distortion characteristics in the mixed audio frame.

[0075] Step 402: Mark the signal components with statistical texture features in the mixed audio frame, and generate a noise suppression mask based on the marking results.

[0076] Among them, statistical texture features are often used to describe the distribution characteristics of noise signals in the time and frequency domain, such as the flatness of its spectrum, the randomness of its energy, and its duration.

[0077] Specifically, in a noise suppression model that includes an STFT module, an energy enhancement module, a feature extraction module, an encoder module, a distortion characteristic identification module, a decoder module, and an ISTFT module, the high-dimensional acoustic feature map output by the encoder and the nonlinear distortion characteristics output by the distortion characteristic identification module are both input to the decoder module. The decoder, by decoding the high-dimensional acoustic feature map and the nonlinear distortion characteristics, can not only label signal components with nonlinear distortion characteristics in the mixed audio frame, but also signal components with statistical texture characteristics. Here, labeling signal components with nonlinear distortion characteristics can be represented by assigning a nonlinear distortion probability value to a specific region in the time-frequency domain, or by directly generating a binary mask indicating which time-frequency units contain significant nonlinear distortion. Labeling signal components with statistical texture characteristics in the mixed audio frame can be represented by identifying and marking those signal components in the mixed audio frame that exhibit statistical regularities or textures specific to noise.

[0078] The decoder module fuses the two types of marking results to construct a noise suppression mask that can comprehensively distinguish nonlinear signal components, noise signal components, and clean audio signal components. This noise suppression mask can be a binary (0 or 1) matrix with the same size as the mixed audio frame. For example, a value of 0 eliminates the corresponding nonlinear echo component or noise signal component, while a value of 1 retains the corresponding clean audio signal component.

[0079] For example, when the encoder module includes N encoder layers, and each encoder layer includes a convolutional layer, a batch normalization layer, and an activation layer, the decoder module also includes N transposed decoder layers, and each transposed decoder layer includes a deconvolutional (Transposed conv2d) layer, a batch normalization (Batch normalization) layer, and an activation (ELU) layer. The deconvolutional layer is used to upsample and restore the dimension of the high-dimensional acoustic feature map and nonlinear distortion characteristics, and gradually maps the abstract features to the original target space; the batch normalization layer performs normalization processing on the features; the activation layer introduces nonlinear transformation to enhance the decoder's ability to model nonlinear interference and constrain output, and finally outputs a noise suppression mask with a linear domain scope.

[0080] It should be noted that the noise suppression model can also include a signal reconstruction path. This path is used to maintain the physical accuracy of waveform synthesis. Its purpose is to treat the original linear amplitude spectrum (i.e., the complex spectrum of the mixed audio frame) as the object of processing, ensuring that subsequent masking operations conform to the principle of signal superposition. Specifically, the signal reconstruction path outputs the complex spectrum of the mixed audio frame to a multiplier, and the input of this multiplier is the complex spectrum of the mixed audio frame and the noise suppression mask output by the encoder. Thus, when the size of the noise suppression mask and the size of the complex spectrum are the same, by multiplying the noise suppression mask element-wise with the complex spectrum of the mixed audio frame, instead of multiplying with the uncompressed complex spectrum (i.e., the spectrum of the complex spectrum of the mixed audio frame after power-law compression), the nonlinear signal components and noise signal components in the mixed audio frame can be eliminated. This decoupling mechanism of "compressed domain identification and linear domain suppression" utilizes the high sensitivity of the compressed domain while reducing the computational power required to restore to the linear domain.

[0081] The noise cancellation method provided in this application refines the noise suppression mask generation process, specifically including marking nonlinear distortion characteristics and statistical texture features. This allows for more accurate identification and differentiation of different signal components in mixed audio frames, significantly improving the accuracy and comprehensiveness of the mask. This provides a more reliable basis for subsequent noise cancellation, enabling more precise identification and differentiation of nonlinear signal components, noise signal components, and clean audio signal components in mixed audio frames. This refined marking process solves the problem of insufficient mask generation in traditional methods, avoiding damage to clean audio signal components or incomplete elimination of interference signals due to inaccurate marking. Consequently, it significantly improves the accuracy and comprehensiveness of noise cancellation, outputs higher-quality clean audio streams, and enhances the voice communication and audio processing performance of the device.

[0082] Based on the above Figure 2 In one example embodiment of the method shown, the noise suppression model in step 102 is trained through [method name missing]. Figure 6Steps 501 and 502 shown are implemented.

[0083] Step 501: Obtain the training sample set. Each training sample in the training sample set includes a mixed audio signal and a target speech signal. The mixed audio signal is determined based on the target speech signal, background noise signal, and nonlinear echo signal. The nonlinear echo signal is determined based on the device acoustic fingerprint characteristics. The device acoustic fingerprint characteristics include the physical nonlinear characteristics of the audio signal transmission link from the device's speaker to the microphone.

[0084] Step 502: Using the target speech signal as a label, train the pre-constructed initial neural network model based on the training sample set to obtain the noise suppression model corresponding to the model training result satisfying the preset convergence condition.

[0085] By setting the mixed audio signal in each training sample to include three key components—target speech, background noise, and nonlinear echo—it is ensured that the training sample set can cover complex scenarios in real-world applications. The nonlinear echo signal simulates the echo generated by the nonlinear distortion of the device itself, and its accuracy is crucial for the model to learn to eliminate real nonlinear echoes.

[0086] Background noise signals can be understood as various environmental noises that exist in the real environment but are neither the target speech signal nor a nonlinear echo signal. Their purpose is to make the training samples closer to real-world application scenarios and improve the model's robustness to noise. Background noise signals can be various noise samples collected from real-world environmental noise databases, or they can be synthetic noise signals with specific statistical characteristics.

[0087] The initial neural network model is a single-channel input deep neural network. The model is trained under supervision using a training dataset, enabling it to learn to recognize and separate nonlinear echo signals, noise signals, and target speech signals.

[0088] The acoustic fingerprint of a device can be the device's impulse response or a library of real echo samples composed of "source speech - nonlinear echo".

[0089] Specifically, for each training sample in the training sample set, the mixed audio signal simulates the multi-component audio captured by a microphone in real-world applications. The mixed audio signal can be fused from clean near-end speech (i.e., the target speech signal), background noise, and nonlinear echo signals. The target speech signal serves as the supervision signal (label) for model training, representing the clean speech content expected to be extracted from the mixed audio signal. When each target speech signal is clean near-end speech, it can be fused with the background noise signal and the pre-simulated nonlinear echo signal to obtain the corresponding mixed audio signal. In the supervised learning training framework, clean near-end speech, as the model label, refers to the original speech signal unaffected by noise, reverberation, or interference signals. This clean near-end speech serves as a reference benchmark, used to compare with the model's output signal to calculate the loss function, thereby guiding the update of model parameters and enabling the model to learn the mapping relationship for recovering the target speech signal from the mixed audio signal.

[0090] For the initial neural network model, it can adopt an encoder-decoder architecture and include a temporal modeling unit (such as LSTM, GRU, self-attention mechanism unit) for extracting long temporal context features, so as to use the harmonic structure features of nonlinear echo signals for signal separation.

[0091] For example, the model structure of the noise suppression model described in this application is as follows: Figure 7 As shown, Figure 7 The Encoder in this context refers to the encoder module. Figure 7 In this context, the Temporal Modeling Unit (TMU) is the temporal modeling unit, also known as the distortion feature recognition module; Feature extraction is the feature extraction module; Decoder is the decoder module; skip represents a jump connection; and y(n) represents the mixed audio frames as input to the model. (n) represents the clean audio frame output by the model. By constructing an initial neural network model that decouples the feature extraction domain from the signal reconstruction domain, the aim is to amplify the weak nonlinear harmonic distortion components in the mixed audio signal, making them more significant in the feature space, so that the model can capture the weak acoustic fingerprint features of the device.

[0092] Finally, the mixed audio signal is used as the input to the initial neural network model, and the target speech signal is used as the model training target, forming paired datasets. That is, each training sample in the training sample set is a paired data consisting of the mixed audio signal and the target speech signal. Furthermore, when each target speech signal in the training sample set is pure near-end speech, the input in the entire model training process does not contain the far-end reference signal corresponding to the pure near-end speech.

[0093] In this way, the target speech signal is used as a label, and the pre-constructed initial neural network model is trained under supervision based on the training sample set. The model parameters are optimized using the signal-to-distortion ratio and frequency domain L1 loss as joint loss functions until the model training result meets the preset convergence condition. For example, if the joint loss value of the network model after training is less than the preset loss threshold, the network model corresponding to the model training result meeting the preset convergence condition is determined as the noise suppression model.

[0094] The noise cancellation method provided in this application determines the nonlinear echo signal based on the device's acoustic fingerprint characteristics and constructs a synthetic training dataset containing the device's unique acoustic fingerprint, instead of using a general linear room impulse response convolution with a reference signal. Here, the linear room impulse response describes the entire acoustic propagation characteristics of sound after it is emitted from the sound source, undergoing one or more reflections, attenuation, and delays from walls, floors, and objects within the room, before finally reaching the pickup point. This ensures that the model training phase can learn the physical nonlinear characteristics of the device's speaker-to-microphone transmission link, fully adapting to the device's unique nonlinear distortion. The model no longer eliminates echoes through subtraction but instead learns the device's acoustic fingerprint characteristics, directly treating nonlinear echoes as background interference with a specific distortion pattern for blind separation. Based on this, the initial neural network model is trained, and the trained noise suppression model can more accurately identify and eliminate nonlinear signal components and noise signal components in mixed audio frames, significantly improving the accuracy of noise cancellation and the quality of the clean audio stream.

[0095] Based on the above Figure 6 In one example embodiment of the method shown, the mixed audio signal in step 501 is determined based on the target speech signal, background noise signal, and nonlinear echo signal. The specific process in this embodiment can be achieved through… Figure 8 Steps 601 and 602 shown are implemented.

[0096] Step 601: Perform convolution processing on the acoustic fingerprint features of the device and the target speech signal to obtain a nonlinear echo signal.

[0097] Step 602: Fuse the target speech signal, nonlinear echo signal, and background noise signal to obtain a mixed audio signal.

[0098] Specifically, in the time domain, nonlinear echo signals can be obtained by performing discrete convolution operations between the device's acoustic fingerprint features (such as the device's impulse response or a real echo sample sampled from a real echo sample library) and the target speech signal. In the frequency domain, a mixed audio signal can also be obtained by multiplying the device's acoustic fingerprint features and the Fourier transform results of the target speech signal, followed by an inverse Fourier transform. The nonlinear echo signal is an echo signal with nonlinear distortion characteristics generated by the device's acoustic fingerprint features after the target speech signal is played through the device's speaker; it serves as a sample for training models to identify and eliminate nonlinear echoes.

[0099] The obtained nonlinear echo signal can be fused with the target speech signal and the background noise signal to obtain a mixed audio signal. For example, the clean near-end speech (i.e., the target speech signal) is represented as s(n), the background noise as v(n), and the nonlinear echo signal as d. NL When (n), the generation model of the mixed audio signal y(n) can be defined as Equation (1).

[0100] y(n) = s(n) + d NL (n) + v(n)(1) In equation (1), n ​​represents the sampling point number in discrete time, that is, n is the number of the nth sampling point, which is used to represent the position of the corresponding signal on the discrete time axis.

[0101] The noise cancellation method provided in this application generates highly realistic nonlinear echo signals by accurately simulating the inherent nonlinear distortion of the device. This allows the training samples to fully reflect the complex acoustic environment in real-world applications, including clean speech, nonlinear echoes, and background noise. Consequently, it significantly improves the performance and robustness of the noise suppression model in handling nonlinear distortion and complex noise environments, avoids strong dependence on external reference signals, and makes model training more autonomous and efficient.

[0102] Based on the above Figure 1 In one example embodiment of the method shown, the mixed audio signal in step 501 is determined based on the target speech signal, background noise signal, and nonlinear echo signal. The specific process in this embodiment can be implemented through the following steps.

[0103] The nonlinear echo signal is subjected to random signal enhancement, and the enhanced nonlinear echo signal, target speech signal and background noise signal are fused to obtain a mixed audio signal; wherein, random signal enhancement includes random resampling and / or amplitude gain perturbation.

[0104] Random resampling involves randomly adjusting the sampling rate of the nonlinear echo signal to simulate problems such as clock drift and sampling rate mismatch that may occur during signal transmission or processing, thereby introducing random distortion on the time axis. Amplitude gain perturbation involves randomly adjusting the amplitude of the nonlinear echo signal to simulate fluctuations in speaker volume, signal attenuation, or amplification, thereby introducing random distortion in energy.

[0105] Both random resampling and amplitude gain perturbation are methods used to more accurately simulate the random variations of nonlinear echo signals in the real world. Each method aims to simulate hardware clock drift and signal level fluctuations in a real communication environment, further enriching the features of the training samples.

[0106] Specifically, random signal enhancement is applied to the simulated nonlinear echo signal to simulate various uncertainties and fluctuations that may occur in nonlinear echo signals in real-world environments. This increases the diversity and complexity of the training data, enabling the model to learn a wider range of nonlinear distortion patterns, thereby improving the model's generalization ability and robustness in practical applications. Based on this, random resampling of the nonlinear echo signal can simulate subtle high-frequency variations caused by hardware and software clock asynchrony, and / or, amplitude gain perturbations can be added, i.e., randomly adjusting the amplitude of the nonlinear echo signal, to simulate volume changes under different interactive scenarios; thus, an enhanced nonlinear echo signal is obtained.

[0107] Then, the nonlinear echo signal, target speech signal, and background noise signal that have undergone random enhancement are superimposed according to certain rules to generate a mixed audio signal. Its purpose is to construct a training sample that is as close as possible to the real application scenario, which includes the target speech signal, background noise signal, and nonlinear echo with random variation characteristics, so that the model can learn how to accurately separate the pure speech signal in a complex environment.

[0108] For example, in addition to random resampling and amplitude gain perturbation, random signal enhancement can also randomly adjust the characteristics of nonlinear echo signals in the time or frequency domain, such as by introducing random delays, phase shifts, or frequency modulations; or, it can generate nonlinear echo signals with random enhancement characteristics by simulating the effects of environmental factors on nonlinear echo signals, such as simulating speaker aging, microphone position changes, or changes in environmental acoustic characteristics.

[0109] The noise cancellation method provided in this application enhances the nonlinear echo signal by random signal enhancement, particularly by introducing random resampling and / or amplitude gain perturbation. This allows the training data to more fully simulate various random changes in the nonlinear echo signal in the real environment, significantly increasing the variability of the training samples. This enables the echo suppression model to learn a wider range of more complex nonlinear distortion patterns during training, thereby ensuring that the noise suppression model trained subsequently can more accurately and effectively eliminate nonlinear echoes and background noise, significantly improving the echo cancellation effect and call quality of the device in voice communication.

[0110] Based on the above Figure 2 In one example embodiment of the method shown, step 501 involves obtaining a training sample set. The specific process of this step can be achieved through… Figure 9 Steps 701 to 703 shown are implemented.

[0111] Step 701: Obtain the device acoustic fingerprint features when the device plays a preset audio signal under different audio playback parameters.

[0112] Step 702: Determine multiple nonlinear echo signals based on the acoustic fingerprint features of multiple devices and the target speech signal.

[0113] Step 703: Obtain the training sample set based on multiple nonlinear echo signals, target speech signals, and background noise signals.

[0114] The number of nonlinear echo signals can be greater than or equal to the number of acoustic fingerprint features of the device.

[0115] Audio playback parameters include at least one of the speaker playback volume and the degree of speaker obstruction. Speaker playback volume is one of the key factors affecting the degree of non-linear distortion of the device. Generally, the higher the volume, the more significant the non-linear distortion. Different playback volumes of the speaker need to be set, and the setting should include the non-linear saturation range (e.g., 80% to 100% of the maximum playback volume).

[0116] The degree of speaker obstruction simulates various physical obstructions a user might encounter while using the device, such as holding it, placing it on a table, or in a pocket. These obstructions alter the speaker's acoustic load and radiation characteristics, thus affecting the nonlinear characteristics of the echo. By systematically changing these parameters, it's possible to simulate various complex acoustic environments the device might encounter in the real world, thereby obtaining more representative acoustic fingerprint characteristics of the device. For example, the playback volume can be changed by adjusting the speaker's output gain via software, and different degrees of obstruction can be simulated by placing obstructions of different materials, shapes, or distances.

[0117] Specifically, to ensure that the trained noise suppression model can adapt to various real-world usage scenarios, it is necessary to acquire the acoustic fingerprint features of the device under different audio playback parameters. This embodiment captures the device's acoustic fingerprint features (such as AI glasses) offline. When the preset audio signal is a preset test stimulus signal, it can be referred to... Figure 10 The method for obtaining the acoustic fingerprint features of a device is shown below. Figure 10 As shown, in a quiet environment (such as an anechoic chamber or a low-noise room), the speaker volume of the AI ​​glasses is adjusted to the maximum level and a preset test stimulus signal is played. The test stimulus signal includes, but is not limited to, logarithmic sine sweep signals, white noise, and speech segments from standard speech libraries (such as LibriSpeech). The audio signal played is recorded simultaneously using the microphone of the AI ​​glasses. By comparing the original test stimulus signal with the recorded signal, the impulse response (RIR) of the AI ​​glasses is extracted, or a real echo sample library composed of "source speech - nonlinear echo" is directly constructed. The acoustic fingerprint features of the device are thus obtained.

[0118] For example, the acoustic fingerprint of a device can be obtained by recording the output of a specific preset audio signal played by a high-precision microphone in an anechoic chamber or echo chamber, and then analyzing its nonlinear distortion characteristics in conjunction with the input signal. Another approach is to indirectly infer or model the acoustic fingerprint characteristics of a device by correlating data from built-in sensors (such as accelerometers and gyroscopes) with the audio signal.

[0119] For multiple audio playback parameters, the playback volume of the device's speakers can be adjusted to different levels in a quiet environment (such as an anechoic chamber or a low-noise room), while still covering the maximum playback volume. In this case, the speakers are often in linear and nonlinear operating regions. The nonlinear operating region will produce clipping, harmonic distortion, and shell resonance. And / or, the speakers can be artificially blocked to different degrees to obtain different audio playback parameters. The linear operating region refers to the working state of the speaker under small signal excitation. At this time, the electroacoustic parameters of the speaker (such as force coefficient, stiffness coefficient, etc.) remain approximately constant, the output sound pressure is linearly proportional to the input voltage, the system follows the superposition principle, and the nonlinear distortion generated can be ignored. The nonlinear operating region refers to the working state of the speaker under large signal excitation (such as maximum volume). At this time, due to the large displacement of the voice coil, the magnetic circuit coupling weakens and the stiffness of the mechanical suspension system changes. The electroacoustic conversion parameters of the system change significantly with the displacement, resulting in nonlinear characteristics such as amplitude compression and high-order harmonic distortion in the output signal.

[0120] By appropriately fusing the generated multiple nonlinear echo signals, the original target speech signal, and various background noise signals, a training sample set containing rich scenarios and complex distortion types can be constructed. For example, the target speech signal, nonlinear echo signal, and background noise signal can be superimposed at different signal-to-noise ratios (SNR) and echo-to-speech ratios (ERL) to simulate the complexity of mixed audio signals in a real environment; the background noise signal can include various environmental noises (such as street noise, office noise, fan noise, etc.) to enhance the model's generalization ability. The training sample set constructed in this way can comprehensively cover various acoustic conditions that the device may encounter in actual use, providing a solid foundation for the effective training of the noise suppression model.

[0121] The noise cancellation method provided in this application systematically acquires the acoustic fingerprint features of the device under different playback volumes and speaker obstruction levels, and generates nonlinear echo signals based on these diverse fingerprint features to construct a more representative and robust training sample set. The training sample set can simulate various complex nonlinear distortion and noise environments that the device may encounter in real-world usage scenarios. This allows the initial neural network model trained based on this sample set to better learn and adapt to these complex nonlinear distortion characteristics, thereby effectively improving the accuracy and stability of noise cancellation and greatly enhancing the model's generalization ability and robustness in complex environments.

[0122] Based on the above Figure 9 The method shown, in one example embodiment, involves obtaining the device acoustic fingerprint characteristics when the device plays a preset audio signal under different audio playback parameters in step 701. The specific process of this step in this embodiment can be achieved through… Figure 11 Steps 801 and 802 shown are implemented.

[0123] Step 801: Align the preset audio signal and the audio signal recorded synchronously during the playback of the preset audio signal under the corresponding audio playback parameters, and determine the total harmonic distortion characteristics and intermodulation distortion characteristics of the speaker based on the two aligned audio signals.

[0124] Step 802: Construct the nonlinear harmonic energy envelope of the loudspeaker based on the total harmonic distortion characteristics and intermodulation distortion characteristics, and normalize the nonlinear harmonic energy envelope to obtain the acoustic fingerprint characteristics of the device.

[0125] Among them, the device's acoustic fingerprint features can be derived from the device's impulse response (RIR) or by directly constructing a real echo sample consisting of "source speech - nonlinear echo". These data fully preserve the physical nonlinearity of the device's speaker-microphone link.

[0126] Specifically, for a large number of defined audio playback parameters, a preset audio signal can be played under each audio playback parameter. During the playback, the device's microphone is used to record the played audio signal simultaneously. Then, the two audio signals are aligned. The purpose of the signal alignment operation is to eliminate the time delay and phase difference between the preset audio signal and the recorded audio signal, so as to ensure the accuracy of subsequent distortion feature analysis.

[0127] The total harmonic distortion (THD) and intermodulation distortion (IMD) characteristics of a loudspeaker are determined based on the two aligned audio signals. THD measures the ratio of the energy of newly added harmonic components to the fundamental frequency energy in the output signal after the signal passes through the loudspeaker, reflecting the loudspeaker's nonlinear response to a single-frequency signal. IMD measures the degree to which new frequency components beyond the original frequency combination are generated in the output signal when the two audio signals pass through the loudspeaker, reflecting the loudspeaker's nonlinear response to complex multi-frequency signals. THD and IMD characteristics directly quantify the degree and type of nonlinear distortion of the loudspeaker under specific audio playback parameters. These characteristics can be determined through spectral analysis, performing a Fourier transform on the two aligned audio signals to identify and calculate the energy of harmonic and intermodulation components; alternatively, distortion can be quantified by comparing the spectrum of the input preset audio signal with the spectrum of the recorded audio signal, thus obtaining the THD and IMD characteristics.

[0128] At this point, by integrating the total harmonic distortion characteristics and intermodulation distortion characteristics and describing the curves of the nonlinear distortion energy distribution of the loudspeaker at different frequencies and amplitudes, the nonlinear harmonic energy envelope of the loudspeaker can be constructed. Further normalization of the nonlinear harmonic energy envelope of the loudspeaker aims to eliminate the dimensional influence caused by different measurement conditions or individual differences in equipment, so that the obtained acoustic fingerprint characteristics of the equipment are comparable and stable, which is convenient for subsequent model training and application.

[0129] The noise cancellation method provided in this application obtains more accurate, stable and representative acoustic fingerprint features of the device by acquiring features under various audio playback parameters and combining signal alignment, determination of total harmonic distortion features and intermodulation distortion features, construction of nonlinear harmonic energy envelope and normalization processing. This makes the subsequent determination of nonlinear echo signals more accurate, thereby significantly improving the performance of the echo suppression model in complex practical application scenarios.

[0130] For example, refer to Figure 12 The flowchart of the noise cancellation method shown is as follows: Figure 12As shown, the process first involves acquiring the acoustic fingerprint features of the target device, then constructing a training sample set containing these features. Next, an initial neural network model is built and trained until a noise suppression model is obtained. Finally, online real-time noise suppression inference is performed. This involves first segmenting the acquired mixed audio stream into frames, then inputting them into the trained noise suppression model for noise suppression, outputting clean audio frames, and then concatenating them sequentially to obtain a clean audio stream. Throughout this process, no reference signal needs to be obtained from the underlying layer. The specific processes involved can be referred to in the aforementioned embodiments, and will not be elaborated further here.

[0131] To verify the effectiveness of the noise suppression method in this application, Figure 13 The comparison of processing results in real-world scenarios is shown. The first channel is the original signal waveform, the second channel is the waveform after model processing, the third channel is the spectrogram of the original signal, and the fourth channel is the spectrogram of the model processing. The spectrogram is a time-frequency distribution image that represents a non-stationary audio signal. It is generated by short-time Fourier transform or other time-frequency analysis methods. Its horizontal axis represents time, and its vertical axis represents frequency. The gray value or color intensity of a pixel represents the energy intensity (or amplitude) of the frequency component corresponding to that time point. It can reveal the dynamic characteristics of the speech signal in both the time and frequency domains.

[0132] The time-domain waveform of the first channel reveals high-amplitude clutter, a chaotic waveform envelope, and significant background noise at speech pauses. Further observation using the spectrogram of the third channel reveals numerous high-energy, broadband vertical stripes. These stripes correspond to echo signals generated by the speaker, spreading across the entire frequency range and exhibiting muddy energy clusters in the low-frequency region, severely obscuring the spectral details of the near-end target speech. This indicates severe time-frequency aliasing between the echo and the human voice before processing. In contrast to the first channel, the background region (i.e., the speechless segment) of the second channel's output waveform is compressed to near zero, exhibiting an excellent signal-to-noise ratio; simultaneously, the waveform envelope of the target speech segment remains intact, with clear start and end points, and no obvious clipping or over-suppression. Comparing to the third channel, the spectrogram of the fourth channel clearly shows that the chaotic vertical stripes representing nonlinear echoes in the original image have been completely filtered out, resulting in a very clean background (deep black area). More importantly, the spectrogram of near-end speech was preserved and restored relatively completely, and the formants were clearly visible. However, there was some distortion in the high-frequency harmonics, and the echo suppression ratio was significantly improved.

[0133] Therefore, the above comparative experiments intuitively demonstrate that the method of this application still achieves excellent noise reduction efficiency and speech fidelity even without a reference signal.

[0134] It should be noted that although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowchart can be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0135] Figure 14 A schematic diagram of a hardware architecture suitable for implementing embodiments of this application is shown, such as... Figure 14 As shown, device 900 specifically includes a processor 901, a communication interface 902, a memory 903, and a communication bus 904. The processor 901, communication interface 902, and memory 903 communicate with each other via the communication bus 904. The memory 903 stores programs that can be executed by the processor 901. The processor 901 executes the programs stored in the memory 903 to implement… Figure 2 The steps of the method shown are as follows.

[0136] The communication bus 904 mentioned in the above-mentioned device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 904 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 14The bus is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Communication interface 902 is used for communication between the aforementioned electronic device and other devices. Memory 903 may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor 901. The aforementioned processor 901 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc., or it may be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0137] In another embodiment of this application, a computer program product is also provided, the computer program product including instructions that, when executed, cause the following as described in the above embodiments: Figure 2 The steps of the method shown are implemented.

[0138] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another, for example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape, etc.), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0139] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A noise cancellation method, characterized in that, The method includes: Acquire a mixed audio stream, divide the mixed audio stream into frames, and obtain multiple mixed audio frames; The mixed audio frame is input into a noise suppression model, and energy enhancement processing is performed on the nonlinear signal components in the mixed audio frame. Distortion characteristics are identified based on the energy-enhanced mixed audio frame. The distortion characteristics identification is used to identify and memorize the nonlinear distortion characteristics of the nonlinear signal components in the mixed audio frame. Based on the result of the distortion characteristic identification, a noise suppression mask is generated. The noise suppression mask is used to distinguish the nonlinear signal component, the noise signal component, and the clean audio signal component. The noise suppression mask is used to perform noise cancellation on the mixed audio frames to eliminate nonlinear signal components and noise signal components in the mixed audio frames, thereby obtaining clean audio frames corresponding to each of the mixed audio frames output by the noise suppression model, and obtaining a clean audio stream based on each of the clean audio frames.

2. The method according to claim 1, characterized in that, The energy enhancement processing of the nonlinear signal components in the mixed audio frame includes: The mixed audio frame is converted to a time-frequency signal to obtain a complex spectrum of the mixed audio frame. The complex spectrum is used to characterize the amplitude and phase of the sound signal at each time-frequency position. Energy enhancement is performed on the low-amplitude spectrum in the complex spectrum to enhance the nonlinear signal components in the mixed audio frame.

3. The method according to claim 2, characterized in that, The distortion characteristic identification based on the hybrid audio frames after energy enhancement processing includes: Feature extraction is performed on the energy-enhanced complex spectrum to obtain spectrum-based acoustic features; The spectrum-based acoustic features and the complex spectrum are fused, and the fused result is encoded to obtain a high-dimensional acoustic feature map. The high-dimensional acoustic feature map includes the nonlinear acoustic features of the nonlinear signal component, the statistical texture features of the noise signal component, and the high-dimensional latent space features of the clean audio signal component. Identify and memorize the nonlinear distortion characteristics of the nonlinear acoustic features in the high-dimensional acoustic feature map as they change over time.

4. The method according to claim 1, characterized in that, The generation of a noise suppression mask based on the result of the distortion characteristic identification includes: Based on the results of the distortion characteristic identification, the signal components with nonlinear distortion characteristics in the mixed audio frame are marked; The signal components with statistical texture features in the mixed audio frame are labeled, and the noise suppression mask is generated based on the labeling results.

5. The method according to claim 1, characterized in that, The training process of the noise suppression model includes: A training sample set is obtained, wherein each training sample in the training sample set includes a mixed audio signal and a target speech signal, the mixed audio signal being determined based on the target speech signal, a background noise signal, and a nonlinear echo signal; the nonlinear echo signal is determined based on the device acoustic fingerprint features; the device acoustic fingerprint features include the physical nonlinear features of the audio signal transmission link from the device's speaker to the microphone; Using the target speech signal as a label, the pre-constructed initial neural network model is trained based on the training sample set to obtain the noise suppression model corresponding to the model training result satisfying the preset convergence condition.

6. The method according to claim 5, characterized in that, The mixed audio signal is determined based on the target speech signal, background noise signal, and nonlinear echo signal, including: The nonlinear echo signal is obtained by convolution processing the acoustic fingerprint features of the device and the target speech signal. The target speech signal, the nonlinear echo signal, and the background noise signal are fused to obtain the mixed audio signal.

7. The method according to claim 5, characterized in that, The mixed audio signal is determined based on the target speech signal, background noise signal, and nonlinear echo signal, including: The nonlinear echo signal is subjected to random signal enhancement, and the enhanced nonlinear echo signal, the target speech signal, and the background noise signal are fused to obtain the mixed audio signal; wherein, the random signal enhancement includes random resampling and / or amplitude gain perturbation.

8. The method according to claim 5, characterized in that, The acquisition of the training sample set includes: Acquire the acoustic fingerprint features of the device when playing a preset audio signal under different audio playback parameters; the audio playback parameters include at least one of the speaker playback volume and the degree of occlusion of the speaker; Multiple nonlinear echo signals are determined based on multiple device acoustic fingerprint features and the target speech signal; The training sample set is obtained based on the multiple nonlinear echo signals, the target speech signal, and the background noise signal.

9. The method according to claim 8, characterized in that, The step of obtaining the device's acoustic fingerprint features when the device plays a preset audio signal under different audio playback parameters includes: The preset audio signal and the audio signal recorded synchronously during the playback of the preset audio signal under the corresponding audio playback parameters are aligned, and the total harmonic distortion characteristics and intermodulation distortion characteristics of the speaker are determined based on the two aligned audio signals. The nonlinear harmonic energy envelope of the loudspeaker is constructed based on the total harmonic distortion characteristics and the intermodulation distortion characteristics, and the nonlinear harmonic energy envelope is normalized to obtain the acoustic fingerprint characteristics of the device.

10. A computer program product, characterized in that, The computer program product includes instructions that, when executed, cause the method as described in any one of claims 1-9 to be implemented.

Images