Speaker intelligent audio signal enhancement method and system based on voice activity detection

Abstract

The present application belongs to the technical field of audio signal processing, and relates to a loudspeaker intelligent audio signal enhancement method and system based on voice activity detection, comprising the following steps: acquiring mixed collection audio stream and internal playback source data, and generating a frequency spectrum through time-frequency domain conversion; analyzing the mixed frequency spectrum, and outputting an activity flag and a frequency band level confidence vector; performing adaptive mapping attenuation on the source frequency spectrum by using a vector mapping penalty coefficient, generating an estimated background frequency spectrum, and calculating the instantaneous signal-to-noise ratio of each subband; introducing the flag, the confidence and the signal-to-noise ratio into a double condition gate to identify a target enhancement frequency band; constructing a gain mask according to the target enhancement frequency band to derive an enhanced frequency spectrum; and performing inverse conversion to generate low-interference continuous audio, synchronously generating a control instruction based on the confidence, and reducing the playback power of a specific frequency band of the loudspeaker. The present application solves the problem that traditional audio enhancement methods cannot fully suppress music noise while avoiding damage to voice signals in a strong non-stationary background music interference environment.

Description

Technical Field

[0001] This invention belongs to the technical field of audio signal processing, and relates to a method and system for intelligent audio signal enhancement for loudspeakers based on speech activity detection. Background Technology

[0002] Smart audio devices with voice interaction capabilities often need to cope with complex acoustic environments when receiving user voice commands. Background music or media audio played by the device's own speakers mixes with the user's voice signal in space, forming a mixed audio signal containing strong correlation interference. This phenomenon, known as the interruption or barge scenario, directly affects the performance of subsequent automatic speech recognition systems, easily leading to decreased recognition rates or wake-up failures.

[0003] Commonly used technical solutions in the industry include acoustic echo cancellation, beamforming, and noise suppression algorithms based on statistical signal processing. Acoustic echo cancellation technology simulates the acoustic path by establishing an adaptive filter and subtracts the estimated echo signal from the acquired signal. Beamforming technology uses the spatial sampling capability of a microphone array to form a pickup beam pointing towards the sound source. Statistical noise suppression methods estimate the power spectrum of background noise and remove it from the power spectrum of the mixed signal.

[0004] The aforementioned traditional methods have technical limitations when dealing with non-stationary and wide-bandwidth background music interference. The linear adaptive filters that acoustic echo cancellation relies on have limited path estimation accuracy when faced with nonlinear distortion of loudspeakers or complex and variable indoor reverberation environments, which can easily lead to incomplete echo suppression. Statistical noise suppression methods rely on the assumption of stationarity of noise signals, while music signals are highly non-stationary and dynamic, making accurate modeling difficult and easily causing damage to the speech signal or producing processing distortion. Beamforming technology also has limited spatial resolution when the user and the interference source are in close proximity or when there is strong reflected sound. Summary of the Invention

[0005] In a first aspect, the present invention provides a method for intelligent audio signal enhancement of a loudspeaker based on voice activity detection, comprising the following steps:

[0006] S1. Acquire the mixed audio stream and the internal playback source data of the synchronized hardware adjustment state, and generate the spectrum of the mixed audio signal and the spectrum of the original background music source signal through time-frequency domain conversion;

[0007] S2. Perform feature analysis on the spectrum of the mixed audio signal, and output a speech activity decision flag representing the global speech state and a frequency band-level speech presence confidence vector representing the local probability.

[0008] S3. Map the frequency band-level speech existence confidence vector to the energy penalty coefficient, perform nonlinear adaptive mapping and attenuation on the spectrum of the original background music source signal, and generate the estimated spectrum of the background music components.

[0009] S4. Compare the spectrum of the mixed audio signal, the spectrum of the original background music source signal, and the spectrum of the estimated background music components, and use the energy separation algorithm to calculate the instantaneous frequency band signal-to-noise ratio estimate of each frequency sub-band.

[0010] S5. The speech activity decision flag, the frequency band-level speech presence confidence vector and the instantaneous frequency band signal-to-noise ratio estimate are introduced into a dual-condition gating mechanism to identify and determine the target enhancement frequency band.

[0011] S6. Based on the instantaneous frequency band signal-to-noise ratio estimate and the target enhanced frequency band, determine the independent compensation parameters and construct the spectrum gain mask. Apply the spectrum gain mask to the spectrum of the mixed audio signal to derive the enhanced spectrum.

[0012] S7. Perform an inverse transformation on the enhanced spectrum to generate a time-series continuous digital audio stream, and simultaneously generate feedback control commands based on the frequency band-level speech presence confidence vector to reduce the playback power of the speaker in a specific frequency band.

[0013] A further aspect of the present invention, step S1, includes the following steps:

[0014] The acquired mixed audio stream and the internal playback source data of the device are subjected to frame-by-frame windowing processing using a preset window function to generate windowed mixed audio frames and windowed playback source data frames.

[0015] Short-time Fourier transform is performed on each frame of the windowed signal to extract the spectrum of the mixed audio signal containing the time-slice frequency domain energy distribution characteristics and the spectrum of the original background music source signal;

[0016] A global frame index counter is used to align the spectrum of the mixed audio signal with the spectrum of the original background music source signal in the time dimension.

[0017] A further aspect of the present invention, step S2, includes the following steps:

[0018] The relative spectral pattern distribution information of the mixed audio signal spectrum in each frequency sub-band is quantitatively analyzed using a spectral feature extraction model.

[0019] The overall activation state of the environment is judged by the top-level global classification node of the spectral feature extraction model, and the speech activity judgment flag established by the binary state parameter is output.

[0020] The probability of the presence of local speech components is predicted by the feedback feature regression nodes of the spectral feature extraction model, generating a frequency band-level speech presence confidence vector composed of a continuous floating-point probability value array.

[0021] A further aspect of the present invention, step S3, includes the following steps:

[0022] Scan the energy parameters of each frequency band of the original background music source signal spectrum, and simultaneously read in the probability value of the frequency band-level speech presence confidence vector corresponding to the position;

[0023] By inputting the probability value into a preset nonlinear mapping function, the energy penalty coefficient characterizing the degree of energy decay is derived.

[0024] The energy amplitude of the corresponding frequency band in the original background music source signal spectrum is proportionally attenuated and scaled using an energy penalty coefficient. The processed frequency band data is then combined in the original frequency order to obtain the estimated background music component spectrum.

[0025] A further aspect of the present invention, step S4, includes the following steps:

[0026] Under the same time frame, the actual mixed observation energy mean of the spectrum of the mixed audio signal and the reference background source energy mean of the spectrum of the original background music source signal are extracted, and the residual noise reference mean of the spectrum of the estimated background music component is obtained simultaneously.

[0027] An acoustic coupling factor is introduced, and based on the energy subtraction logic, the reference background source energy mean weighted by the acoustic coupling factor is removed from the actual mixed audio observation energy mean. The residual noise reference mean weighted by the acoustic coupling factor is used as the energy lower limit constraint to restore the approximate speech target energy intensity.

[0028] The ratio of the energy intensity of the approximate speech target to the mean energy of the reference background source after being weighted by the acoustic coupling factor is quantized and logarithmically calculated to synthesize the instantaneous frequency band signal-to-noise ratio estimate.

[0029] A further aspect of the present invention, step S5, includes the following steps:

[0030] Monitor the voice activity decision flag. When the decision flag is in the valid voice input state, start the composite evaluation for each frequency sub-band.

[0031] Determine whether the continuous floating-point probability value is greater than the first confidence threshold, and simultaneously determine whether the instantaneous frequency band signal-to-noise ratio estimate is lower than the second signal-to-noise ratio threshold;

[0032] In the case where the continuous floating-point probability value is greater than the first confidence threshold and the instantaneous frequency band signal-to-noise ratio estimate is lower than the second signal-to-noise ratio threshold, the corresponding frequency band is recorded as the target enhancement frequency band.

[0033] A further aspect of the present invention, step S6, includes the following steps:

[0034] Calculate the difference offset between the estimated instantaneous frequency band signal-to-noise ratio within the target enhancement band and the target preferred by the predetermined sharpness index;

[0035] The amplitude of the nonlinear boost coefficient required to map the target frequency sub-band based on the difference offset;

[0036] The nonlinear boosting coefficient amplitude is assigned to the position corresponding to the target enhanced frequency band, and the gain value of the non-target enhanced frequency band is set as a constant coefficient to assemble and generate a spectrum gain mask.

[0037] A further aspect of the present invention, step S7, includes the following steps:

[0038] The original phase information of the mixed audio signal spectrum is preserved, and an inverse short-time Fourier transform is performed in combination with the enhanced amplitude information to obtain the time-domain signal frame.

[0039] A sequential digital audio stream is reconstructed by overlapping and adding consecutive time-domain signal frames.

[0040] The frequency band intervals that exhibit continuous probability values ​​in the confidence vector of frequency band-level speech are identified, and feature indicator code blocks containing frequency band index information are generated.

[0041] A dynamic modification command containing feature indicator code block information is issued through the reverse hardware communication channel, driving the speaker to adjust the drive current parameters within the frequency band covered by the feature indicator code block.

[0042] A further aspect of the present invention includes generating a feature indicator code block comprising:

[0043] Based on the preset third confidence threshold and the preset consecutive frequency band number threshold, search for frequency band sequences in the frequency band-level speech existence confidence vector that satisfy the condition that all consecutive probability values ​​are higher than the third confidence threshold;

[0044] Extract the start and end frequency band indices of the frequency band sequence and encapsulate them to generate feature indicator code blocks.

[0045] Secondly, the present invention provides a speaker intelligent audio signal enhancement system based on voice activity detection, comprising the following modules:

[0046] The signal acquisition and transformation module is used to acquire the mixed acquisition audio stream and the internal playback source data with synchronized hardware adjustment status, and generate the spectrum of the mixed audio signal and the spectrum of the original background music source signal through time-frequency domain transformation;

[0047] The speech activity analysis module is used to perform feature analysis on the spectrum of mixed audio signals and output a speech activity decision flag representing the global speech state and a frequency band-level speech presence confidence vector representing the local probability.

[0048] The background spectrum estimation module is used to map the energy penalty coefficient to the frequency band-level speech existence confidence vector, perform nonlinear adaptive mapping and attenuation on the spectrum of the original background music source signal, and generate the estimated spectrum of the background music components.

[0049] The signal-to-noise ratio (SNR) calculation module is used to compare the spectrum of the mixed audio signal, the spectrum of the original background music source signal, and the spectrum of the estimated background music components, and to generate the instantaneous frequency band SNR estimate for each frequency sub-band through an energy separation algorithm.

[0050] The target frequency band identification module is used to introduce the speech activity decision flag, the frequency band-level speech presence confidence vector and the instantaneous frequency band signal-to-noise ratio estimate into a dual-condition gating mechanism to identify and determine the target enhancement frequency band.

[0051] The spectrum enhancement module determines independent compensation parameters based on the instantaneous frequency band signal-to-noise ratio estimate and the target enhancement frequency band, constructs a spectrum gain mask, applies the spectrum gain mask to the spectrum of the mixed audio signal, and derives the enhanced spectrum.

[0052] The audio synthesis and feedback module is used to perform an inverse transformation on the enhanced spectrum to generate a time-series continuous digital audio stream, and simultaneously generate feedback control commands based on the frequency band-level speech presence confidence vector to reduce the playback power of the speaker in a specific frequency band.

[0053] In summary, the present invention has the following beneficial technical effects:

[0054] 1. This invention obtains the frequency band-level speech presence confidence through a deep learning network and introduces it as a dynamic weight into the nonlinear mapping process of the original playback source signal. Compared with traditional noise estimation methods based on full-band statistical characteristics, this mechanism can calculate the energy ratio of speech and music in different frequency bands, making the generated estimated background music spectrum closer to the real interference components in the mixed signal in terms of time-frequency distribution. This provides a more physically realistic reference benchmark for subsequent signal separation and signal-to-noise ratio calculation.

[0055] 2. This invention introduces a dual-condition gating mechanism to accurately locate the target enhancement frequency band by combining band-level confidence with instantaneous signal-to-noise ratio (SNR). This logic ensures that spectral gain compensation only applies to specific regions containing speech components and with low SNR, avoiding blind amplification of purely noisy frequency bands and reducing excessive intervention in high-SNR, clear speech regions. This on-demand enhancement strategy helps to improve the intelligibility of low-quality speech while reducing the probability of generating musical noise or speech waveform distortion during processing.

[0056] 3. This invention utilizes speech detection results to construct a feedback path pointing to the acoustic playback end, driving the speaker to perform current limiting adjustment in the high-frequency speech region through feature indicator code blocks. This mechanism changes the passive processing method that relies solely on backend algorithms for filtering, achieving targeted weakening of interference sources at the physical level. By improving the initial signal-to-noise ratio of the microphone before spatial aliasing, the convergence difficulty and computational pressure of subsequent digital signal processing algorithms are reduced, improving the overall robustness of the system in environments with strong background music interference. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. The drawings are used to provide a further understanding of the present invention.

[0058] Figure 1 A flowchart illustrating an embodiment of this application is disclosed.

[0059] Figure 2 Structural schematic diagrams of embodiments of this application are disclosed. Detailed Implementation

[0060] The following is in conjunction with the appendix Figure 1 - Figure 2 A preferred description of the present invention is provided below.

[0061] See attached document Figure 1 This invention proposes a smart audio signal enhancement method for loudspeakers based on speech activity detection, comprising the following steps:

[0062] S1. Acquire the mixed audio stream and the internal playback source data of the synchronized hardware adjustment state, and generate the spectrum of the mixed audio signal and the spectrum of the original background music source signal through time-frequency domain conversion;

[0063] S2. Perform feature analysis on the spectrum of the mixed audio signal, and output a speech activity decision flag representing the global speech state and a frequency band-level speech presence confidence vector representing the local probability.

[0064] S3. Map the frequency band-level speech existence confidence vector to the energy penalty coefficient, perform nonlinear adaptive mapping and attenuation on the spectrum of the original background music source signal, and generate the estimated spectrum of the background music components.

[0065] S4. Compare the spectrum of the mixed audio signal, the spectrum of the original background music source signal, and the spectrum of the estimated background music components, and use the energy separation algorithm to calculate the instantaneous frequency band signal-to-noise ratio estimate of each frequency sub-band.

[0066] S5. The speech activity decision flag, the frequency band-level speech presence confidence vector and the instantaneous frequency band signal-to-noise ratio estimate are introduced into a dual-condition gating mechanism to identify and determine the target enhancement frequency band.

[0067] S6. Based on the instantaneous frequency band signal-to-noise ratio estimate and the target enhanced frequency band, determine the independent compensation parameters and construct the spectrum gain mask. Apply the spectrum gain mask to the spectrum of the mixed audio signal to derive the enhanced spectrum.

[0068] S7. Perform an inverse transformation on the enhanced spectrum to generate a time-series continuous digital audio stream, and simultaneously generate feedback control commands based on the frequency band-level speech presence confidence vector to reduce the playback power of the speaker in a specific frequency band.

[0069] In one embodiment of the present invention, step S1 includes the following steps:

[0070] The acquired mixed audio stream and the internal playback source data of the device are subjected to frame-by-frame windowing processing using a preset window function to generate windowed mixed audio frames and windowed playback source data frames.

[0071] Short-time Fourier transform is performed on each frame of the windowed signal to extract the spectrum of the mixed audio signal containing the time-slice frequency domain energy distribution characteristics and the spectrum of the original background music source signal;

[0072] A global frame index counter is used to align the spectrum of the mixed audio signal with the spectrum of the original background music source signal in the time dimension.

[0073] The processor of a smart audio device uses its built-in microphone array to capture sound in the near-field acoustic space in real time, generating a first digital audio signal stream, denoted as the hybrid acquisition audio stream. It should be noted that the near-field acoustic space refers to the sound field range within which the user and the smart audio device's speakers and microphone array are located in the same indoor environment, typically within a few meters. The hybrid acquisition audio stream, captured by the microphone array, is a sequence of digital audio signals formed by the coherent superposition of the user's voice and background music played by the device's own speakers in the air. The sampling rate of the microphone array is set to... Simultaneously, the processor extracts a block of digital audio data from the memory buffer of the audio playback module. This data is currently at the output end of the digital signal processor, has been processed by the user's physical volume gain and the subsequent dynamic current limiting adjustment instruction in step S7, but has not yet undergone digital-to-analog conversion. This data is used to generate a second digital audio stream, denoted as the device's internal playback source data. In this embodiment, the device's internal playback source data refers to the pure digital audio data in the audio playback link of the intelligent audio device, which has crossed control loop logic conflicts and faithfully reflects the current actual driving reference state of the speaker before digital-to-analog conversion.

[0074] The sampling rate of the playback source data inside the device and the sampling rate of the mixed acquisition audio stream. To maintain consistency, the processor then performs frame-by-frame windowing on the mixed audio stream for time-frequency domain transformation. It should be understood that time-frequency domain transformation refers to mapping a time-domain signal into a two-dimensional representation that simultaneously contains time and frequency dimensions through mathematical transformation; in this step, it specifically refers to the short-time Fourier transform. Frame-by-frame windowing is a process that overcomes the limitations of the Fourier transform in analyzing non-stationary signals by dividing the continuous audio stream into short time intervals and applying a window function to reduce spectral leakage.

[0075] Specifically, a Hanning window function with a length of N sampling points is used to frame the mixed audio stream, with an overlap interval of R sampling points between adjacent frames; the Hanning window is a commonly used window function, and its expression is: Used to smooth signal edges during framing. Frame length. This determines the frequency resolution, typically 512 or 1024 sampling points. Frame shift The time resolution is determined and is usually set to the frame length. Half of, that is Based on the assumption of smart speakers in typical living room application scenarios, the supplemented audio sampling rate is... The range is 16 kHz to 48 kHz; frame length The basis for completion is to balance frequency resolution and time resolution. At kHz, The point corresponds to a time window length of 32 ms, and the frequency resolution is approximately 31.25 Hz.

[0076] Next, a short-time Fourier transform is performed on the windowed time-domain signal for each frame, yielding a two-dimensional matrix containing complex elements. This matrix defines the amplitude and phase information of the signal at each time slice and frequency point; this matrix is ​​defined as the spectrum of the mixed audio signal. Furthermore, the calculation process of the short-time Fourier transform can be defined by the following mathematical formula:

[0077]

[0078] in, The frequency index represents the spectrum obtained after the transformation. Time frame index Complex values ​​at the location. This represents the starting position in the original time-domain audio signal. The The first frame within the frame One sampling point. It is a length of The window function, in this step, specifically refers to the Hanning window. The frame length for performing the Fourier transform also determines the frequency resolution. It is the frame shift length. It is the imaginary unit. The resulting mixed audio signal spectrum is a two-dimensional complex matrix, where the row index corresponds to the frequency point, the column index corresponds to the time frame, and the magnitude of each element in the matrix represents the signal energy amplitude of that time-frequency unit.

[0079] Simultaneously, the processor performs the same time-frequency mapping transformation on the playback source data within the device. To ensure time-frequency domain alignment, the playback source data is framed and windowed using the exact same frame length N, frame shift R, and Hanning window function as the mixed audio stream. A short-time Fourier transform is performed on each frame to generate another complex two-dimensional matrix. This matrix reflects the energy distribution of the pure background music signal at various time and frequency points, unaffected by spatial acoustic path contamination and mixing. This matrix is ​​defined as the original background music source signal spectrum. Clearly, the original background music source signal spectrum is a two-dimensional complex matrix with the same number of time frames and frequency points as the mixed audio signal spectrum. The system maintains a global frame index counter to ensure that the mixed audio signal spectrum and the original background music source signal spectrum strictly correspond to the same set of time windows in the time dimension.

[0080] For example, suppose the system sets the audio sampling rate. 16 kHz, frame length Set to 512 sampling points, frame shift The sampling point was set to 256. First, the microphone array acquired a frame of time-domain signal, which was the spatial mixture of the user's voice saying "play music" and the music from a popular song being played by the device. This time-domain sequence is denoted as... A total of 512 points. Synchronously, a clean digital audio sequence of the song to be played, corresponding to the same time period, is extracted from the playback buffer and denoted as... It is also 512 points. Next, the system generates a 512-point Hanning window sequence. .right Add a window: ,in Then, regarding Perform a 512-point Fast Fourier Transform to obtain its spectrum. Only the positive frequency portion is taken. It is a complex number that contains both a real part and an imaginary part, for example in This corresponds to a frequency of approximately 312.5 Hz. The modulus is Using the exact same frame length, frame shift, and window function... The spectrum was obtained by processing. In the same At the frequency point, The modulus is Finally, after processing the first frame, the system obtained the first column of data from the spectrum of the mixed audio signal. The first column of data and the spectrum of the original background music source signal And establish timestamp alignment relationships in memory.

[0081] In one embodiment of the present invention, step S2 includes the following steps:

[0082] The relative spectral pattern distribution information of the mixed audio signal spectrum in each frequency sub-band is quantitatively analyzed using a spectral feature extraction model.

[0083] The overall activation state of the environment is judged by the top-level global classification node of the spectral feature extraction model, and the speech activity judgment flag established by the binary state parameter is output.

[0084] The probability of the presence of local speech components is predicted by the feedback feature regression nodes of the spectral feature extraction model, generating a frequency band-level speech presence confidence vector composed of a continuous floating-point probability value array.

[0085] The processor invokes a lightweight spectral feature extraction model preloaded in system memory, using the mixed audio signal spectrum generated in the previous step S1 as input to the model. It should be noted that this model is a pre-trained deep learning network layer, meaning it has been trained offline during the development phase on a dataset containing a large amount of mixed audio of speech and background music, along with their corresponding labeled data, enabling its internal weight parameters to effectively distinguish between the features of speech and background music.

[0086] The spectral feature extraction model is a computationally inefficient convolutional neural network model customized for the aforementioned multi-task features, suitable for real-time operation on resource-constrained embedded devices. Specifically, to enable the model to have dual output capabilities, the construction and training methods for the offline training dataset include: collecting clean speech and clean background music under different signal-to-noise ratio scenarios, and proportionally superimposing them at the time or frequency domain to generate an input training set containing the spectrum of the mixed audio signal; in the label construction stage, for each frame of mixed audio, if actual spoken speech exists within that frame, the global speech activity ground truth label is set to 1, otherwise it is set to 0; simultaneously, the ratio of the true frequency band energy of the individual source speech to the corresponding frequency band energy of the mixed audio is calculated and normalized to generate an array of energy distribution proportions for each frequency sub-band, which is used as the ground truth label for the frequency band-level local existence confidence; during training, a binary cross-entropy loss function is used for joint supervision and weight updates of the global and local branches.

[0087] Specifically, the amplitude information of the mixed audio signal spectrum is extracted and organized into a two-dimensional or three-dimensional tensor. For example, the amplitude spectra of 256 nodes from multiple consecutive frames are concatenated into a 5×256 matrix combination. This tensor represents the distribution of signal energy at various frequency points within a specific time window. This tensor is fed into the input layer of the spectral feature extraction model and undergoes forward propagation calculation through the multi-layer convolutional neural network layers contained within the model. In a typical and specific network structure embodiment, the core backbone network extraction layer of this multi-convolutional layer includes: firstly, a shallow feature layer is constructed by a standard 3×3 two-dimensional convolutional layer for joint feature mapping in the time and frequency domains; subsequently, to control the model parameters, a depthwise separable convolutional module containing 3 to 5 layers is cascaded backward. Each depthwise separable module consists of a 3×3 deep spatial convolutional layer, a batch normalization layer, a ReLU activation function, and a 1×1 pointwise convolutional layer connected in sequence. These convolutional layers, pre-trained with weight parameters through the aforementioned supervision mechanism, filter the input spectrogram to extract local spectral texture features that distinguish the speech signal from background music, such as harmonic structures and formants. This is the relative spectral texture feature distribution information, which refers to the energy distribution patterns of the speech signal in the spectrum that the model can learn to recognize, such as the harmonic peaks that appear in specific frequency regions and their changes over time. These patterns are significantly different from the spectral features of background music.

[0088] Following the convolutional layers, the model uses pooling layers to reduce the dimensionality of the feature maps, and finally passes the extracted deep features to two parallel output branches:

[0089] The first branch is a top-level global classification node. This node performs global average pooling on the deep feature map, then passes it through a fully connected layer and a sigmoid activation function, outputting a scalar value between zero and one. This value represents the global probability of speech existing within the current time window. This probability value is compared with a preset decision threshold to generate a binary state parameter, which is the speech activity decision flag. The speech activity decision flag is a binary variable, for example, denoted as... A value of 1 indicates that the system determines the current audio frame contains valid speech activity, while a value of 0 indicates it does not. This flag is based on the probability value output by the top-level global classification node. With the decision threshold The comparison shows that, if Greater than ,but It is 1 if it is not 0 otherwise. The setting is based on experimental statistics that balance the false alarm rate and the false alarm rate on the validation set, and is usually set between 0.5 and 0.7.

[0090] The second branch is the feedback feature regression node. This node processes deep features through an additional fully connected layer or a 1×1 convolutional layer. Its output dimension is consistent with the preset number of frequency bands, and each output value is mapped to the interval between zero and one using the sigmoid activation function, thereby generating an array of continuous floating-point probability values. This array is defined as the frequency band-level speech presence confidence vector. This frequency band-level speech presence confidence vector is a one-dimensional floating-point array, denoted as... Its dimension is equal to the number of subbands into which the spectrum is divided, for example, 64 subbands. Each element in the vector... The value of is in the range of 0 to 1, indicating that the value of is in the th... The probability or confidence level of the presence of a speech signal within a frequency sub-band.

[0091] For example, following the example of step S1, the system obtains the first frame of data of the mixed audio signal spectrum, that is, a vector containing 256 complex values. First, the processor calculates the amplitude of the spectral vector, obtaining a 256-point amplitude spectrum. Assuming the spectral feature extraction model requires input containing context from five consecutive frames, the system caches the amplitude spectra of these five frames, forming an input tensor of size 256×5. This tensor is then fed into the model for inference. After the model's forward computation, its top-level global classification node outputs the global speech activity probability value. The value is 0.92. This value is then compared with the preset decision threshold. The value is compared to 0.6. Since 0.92 is greater than 0.6, the voice activity decision flag is set. Set to 1. Meanwhile, the model's feedback feature regression node outputs a 64-dimensional frequency band-level speech presence confidence vector. In this vector, the mid-frequency bands that are strongly correlated with the fundamental frequency and formants of human speech, such as the band corresponding to 1 kHz to 4 kHz, have higher confidence values, for example, in the vector... The values ​​of the 9th to 32nd elements in the vector could be [0.88, 0.93, ..., 0.90]; while the low-frequency and high-frequency bands, which are strongly correlated with the background music, have lower confidence values, such as those in the vector. The values ​​of the first and last few elements in the middle may be [0.05, 0.12, ..., 0.08]. Finally, the processor will use the generated voice activity decision flag. Equal to 1 and the existence of a confidence vector for frequency band-level speech Store it at the specified memory address for later use in subsequent steps.

[0092] In one embodiment of the present invention, step S3 includes the following steps:

[0093] Scan the energy parameters of each frequency band of the original background music source signal spectrum, and simultaneously read in the probability value of the frequency band-level speech presence confidence vector corresponding to the position;

[0094] By inputting the probability value into a preset nonlinear mapping function, the energy penalty coefficient characterizing the degree of energy decay is derived.

[0095] The energy amplitude of the corresponding frequency band in the original background music source signal spectrum is proportionally attenuated and scaled using an energy penalty coefficient. The processed frequency band data is then combined in the original frequency order to obtain the estimated background music component spectrum.

[0096] The processor uses the frequency band-level speech presence confidence vector obtained in step S2 as dynamic weights to perform nonlinear adaptive mapping and attenuation operations on the original background music source signal spectrum obtained in step S1. The purpose is to reconstruct the spectral representation of the background music component in the estimated mixed signal. It should be noted that nonlinear adaptive mapping and attenuation is a signal processing operation that nonlinearly and adaptively adjusts the energy of each frequency band in the target signal spectrum based on an external control signal, i.e., the frequency band-level speech presence confidence vector. Specifically, the processor initiates an iterative processing flow per frequency sub-band. For each frequency sub-band in the original background music source signal spectrum, the system first reads its corresponding energy amplitude. Simultaneously, it reads the continuous floating-point probability value at the index position that perfectly corresponds to the currently processed frequency sub-band from the frequency band-level speech presence confidence vector.

[0097] Next, the probability value is input into a preset nonlinear mapping function to derive the energy penalty coefficient. In this embodiment, the energy penalty coefficient is a scaling factor between zero and one, calculated based on continuous floating-point probability values, used to determine the degree of attenuation of the original background music energy. When the continuous floating-point probability value is high, it indicates that this frequency band is highly likely to be occupied by user voice signals, and the derived energy penalty coefficient is correspondingly close to zero. The processor then multiplies this coefficient with the energy amplitude of the original background music source signal spectrum in this frequency band, thereby achieving a significant attenuation of the background music energy. Conversely, if the continuous floating-point probability value is low, it indicates that this frequency band is dominated by background music, and the derived energy penalty coefficient is close to one. After the processor performs the multiplication operation, the energy amplitude of the original background music source signal spectrum in this frequency band is basically maintained, thereby protecting the music components. Furthermore, the core calculation process of nonlinear adaptive mapping and attenuation can be described by the following formula:

[0098]

[0099] in, This step calculates the estimated spectrum of the background music component in the mixed signal at the frequency index. Time frame index The energy amplitude at that location. It is the energy amplitude of the unit at the same frequency of the spectrum of the original background music source signal. The frequency band-level speech exists in the confidence vector, along with the frequency index. The corresponding number The confidence values ​​for each frequency sub-band, here. Indicates frequency index To frequency subband index The mapping relationship, since the spectral resolution is usually higher than the band resolution of the confidence vector, involves multiple adjacent frequency indices. They will be mapped to the same frequency subband index. .index It is a non-linear shrinkage coefficient used to adjust the intensity of the penalty. When When it equals 1, it exhibits linear decay; when... When the value is greater than 1, the attenuation curve is steeper, allowing for stronger penalties to be imposed on frequency bands with medium to high confidence. Based on statistical data from experimental data of typical speech enhancement applications, in order to achieve a balance between preserving music and accurately estimating background noise, The value range is usually set to 1.5 to 2.5.

[0100] After the processor traverses all frequency sub-bands and performs attenuation or protection operations, it recombines all processed frequency band data according to the original frequency order, ultimately forming a complete two-dimensional matrix with the same dimensions as the original spectrum. This matrix is ​​the estimated spectrum of the background music component in the mixed signal. This estimated spectrum of the background music component in the mixed signal is a dynamic estimate of the spectral shape of the background music part in the actual mixed audio signal. It is not the actual separated background music, but rather the theoretical shape after attenuating the original pure background music based on the probability of speech presence.

[0101] For example, following the previous steps, the processor obtains a 64-dimensional frequency band-level speech presence confidence vector. The first frame of the original background music source signal spectrum. Assume the system has a preset nonlinear shrinkage index. It's 2.0. First, the processor processes the low-frequency portion, for example, the portion corresponding to... The second element The frequency sub-band. This sub-band covers the frequency index. arrive .for Original background music energy amplitude The system calculates its energy penalty coefficient as follows: Next, the estimated background music energy amplitude was calculated to be... This value is close to the original value of 0.75, reflecting the protection of musical components. Subsequently, the processor processes the mid-frequency range, for example, the range corresponding to... The thirty-first element The frequency sub-band may coincide with the formants of the user's speech. Assume that within this sub-band... Original background music energy amplitude The system calculates its energy penalty coefficient as follows: The calculated estimated background music energy amplitude is... This value is much smaller than the original value of 0.60, reflecting a significant attenuation of the background music energy in the frequency bands heavily polluted by speech. Finally, the processor repeats this process for all 256 frequency points, calculating the 256... The values ​​are combined to form the first frame of data for the predicted spectrum of the background music component in the mixed signal.

[0102] In one embodiment of the present invention, step S4 includes the following steps:

[0103] Under the same time frame, the actual mixed observation energy mean of the spectrum of the mixed audio signal and the reference background source energy mean of the spectrum of the original background music source signal are extracted, and the residual noise reference mean of the spectrum of the estimated background music component is obtained simultaneously.

[0104] An acoustic coupling factor is introduced, and based on the energy subtraction logic, the reference background source energy mean weighted by the acoustic coupling factor is removed from the actual mixed audio observation energy mean. The residual noise reference mean weighted by the acoustic coupling factor is used as the energy lower limit constraint to restore the approximate speech target energy intensity.

[0105] The ratio of the energy intensity of the approximate speech target to the mean energy of the reference background source after being weighted by the acoustic coupling factor is quantized and logarithmically calculated to synthesize the instantaneous frequency band signal-to-noise ratio estimate.

[0106] The processor, based on the energy separation and comparison algorithm, compares and analyzes the spectrum of the background music component in the estimated mixed signal generated in step S3 with the spectrum of the mixed audio signal generated in step S1 to quantify and determine the instantaneous frequency band signal-to-noise ratio estimate specific to each frequency sub-band. It should be understood that the energy separation and comparison algorithm here specifically refers to the processing flow of estimating speech energy through spectral subtraction and calculating its ratio to the background energy. Specifically, under the same time frame index, the processor performs the following operations for each frequency sub-band:

[0107] First, the actual observed energy value of the current sub-band is extracted from the spectrum of the mixed audio signal. This value is obtained by calculating the square of the spectral amplitude within the corresponding frequency band. This is the actual observed energy mean, which refers to the power spectral density value calculated from the complex value of the spectrum of the mixed audio signal at a specific time-frequency unit. Next, the reference background source energy mean is extracted from the spectrum of the original background music source signal. This is the reference background source energy mean, which refers to the power spectral density value calculated from the amplitude of the spectrum of the original background music source signal without attenuation at the same time-frequency unit. Simultaneously, the residual noise floor reference mean in the same frequency band of the estimated background music component spectrum is extracted.

[0108] Subsequently, following the principle of energy subtraction, the processor subtracts the reference background source energy mean after acoustic coupling factor weighting from the actual mixed observation energy value, which is the physical removal of the real echo intensity of the physical sound field. Furthermore, it uses the residual noise reference mean containing the deep learning confidence distribution to construct an adaptive tolerance protection mechanism to prevent spectral subtraction negative value truncation distortion and excessive erasure. The final result is regarded as the approximate speech target energy intensity in this frequency band.

[0109] Finally, the processor constructs an approximate ratio of the target speech energy intensity to the energy value corresponding to the spectrum of the original background music source signal after compensating for spatial propagation attenuation, and performs a logarithmic operation on this ratio to convert the linear energy ratio into a quantization parameter in the logarithmic domain. Specifically, the calculation process of the instantaneous frequency band signal-to-noise ratio estimate can be defined by the following modified formula:

[0110]

[0111] Among them, the energy intensity of the approximate speech target The calculation of the energy adaptive lower bound constraint is expanded as follows:

[0112]

[0113] Here, It is in the The instantaneous signal-to-noise ratio of each frequency sub-band. It is a mixed signal energy. It is the original background music source energy generated in step S1. It is the acoustic coupling factor, which is used as the energy gain ratio, and is directly multiplied by... And to unify the dimensions before estimating the background energy. This refers to the dynamic signal-to-noise ratio (SNR) minimum constraint established by inferring probabilities using neural networks. When the confidence level is high, its natural decline ensures the depth of denoising. For smoothing hyperparameters, for example, take 0.01. The function is used to ensure that the numerator is not negative. It is a preset noise energy lower limit, such as 1e-12, used to prevent taking the logarithm of negative numbers or zero. Its value is set according to the system's background noise level. It is a preset denominator stabilization factor, which is also a very small positive number, such as 1e-12, used to prevent division by zero errors when the background energy estimate is zero.

[0114] In one embodiment of the present invention, as a preferred implementation, the adaptive acoustic coupling factor is dynamically updated during non-speech activities. The specific algorithm logic includes:

[0115] The processor continuously monitors the voice activity decision flag output in step S2 above; when the voice activity decision flag indicates that the current frame is in a non-voice activity period, that is, it does not contain valid user voice, and the system mainly collects background music played by the speaker and environmental noise at this time, the wideband observation total energy of the mixed audio signal spectrum of the current frame is extracted. And the broadband reference total energy of the original background music source signal spectrum in the current frame. ;

[0116] Subsequently, the current acoustic coupling factor is calculated and updated using a first-order recursive smoothing algorithm. The calculation formula is as follows:

[0117]

[0118] in, Indicates the current number The acoustic coupling factor after frame update This represents the acoustic coupling factor of the previous frame; The preset time-smoothing decay constant is used to ensure the stability of the estimation and avoid short-term fluctuations. Its value range is preferably 0.90 to 0.99. This is the instantaneous ratio of the actual energy collected in the current frame to the internal reference energy. Through the above dynamic update and iteration logic, the system can enable the acoustic coupling factor to track changes in the indoor environment in real time, such as changes in the acoustic path caused by people walking or the opening and closing of doors and windows, ensuring the physical accuracy of subsequent signal-to-noise ratio calculations.

[0119] Finally, the processor combines the logarithmic results of all frequency bands covering the full spectrum into a vector, which is the final output of this step: the instantaneous frequency band signal-to-noise ratio estimate. This instantaneous frequency band signal-to-noise ratio estimate is a vector, where each element represents the ratio of the speech signal energy to the background music energy in a specific frequency band within the current analysis frame, and is logarithmically quantized in decibels.

[0120] For example, following the previous steps, we now need to calculate the instantaneous band signal-to-noise ratio estimate. Assume the acoustic coupling factor is [missing information - likely a factor in the original text]. Approximately 1.0. A dynamic lower bound smoothing factor is set. For low-frequency band frequency index The estimated background music energy amplitude has already been obtained in the previous steps. The corresponding energy is 0.3373, and the amplitude of the mixed signal energy is... Energy 0.5625; the original background music amplitude at this location is The energy is 0.5625. First, the processor, based on the new tolerance formula, evaluates the result of the subtraction as follows: The adaptive lower bound constraint is The maximum value of the two values ​​is taken to obtain an approximate speech target energy of 0.0034. Next, the physical signal-to-noise ratio for this frequency band is calculated. .

[0121] Conversely, for mid-frequency indexes that are highly likely to contain speech activity... At this point, the estimated background energy amplitude is greatly attenuated. The corresponding energy is extremely close to 0, and the mixed speech signal experiences a maximum jump in amplitude. Energy 1.0000, the original background energy remains constant. The processor evaluates the reduction through the main physical subtraction branch. Since the lower limit constraint is 0 and therefore no intervention is allowed, the value is set to 0.6400. Subsequently, the signal-to-noise ratio of the core audio band is... The processor sequentially combines the full vectors and outputs the instantaneous signal-to-noise ratio estimate.

[0122] In one embodiment of the present invention, step S5 includes the following steps:

[0123] Monitor the voice activity decision flag. When the decision flag is in the valid voice input state, start the composite evaluation for each frequency sub-band.

[0124] Determine whether the continuous floating-point probability value is greater than the first confidence threshold, and simultaneously determine whether the instantaneous frequency band signal-to-noise ratio estimate is lower than the second signal-to-noise ratio threshold;

[0125] In the case where the continuous floating-point probability value is greater than the first confidence threshold and the instantaneous frequency band signal-to-noise ratio estimate is lower than the second signal-to-noise ratio threshold, the corresponding frequency band is recorded as the target enhancement frequency band.

[0126] The processor incorporates the frequency band-level speech presence confidence vector obtained in step S2 and the instantaneous frequency band signal-to-noise ratio (SNR) estimate obtained in step S4 into a dual-condition gating mechanism model to achieve accurate identification and localization of the target enhanced frequency band. It should be noted that the dual-condition gating mechanism model is a logical judgment framework that requires two or more independent conditions to be met simultaneously to trigger subsequent operations. In this embodiment, these two conditions are the speech presence confidence condition and the SNR condition. This process is first triggered by a global activation condition. The processor continuously monitors the speech activity decision flag output in step S2. Only when the value of this flag is confirmed as a valid speech input state does the processor initiate a composite evaluation process for each independent candidate analysis frequency band. Here, the candidate analysis frequency band refers to all available frequency sub-bands for analysis before recognition. For each candidate analysis frequency band, the processor reads the continuous floating-point probability value corresponding to its position from the frequency band-level speech presence confidence vector and compares it with a first confidence threshold preset by the system. The first confidence threshold is a preset scalar value, denoted as […]. This threshold is used to filter out frequency bands with sufficient probability of speech. The threshold is set based on receiver operating characteristic curve analysis on a dataset labeled with speech and non-speech segments, seeking a balance point that achieves the highest detection rate at an acceptable false alarm rate; its typical value ranges from 0.7 to 0.9.

[0127] Simultaneously, the processor reads the signal-to-noise ratio (SNR) value at the same frequency band location from the instantaneous frequency band SNR estimation vector and compares it with a second SNR threshold preset by the system. The second SNR threshold is a preset scalar value in decibels, denoted as […]. This threshold is used to identify frequency bands where the speech signal quality has deteriorated to the point where enhancement is needed. The threshold is set based on research into the performance of speech recognition engines, determining the minimum signal-to-noise ratio level required to guarantee basic intelligibility, with a typical range of 0 dB to 5 dB.

[0128] Next, a logical AND gate jointly evaluates the two comparison results. The gate outputs a true value only when both conditions are met: the continuous floating-point probability value of a frequency band is greater than the first confidence threshold, and its signal-to-noise ratio (SNR) value is lower than the second SNR threshold. This satisfies the multidimensional intersection constraint. The indices of all candidate analysis frequency bands that output true values ​​are recorded in a dynamic list, which ultimately constitutes the set of target enhancement frequency bands eligible for activation-triggered adjustment. In this embodiment, the target enhancement frequency band refers to a specific frequency range in the mixed audio signal that is identified by the system as containing both high-probability speech components and being subject to severe background noise interference, thus requiring priority spectral enhancement processing.

[0129] For example, following the previous steps, the processor first checks the voice activity decision flag. Its value is 1, confirming that the current period is a valid voice input activation period, thus initiating the frequency band evaluation process. The system's preset first confidence threshold... The second signal-to-noise ratio threshold is 0.8. The signal-to-noise ratio (SNR) is 3 dB. First, the low-frequency band is evaluated. The continuous floating-point probability value corresponding to this band is 0.12, and the estimated instantaneous SNR is -22.2 dB. Since the probability value of 0.12 is less than the first confidence threshold of 0.8, the first condition is not met. Therefore, even though its SNR of -22.2 dB is lower than the second SNR threshold of 3 dB, this band will not be selected as the target enhancement band. Next, the mid-frequency band calculated in step S4 is evaluated. The continuous floating-point probability value corresponding to this band is 0.93, and according to the calculation result of S4, its estimated instantaneous SNR is 2.5 dB. In this case, the probability value of 0.93 is greater than the first confidence threshold of 0.8, satisfying condition one; at the same time, the SNR of 2.5 dB is lower than the second SNR threshold of 3 dB, satisfying condition two. Since both conditions are met simultaneously, the processor formally records the index number corresponding to the mid-frequency band, for example, the 31st band corresponding to the k=121 frequency point in the example of step S4, into the list of target enhancement bands. After the processor traverses all 64 frequency bands, the list contains information on all frequency bands that require enhancement processing.

[0130] In one embodiment of the present invention, step S6 includes the following steps:

[0131] Calculate the difference offset between the estimated instantaneous frequency band signal-to-noise ratio within the target enhancement band and the target preferred by the predetermined sharpness index;

[0132] The amplitude of the nonlinear boost coefficient required to map the target frequency sub-band based on the difference offset;

[0133] The nonlinear boosting coefficient amplitude is assigned to the position corresponding to the target enhanced frequency band, and the gain value of the non-target enhanced frequency band is set as a constant coefficient to assemble and generate a spectrum gain mask.

[0134] The processor performs independent compensation parameter verification calculations for all target enhancement frequency bands identified in step S5, and constructs a spectral gain mask based on these parameters. Finally, this mask is loaded onto the mixed audio signal spectrum to derive the enhanced spectrum. The spectral gain mask is a real-number matrix with the same size as the spectrum to be processed, where each element represents a gain coefficient applied to the corresponding time-frequency unit. In this embodiment, the independent compensation parameters are represented as the magnitude of the nonlinear boost coefficient calculated for each target frequency band.

[0135] Specifically, the processor first iterates through the list of target enhancement frequency bands. For each frequency band in the list, the processor extracts its dedicated instantaneous frequency band signal-to-noise ratio estimate calculated in step S4, and compares this value with a system-preset, predetermined clarity index preference target representing ideal speech clarity, calculating the difference offset between the two. This is the minimum expected speech signal-to-noise ratio (SNR) of the system. Its setting is based on the minimum SNR condition required to ensure the subsequent speech recognition engine achieves a specific recognition accuracy, and is typically set to 15 dB to 20 dB. The difference offset refers to... With actual measurement The difference between them.

[0136] Subsequently, the processor uses this difference offset as an independent variable and substitutes it into a preset gain mapping function to calculate the required nonlinear boost coefficient amplitude for the target frequency sub-band. After completing the coefficient calculation for all target enhancement frequency bands, the processor initializes a full-band gain mask matrix with the same spectral dimension as the mixed audio signal and sets the initial values ​​of all elements to unmodified direct pass-through constant coefficients. These unmodified direct pass-through constant coefficients refer to a gain value of 1.0, indicating that no amplitude modification is performed on the corresponding frequency band. Next, the processor updates the nonlinear boost coefficient amplitudes calculated in the previous step to the positions in the mask matrix corresponding to the target enhancement frequency bands. In this embodiment, the nonlinear boost coefficient amplitude... The calculation of spectral gain mask and its application can be described by the following set of formulas:

[0137] First, calculate the gain value for each target frequency band, i.e., the magnitude of the nonlinear boost factor. :

[0138]

[0139] This formula applies only to frequency bands identified as the target enhancement band. For other non-target enhancement frequency bands, their gain values... It is always equal to 1. In the formula, It is in the The first time frame, the first Gain value of frequency sub-band. It is a predetermined clarity index preference target. It is the signal-to-noise ratio of the corresponding frequency-time unit obtained from the instantaneous frequency band signal-to-noise ratio estimate. It is a gain control factor used to control the sensitivity slope of the mapping from logarithmic domain gain compensation to linear amplitude enhancement. Its value is adjusted based on auditory perception experiments and typically ranges from 0.1 to 0.5. It is the maximum gain limit, which is the cap on the gain value to prevent excessive amplification of frequency bands with extremely low signal-to-noise ratios, thus preventing the generation of unnatural musical noise. Its value is usually set to 8 to 16 times the linear amplitude, corresponding to a gain of about 18 to 24 dB.

[0140] Then, this gain mask is applied to the mixed signal spectrum to obtain the enhanced spectrum. :

[0141]

[0142] in, It is the frequency-time unit of the enhanced spectrum output. Complex values ​​at the location. It is the complex value of the spectrum of the input mixed audio signal at the same position. Represents frequency index The frequency sub-band index to which it belongs.

[0143] Finally, the processor performs an array dot product operation on the corresponding elements of the assembled spectral gain mask and the complex value of the mixed audio signal spectrum. This operation applies the gain factor to the amplitude of the spectrum, thereby stretching the energy of the weak subband of the core speech, completing the correction and outputting the enhanced spectrum.

[0144] For example, continuing from the previous example, the processor has identified the mid-frequency band with index number 31 as the target enhancement band, and the instantaneous signal-to-noise ratio estimate for this band is 2.5 dB. The system's preset parameters are: a predetermined sharpness index preference target. 15 dB, gain control factor The maximum gain is 0.3. The value is 8.0. First, the processor calculates the difference offset for this frequency band as... dB. Next, the initial nonlinear boost factor magnitude is calculated as follows: Since 1.54 is less than the maximum gain limit of 8.0, the gain value for this frequency band is ultimately determined to be 1.54. Subsequently, the processor creates a 64-dimensional spectral gain mask vector, with its 31st element set to 1.54, while all other 63 elements are set to the unmodified pass-through constant coefficient of 1.0. Finally, the processor applies this mask to the first frame of the mixed audio signal spectrum. For all frequency points belonging to the frequency band corresponding to index number 31. ,For example 121, its corresponding complex spectral value will be multiplied by 1.54. Assume... The value is If its amplitude is 0.8, then the enhanced spectral value is... Will become Its amplitude increased to 1.232. For frequency points that do not belong to the target enhancement band, their spectral values ​​remain unchanged. After processing all frequency points, the first frame of enhanced spectrum data is obtained.

[0145] In one embodiment of the present invention, step S7 includes the following steps:

[0146] The original phase information of the mixed audio signal spectrum is preserved, and an inverse short-time Fourier transform is performed in combination with the enhanced amplitude information to obtain the time-domain signal frame.

[0147] A sequential digital audio stream is reconstructed by overlapping and adding consecutive time-domain signal frames.

[0148] The frequency band intervals that exhibit continuous probability values ​​in the confidence vector of frequency band-level speech are identified, and feature indicator code blocks containing frequency band index information are generated.

[0149] A dynamic modification command containing feature indicator code block information is issued through the reverse hardware communication channel, driving the speaker to adjust the drive current parameters within the frequency band covered by the feature indicator code block.

[0150] The processor executes two parallel tasks: synthesizing the enhanced audio and establishing a feedback control loop to the playback end. In the audio synthesis task, the processor receives the corrected enhanced spectrum output from step S6. Since the array multiplication operation in step S6 only changes the amplitude of the spectrum, the enhanced spectrum completely retains the original phase information from the mixed audio signal spectrum in step S1. The processor substitutes this complex spectrum data, containing the updated amplitude and the original phase, into the overlap-preserving structure computing platform, performing an inverse short-time Fourier transform on each time frame to decode the frequency domain data back into a time domain signal frame. Subsequently, by performing overlap and addition on consecutive time domain signal frames, a temporally continuous and smooth digital audio stream is regenerated. This audio stream is the final output target speech command segment, which can be directly transmitted to the front-end semantic recognition server. It should be noted that the inverse time-domain dimension transformation synthesis process is mainly implemented by the inverse short-time Fourier transform and the overlap and addition method. Its core reconstruction formula is as follows:

[0151]

[0152] in, It is the final synthesized digital audio stream. It is the frame shift step size. It is a composite window function. Indicates the first The inverse transform result of the frame is aligned and shifted on the time axis and superimposed onto the corresponding position in the output sequence.

[0153] In the feedback control task, the processor separates and extracts the frequency band-level speech presence confidence vector generated in step S2. The processor scans this vector to identify intervals consisting of multiple consecutive frequency bands with probability values ​​all higher than a specific third confidence threshold, and combines the start and end frequency band indices of these intervals into one or more feature indicator code blocks. This process establishes the frequency band selection power adjustment feedback inner loop network, a closed-loop control system in which the speech detection results from the microphone acquisition end are used to adjust the output power of specific frequency bands at the speaker playback end in real time. The identified "extremely dense, continuous, high-probability detection statistical pattern" refers to the phenomenon in the frequency band-level speech presence confidence vector where the confidence values ​​of multiple consecutive frequency bands all exceed a certain high threshold, which typically corresponds to a complete formant structure in the speech signal. The feature indicator code block is a data structure used to encapsulate the frequency range information that needs power attenuation, such as a tuple containing start and end frequency band indices.

[0154] Finally, the processor, through the device's internal reverse hardware communication channel, sends a dynamic modification command containing feature indicator code block information to the logical register area of ​​the local sound playback controller. This forces the controller to selectively attenuate the currently playing internal playback source data in a frequency band that coincides with the frequency boundary covered by the feature indicator code block. To address the timing delay caused by the processing time of the algorithm link and to avoid discontinuous sonic pop distortion caused by transient frequency band gain changes, the speaker introduces a preset time smoothing window when performing the limiting adjustment to reduce playback power, such as a transition time set to 10ms~20ms. This dynamically transitions to the target current-limiting state through envelope gradation, thereby predicting and reducing musical interference emitted by the speaker in the relevant audio segments of the next time period while ensuring a smooth subjective listening experience for the remaining background music. It should be noted that the reverse device-level hardware communication channel refers to the bus within the device used to send control commands from the main processor to peripherals such as audio codecs or digital signal processors, such as the I2C or SPI bus. The implementation of reducing current limiting by reducing drive current parameters at the signal processing level involves applying an adjustable multi-band equalizer or a set of band-stop filters to the audio playback signal, applying a preset attenuation amount to the specified frequency band. The setting of this attenuation amount needs to strike a balance between effectively reducing interference and avoiding excessive impact on the music experience. The typical attenuation range is -6dB to -12dB.

[0155] For example, following the example of step S6, the processor receives the first frame of the enhanced spectrum. First, the processor performs a 512-point inverse fast Fourier transform on the frame to obtain a 512-point time-domain signal frame. This frame is then added to the corresponding position in the output buffer and summed with the 256 overlapping samples from the previous frame's processing result, thus smoothly reconstructing the final enhanced audio stream. Simultaneously, the processor analyzes the frequency band-level speech existence confidence vector from step S2. Assuming the third confidence threshold is 0.85 and the consecutive frequency band threshold is 3, the processor finds that the confidence levels of frequency bands 7, 8, 9, and 10 in the vector are [0.89, 0.93, 0.91, 0.88], all higher than 0.85 and appearing consecutively for four times. Therefore, the system generates a feature indicator code block with the content (7, 10). This code block is encoded into an instruction. Subsequently, the processor sends this instruction to the audio DSP via the I2C bus. Upon receiving the instruction, the audio DSP immediately applies a -9 dB attenuation filter to the frequency range corresponding to frequency bands 7 to 10, approximately 875 Hz to 1250 Hz, on its internal playback signal processing link. Therefore, at the next moment, the volume of music played by the speaker within this frequency range will be significantly reduced, creating a more favorable acoustic environment for the microphone to capture user speech in the next frame, thus achieving feedback closed-loop control.

[0156] See appendix Figure 2 The present invention also proposes a speaker intelligent audio signal enhancement system based on voice activity detection, comprising the following modules:

[0157] The signal acquisition and transformation module is used to acquire the mixed acquisition audio stream and the internal playback source data with synchronized hardware adjustment status, and generate the spectrum of the mixed audio signal and the spectrum of the original background music source signal through time-frequency domain transformation;

[0158] The speech activity analysis module is used to perform feature analysis on the spectrum of mixed audio signals and output a speech activity decision flag representing the global speech state and a frequency band-level speech presence confidence vector representing the local probability.

[0159] The background spectrum estimation module is used to map the energy penalty coefficient to the frequency band-level speech existence confidence vector, perform nonlinear adaptive mapping and attenuation on the spectrum of the original background music source signal, and generate the estimated spectrum of the background music components.

[0160] The signal-to-noise ratio (SNR) calculation module is used to compare the spectrum of the mixed audio signal, the spectrum of the original background music source signal, and the spectrum of the estimated background music components, and to generate the instantaneous frequency band SNR estimate for each frequency sub-band through an energy separation algorithm.

[0161] The target frequency band identification module is used to introduce the speech activity decision flag, the frequency band-level speech presence confidence vector and the instantaneous frequency band signal-to-noise ratio estimate into a dual-condition gating mechanism to identify and determine the target enhancement frequency band.

[0162] The spectrum enhancement module determines independent compensation parameters based on the instantaneous frequency band signal-to-noise ratio estimate and the target enhancement frequency band, constructs a spectrum gain mask, applies the spectrum gain mask to the spectrum of the mixed audio signal, and derives the enhanced spectrum.

[0163] The audio synthesis and feedback module is used to perform an inverse transformation on the enhanced spectrum to generate a time-series continuous digital audio stream, and simultaneously generate feedback control commands based on the frequency band-level speech presence confidence vector to reduce the playback power of the speaker in a specific frequency band.

[0164] Each of the modules can be implemented in whole or in part through software, hardware, or a combination thereof. It supports hardware embedded in or independent of the processor in the computer device, and also supports software stored in the memory of the computer device, so that the processor can call and execute the operations corresponding to each of the above modules.

[0165] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for intelligent audio signal enhancement for loudspeakers based on speech activity detection, characterized in that, Includes the following steps: S1. Acquire the mixed audio stream and the internal playback source data of the synchronized hardware adjustment state, and generate the spectrum of the mixed audio signal and the spectrum of the original background music source signal through time-frequency domain conversion; S2. Perform feature analysis on the spectrum of the mixed audio signal, and output a speech activity decision flag representing the global speech state and a frequency band-level speech presence confidence vector representing the local probability. S3. Map the frequency band-level speech existence confidence vector to the energy penalty coefficient, perform nonlinear adaptive mapping and attenuation on the spectrum of the original background music source signal, and generate the estimated spectrum of the background music components. S4. Compare the spectrum of the mixed audio signal, the spectrum of the original background music source signal, and the spectrum of the estimated background music components, and use the energy separation algorithm to calculate the instantaneous frequency band signal-to-noise ratio estimate of each frequency sub-band. S5. The speech activity decision flag, the frequency band-level speech presence confidence vector and the instantaneous frequency band signal-to-noise ratio estimate are introduced into a dual-condition gating mechanism to identify and determine the target enhancement frequency band. S6. Based on the instantaneous frequency band signal-to-noise ratio estimate and the target enhanced frequency band, determine the independent compensation parameters and construct the spectrum gain mask. Apply the spectrum gain mask to the spectrum of the mixed audio signal to derive the enhanced spectrum. S7. Perform an inverse transformation on the enhanced spectrum to generate a time-series continuous digital audio stream, and simultaneously generate feedback control commands based on the frequency band-level speech presence confidence vector to reduce the playback power of the speaker in a specific frequency band.

2. The intelligent audio signal enhancement method for loudspeakers based on speech activity detection according to claim 1, characterized in that, Step S1 includes the following steps: The acquired mixed audio stream and the internal playback source data of the device are subjected to frame-by-frame windowing processing using a preset window function to generate windowed mixed audio frames and windowed playback source data frames. Short-time Fourier transform is performed on each frame of the windowed signal to extract the spectrum of the mixed audio signal containing the time-slice frequency domain energy distribution characteristics and the spectrum of the original background music source signal; A global frame index counter is used to align the spectrum of the mixed audio signal with the spectrum of the original background music source signal in the time dimension.

3. The intelligent audio signal enhancement method for loudspeakers based on speech activity detection according to claim 1, characterized in that, Step S2 includes the following steps: The relative spectral pattern distribution information of the mixed audio signal spectrum in each frequency sub-band is quantitatively analyzed using a spectral feature extraction model. The overall activation state of the environment is judged by the top-level global classification node of the spectral feature extraction model, and the speech activity judgment flag established by the binary state parameter is output. The probability of the presence of local speech components is predicted by the feedback feature regression nodes of the spectral feature extraction model, generating a frequency band-level speech presence confidence vector composed of a continuous floating-point probability value array.

4. The intelligent audio signal enhancement method for loudspeakers based on speech activity detection according to claim 1, characterized in that, Step S3 includes the following steps: Scan the energy parameters of each frequency band of the original background music source signal spectrum, and simultaneously read in the probability value of the frequency band-level speech presence confidence vector corresponding to the position; By inputting the probability value into a preset nonlinear mapping function, the energy penalty coefficient characterizing the degree of energy decay is derived. The energy amplitude of the corresponding frequency band in the original background music source signal spectrum is proportionally attenuated and scaled using an energy penalty coefficient. The processed frequency band data is then combined in the original frequency order to obtain the estimated background music component spectrum.

5. The intelligent audio signal enhancement method for loudspeakers based on speech activity detection according to claim 1, characterized in that, Step S4 includes the following steps: Under the same time frame, the actual mixed observation energy mean of the spectrum of the mixed audio signal and the reference background source energy mean of the spectrum of the original background music source signal are extracted, and the residual noise reference mean of the spectrum of the estimated background music component is obtained simultaneously. An acoustic coupling factor is introduced, and based on the energy subtraction logic, the reference background source energy mean weighted by the acoustic coupling factor is removed from the actual mixed audio observation energy mean. The residual noise reference mean weighted by the acoustic coupling factor is used as the energy lower limit constraint to restore the approximate speech target energy intensity. The ratio of the energy intensity of the approximate speech target to the mean energy of the reference background source after being weighted by the acoustic coupling factor is quantized and logarithmically calculated to synthesize the instantaneous frequency band signal-to-noise ratio estimate.

6. The intelligent audio signal enhancement method for loudspeakers based on speech activity detection according to claim 1, characterized in that, Step S5 includes the following steps: Monitor the voice activity decision flag. When the decision flag is in the valid voice input state, start the composite evaluation for each frequency sub-band. Determine whether the continuous floating-point probability value is greater than the first confidence threshold, and simultaneously determine whether the instantaneous frequency band signal-to-noise ratio estimate is lower than the second signal-to-noise ratio threshold; In the case where the continuous floating-point probability value is greater than the first confidence threshold and the instantaneous frequency band signal-to-noise ratio estimate is lower than the second signal-to-noise ratio threshold, the corresponding frequency band is recorded as the target enhancement frequency band.

7. The intelligent audio signal enhancement method for loudspeakers based on speech activity detection according to claim 1, characterized in that, Step S6 includes the following steps: Calculate the difference offset between the estimated instantaneous frequency band signal-to-noise ratio within the target enhancement band and the target preferred by the predetermined sharpness index; The amplitude of the nonlinear boost coefficient required to map the target frequency sub-band based on the difference offset; The nonlinear boosting coefficient amplitude is assigned to the position corresponding to the target enhanced frequency band, and the gain value of the non-target enhanced frequency band is set as a constant coefficient to assemble and generate a spectrum gain mask.

8. The intelligent audio signal enhancement method for loudspeakers based on speech activity detection according to claim 1, characterized in that, Step S7 includes the following steps: The original phase information of the mixed audio signal spectrum is preserved, and an inverse short-time Fourier transform is performed in combination with the enhanced amplitude information to obtain the time-domain signal frame. A sequential digital audio stream is reconstructed by overlapping and adding consecutive time-domain signal frames. The frequency band intervals that exhibit continuous probability values ​​in the confidence vector of frequency band-level speech are identified, and feature indicator code blocks containing frequency band index information are generated. A dynamic modification command containing feature indicator code block information is issued through the reverse hardware communication channel, driving the speaker to adjust the drive current parameters within the frequency band covered by the feature indicator code block.

9. The intelligent audio signal enhancement method for loudspeakers based on speech activity detection according to claim 8, characterized in that, The generation of feature indicator code blocks includes: Based on the preset third confidence threshold and the preset consecutive frequency band number threshold, search for frequency band sequences in the frequency band-level speech existence confidence vector that satisfy the condition that all consecutive probability values ​​are higher than the third confidence threshold; Extract the start and end frequency band indices of the frequency band sequence and encapsulate them to generate feature indicator code blocks.

10. A speaker intelligent audio signal enhancement system based on voice activity detection, characterized in that, Includes the following modules: The signal acquisition and transformation module is used to acquire the mixed acquisition audio stream and the internal playback source data with synchronized hardware adjustment status, and generate the spectrum of the mixed audio signal and the spectrum of the original background music source signal through time-frequency domain transformation; The speech activity analysis module is used to perform feature analysis on the spectrum of mixed audio signals and output a speech activity decision flag representing the global speech state and a frequency band-level speech presence confidence vector representing the local probability. The background spectrum estimation module is used to map the energy penalty coefficient to the frequency band-level speech existence confidence vector, perform nonlinear adaptive mapping and attenuation on the spectrum of the original background music source signal, and generate the estimated spectrum of the background music components. The signal-to-noise ratio (SNR) calculation module is used to compare the spectrum of the mixed audio signal, the spectrum of the original background music source signal, and the spectrum of the estimated background music components, and to generate the instantaneous frequency band SNR estimate for each frequency sub-band through an energy separation algorithm. The target frequency band identification module is used to introduce the speech activity decision flag, the frequency band-level speech presence confidence vector and the instantaneous frequency band signal-to-noise ratio estimate into a dual-condition gating mechanism to identify and determine the target enhancement frequency band. The spectrum enhancement module determines independent compensation parameters based on the instantaneous frequency band signal-to-noise ratio estimate and the target enhancement frequency band, constructs a spectrum gain mask, applies the spectrum gain mask to the spectrum of the mixed audio signal, and derives the enhanced spectrum. The audio synthesis and feedback module is used to perform an inverse transformation on the enhanced spectrum to generate a time-series continuous digital audio stream, and simultaneously generate feedback control commands based on the frequency band-level speech presence confidence vector to reduce the playback power of the speaker in a specific frequency band.

Images