A method and apparatus for enhancing sound effects

Abstract

The application relates to a sound effect enhancement method and device. The method comprises the following steps: performing time-frequency feature extraction and time-frequency feature masking on original audio, and performing time-frequency feature separation to obtain background audio spectrum and vocal spectrum; extracting an observation sequence according to the background audio spectrum, performing time sequence dynamic scene calculation on the observation sequence by using a hidden Markov model, and obtaining a continuously changing scene state vector; performing RAG sound effect retrieval based on acoustic fingerprint double constraints of the background audio spectrum, and generating a target enhancement sound effect; performing multi-frequency band side chain on the original background audio spectrum and the enhancement sound effect to obtain a gain adjustment enhancement sound effect; positioning the original sound image according to the background audio feature, recalculating a signal-to-mask ratio according to space rendering of the scene state vector, and finally performing frequency domain fusion on the signal-to-mask ratio and the vocal spectrum. The application realizes dynamic, intelligent and seamless deep fusion of the enhancement sound effect and the original audio in the time, frequency spectrum and space dimensions, and significantly improves the immersion, authenticity and clarity of sound enhancement.

Description

Technical Field

[0001] This application relates to the field of audio signal processing and sound enhancement technology, specifically to a method and apparatus for enhancing sound effects. Background Technology

[0002] In multimedia applications such as film, games, and music playback, sound enhancement processing is often required to improve the auditory experience and create an immersive sound field environment that better matches the visual content. Traditional sound enhancement techniques typically process mixed audio directly or rely on simple semantic tag retrieval and overlay of pre-made environmental sound effects. These methods often face the following technical problems when processing complex audio containing human voices: First, human voices overlap with environmental audio segments in the mixed audio, potentially causing severe interference with scene perception and leading to inaccurate scene recognition. Second, sound effect retrieval based solely on semantic tags easily overlooks the matching degree between the enhanced sound and the original sound in terms of acoustic physical characteristics, resulting in a disconnect between the retrieved enhanced sound effects and the original sound environment, resulting in a harsh listening experience. Third, directly overlaying enhanced sound effects with the original sound may cause conflicts in frequency bands where spectral energy overlaps, resulting in muddy and distorted sound. Finally, if the spatial sound image distribution of the enhanced sound effects is inconsistent with the sound field positioning of the original background sound, it can create a sense of spatial disjointedness, disrupting the immersive experience. Therefore, how to achieve dynamic perception, physical adaptation, and seamless integration of sound enhancement has become an urgent technical problem to be solved. Summary of the Invention

[0003] One of the purposes of this application is to provide a sound enhancement method and apparatus to solve the technical problems in the prior art, such as the interference of human voices in scene perception, the disconnect between sound effect retrieval and physical environment, the spectrum conflict caused by sound effect superposition, and the fragmentation of spatial sound image distribution, which result in unnatural and unimmersive sound enhancement effects. This application achieves dynamic, intelligent and seamless deep integration of enhanced sound effects and original audio in the time, spectrum and spatial dimensions.

[0004] To solve the above-mentioned technical problems, the embodiments of the present invention are implemented as follows: Firstly, a method for enhancing sound effects is provided, the steps of which include: Step S001: Obtain the original audio frequency, extract and mask the time-frequency features of the original audio frequency, and separate the time-frequency features to obtain the background audio spectrum and the human voice spectrum; Step S002: Extract the observation sequence based on the background audio spectrum, and use the Hidden Markov Model to perform time-series dynamic scene calculation on the observation sequence to obtain a continuously changing scene state vector; Step S003: Based on the acoustic fingerprint dual constraint of the background audio spectrum, perform RAG sound effect retrieval and generate target enhanced sound effect. Combine the original background audio spectrum with the enhanced sound effect through multi-band sidechaining to obtain gain-adjusted enhanced sound effect. Step S004: Based on the background sound features, perform original sound image localization, then perform spatial rendering based on the scene state vector, calculate the signal-to-mask ratio, and finally perform frequency domain fusion with the human voice spectrum to obtain the final output audio.

[0005] The second aspect also discloses a sound enhancement device, which includes: The spectrum separation module is used to acquire the original audio frequency, extract and mask its time-frequency features, and separate the time-frequency features to obtain the background audio spectrum and the human voice spectrum. The scene state vector module is used to extract the observation sequence based on the background audio spectrum, and to perform time-series dynamic scene calculation on the observation sequence using a hidden Markov model to obtain a continuously changing scene state vector. The sound enhancement module is used to perform RAG sound effect retrieval based on the acoustic fingerprint dual constraint of the background audio spectrum and generate the target enhanced sound effect. The original background audio spectrum and the enhanced sound effect are side-chained in multiple frequency bands to obtain the gain-adjustable enhanced sound effect. The frequency domain fusion module is used to locate the original sound image based on the background sound features, then perform spatial rendering based on the scene state vector and calculate the signal-to-mask ratio, and finally perform frequency domain fusion with the human voice spectrum to obtain the final output audio.

[0006] The beneficial effects of the above-described technical solutions provided in the embodiments of the present invention include at least the following: In the sound enhancement method and apparatus of this invention, firstly, an ideal ratio mask is predicted using a deep neural network to accurately separate the original audio frequency into pure background sound and human voice, eliminating the interference of human voice on subsequent scene perception from the source; then, a hidden Markov model is used to calculate the temporal features of the background sound to obtain a continuously changing scene state vector, realizing dynamic and smooth scene perception and avoiding abrupt sound effect switching; finally, a dual-constraint retrieval (RAG) combining scene semantics and background sound physical acoustic features is performed, and a lightweight neural acoustic... The encoder performs rapid feature transfer to generate target enhanced sound effects, ensuring that the enhanced sound effects are highly consistent with the original scene in terms of semantics and acoustic physics. Simultaneously, a multi-band sidechain dynamic gain mechanism is used to adaptively adjust the gain of the enhanced sound effects based on the energy of each frequency band of the original background sound, effectively avoiding spectral conflicts and local masking. Finally, based on the original sound image localization and scene state vector, the enhanced sound effects are dynamically spatially rendered to align with the spatial distribution of the original sound field. The signal-to-mask ratio is calculated based on a psychoacoustic model, and adaptive avoidance fusion of the enhanced sound effects in the frequency domain is achieved through dynamic fusion weights. Therefore, this application fundamentally solves the prior art problems mentioned in the background, achieving dynamic, intelligent, and seamless deep fusion of enhanced sound effects and original audio in the temporal, spectral, and spatial dimensions, significantly improving the immersiveness, realism, and clarity of sound enhancement.

[0007] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0008] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0009] Figure 1 A flowchart illustrating a sound effect enhancement method provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating time-frequency feature separation according to an embodiment of the present invention; Figure 3 This is a flowchart illustrating the process of obtaining the scene state vector in an embodiment of the present invention. Figure 4 This is a flowchart illustrating the adjustment and enhancement of sound effects in an embodiment of the present invention; Figure 5 This is a schematic diagram of the sound enhancement method of the present invention. Detailed Implementation

[0010] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0011] The embodiments of this application can be applied to supply chain management scenarios for various enterprises, especially those requiring the integration of multiple heterogeneous supply chain systems and intelligent demand matching and global optimization. For ease of understanding, key terms involved in this application are first explained. These explanations are only for illustrating their meaning in the context of this application and should not be construed as an undue narrowing of the scope of the claims.

[0012] Before describing the specific embodiments in detail, some key terms used in the claims of this application will be explained first: Time-frequency characteristics: refers to the spectral representation obtained after converting a time-domain audio signal to the frequency domain, which includes information such as the energy and phase of the signal at different frequencies in continuous time frames.

[0013] Ideal ratio masking: refers to a computational framework used to isolate a specific sound source from a mixed audio. It is usually a matrix with values ​​ranging from 0 to 1, where each value represents the ideal proportion of the target sound source energy to the mixed energy in a specific time-frequency unit.

[0014] Hidden Markov Model: A statistical model used to describe Markov processes containing hidden unknown parameters. In this application, it is used to estimate the implicit scene state and its evolution probability over time based on observable acoustic sequences.

[0015] Scene state vector: refers to a multi-dimensional probability distribution vector, where each dimension represents the posterior probability of a predefined scene type. It can continuously and dynamically describe the environment category in which the current audio is located, such as indoor, outdoor, wind and snow.

[0016] Acoustic fingerprint: refers to the set of quantitative features extracted from an audio signal that can characterize its physical acoustic environment, such as reverberation time, spectral centroid, spectral flux, etc., rather than semantic labels.

[0017] RAG sound effect retrieval: refers to a method of retrieving enhanced generation (RAG) sound effects, which retrieves basic sound effect resources related to the current scene from the database through joint constraints.

[0018] Multi-band sidechain: refers to a dynamic processing technique that divides the spectrum of an audio signal into multiple frequency bands and uses the energy envelope of one signal (the sidechain signal, in this case, the original background sound) to independently control the dynamic gain of another signal (in this case, the enhanced sound effect) in each frequency band.

[0019] Before describing the embodiments of the present invention in detail, the design concept of the present invention will be summarized below.

[0020] This invention provides a method for enhancing sound effects, the specific implementation of which is as follows. Figure 1 As shown, the core of the inventive concept of this invention includes: firstly, performing time-frequency transformation and deep soundscape separation on the original audio frequency to accurately separate the pure background sound and human voice; then, using a hidden Markov model to perform temporal dynamic scene perception on the background sound and capture the scene evolution trajectory; subsequently, based on acoustic feature dual-constraint retrieval enhancement generation (RAG) and multi-band dynamic gain mechanism, synthesizing enhanced sound effects that fit the current scene and have appropriate spectral energy; finally, introducing sound image distribution alignment and psychoacoustic masking effects to adaptively avoid and fuse the enhanced sound effects with the original audio in the spatial and frequency domains, thereby achieving a leap in sound enhancement from static direct superposition to dynamic perception generation, spatial alignment, and psychoacoustic adaptive fusion.

[0021] First embodiment: Please refer to the first aspect. Figure 1A flowchart illustrating methods for enhancing sound effects is shown. For example... Figure 1 As shown in the figure, this application embodiment provides a method for enhancing sound effects, the method comprising the following steps: Step S001: Obtain the original audio frequency, extract and mask the time-frequency features of the original audio frequency, and separate the time-frequency features to obtain the background audio spectrum and the human voice spectrum; Step S002: Extract the observation sequence based on the background audio spectrum, and use the Hidden Markov Model to perform time-series dynamic scene calculation on the observation sequence to obtain a continuously changing scene state vector; Step S003: Based on the acoustic fingerprint dual constraint of the background audio spectrum, perform RAG sound effect retrieval and generate target enhanced sound effect. Combine the original background audio spectrum with the enhanced sound effect through multi-band sidechaining to obtain gain-adjusted enhanced sound effect. Step S004: Based on the background sound features, perform original sound image localization, then perform spatial rendering based on the scene state vector, calculate the signal-to-mask ratio, and finally perform frequency domain fusion with the human voice spectrum to obtain the final output audio.

[0022] In this embodiment, step S001 first deeply decouples the mixed audio in the time spectrum, accurately separating the pure background sound component and the human voice component, providing a clean signal source for subsequent processing in different directions. Then, step S002 focuses on the separated background sound, using a Hidden Markov Model to dynamically calculate its temporal features, thereby transforming discrete scene recognition into a continuous, smoothly evolving scene state vector to perceive dynamic changes in the environment. Next, step S003 uses scene semantics and background sound physical acoustic features for dual constraints, ensuring that the retrieved enhanced sound effect not only conforms to scene semantics but also highly matches the original environment in acoustic characteristics. Furthermore, through a multi-band sidechain mechanism, the gain of the enhanced sound effect is adaptively adjusted according to the real-time energy distribution of the original background sound, avoiding spectral conflicts. Finally, step S004 performs spatial rendering on the enhanced sound effect based on the original sound image localization and scene state, aligning it with the original sound field in spatial orientation. By calculating the signal-to-mask ratio, which reflects the degree of perceived masking, the frequency domain fusion weight is driven, enabling the enhanced sound effect to intelligently avoid and merge with the original sound in the frequency domain region that is sensitive to the human ear, ultimately outputting high-definition, highly immersive enhanced audio.

[0023] For example: Imagine a user watching a movie where the protagonist walks out of a secluded basement into a blizzard. The original audio signal contains the protagonist's footsteps, voice, and ambient sounds. First, this original audio signal is acquired, and through time-frequency feature extraction and masking, the muffled background audio spectrum of the basement is separated from the clear human voice spectrum. Next, observation sequences are extracted from the changing background audio spectrum. A Hidden Markov Model (HMM) detects the increasing wind and ambient energy, and its output scene state vector smoothly transitions from the closed-room state to the blizzard state. Then, based on the current high-probability blizzard scene state vector and the short reverberation acoustic fingerprint reflected in the background audio spectrum, the best-matching blizzard sound effect is retrieved from the database as a reference, and the target enhanced sound effect is generated. Simultaneously, a gain adjustment mechanism analyzes the energy of the original background sound. If the original background sound already contains strong low-frequency components, the low-frequency gain of the enhanced sound effect is automatically reduced to prevent energy buildup. Finally, the original background sound characteristics were analyzed to determine that the sound of wind and snow should be located on the left side of the screen, and the enhanced sound effects were spatially rendered so that they came out from the left channel. During the blending process, the masking threshold of the human ear for the current sound was calculated. When the main character speaks, the enhanced sound effects in the corresponding frequency band are intelligently reduced to avoid drowning out the human voice. In the gaps between wind sounds or in the high-frequency parts without dialogue, the enhanced sound effects are fully released, ultimately outputting an immersive audio with a realistic sound field and clear dialogue.

[0024] In a further embodiment, regarding the aforementioned step S001, step S001 specifically includes: Step S011: Obtain the original audio frequency, which includes background noise and human voice; extract time-frequency features from the original audio frequency using short-time Fourier transform to obtain the complex spectrum matrix of the mixed audio.

[0025] In this embodiment, the raw audio signal input from the player is received. Performing a short-time Fourier transform (STFT) on the mixture yields the complex spectral matrix of the mixed audio. The formula is: .in, For time frames, For frequency slots, the complex spectrum matrix It contains phase and amplitude information of the mixed audio.

[0026] Step S012: Mask the complex spectrum matrix by predicting the ideal ratio of time-frequency features through a deep neural network to obtain the masking matrix.

[0027] In the example, the complex spectrum matrix The amplitude spectrum is input into the trained deep neural network to predict the ideal ratio mask for the background sound. The prediction formula for the ideal ratio mask is: , in, The background noise masking matrix is ​​for prediction, with a value range of [0,1]. Use the Sigmoid activation function; and These are the trainable weight parameters and biases of the neural network; The amplitude of the mixed spectrum.

[0028] Step S013: Separate the masking matrix using the separation formula to obtain the separated background audio spectrum and human voice spectrum.

[0029] In a further embodiment, the separation formula is:

[0030] in, This is the separated background audio spectrum. For the separated human voice spectrum, ⊙ denotes element-wise multiplication. The embodiment defines the specific mathematical form of the masking and separation process. In the masking prediction formula, after linearly weighting and biasing the amplitude of the mixed spectrum, it is compressed to the [0,1] interval via the Sigmoid function. Its output value can smoothly and continuously represent the probability of the presence of background sound, rather than a hard binary determination. This helps to achieve a more natural transition in critical regions where sound overlap is severe. The trainable weights of the neural network... and bias Driven by massive amounts of data, it is possible to learn the deep spectral differences between background noise and human voice in complex acoustic environments. In the separation formula, multiplying the mixed spectrum element-wise with the masking matrix and its probability complement is equivalent to performing a soft filter. This precise element-wise operation maximizes the preservation of details in complex spectral structures while effectively separating overlapping background noise from human voice, providing a high-fidelity separation signal for subsequent steps.

[0031] In this example, the problem arises because the original audio contains a mixture of speech and ambient sound, with the human voice frequency band occupying the majority of the energy. Direct scene analysis can easily lead to misjudgment, and subsequent direct superposition of sound effects can cause serious conflicts with the human voice frequency band, resulting in muddy sound.

[0032] The embodiment clarifies the specific technical path for achieving audio separation. A short-time Fourier transform converts the time-domain signal into a complex spectral matrix that simultaneously represents time and frequency characteristics, providing structured input features for subsequent masking prediction. This complex spectral matrix is ​​then fed into a deep neural network for inference. Based on the spectral distribution patterns of background noise and human voice learned during training, the network outputs a masking value between 0 and 1 for each time-frequency unit, forming a masking matrix. This matrix quantifies the probability that each time-frequency unit belongs to the background noise. Finally, using the separation formula, the masking matrix and its complement are performed element-wise with the original complex spectral matrix, achieving precise segmentation of background noise and human voice at the spectral level. This method eliminates human voice interference at its source, providing a clean and interference-free input for the subsequent scene calculation step S002, which analyzes background noise separately. This significantly improves the accuracy of scene perception while preserving the complete human voice components, laying the foundation for final distortion-free fusion.

[0033] In a further embodiment, regarding the aforementioned step S002, step S002 includes: Step S021: Extract the Mel frequency cepstral coefficients and spectral centroids from the background sound as the observation sequence.

[0034] In this embodiment, the background audio spectrum output in step S001 is... Mel-frequency cepstral coefficients (MFCCs) and spectral centroids are extracted and combined to form the acoustic observation sequence for the current time frame. .

[0035] In this embodiment, the combination constitutes the acoustic observation sequence of the current time frame. Yes, by applying a Mel filter bank to the amplitude spectrum, taking the logarithm of the energy, and then performing a discrete cosine transform (DCT), a 13-dimensional MFCC vector is obtained. Simultaneously calculate the spectral centroid: .

[0036] Finally, the MFCC vector and the spectral centroid are concatenated to form the acoustic observation sequence for the current time frame. .

[0037] Step S022: Construct a Hidden Markov Model (HMM) and calculate the probability of being in the current scene state using the HMM to obtain the scene evolution probability.

[0038] In this example, based on scene state probability deduction and vector mapping, the scene evolution is smoothly predicted and a probability vector is output through temporal constraints, avoiding misjudgments caused by relying solely on single-frame features. Specifically, a Hidden Markov Model based on HMM is constructed, and this model is used to calculate the scene state at the current moment. The probability is calculated, and the scene probability distribution vector at the current moment is output by the model. .

[0039] In this example, a set of discrete scene states is predefined. First, the emission probability is calculated, which involves analyzing the acoustic observation features of the current frame. The system compares the current feature with pre-stored feature distribution models for each scene to determine the probability that the current feature is generated by each scene. For example, a feature with a sudden increase in high-frequency energy corresponds to a higher probability of emission in snowy outdoor conditions. Secondly, a state transition probability is introduced. This probability is based on the physical laws of real-world environmental evolution and is used to constrain the rationality of scene transitions. For example, the probability of transitioning from a confined room to snowy outdoor conditions is higher than directly transitioning to underwater conditions. Finally, a temporal recursion is performed. The system weightedly fuses the emission probability of the current frame with the probabilities of each scene and their transition probabilities from the previous time step to obtain the comprehensive confidence score of the current scene. This confidence score is then normalized to ensure that the sum of the probabilities of each scene is 1. Finally, the system arranges the normalized confidence scores of each scene in order and outputs the scene probability distribution vector for the current time step. .

[0040] Furthermore, in this embodiment, the formula for the state transition probability can be the probability calculation formula of a hidden Markov model:

[0041] in, and These represent the current and previous scene states, respectively. This represents the probability of launch. This represents the scene transition probability.

[0042] This implementation clarifies how to transform the separated background sound into a scene state vector. Mel-frequency cepstral coefficients simulate the nonlinear perception of frequency by the human ear, effectively characterizing the macroscopic features of timbre and ambient sound. The spectral centroid reflects the brightness and energy concentration areas of the sound; the observation sequence formed by these two components provides robust acoustic cues for scene identification. The calculation is performed using a Hidden Markov Model, whose core advantage lies in introducing temporal dynamics. This model not only evaluates the matching degree (emission probability) between the current observation sequence and a particular scene but also considers the temporal transition logic between scenes; for example, the probability of transitioning from "indoor" to "outdoor" is usually higher than the probability of a direct jump to "underwater." By utilizing the scene state probability of the previous time step and the current observation, the model can output a smooth, continuous scene evolution probability distribution, i.e., the scene state vector. This approach effectively solves the problem of discontinuous output on the time axis in traditional static classifiers, leading to abrupt changes in subsequent sound effects. It makes scene determination predictable and smooth, providing a reliable basis for enhancing the dynamic tracking of sound effects.

[0043] In a further embodiment, regarding the aforementioned step S003, step S003 includes: Step S031: Based on the acoustic fingerprint of the background audio spectrum as a dual constraint, RAG sound effect retrieval is performed through a joint retrieval scoring formula. The candidate sound effects related to the current scene are comprehensively ranked from the database, and the highest score is used as the reference sound effect. The acoustic fingerprint includes the state scene probability and the background audio spectrum.

[0044] Furthermore, in the embodiment, the joint retrieval scoring formula is as follows:

[0045] in, The overall relevance of candidate sound effects; As weight; These are the semantic vector of the current scene and the semantic tag vector of the candidate sound effects in the database, respectively. These are the acoustic feature vectors of the original background sound and the candidate sound effect, respectively. The square of the Euclidean distance.

[0046] In this embodiment, the acoustic fingerprint of the background audio spectrum output in step S001, such as reverberation time and spectral centroid, is extracted and combined with the scene probability distribution vector output in step S002. The constructed joint retrieval scoring formula is logically sound, as it uses a unified weighting system for semantic similarity and acoustic physical similarity. The Top-K sound effects with the highest overall relevance among the candidate sound effects are selected as the reference base.

[0047] Step S032: Using the reference sound effect as the style target, a lightweight neural vocoder is used for fast feature transfer to generate the target enhanced sound effect.

[0048] In a further embodiment, the generation formula of the neural vocoder is:

[0049] in, Enhance the sound effects of the generated target; For vocoder codec; Frequency domain style characteristics for reference sound effects; This is a style feature transformation function.

[0050] In the example, the frequency domain stylistic features of the reference sound effect are used. The goal is to achieve fast feature transfer using a lightweight neural vocoder. This mechanism completes style enhancement with only one forward pass, meeting real-time requirements.

[0051] Step S033: Divide the background sound and the target enhanced sound effect into multiple frequency bands, and perform dynamic gain calculations on each to obtain the gain adjustment enhanced sound effect.

[0052] In a further embodiment, the formula for calculating the gain is:

[0053] in, For the first Frequency band dynamic gain; Basic gain; Side chain inhibition sensitivity; The original background sound was in the first... Short-term energy in the frequency band; This indicates the final gain adjustment to enhance the sound effect.

[0054] This set of formulas clarifies the internal calculation logic of each step. In the joint retrieval scoring formula, weights are used... The cosine similarity of semantic vectors is linked to the Euclidean distance of acoustic vectors. It measures the semantic fit of the scene, and The acoustic feature distance is then mapped to a physical similarity score between 0 and 1. Through weighted summation, the system can select the candidate sound effect with the best overall performance. In the vocoder formula, the style feature transformation of the reference sound effect is superimposed on the encoding of the original background sound, resulting in a higher output from the decoder. It retains the original background sound while also possessing the enhanced characteristics of a reference sound effect. In the gain calculation, this formula embodies the concept of sidechain compression: when the original background sound has short-time energy in a certain frequency band... When increased, gain Based on sensitivity Automatic attenuation, and when the original background sound energy is low, the gain is close to the base gain. Ultimately, the enhanced sound effects from all frequency bands after gain adjustment are summed to generate a dynamically adapted sound. .

[0055] The example details the process of acquiring and adjusting the gain of sound effects. First, the dual constraints for RAG sound effect retrieval are determined by the scene's semantic vector (by the state scene probability) and the physical acoustic fingerprint of the background sound (by acoustic features extracted from the background audio spectrum). This ensures that the retrieved reference sound effects not only conform to the scene description in terms of category, but also that their inherent acoustic properties (such as reverberation characteristics) match the real environment, fundamentally solving the problem of a "disconnect" between sound effects and the perceived sound. Second, when generating enhanced sound effects, instead of using a time-consuming generation model from scratch, a lightweight neural vocoder is used to quickly "transfer" the frequency domain style features of the retrieved reference sound effects to the current background sound base. This process can be completed with a single forward propagation, avoiding the high latency problem of traditional AI generation models and meeting the requirements of real-time applications. Finally, both the original background sound and the generated target enhanced sound effects are divided into multiple independent frequency bands, and independent dynamic gain calculations are performed on each frequency band. This processing method can precisely adjust the gain of the enhanced sound effect in real time according to the energy of the original background sound in different frequency bands. It perfectly solves the problem that traditional single-band gain will mistakenly suppress the volume of the entire frequency band when the original sound and the enhanced sound effect only conflict in a specific frequency band, and achieves surgical-like precise spectrum avoidance.

[0056] In one embodiment, step S004 mentioned above includes: In this embodiment, firstly, the azimuth angle of the original sound image is extracted based on the background sound features separated in step S001, and then the enhanced sound effect output in step S003 is dynamically rendered in space by combining the scene vector from step S002; subsequently, the signal-to-mask ratio is calculated by comparing the synthesized masking threshold of human voice and background sound separated in step S001 with the rendered enhanced sound effect; finally, the frequency domain fusion weight is generated based on the signal-to-mask ratio to complete the adaptive fusion and output of multiple audio tracks.

[0057] Step S041: Determine the azimuth angle of the original sound image based on the background sound characteristics, and then perform dynamic spatial rendering of the enhanced sound effect based on the scene distribution vector to obtain the enhanced sound effect with precise spatial positioning.

[0058] In a further embodiment, the formula for calculating the acoustic image azimuth angle is:

[0059] in, The horizontal azimuth angle of the original background sound image in the current time frame. For short-term energy of the left and right channels, This is the maximum coverage angle.

[0060] Step S042: Using human voice and background sound as the original audio tracks, calculate the joint masking threshold through a psychoacoustic model, and compare the spatial enhancement sound effect with the masking threshold to obtain the signal-to-mask ratio.

[0061] In a further embodiment, the enhanced sound effect rendering formula is:

[0062] in, To enhance the time domain signal of the audio effect, For dynamic azimuth angle The corresponding left and right ear-related transfer function impulse responses, Enhance the sound effects of the left and right channels to give them a sense of spatial orientation after rendering.

[0063] Step S043: Based on the signal-to-mask ratio, use the Sigmoid function to generate smooth frequency domain fusion weights, then superimpose them with the audio track, and reconstruct them into a time domain signal by performing an inverse short-time Fourier transform to output the final enhanced sound.

[0064] In this embodiment, the human voice spectrum separated and preserved in step S001 is... With background audio spectrum Calculate the joint masking threshold using a psychoacoustic model. Subsequently, based on spatial enhancement sound effects The signal-to-mask ratio is calculated by comparing it with the masking threshold.

[0065] In a further embodiment, the formula for calculating the signal-to-mask ratio is:

[0066] in, The masking threshold, calculated based on both human voice and original background noise, represents the lowest perceptible frequency energy level within the critical frequency band of human hearing. To enhance the spectral amplitude of spatial sound effects in the time-frequency unit, The signal-to-masking ratio reflects the ability of enhanced sound effects to penetrate the original sound masking. When the sound effect is masked by the original sound, it means that the sound effect is inaudible to the human ear.

[0067] This set of formulas details the precise calculation logic for spatial localization and masking quantization. In the sound image azimuth calculation, by comparing the short-time energy differences between the left and right channels and applying the sine law, the dynamic position of the sound source on the horizontal plane can be accurately quantified. This calculation is entirely based on existing background sound characteristics and requires no additional hardware. During rendering, by convolving the enhanced sound effect with the head-related transfer function corresponding to the time-varying azimuth angle, the process of sound entering the human ear from a specific direction is simulated, generating a highly directional stereo enhanced sound effect. In the signal-to-mask ratio calculation, by comparing the spectral amplitude of the spatially rendered enhanced sound effect point-by-point with the joint masking threshold calculated based on the original sound track, a parameter that accurately reflects the audibility of the enhanced sound effect at each time-frequency unit is obtained. When this ratio is less than 1, it clearly indicates that the spectral components of the enhanced sound effect are below the human ear's perception threshold, providing both physical and physiological justification for subsequent weighting.

[0068] In a further embodiment, regarding the aforementioned method of generating and superimposing fusion weights using the Sigmoid function, the dynamic fusion weight formula is as follows:

[0069] The formula for the final fused output frequency domain signal is:

[0070] This dynamic fusion weight utilizes the saturation property of the Sigmoid function; when the original sound masking is strong... Hour, Approaching 0, sound enhancement automatically yields; when the original sound is unmasked... When it is big, Approaching 1, the sound effects are fully released. Among them, The weights are dynamically fused, with values ​​ranging from [0,1]. The steepness coefficient controls the smoothness of the concession transition. The preset signal-to-mask ratio threshold, The final frequency domain signal output by fusion This is for pure vocals and original background music.

[0071] The example clarifies the specific methods for achieving adaptive spectrum fusion. The Sigmoid function in the formula implements a nonlinear mapping, and the parameters... The weight is determined at the threshold. The degree of abrupt change in the surrounding area. Utilizing this smooth transition characteristic avoids the audible audio flickering or clipping caused by traditional logical "switches." In the fusion output formula, the pure vocal spectrum... and the original background audio spectrum Directly participating in the summation ensures 100% preservation of the original sound signal without attenuation, guaranteeing the clarity of the basic content. Meanwhile, spatial enhancement sound effects... The signal is dynamically weighted, and its energy is injected only in the time-frequency region where the signal-to-mask ratio is above a threshold, while it remains almost silent in the region masked by the original sound. This mechanism transforms the fusion process into an intelligent mixing constrained by physiological perception, avoiding interference from enhanced sound effects with key vocals and dialogue, while also creating a sufficient environmental atmosphere at the appropriate time, significantly improving the auditory comfort and information clarity of the final audio.

[0072] The example illustrates the steps for achieving spatial alignment and adaptive avoidance fusion. First, the azimuth angle of the original sound image is calculated based on background sound features. This automatically extracts the spatial location information of the ambient sound from the original stereo signal, allowing the subsequently generated enhanced sound effects to be rendered in a location consistent with the visuals and original sound, resolving the disjointed feeling caused by spatial sound image mismatch. Next, a joint masking threshold is calculated based on the two original sound tracks: human voice and background sound. This threshold represents the lowest sound level that the human ear can just barely perceive at any given time and frequency. The spatially rendered enhanced sound effect is compared to this threshold, and the resulting signal-to-mask ratio directly quantifies the degree to which the enhanced sound effect is masked by the original sound track at each time-frequency unit. When the frequency domain energy of the enhanced sound effect is greater than the masking threshold (high signal-to-mask ratio), it is audible; conversely, it is masked by the original sound and cannot be perceived. Finally, the sigmoid function is used to generate fusion weights based on the signal-to-mask ratio. Its key feature is that when the signal-to-mask ratio is below a set threshold, the weights smoothly approach zero, automatically suppressing inaudible enhancement energy that only causes muddiness; while when the signal-to-mask ratio is higher, the weights approach one, allowing the enhancement effects to be fully displayed. This mechanism intelligently achieves "active concession" of enhancement effects in the frequency domain, ultimately superimposing, fusing, and inversely transforming the output, resulting in spatially harmonious and unified audio that is clear, transparent, and rich in detail.

[0073] It should be noted that in some optional implementations, the neural network training process and the specific parameter configuration of the acoustic model in each of the above steps can be optimized and adjusted. As a specific implementation, the deep neural network used to predict the ideal ratio mask in step S001 can adopt a network structure based on recurrent networks or attention mechanisms to better capture the temporal dependencies of the audio signal. The number of scene states, the transition probability matrix, and the emission probability model in the hidden Markov model of step S002 can be pre-trained based on thousands of hours of audio data with scene annotations to cover a wider range of scene categories. The database in step S003 can contain a massive amount of high-quality environmental sound effect material with semantic tags and pre-computed acoustic feature vectors. In addition to reverberation time and spectral centroid, the extraction of acoustic fingerprints can also include features such as loudness range and spectral flux to improve the robustness of physical environment matching. The head-related transfer function in step S004 can use a publicly available measurement database or be custom-rendered for the speaker layout of the playback device. (Steepness coefficient) And signal masking threshold Calibration can be performed before shipment based on subjective listening evaluation. In some alternative embodiments, when the original audio frequency is mono, the sound image azimuth angle calculation and spatial rendering steps in step S004 can be omitted or simplified to fixed sound field expansion, while the frequency domain fusion mechanism of enhanced sound effects and human voices can still function normally. These adjustments and additions to details do not depart from the core steps and logic defined by the claims of this application.

[0074] For example: Imagine a user is watching a suspense movie. The current scene shows the protagonist walking out of a basement, like a locked room, into a blizzard outdoors.

[0075] In step S001, the system acquires the raw audio signal. After STFT conversion, the deep network predicts the ideal ratio masking for the background noise. Successfully transformed the dull basement echo, i.e., the background audio spectrum. The sound spectrum of human voices includes the protagonist's footsteps and dialogue. Separation.

[0076] In step S002, the background audio spectrum is extracted. The MFCC features are input into the HMM model. As the door opens and the wind begins to blow, the HMM calculates the scene transition probability and outputs a scene probability distribution vector. The scene transitions smoothly from [Secret Room: 0.9, Snowstorm: 0.1] to [Secret Room: 0.1, Snowstorm: 0.9], recognizing the dynamic evolution of the scene state.

[0077] In step S003, the system uses the scene probability distribution vector and background audio spectrum Using a shorter reverberation time acoustic fingerprint as a condition, the "short reverberation wind and snow sound effect" was calculated to have the highest score through a double-constraint RAG formula, thus avoiding the sound from being mixed into the hall reverberation; the vocoder instantaneously migrated to generate the wind and snow sound. Meanwhile, due to the strong low-frequency boom in the original background sound, the low-frequency gain was calculated using a multi-band gain formula. Minimal, mid-to-high frequency gain Larger, automatically suppresses and enhances the low frequencies of wind and snow sounds, compensates for high-frequency tearing, and outputs... .

[0078] In step S004, the system first extracts the background audio spectrum. The left and right channel energy was used to calculate the azimuth angle of the sound image on the left side of the screen where the sound of wind and snow should be. By using HRTF convolution Rendered to the left channel bias position Subsequently, based on the human voice spectrum and background audio spectrum Calculate the masking threshold and, within the main character's dialogue frequency band, derive the fusion weights. Automatically reduced noise levels; wind and snow sounds are intelligently muted to prevent masking conversations; and in high-frequency areas where there is no conversation, Approaching 1, the sound of wind and snow is fully released. Finally, after being merged and inversely transformed, the output delivers an immersive sound effect where footsteps and dialogue are clear, sound image location is accurate, and the sound reverberates in a realistic wind and snow.

[0079] This application also provides a sound effect enhancement device. For example... Figure 5 As shown, the device includes: The spectrum separation module 10 is used to acquire the original audio frequency, extract and mask the time-frequency features of the original audio frequency, and separate the time-frequency features to obtain the background audio spectrum and the human voice spectrum; Scene state vector module 20 is used to extract the observation sequence based on the background audio spectrum, and to perform time-series dynamic scene calculation on the observation sequence using a hidden Markov model to obtain a continuously changing scene state vector. The sound enhancement module 30 is used to perform RAG sound effect retrieval based on the acoustic fingerprint dual constraint of the background audio spectrum and generate the target enhanced sound effect. The original background audio spectrum and the enhanced sound effect are side-chained in multiple frequency bands to obtain the gain-adjusted enhanced sound effect. The frequency domain fusion module 40 is used to locate the original sound image based on the background sound features, then perform spatial rendering based on the scene state vector and calculate the signal-to-mask ratio, and finally perform frequency domain fusion with the human voice spectrum to obtain the final output audio.

[0080] Each module in the device corresponds one-to-one with each step in the aforementioned sound enhancement method, and can collaboratively execute corresponding software instructions or be implemented through hardware circuits, thereby achieving all the beneficial effects described in this application in any device that requires audio processing, which will not be elaborated further here.

[0081] Through the coordinated operation of the aforementioned modules, this device can achieve all the functions and beneficial effects described in the above-mentioned method embodiments.

[0082] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the sound enhancement method as described in any of the above embodiments. This electronic device has the same technical effects as the corresponding method.

[0083] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the sound effect enhancement method as described in any of the above embodiments. This computer-readable storage medium has the same technical effects as the corresponding method.

[0084] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0085] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. This disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims. Thus, if these modifications and variations of the invention fall within the scope of the claims of the invention and their equivalents, the invention is also intended to include these modifications and variations.

Claims

1. A method for enhancing sound effects, characterized in that the steps include... include: Step S001: Obtain the original audio frequency, extract and mask the time-frequency features of the original audio frequency, and separate the time-frequency features to obtain the background audio spectrum and the human voice spectrum; Step S002: Extract the observation sequence based on the background audio spectrum, and use the Hidden Markov Model to perform time-series dynamic scene calculation on the observation sequence to obtain a continuously changing scene state vector; Step S003: Based on the acoustic fingerprint dual constraint of the background audio spectrum, perform RAG sound effect retrieval and generate target enhanced sound effect. Combine the original background audio spectrum with the enhanced sound effect through multi-band sidechaining to obtain gain-adjusted enhanced sound effect. Step S004: Based on the background sound features, perform original sound image localization, then perform spatial rendering based on the scene state vector, calculate the signal-to-mask ratio, and finally perform frequency domain fusion with the human voice spectrum to obtain the final output audio.

2. The sound enhancement method according to claim 1, characterized in that, Step S001 includes: Acquire the raw audio frequency, which includes background noise and human voice; The time-frequency features are extracted by performing a short-time Fourier transform on the original audio frequency to obtain the complex spectrum matrix of the mixed audio. The ideal ratio of the time-frequency features predicted by the complex spectral matrix is ​​masked by a deep neural network to obtain the masking matrix; By separating the masking matrix using the separation formula, the separated background audio spectrum and human voice spectrum are obtained.

3. The sound enhancement method according to claim 2, characterized in that, The ideal ratio masking formula is: ； in, The background noise masking matrix is ​​for prediction, with a value range of [0,1]. Use the Sigmoid activation function; and These are the trainable weight parameters and biases of the neural network; The amplitude of the mixed spectrum; The separation formula is: ； in, This is the separated background audio spectrum. The separated human voice spectrum is represented by ⊙, which indicates element-wise multiplication.

4. The sound enhancement method according to claim 1, characterized in that, Step S002 includes: Mel frequency cepstral coefficients and spectral centroids were extracted from the background sound as an observation sequence; Construct a Hidden Markov Model (HMM) and use it to calculate the probability of the current state of the scene, thus obtaining the probability of scene evolution.

5. The sound enhancement method according to claim 1, characterized in that, Step S003 includes: Based on the acoustic fingerprint of the background audio spectrum as a dual constraint, RAG sound effect retrieval is performed through a joint retrieval scoring formula. The candidate sound effects related to the current scene are obtained from the database and ranked comprehensively. The highest score is used as the reference sound effect. The acoustic fingerprint includes the state scene probability and the background audio spectrum. Using reference sound effects as style targets, a lightweight neural vocoder is used for fast feature transfer to generate target-enhanced sound effects; The background sound and the target enhanced sound effect are divided into multiple frequency bands, and dynamic gain calculations are performed on each band to obtain the gain adjustment enhanced sound effect.

6. The sound enhancement method according to claim 5, characterized in that, The joint retrieval scoring formula is as follows: ； in, The overall relevance of candidate sound effects; As weight; These are the semantic vector of the current scene and the semantic tag vector of the candidate sound effects in the database, respectively. These are the acoustic feature vectors of the original background sound and the candidate sound effect, respectively. The square of the Euclidean distance; The formula for the neural vocoder is: ； in, Enhance the sound effects of the generated target; For vocoder codec; For reference sound effects, the frequency domain style characteristics; This is a style feature transformation function.

7. The sound enhancement method according to claim 1, characterized in that, Step S004 includes: The azimuth angle of the original sound image is determined based on the background sound characteristics. Then, the enhanced sound effect is dynamically rendered in space based on the scene distribution vector to obtain the enhanced sound effect with precise spatial positioning. The human voice and background sound are used as the original audio tracks. The joint masking threshold is calculated through a psychoacoustic model. The spatial enhancement sound effect is compared with the masking threshold to obtain the signal-to-mask ratio. Based on the signal-to-mask ratio, a smooth frequency domain fusion weight is generated using the Sigmoid function, which is then superimposed on the audio track. The signal is then reconstructed into a time domain signal through an inverse short-time Fourier transform, and the final enhanced sound is output.

8. The sound enhancement method according to claim 7, characterized in that, The formula for calculating the azimuth angle of the acoustic image is as follows: ； in, The horizontal azimuth angle of the original background sound image in the current time frame. For short-term energy of the left and right channels, This is the maximum coverage angle.

9. The sound enhancement method according to claim 7, characterized in that, The dynamic fusion weight formula is as follows: ； The formula for the final fused output frequency domain signal is as follows: ； in, The weights are dynamically fused, with values ​​ranging from [0,1]. The steepness coefficient controls the smoothness of the concession transition. The preset signal-to-mask ratio threshold, The final frequency domain signal output by fusion This is for pure vocals and original background music.

10. A sound enhancement device, characterized in that, include: The spectrum separation module is used to acquire the original audio frequency, extract and mask its time-frequency features, and separate the time-frequency features to obtain the background audio spectrum and the human voice spectrum. The scene state vector module is used to extract the observation sequence based on the background audio spectrum, and to perform time-series dynamic scene calculation on the observation sequence using a hidden Markov model to obtain a continuously changing scene state vector. The sound enhancement module is used to perform RAG sound effect retrieval based on the acoustic fingerprint dual constraint of the background audio spectrum and generate the target enhanced sound effect. The original background audio spectrum and the enhanced sound effect are side-chained in multiple frequency bands to obtain the gain-adjustable enhanced sound effect. The frequency domain fusion module is used to locate the original sound image based on the background sound features, then perform spatial rendering based on the scene state vector and calculate the signal-to-mask ratio, and finally perform frequency domain fusion with the human voice spectrum to obtain the final output audio.

Images