A multi-modal reverberation effect generation method and system

Abstract

The application discloses a kind of multi-modal reverberation effect generation methods, comprising the following steps: S1.multimodal feature extraction: based on the original audio of input, generate multiple reverberation modes, and extract the time-domain feature and frequency-domain feature of original audio, and the modal feature of reverberation mode, fusion forms multidimensional feature vector;S2.Adaptive weight distribution: input multidimensional feature vector into MLP attention network, based on the content type of original audio, distribute the weight of each reverberation mode;S3.time domain-frequency domain joint mixing: the time-domain signal of multiple reverberation modes is weighted and superimposed, and time-domain mixed signal is generated;The time-domain signal of multiple reverberation modes is converted into frequency-domain signal, and cross fusion is realized by complex conjugate multiplication, and frequency-domain mixed signal is generated;The time domain and frequency domain mixed signal are fused to form a preliminary reverberation signal;S4.Spectrum calibration: spectrum power analysis is carried out on the preliminary reverberation signal, low frequency turbidity or high frequency is eliminated by equalization filtering, and the final reverberation signal is output.

Description

Technical Field

[0001] This invention relates to the field of audio signal processing technology, and more specifically, to a method and system for generating multimodal reverberation effects. Background Technology

[0002] Reverb plays a central role in audio processing, significantly enhancing the spatiality, atmosphere, and immersive experience of audio by simulating the reflection, scattering, and attenuation of sound in physical space. Current reverb techniques are mainly divided into two categories: single-modal reverb and simple mixed reverb.

[0003] Single-modal reverberation has significant functional limitations: while plate reverberation can provide a bright high-frequency response and high clarity, its spatial performance is insufficient, making it difficult to create an open sound field environment; spring reverberation can produce a unique vibrato texture, suitable for retro sound effects creation, but it is accompanied by problems of low-frequency muddiness and uneven attenuation, affecting the purity of the sound; hall reverberation can simulate the acoustic characteristics of a large space, with a long tail and strong spatial sense, but it significantly damages the clarity of speech signals, especially in vocal processing, which can easily lead to a decrease in intelligibility; room reverberation presents natural acoustic characteristics, but the sense of atmosphere is relatively weak, and it cannot meet the needs of high immersion; church reverberation is characterized by long reverberation time and rich atmosphere, but it is prone to sound blurring and weakens the ability to reproduce audio details.

[0004] Traditional mixed reverberation methods generally employ a fixed-ratio superposition strategy, such as a pre-set combination of 50% hall reverberation and 50% plate reverberation. This type of method has three major drawbacks:

[0005] First, the rigid weighting mechanism cannot dynamically adapt to the characteristics of the input audio content (such as the differences between speech and music, human voice and instrumental music), resulting in loss of clarity in live broadcasts and other vocal scenarios, or loss of detail in music production. Second, relying solely on the simple superposition of time-domain signals and lacking deep integration of frequency-domain characteristics causes spectral distortion and timbre fragmentation, making the mixing effect appear unnatural. Third, the spectral calibration process is neglected in the mixing process, which easily leads to low-frequency energy accumulation causing muddiness or high-frequency energy attenuation imbalance. At the same time, the problem of early reflection energy conflict is prominent, and the final effect is often less accurate and reliable than that of a single mode.

[0006] With the rapid expansion of audio application scenarios, the live streaming field requires both vocal clarity and spatiality, while music production needs to balance atmosphere and detail reproduction. Existing technologies struggle to dynamically integrate the advantages of multiple reverberation modes and cannot simultaneously meet the comprehensive requirements of clarity, spatiality, and atmosphere. There is an urgent need for a reverberation generation solution that can intelligently integrate multimodal characteristics, adaptively adapt to audio content, and ensure spectral integrity.

[0007] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention

[0008] The purpose of this invention is to provide a method for generating multimodal reverberation effects, which has the advantages of strong adaptability, high spectral integrity, and natural reverberation effects.

[0009] The above-mentioned technical objective of the present invention is achieved through the following technical solution:

[0010] A method for generating multimodal reverberation effects includes the following steps:

[0011] S1. Multimodal feature extraction: Based on the input original audio, generate multiple reverberation modes, and extract the time domain and frequency domain features of the original audio, as well as the modal features of the reverberation modes, and fuse them to form a multidimensional feature vector;

[0012] S2. Adaptive weight allocation: Input the multidimensional feature vector into the pre-trained MLP attention network, and allocate weights for each reverberation mode based on the content type of the original audio.

[0013] S3. Time-frequency domain joint hybrid:

[0014] S31. Time-domain weighted mixing: Weighted superposition of time-domain signals of multiple reverberation modes to generate a time-domain mixed signal;

[0015] S32. Frequency Domain Cross-fusion: Convert time-domain signals of multiple reverberation modes into frequency-domain signals, and achieve cross-fusion through complex conjugate multiplication to generate a frequency-domain mixed signal;

[0016] S33. Time-domain-frequency domain fusion: fusing time-domain mixed signals with frequency-domain mixed signals to form a preliminary reverberation signal;

[0017] S4. Spectrum Calibration: Perform spectrum power analysis on the preliminary reverberation signal, eliminate low-frequency muddiness or high-frequency muddiness through equalization filtering, and output the final reverberation signal.

[0018] Furthermore, the multiple reverberation modes include plate reverberation, spring reverberation, hall reverberation, room reverberation, and church reverberation modes; the multidimensional feature vector is a 64-dimensional feature vector, which includes 2-dimensional time-domain features, 40-dimensional frequency-domain features, and 22-dimensional modal features.

[0019] Furthermore, the time-domain features in step S1 are extracted using the following formula:

[0020] Peak amplitude:

[0021]

[0022] Zero crossing rate:

[0023]

[0024] Among them, A peak is the peak amplitude; n is the number of sampling points; x(n) represents the sample value of the nth sampling point of the input audio signal; max() represents the maximum value of all sample values ​​within the current analysis frame;

[0025] ZCR is the zero-crossing rate; N is the frame length; sgn() is the function symbol; x(n-1) represents the sampled value of the previous sampled point n.

[0026] Furthermore, the frequency domain features in step S1 are extracted using the following formula:

[0027]

[0028] in, This represents the complex spectrum obtained after the audio signal undergoes a short-time Fourier transform, specifically the complex spectrum of the m-th frame signal at the k-th frequency point. This represents the summation of the weighted contributions of all sampling points within a frame; R represents the time-domain sample value, the position of the nth sample point in the audio signal within the m-th frame, and R is the frame shift. This is a Hamming window function used to suppress spectral leakage; It is a complex exponential kernel used to project time-domain signals into the frequency domain.

[0029] Furthermore, the modal features in step S1 are extracted using the following formula;

[0030] Reverberation time T 60 The time required for the signal energy to decay by 60dB is obtained by fitting the energy decay curve.

[0031] Decay rate:

[0032]

[0033] Wherein, the decay rate Decay represents the amount of signal energy attenuated per unit time. The initial reflected energy, For the energy of the final note;

[0034] Early reflected energy :

[0035]

[0036] Among them, early reflection energy This represents the sum of all reflected sound energy within a 100ms time window after the direct sound; n is the number of sampling points; This represents the direct sound reaching the receiving point; 100ms is the early reflection time window, indicating the time from... Start extending the time range backward by 100ms; This is the reverberation signal, representing the sampled values ​​of direct sound and reflected sound.

[0037] Furthermore, in step S2, the adaptive weight allocation method is as follows:

[0038] S21. Attention Weight Calculation

[0039] The weights of each reverberation mode are calculated using the following formula:

[0040]

[0041] in, represents the weight coefficient of the i-th reverberation mode, used to quantify the proportion of this reverberation mode in the final reverberation effect; exp() is an exponential function used to amplify similarity differences; MLP() is a multilayer perceptron used to process the input 64-dimensional feature vector and output a feature vector that is strongly correlated with the reverberation effect. It is a 64-dimensional feature vector; Let i be the feature vector of the i-th reverberation mode; Let be the eigenvector of the j-th reverberation mode; T represents the transpose of the matrix;

[0042] S22. Weight Normalization

[0043] The weights of each reverberation mode are normalized using the following formula:

[0044]

[0045] Normalization helps prevent a single reverberation mode from dominating the reverberation effect.

[0046] Furthermore, in step S31, the formula for time-domain weighted mixing is:

[0047]

[0048] in, This represents the generated time-domain mixed signal; Represents the weighting coefficient of the i-th reverberation mode; This represents the time-domain mixing signal of the i-th reverberation mode; Indicates the retention factor; Represents the original audio signal;

[0049] In step S32, the formula for frequency domain cross-fusion is:

[0050]

[0051] in, Represents a frequency domain mixed signal; The frequency domain signal representing the i-th reverberation mode; Represents the complex conjugate of the frequency domain signal of the j-th reverberation mode; Indicates the frequency band adaptive coefficient;

[0052] Step S33 includes the following method:

[0053] For frequency domain mixed signals Perform an inverse STFT transform to generate a time-domain mixed signal xmix(n); combine the time-domain mixed signal xmix(n) with the time-domain mixed signal generated in step S31. Weighted average is used to obtain the preliminary reverberation signal.

[0054] Furthermore, the specific method for step S4 is as follows:

[0055] S41. Calculation of spectral power density:

[0056] Calculate the spectral power density of the initial reverberant signal:

[0057]

[0058] Where k is the frequency point, N is the number of FFT points, m is the time-domain frame, and M is the total number of frames. Represents a frequency domain mixed signal;

[0059] Calculate the target spectral power density:

[0060]

[0061] in, The spectral power density of the standard reference audio at the k-th frequency point; The auditory weighting coefficients for the Mel domain;

[0062] S42. Equalization Filter Design:

[0063] Constructing the equalization filter transfer function:

[0064]

[0065] in, For frequency smoothing coefficients, For the target spectral power density, The spectral power density of the initial reverberation signal;

[0066] S43. Final reverberation signal generation:

[0067] The inverse short-time Fourier transform of the calibrated frequency domain signal is performed using the following formula to obtain the final reverberant audio signal;

[0068]

[0069] in, This is the calibrated frequency domain signal; The transfer function of the equalization filter; This represents a frequency domain mixed signal.

[0070] A multimodal reverberation effect generation system, comprising:

[0071] The feature extraction module is configured as follows:

[0072] Receive raw audio input;

[0073] Generate time-domain signals with multiple reverberation modes;

[0074] Extract the time-domain and frequency-domain features of the original audio;

[0075] Extract modal features for each reverberation mode;

[0076] The time-domain features, frequency-domain features, and modal features are fused to form a multi-dimensional feature vector;

[0077] The weight allocation module, connected to the feature extraction module, is configured as follows:

[0078] Receive the multidimensional feature vector;

[0079] The weight coefficients of each reverberation mode are calculated using a pre-trained MLP attention network;

[0080] Normalize the weighting coefficients;

[0081] The time-frequency mixing module, connected to the weight allocation module, is configured as follows:

[0082] Temporal mixing unit: Based on weighting coefficients, the temporal signals of multiple reverberation modes are weighted and superimposed, and then mixed with the original audio signal according to the retention coefficients to generate a temporal mixed signal;

[0083] Frequency domain fusion unit: Converts time-domain signals of multiple reverberation modes into frequency-domain signals, and achieves cross-fusion through complex conjugate multiplication to generate a frequency-domain mixed signal;

[0084] Signal synthesis unit: Performs inverse short-time Fourier transform on the frequency domain mixed signal and weights it with the time domain mixed signal to generate a preliminary reverberation signal;

[0085] The spectrum calibration module, connected to the time-frequency mixing module, is configured as follows:

[0086] Calculate the spectral power density of the preliminary reverberation signal;

[0087] Construct the transfer function of the equalization filter based on the target spectral power density and Mel domain auditory weighting coefficients;

[0088] The frequency domain mixed signal is calibrated using the transfer function of the equalization filter described above.

[0089] Perform an inverse short-time Fourier transform on the calibrated frequency domain signal to output the final reverberation signal.

[0090] In summary, the present invention has the following beneficial effects:

[0091] By extracting multimodal features, the system achieves a precise correlation between audio content and reverberation characteristics. It dynamically adapts to different audio types using adaptive weight allocation, and combines time-domain and frequency-domain joint mixing technology to ensure the naturalness of signal fusion. Furthermore, it eliminates spectral distortion through spectral calibration, effectively solving problems such as unnatural reverberation effects, insufficient clarity, and lack of spectral integrity in existing technologies. It has advantages such as strong adaptability, high spectral integrity, and natural reverberation effects. Attached Figure Description

[0092] Figure 1 This is a flowchart of the multimodal reverberation effect generation method described in this invention. Detailed Implementation

[0093] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below with reference to the figures and specific embodiments.

[0094] like Figure 1 As shown, the present invention proposes a method for generating multimodal reverberation effects, comprising the following steps:

[0095] S1. Multimodal feature extraction: Based on the input original audio, generate multiple reverberation modes, and extract the time domain and frequency domain features of the original audio, as well as the modal features of the reverberation modes, and fuse them to form a multidimensional feature vector;

[0096] S2. Adaptive weight allocation: Input the multidimensional feature vector into the pre-trained MLP attention network and allocate weights for each reverberation mode based on the content type of the original audio.

[0097] S3. Time-frequency domain joint hybrid:

[0098] S31. Time-domain weighted mixing: Weighted superposition of time-domain signals of multiple reverberation modes to generate a time-domain mixed signal;

[0099] S32. Frequency Domain Cross-fusion: Convert time-domain signals of multiple reverberation modes into frequency-domain signals, and achieve cross-fusion through complex conjugate multiplication to generate a frequency-domain mixed signal;

[0100] S33. Time-domain-frequency domain fusion: fusing time-domain mixed signals with frequency-domain mixed signals to form a preliminary reverberation signal;

[0101] S4. Spectrum Calibration: Perform spectrum power analysis on the preliminary reverberation signal, eliminate low-frequency muddiness or high-frequency muddiness through equalization filtering, and output the final reverberation signal.

[0102] Multimodal feature extraction can be understood as extracting various types of feature information from the original audio to characterize its content. For example, time-domain features can be extracted using short-time energy and zero-crossing rate, frequency-domain features can be extracted using Fourier transform or wavelet transform, and modal features can be obtained by analyzing parameters such as reverberation time and decay rate. These feature extraction methods can be implemented using different algorithms, such as using Mel-frequency cepstral coefficients (MFCC) to replace traditional frequency-domain feature extraction methods, or calculating the energy distribution characteristics of reverberation modes using statistical methods.

[0103] Adaptive weighting is used to dynamically adjust the weights of each reverb modality based on the content type of the input audio. The weighting implementation can also incorporate a rule engine, which uses a pre-defined rule base to classify different audio types and assign weights accordingly.

[0104] The innovation of this invention lies in its collaborative mechanism of multimodal feature extraction, adaptive weight allocation, time-domain-frequency domain joint mixing, and spectrum calibration, which solves the problems of single-modal effect limitations, spectral distortion, and poor adaptability in traditional reverberation techniques. Compared to the fixed-ratio superposition of reverberation modes in existing technologies, this invention achieves dynamic weight adjustment based on audio content, avoiding spectral distortion and timbre fragmentation. Simultaneously, the spectrum calibration step optimizes the balance between high and low frequency bands, further improving the overall quality of the reverberation effect.

[0105] The working principle of this invention is as follows:

[0106] A multimodal reverberation effect generation method addresses the limitations of single reverberation modal effects and the spectral distortion and poor adaptability of traditional mixing methods through the coordinated operation of multiple steps. In the multimodal feature extraction stage, multiple reverberation modalities are generated based on the input original audio, and the temporal and frequency domain features of the original audio, as well as the modal features of the reverberation modalities, are extracted. These features are fused to form a multidimensional feature vector, providing a comprehensive data foundation for subsequent dynamic weight allocation. In the adaptive weight allocation stage, the multidimensional feature vector is input into a pre-trained MLP attention network, and the weights of each reverberation modality are dynamically allocated according to the content type of the original audio. This avoids the insufficient adaptability problem caused by traditional fixed weights and ensures a reasonable proportion of each modality under different audio types.

[0107] In the time-domain-frequency domain joint mixing stage, the optimized fusion of signals is achieved through three sub-steps. First, time-domain weighted mixing generates a time-domain mixed signal by weighted superposition of time-domain signals from multiple reverberation modes, preserving the temporal structure and maintaining the natural decay characteristics of reverberation. Second, frequency-domain cross-fusion converts the time-domain signals of multiple reverberation modes into frequency-domain signals and achieves cross-fusion through complex conjugate multiplication, generating a frequency-domain mixed signal, thereby reducing frequency-domain distortion and enhancing timbre coherence. Third, time-domain-frequency fusion combines the time-domain mixed signal with the frequency-domain mixed signal to form a preliminary reverberation signal, fully utilizing the time precision of the time domain and the spectral details of the frequency domain to overcome the spectral fragmentation problem caused by time-domain mixing alone.

[0108] Finally, in the spectrum calibration stage, spectral power analysis is performed on the preliminary reverberation signal, and low-frequency muddiness or high-frequency imbalance is eliminated through equalization filtering. The spectrum distribution is calibrated to ensure the balance of the output signal in the high and low frequency bands. In the overall technical solution, each step works closely together. Multimodal feature extraction provides the basis for adaptive weight allocation, which guides the time-domain and frequency-domain joint mixing process. The time-domain and frequency-domain joint mixing generates a high-quality preliminary signal, and finally, the output effect is optimized through spectrum calibration, achieving a comprehensive improvement in the reverberation effect in terms of spatiality, clarity, and atmosphere.

[0109] This invention further proposes multiple reverberation modes, including plate reverberation, spring reverberation, hall reverberation, room reverberation, and church reverberation modes; the multidimensional feature vector is a 64-dimensional feature vector, which includes 2-dimensional time-domain features, 40-dimensional frequency-domain features, and 22-dimensional modal features.

[0110] Plate reverberation can be a type of reverberation that simulates high-frequency response based on the vibration of a metal plate, used to improve audio clarity; spring reverberation can be a type of reverberation that uses spring vibration to create a vibrato texture, enhancing retro sound effects; hall reverberation can be a type of reverberation that simulates the reflection characteristics of a large space, strengthening the sense of space and the long reverberation effect; room reverberation can be a type of reverberation based on the acoustic characteristics of a small, enclosed space, ensuring a natural and realistic feel; church reverberation can be a type of reverberation that simulates a long reverberation time, creating a long reverberation atmosphere. The selection of these reverberation modes aims to cover the acoustic needs of different scenarios, avoiding limitations in effects due to missing modes or selection bias. Multidimensional feature vectors refer to the set of features generated after multidimensional analysis of the original audio signal. Specifically, 2D time-domain features can be the quantified results of instantaneous dynamic attributes such as peak amplitude and zero-crossing rate, used to capture the time-domain variation characteristics of the signal; 40D frequency-domain features can be the spectral information extracted through short-time Fourier transform, providing detailed spectral analysis; 22D modal features can be the reverberation time T... 60 The quantization results, along with acoustic parameters such as decay rate, are used to describe the specific characteristics of the reverberation modes. This dimensionality allocation ensures that the eigenvectors comprehensively capture signal characteristics while avoiding redundancy.

[0111] The above scheme addresses the issues of ambiguous mode selection and inconsistent feature extraction by clearly defining the specific types of reverberation modes and the precise dimensions of feature vectors. Regarding reverberation modes, the combination of plate reverberation, spring reverberation, hall reverberation, room reverberation, and church reverberation modes covers a wide range of acoustic characteristics, from high-frequency response to low-frequency muddiness, and from short to long reverberation, enabling the system to dynamically integrate the advantages of each mode based on the audio content. In terms of feature vectors, 2D time-domain features simplify key dynamic information, 40D frequency-domain features support detailed spectral processing, and 22D modal features accurately describe reverberation characteristics, thus providing a structured input basis for adaptive weight allocation. Based on this, the above scheme, combined with multimodal feature extraction, adaptive weight allocation, time-domain-frequency domain joint mixing, and spectral calibration, forms a complete reverberation effect generation process. This approach not only improves the comprehensiveness and stability of reverberation effect generation but also effectively balances spatiality, clarity, and atmosphere, ultimately achieving adaptation and optimization for different types of audio signals.

[0112] The present invention further proposes that the temporal features in step S1 be extracted using the following formula:

[0113] Peak amplitude:

[0114]

[0115] Zero crossing rate:

[0116]

[0117] Where Apeak is the peak amplitude; n is the number of sampling points; x(n) represents the sample value of the nth sampling point of the input audio signal; max() represents the maximum value of all sample values ​​in the current analysis frame; ZCR is the zero-crossing rate; N is the frame length; sgn() is the function symbol; and x(n-1) represents the sample value of the previous sampling point n.

[0118] Peak amplitude refers to the maximum absolute value of all sampled values ​​within the current analysis frame. It can be achieved by iterating through all sampled points within the frame and recording the maximum value. The purpose of this feature is to accurately capture the instantaneous dynamic range of the audio signal and avoid feature distortion caused by signal peak fluctuations. Zero-crossing rate can be understood as the frequency of signal symbol changes per unit time. It can be achieved by counting the number of symbol transitions between adjacent sampled points. The purpose of this feature is to quantify the temporal roughness of the audio signal, thereby providing a basis for distinguishing between speech and music signals.

[0119] The above technical solution solves the problem of inaccurate feature quantization by constructing a standardized temporal feature extraction mechanism. In actual operation, the peak amplitude is first calculated based on all sampled values ​​within the current analysis frame to ensure that dynamic range information can be reliably integrated into the multi-dimensional feature vector. Subsequently, the zero-crossing rate is calculated by statistically analyzing the sign changes of adjacent sampled points to reflect the temporal characteristics of the signal. The introduction of the frame length N constrains the fixed duration of the analysis window, ensuring that feature calculation is performed on a uniform time scale and improving the stability and repeatability of the features. In addition, the association design between sampled point n and the previous sampled point x(n-1) effectively reflects the drastic degree of instantaneous changes in the signal. These features work together to enable the temporal features to more comprehensively and consistently represent the essential attributes of the original audio, providing high-precision input for the adaptive weight allocation of the subsequent MLP attention network.

[0120] The present invention further proposes that the frequency domain features in step S1 be extracted using the following formula:

[0121]

[0122] in, This represents the complex spectrum obtained after the audio signal undergoes a short-time Fourier transform, specifically the complex spectrum of the m-th frame signal at the k-th frequency point. This represents the summation of the weighted contributions of all sampling points within a frame; R represents the time-domain sample value, the position of the nth sample point in the audio signal within the m-th frame, and R is the frame shift. This is a Hamming window function used to suppress spectral leakage; It is a complex exponential kernel used to project time-domain signals into the frequency domain.

[0123] Complex spectrum refers to the frequency domain representation of an audio signal, which can be implemented using Discrete Fourier Transform (DFT) or Fast Fourier Transform (FFT). Its purpose is to capture the frequency components and phase information of the audio signal. The Hamming window function is a smoothly decaying window function that can be adjusted to adapt to the characteristics of different audio signals, aiming to reduce spectral leakage and improve the accuracy of frequency domain features. Furthermore, the complex exponential kernel is the core computational tool of the Short Time Fourier Transform (SFT), which can convert time-domain signals into frequency-domain signals through mathematical projection, thereby ensuring high resolution of frequency domain features.

[0124] This technical solution extracts frequency domain features using a short-time Fourier transform-based method. First, a Hamming window function is used to weight the time-domain signal to reduce edge abrupt changes caused by signal truncation, effectively suppressing spectral leakage. Then, a complex exponential kernel is used to project the time-domain signal into the frequency domain, generating a high-resolution complex spectrum. This process accurately captures the phase and amplitude information of the audio signal, providing a reliable spectral foundation for subsequent frequency domain cross-fusion. Simultaneously, the frame shift R optimizes the dynamic balance between time and frequency resolution. By controlling the overlap between adjacent frames, it adapts to the time-varying characteristics of the audio signal, avoiding problems of insufficient time resolution or excessively low spectral resolution. Finally, by summing the weighted contributions of all sampling points within a frame, the window function and kernel function are integrated to generate a stable complex spectral representation, ensuring the sensitive capture of local characteristics of the audio signal by the frequency domain features.

[0125] The above technical solution solves the problems of spectral leakage and feature distortion caused by inaccurate frequency domain feature extraction, improves the robustness and adaptability of frequency domain feature extraction, provides high-quality input for adaptive weight allocation and spectral calibration, and thus significantly improves the naturalness and spatiality of the reverberation effect.

[0126] The present invention further proposes that the modal features in step S1 be extracted using the following formula;

[0127] Reverberation time T 60 The time required for the signal energy to decay by 60dB is obtained by fitting the energy decay curve.

[0128] Decay rate:

[0129]

[0130] Wherein, the decay rate Decay represents the amount of signal energy attenuated per unit time. The initial reflected energy, For the energy of the final note;

[0131] Early reflected energy :

[0132]

[0133] Among them, early reflection energy This represents the sum of all reflected sound energy within a 100ms time window after the direct sound; n is the number of sampling points; This represents the direct sound reaching the receiving point; 100ms is the early reflection time window, indicating the time from... Start extending the time range backward by 100ms; This is the reverberation signal, representing the sampled values ​​of direct sound and reflected sound.

[0134] Reverberation time T 60 The decay rate (Decay) refers to the time required for signal energy to decay from its initial value to below 60 dB. It can be calculated by fitting a logarithmic energy decay curve, aiming to quantify the temporal characteristics of different reverberation modes. The decay rate (Decay) can be understood as the rate of change of signal energy per unit time. It can be dynamically derived by analyzing the difference between the initial reflection energy and the tail sound energy, aiming to capture the differences in the instantaneous decay behavior of reverberation modes. Early reflection energy refers to the cumulative value of all reflected sound energy within a 100 ms time window after the arrival of the direct sound. It is achieved by locating and summing the reflected sound sample values ​​through a standardized time window, aiming to ensure the quantification consistency of early reflection intensity.

[0135] The above technical solution defines the reverberation time T. 60 The specific extraction formulas for decay rate Decay and early reflection energy solve the problem of inaccurate modal feature quantization. First, the reverberation time T... 60 Based on energy decay curve fitting, the subjective bias of manual measurement is avoided, allowing for the objective quantification of the temporal characteristics of different reverberation modes, thus providing a reliable temporal dimension for subsequent weight allocation. Secondly, the decay rate (Decay), through dynamic analysis of initial reflection energy and tail sound energy, accurately reflects the differences in decay behavior of reverberation modes, such as the uneven decay characteristics of spring reverberation, thereby distinguishing the adaptation requirements of different modes to audio content in adaptive weight allocation. Finally, early reflection energy is extracted using standardized time windows and direct sound point localization, ensuring the quantification consistency of early reflection intensity. For example, the strong early reflection characteristics of plate reverberation can significantly improve audio clarity, while avoiding feature distortion caused by ambiguous time window definitions. Overall, these formulas transform the physical characteristics of reverberation modes into calculable numerical features, making the modal feature extraction process repeatable and verifiable, significantly improving the representativeness and robustness of multi-dimensional feature vectors, laying a precise feature foundation for adaptive weight allocation, and ultimately achieving a dynamic balance of reverberation effects in terms of spatiality, clarity, and atmosphere.

[0136] The present invention further proposes the following method for adaptive weight allocation in step S2:

[0137] S21. Attention Weight Calculation

[0138] The weights of each reverberation mode are calculated using the following formula:

[0139]

[0140] in, represents the weight coefficient of the i-th reverberation mode, used to quantify the proportion of this reverberation mode in the final reverberation effect; exp() is an exponential function used to amplify similarity differences; MLP() is a multilayer perceptron used to process the input 64-dimensional feature vector and output a feature vector that is strongly correlated with the reverberation effect. It is a 64-dimensional feature vector; Let i be the feature vector of the i-th reverberation mode; Let be the eigenvector of the j-th reverberation mode; T represents the transpose of the matrix;

[0141] S22. Weight Normalization

[0142] The weights of each reverberation mode are normalized using the following formula:

[0143]

[0144] Normalization helps prevent a single reverberation mode from dominating the reverberation effect.

[0145] A multilayer perceptron is a nonlinear mapping structure composed of multiple neural network layers. It can be implemented using fully connected neural networks, convolutional neural networks, or recurrent neural networks, with the aim of extracting deep features strongly correlated with reverberation effects. The exponential function is a mathematical operation that amplifies the similarity differences between feature vectors, ensuring that the weights of highly matched modes are significantly enhanced. Weight normalization is a standardization method that aims to standardize each weight coefficient by dividing it by the sum of the weights, preventing a single mode from excessively dominating the reverberation effect.

[0146] This scheme first generates weight coefficients based on the similarity between the multidimensional feature vector and the reverberation modal feature vector. In the attention weight calculation stage, a multilayer perceptron is used to perform a nonlinear mapping on the 64-dimensional feature vector, extracting deep features strongly correlated with the reverberation effect. This processing method effectively eliminates irrelevant noise, ensuring that the output feature vector accurately reflects the correlation between the audio content type and the reverberation requirement. Subsequently, dot product operations are used to quantify the matching degree between different modalities and the current audio, and an exponential function is used to further amplify the weight differences of highly matched modalities. In the weight normalization stage, each weight coefficient is standardized by dividing it by the sum of weights, forcing the total weight to be one, effectively constraining the upper limit of the weight of a single modality and preventing it from excessively dominating the reverberation effect.

[0147] Building upon this foundation, the scheme combines feature-driven dynamic weight generation with a stability guarantee mechanism based on mathematical constraints. This allows the reverberation mixing process to respond to subtle changes in audio content while mitigating the risk of extreme weighting. For example, when the input is a human voice signal, the high-frequency clarity characteristics of the plate reverberation are more similar to the feature vector, resulting in a significant increase in its weight, while the weight of the hall reverberation is relatively suppressed. This ensures that the weight allocation is strictly adjusted dynamically according to the audio content. Simultaneously, the normalization operation effectively prevents numerical imbalance caused by a high similarity between a modal feature vector and the processed feature vector, maintaining a balance between the spatial sense and clarity of the reverberation effect and providing a reliable foundation for subsequent time-domain and frequency-domain joint mixing.

[0148] In the further step S31 of this invention, the formula for time-domain weighted mixing is:

[0149]

[0150] in, This represents the generated time-domain mixed signal; Represents the weighting coefficient of the i-th reverberation mode; This represents the time-domain mixing signal of the i-th reverberation mode; Indicates the retention factor; Represents the original audio signal;

[0151] In step S32, the formula for frequency domain cross-fusion is:

[0152]

[0153] in, Represents a frequency domain mixed signal; The frequency domain signal representing the i-th reverberation mode; Represents the complex conjugate of the frequency domain signal of the j-th reverberation mode; Indicates the frequency band adaptive coefficient;

[0154] Step S33 includes the following method:

[0155] For frequency domain mixed signals Perform an inverse STFT transform to generate a time-domain mixed signal xmix(n); combine the time-domain mixed signal xmix(n) with the time-domain mixed signal generated in step S31. Weighted average is used to obtain the preliminary reverberation signal.

[0156] Temporal weighted mixing is a technique that involves weighting and superimposing multiple reverberation modes' temporal signals by introducing preservation coefficients and the original audio signal. Its purpose is to prevent the reverberation modes from completely obscuring the transient details of the original audio. In practical applications, the preservation coefficients can be dynamically adjusted, for example, by automatically adjusting the participation ratio of the original signal based on the audio content type or signal energy distribution, thereby protecting the clarity of the initial parts of speech or instruments. If the original signal preservation mechanism is ignored, high-frequency transient information is easily smoothed, leading to the blurring of sibilance in vocals or the plucked sounds of instruments.

[0157] Frequency domain cross-fusion refers to a technique that uses complex conjugate multiplication to achieve frequency domain signal interaction between different reverberation modes, aiming to solve the spectral distortion problem caused by phase differences. In practical applications, complex conjugate operations can effectively cancel phase conflicts through mathematical operations, allowing for a smooth transition in the frequency domain between the bright high frequencies of slab reverberation and the lingering tails of church reverberation. If the spectrum is directly superimposed without addressing the phase relationship, the phase conflict between different modes will cause abnormal distribution of spectral energy, resulting in a spring-like reverberation effect.

[0158] Time-domain-frequency fusion refers to balancing the transient response characteristics of the time-domain signal with the spectral continuity of the frequency-domain signal by weighted averaging the time-domain signal generated by the inverse STFT transform and the time-domain weighted mixing result. In practical applications, the weighted averaging mechanism can employ a dynamic weight allocation strategy based on signal energy distribution or spectral characteristics, thereby avoiding waveform discontinuities caused by simple splicing. If the time-domain and frequency-domain processing results are directly concatenated, the natural decay of room reverberation and the rapid reflection of plate reverberation will cause abrupt changes at the signal boundaries, disrupting the continuity of the reverberation effect.

[0159] Specifically, the above scheme systematically solves the problems of signal integrity and spectral continuity in the reverberation mixing process by meticulously designing the implementation mechanisms of time-domain weighted mixing, frequency-domain cross-fusion, and time-domain-frequency fusion. First, in the time-domain weighted mixing stage, the participation ratio of the original signal is dynamically adjusted by the retention coefficients to ensure the preservation of the semantic key information of the original audio, which is particularly crucial for protecting the clarity of the beginning parts of speech or instruments. Second, in the frequency-domain cross-fusion stage, phase differences are effectively offset by complex conjugation operations, organically fusing the spectral characteristics of different reverberation modes and avoiding abnormal spectral energy distribution. Finally, in the time-domain-frequency fusion stage, a weighted averaging mechanism balances the transient response characteristics of the time-domain signal and the spectral continuity of the frequency-domain signal, avoiding abrupt changes in signal boundaries. Overall, these technical features work together to preserve the key information of the original audio while achieving the organic fusion of multi-modal spectral characteristics, laying a high-quality signal foundation for subsequent spectral calibration.

[0160] The above technical solution not only solves the problem of the limitations of a single reverberation mode effect, but also overcomes the defects of traditional mixing methods such as spectral distortion and poor adaptability, generating a comprehensive reverberation effect with spatial sense, clarity and atmosphere. It can be widely used in audio tuning, music production, live broadcasting, voice processing and other scenarios, supports real-time processing and is compatible with different types of audio signals.

[0161] The present invention further proposes the following specific method for step S4:

[0162] S41. Calculation of spectral power density:

[0163] Calculate the spectral power density of the initial reverberant signal:

[0164]

[0165] Where k is the frequency point, N is the number of FFT points, m is the time-domain frame, and M is the total number of frames. Represents a frequency domain mixed signal;

[0166] Calculate the target spectral power density:

[0167]

[0168] in, The spectral power density of the standard reference audio at the k-th frequency point; The auditory weighting coefficients in the Mel domain;

[0169] S42. Equalization Filter Design:

[0170] Constructing the transfer function of the equalization filter:

[0171]

[0172] in, For frequency smoothing coefficients, For the target spectral power density, The spectral power density of the initial reverberation signal;

[0173] S43. Final reverberation signal generation:

[0174] The inverse short-time Fourier transform of the calibrated frequency domain signal is performed using the following formula to obtain the final reverberant audio signal;

[0175]

[0176] in, This is the calibrated frequency domain signal; The transfer function of the equalization filter; This represents a frequency domain mixed signal.

[0177] Spectral power density refers to the power distribution characteristics obtained through statistical analysis of the signal's frequency domain representation. It can be achieved using time-frequency analysis methods such as short-time Fourier transform or wavelet transform, aiming to provide an objective quantitative benchmark for spectrum calibration. Target spectral power density can be understood as an ideal spectral distribution designed in conjunction with a human auditory perception model. It simulates the differences in human ear sensitivity to different frequencies by introducing Mel-domain auditory weighting coefficients, thereby ensuring that the calibration process closely matches actual auditory needs. The equalization filter transfer function is the core tool for achieving spectrum compensation. It can be implemented using IIR or FIR filter structures, aiming to ensure the smoothness and naturalness of the spectrum adjustment process.

[0178] This solution addresses the spectral imbalance problem of reverberation signals through a systematic spectral calibration process. First, in the spectral power density calculation stage, the spectral power density is calculated based on the frequency domain mixture of the initial reverberation signal, providing a reliable objective benchmark for subsequent calibration. Simultaneously, the target spectral power density is calculated by introducing the spectral power density of a standard reference audio and Mel-domain auditory weighting coefficients, making the calibration target more aligned with actual human hearing requirements. Second, in the equalization filter design stage, the equalization filter transfer function is constructed based on the ratio of the target spectral power density to the initial reverberation signal's spectral power density, incorporating frequency smoothing coefficients. This not only ensures a smooth transition in the filter response but also effectively avoids timbre fragmentation caused by spectral abrupt changes. Finally, in the final reverberation signal generation stage, an inverse short-time Fourier transform is performed on the calibrated frequency domain signal and the equalization filter transfer function, seamlessly converting the optimized spectral characteristics into a time-domain signal, thereby outputting high-fidelity reverberation audio.

[0179] The above technical solution not only solves the calibration deviation problem caused by relying solely on experience in traditional methods, but also achieves a more natural spectral adjustment effect by introducing a human auditory perception model. Furthermore, this solution is organically combined with the aforementioned multimodal feature extraction, adaptive weight allocation, and time-domain-frequency domain joint mixing steps to form a complete reverberation effect generation system, significantly improving the spatial sense, clarity, and atmosphere of the reverberation signal.

[0180] The present invention also provides a multimodal reverberation effect generation system, comprising:

[0181] The feature extraction module is configured as follows:

[0182] Receive raw audio input;

[0183] Generate time-domain signals with multiple reverberation modes;

[0184] Extract the time-domain and frequency-domain features of the original audio;

[0185] Extract modal features for each reverberation mode;

[0186] The time-domain features, frequency-domain features, and modal features are fused to form a multi-dimensional feature vector;

[0187] The weight allocation module, connected to the feature extraction module, is configured as follows:

[0188] Receive the multidimensional feature vector;

[0189] The weight coefficients of each reverberation mode are calculated using a pre-trained MLP attention network;

[0190] Normalize the weighting coefficients;

[0191] The time-frequency mixing module, connected to the weight allocation module, is configured as follows:

[0192] Temporal mixing unit: Based on weighting coefficients, the temporal signals of multiple reverberation modes are weighted and superimposed, and then mixed with the original audio signal according to the retention coefficients to generate a temporal mixed signal;

[0193] Frequency domain fusion unit: Converts time-domain signals of multiple reverberation modes into frequency-domain signals, and achieves cross-fusion through complex conjugate multiplication to generate a frequency-domain mixed signal;

[0194] Signal synthesis unit: Performs inverse short-time Fourier transform on the frequency domain mixed signal and weights it with the time domain mixed signal to generate a preliminary reverberation signal;

[0195] The spectrum calibration module, connected to the time-frequency mixing module, is configured as follows:

[0196] Calculate the spectral power density of the preliminary reverberation signal;

[0197] Construct the transfer function of the equalization filter based on the target spectral power density and Mel domain auditory weighting coefficients;

[0198] The frequency domain mixed signal is calibrated using the transfer function of the equalization filter described above.

[0199] Perform an inverse short-time Fourier transform on the calibrated frequency domain signal to output the final reverberation signal.

[0200] The multimodal reverberation effect generation system is used to implement the above-mentioned multimodal reverberation effect generation method.

[0201] In this document, the terms "upper," "lower," "front," "back," "left," "right," "top," "bottom," "inner," "outer," "vertical," and "horizontal," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only used for the clarity of expressing the technical solution and for the convenience of description, and therefore should not be construed as limiting the present invention.

[0202] In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, which includes not only the elements listed but also other elements not expressly listed.

[0203] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for generating multimodal reverberation effects, characterized in that, Includes the following steps: S1. Multimodal feature extraction: Based on the input original audio, generate multiple reverberation modes, and extract the time domain and frequency domain features of the original audio, as well as the modal features of the reverberation modes, and fuse them to form a multidimensional feature vector; S2. Adaptive weight allocation: Input the multidimensional feature vector into the pre-trained MLP attention network, and allocate weights for each reverberation mode based on the content type of the original audio. In step S2, the adaptive weight allocation method is as follows: S21. Attention Weight Calculation The weights of each reverberation mode are calculated using the following formula: ； in, represents the weight coefficient of the i-th reverberation mode, used to quantify the proportion of this reverberation mode in the final reverberation effect; exp() is an exponential function used to amplify similarity differences; MLP() is a multilayer perceptron used to process the input 64-dimensional feature vector and output a feature vector that is strongly correlated with the reverberation effect. It is a 64-dimensional feature vector; Let i be the feature vector of the i-th reverberation mode; Let be the eigenvector of the j-th reverberation mode; T represents the transpose of the matrix; S22. Weight Normalization The weights of each reverberation mode are normalized using the following formula: ； Normalization helps prevent a single reverberation mode from dominating the reverberation effect; S3. Time-frequency domain joint hybrid: S31. Time-domain weighted mixing: Weighted superposition of time-domain signals of multiple reverberation modes to generate a time-domain mixed signal; In step S31, the formula for time-domain weighted mixing is: ； in, This represents the generated time-domain mixed signal; This represents the time-domain mixing signal of the i-th reverberation mode; Indicates the retention factor; S32. Frequency Domain Cross-fusion: Converts time-domain signals of multiple reverberation modes into frequency-domain signals, and achieves cross-fusion through complex conjugate multiplication to generate a frequency-domain mixed signal; In step S32, the formula for frequency domain cross-fusion is: ； in, Represents a frequency domain mixed signal; The frequency domain signal representing the i-th reverberation mode; Represents the complex conjugate of the frequency domain signal of the j-th reverberation mode; This represents the frequency band adaptive coefficient; m and k represent the m-th frame signal and the k-th frequency point, respectively. S33. Time-domain-frequency domain fusion: fusing time-domain mixed signals with frequency-domain mixed signals to form a preliminary reverberation signal; Step S33 includes the following method: For frequency domain mixed signals Perform an inverse STFT transform to generate a time-domain mixed signal xmix(n); combine the time-domain mixed signal xmix(n) with the time-domain mixed signal generated in step S31. Weighted averaging yields the preliminary reverberation signal; S4. Spectrum Calibration: Perform spectrum power analysis on the preliminary reverberation signal, eliminate low-frequency muddiness or high-frequency muddiness through equalization filtering, and output the final reverberation signal.

2. The method for generating multimodal reverberation effects according to claim 1, characterized in that, The various reverberation modes include plate reverberation, spring reverberation, hall reverberation, room reverberation, and church reverberation modes; the multidimensional feature vector is a 64-dimensional feature vector, which includes 2-dimensional time-domain features, 40-dimensional frequency-domain features, and 22-dimensional modal features.

3. The method for generating multimodal reverberation effects according to claim 1, characterized in that, The temporal features in step S1 are extracted using the following formula: Peak amplitude: ； Zero crossing rate: ； Among them, A peak is the peak amplitude; n is the number of sampling points; x(n) represents the sample value of the nth sampling point of the input audio signal; max() represents the maximum value of all sample values ​​within the current analysis frame; ZCR is the zero-crossing rate; N is the frame length; sgn() is the function symbol; x(n-1) represents the sampled value of the previous sampled point n.

4. The method for generating multimodal reverberation effects according to claim 1, characterized in that, The frequency domain features in step S1 are extracted using the following formula: ； in, This represents the complex spectrum obtained after the audio signal undergoes a short-time Fourier transform, specifically the complex spectrum of the m-th frame signal at the k-th frequency point. This represents the summation of the weighted contributions of all sampling points within a frame; R represents the time-domain sample value, the position of the nth sample point in the audio signal within the m-th frame, and R is the frame shift. This is a Hamming window function used to suppress spectral leakage; It is a complex exponential kernel used to project time-domain signals into the frequency domain.

5. The method for generating multimodal reverberation effects according to claim 1, characterized in that, The modal features in step S1 are extracted using the following formula; Reverberation time T 60 The time required for the signal energy to decay by 60dB is obtained by fitting the energy decay curve. Decay rate: ； Wherein, the decay rate Decay represents the amount of signal energy attenuated per unit time. The initial reflected energy, For the energy of the final note; Early reflected energy : ； Among them, early reflection energy This represents the sum of all reflected sound energy within a 100ms time window after the direct sound; n is the number of sampling points; This represents the direct sound reaching the receiving point; 100ms is the early reflection time window, indicating the time from... Start extending the time range backward by 100ms; This is the reverberation signal, representing the sampled values ​​of direct sound and reflected sound.

6. The method for generating multimodal reverberation effects according to claim 1, characterized in that, The specific method for step S4 is as follows: S41. Calculation of spectral power density: Calculate the spectral power density of the initial reverberant signal: ； Where N is the number of FFT points and M is the total number of frames; Calculate the target spectral power density: ； in, The spectral power density of the standard reference audio at the k-th frequency point; The auditory weighting coefficients in the Mel domain; S42. Equalization Filter Design: Constructing the transfer function of the equalization filter: ； in, For frequency smoothing coefficients, For the target spectral power density, The spectral power density of the initial reverberation signal; S43. Final reverberation signal generation: The inverse short-time Fourier transform of the calibrated frequency domain signal is performed using the following formula to obtain the final reverberant audio signal; ; ； in, This is the calibrated frequency domain signal; The transfer function of the equalization filter; This represents a frequency domain mixed signal.

7. A multimodal reverberation effect generation system, characterized in that, include: The feature extraction module is configured as follows: Receive raw audio input; Generate time-domain signals with multiple reverberation modes; Extract the time-domain and frequency-domain features of the original audio; Extract modal features for each reverberation mode; The time-domain features, frequency-domain features, and modal features are fused to form a multi-dimensional feature vector; The weight allocation module, connected to the feature extraction module, is configured as follows: Receive the multidimensional feature vector; The weight coefficients of each reverberation mode are calculated using a pre-trained MLP attention network; Normalize the weighting coefficients; The time-frequency mixing module, connected to the weight allocation module, is configured as follows: Temporal mixing unit: Based on weighting coefficients, the temporal signals of multiple reverberation modes are weighted and superimposed, and then mixed with the original audio signal according to the retention coefficients to generate a temporal mixed signal; Frequency domain fusion unit: Converts time-domain signals of multiple reverberation modes into frequency-domain signals, and achieves cross-fusion through complex conjugate multiplication to generate a frequency-domain mixed signal; Signal synthesis unit: Performs inverse short-time Fourier transform on the frequency domain mixed signal and weights it with the time domain mixed signal to generate a preliminary reverberation signal; The spectrum calibration module, connected to the time-frequency mixing module, is configured as follows: Calculate the spectral power density of the preliminary reverberation signal; Construct the transfer function of the equalization filter based on the target spectral power density and Mel domain auditory weighting coefficients; The frequency domain mixed signal is calibrated using the transfer function of the equalization filter described above. Perform an inverse short-time Fourier transform on the calibrated frequency domain signal to output the final reverberation signal.

Images