Audio reverberation method and system

Abstract

An audio reverberation method and system in the technical field of audio reverberation. The method comprises: performing pre-processing on an input audio signal to obtain a reverberation input signal (S110); performing reverberation processing on the reverberation input signal to generate an initial reverberation audio signal of a target scenario (S120); analyzing audio content of the input audio signal to obtain an audio content feature of the input audio signal (S130); determining a content adaptive masking matrix on the basis of the audio content feature (S140); performing weighted mixing on the content adaptive masking matrix and the initial reverberation audio signal to obtain a content adaptive reverberation signal (S150); and performing weighted mixing on the content adaptive reverberation signal and the reverberation input signal according to a preset proportion, to obtain a final reverberation audio signal (S160). Therefore, different reverberation effects are adapted for audio signals having different audio content, thereby avoiding the problem of distortion of a final reverberation audio signal.

Description

Audio reverberation method and system Technical Field

[0001] The present disclosure relates to the technical field of audio reverberation, and in particular to an audio reverberation method and system. Background Art

[0002] In sound design, existing technologies usually simulate the listening effects in different acoustic scenarios by superimposing specific reverberation signals on the original audio.

[0003] Common reverberation generation methods include convolution reverberation and artificial reverberation. The convolution reverberation method measures the room impulse response (RIR) of an actual scene, then convolves the target audio with the RIR during sound generation to reproduce the corresponding reverberation effect. The convolution reverberation method can produce a realistic confusion effect, but it is relatively complex. Artificial reverberation uses a model to simulate reverberation generation, including early radiation, late reverberation, time delay, and frequency attenuation characteristics, to emulate the reverberation effect of the target scene. Artificial reverberation methods are more flexible and less complex.

[0004] The effects of adding reverberation to different music content can vary. Some music styles, such as those containing fast-paced drum beats, can easily produce a chaotic effect when overlaid with a large amount of reverberation. Additionally, some audio content already has reverberation, and adding reverberation on top of that can result in an unexpectedly high reverberation volume.

[0005] Existing technology has proposed a virtual acoustic scene solution for the in-vehicle environment. In this solution, the input microphone signal and music signal are processed by a pre-processing module (including extracting the dry sound signal) to obtain the input signal required for artificial reverberation. The input signal is then processed using a reverberation generation algorithm to obtain a multi-channel artificial reverberation signal. Finally, the multi-channel artificial reverberation signal is processed by post-processing modules such as time delay, gain control, and dry-wet sound ratio mixing to obtain the final virtual scene audio output.

[0006] However, while these existing solutions can ensure that double reverberation is not superimposed on the reverberation components of the input signal through dry sound extraction, their algorithms for directly extracting the dry sound signal generally suffer from distortion. The resulting reverberation generation and mixing of dry and wet sounds can also result in distortion in the final output reverberation signal. Furthermore, these solutions fail to consider the reverberation effects of different musical styles. Summary of the Invention

[0007] The present disclosure aims to solve at least one of the problems existing in the prior art and provide an audio reverberation method and system.

[0008] In one aspect of the present disclosure, an audio reverberation method is provided, the audio reverberation method comprising:

[0009] Pre-process the input audio signal to obtain a reverberation input signal;

[0010] Performing reverberation processing on the reverberation input signal to generate an initial reverberation audio signal of a target scene;

[0011] Performing audio content analysis on the input audio signal to obtain audio content features of the input audio signal;

[0012] determining a content-adaptive masking matrix based on the audio content characteristics;

[0013] Performing weighted mixing of the content-adaptive masking matrix and the initial reverberation audio signal to obtain a content-adaptive reverberation signal;

[0014] The content adaptive reverberation signal and the reverberation input signal are weighted mixed according to a preset ratio to obtain a final reverberation audio signal.

[0015] Optionally, the audio content feature includes a music style; and performing audio content analysis on the input audio signal to obtain the audio content feature of the input audio signal includes:

[0016] Obtaining the music style of the input audio signal based on the audio tag of the input audio signal; or,

[0017] Audio features are obtained based on the music spectrum of the input audio signal, and the audio features are input into a pre-trained music classification model to obtain the music style of the input audio signal; wherein the music classification model is trained using audio features and corresponding classification labels.

[0018] Optionally, the audio content feature includes a density of drum beats; and performing audio content analysis on the input audio signal to obtain the audio content feature of the input audio signal includes:

[0019] Based on a note starting point detection scheme, energy or spectrum change information is used to determine the mutation point of the input audio signal, the mutation point with a mutation degree greater than a preset mutation threshold is regarded as a drum beat, and the drum beat density of the input audio signal is obtained based on the number of drum beats within a preset duration; or

[0020] The multi-frame spectrum of the input audio signal is input into a pre-trained drum beat detection model to obtain the probability of each time point corresponding to a drum sound. The time point with the probability greater than a preset probability threshold is regarded as a drum beat. Based on the number of drum beats within a preset duration, the drum beat density of the input audio signal is obtained.

[0021] Optionally, the audio content feature includes a reverberation degree; and performing audio content analysis on the input audio signal to obtain the audio content feature of the input audio signal includes:

[0022] Inputting the signal spectrum of the input audio signal into a pre-trained de-reverberation model to obtain a time-frequency masking matrix corresponding to the de-reverberation; or,

[0023] performing dry sound and wet sound separation on a signal spectrum of the input audio signal, and using a ratio of the separated dry sound signal spectrum to the signal spectrum of the input audio signal as a time-frequency masking matrix;

[0024] The time-frequency masking matrix is ​​used to characterize the reverberation degree at each time-frequency point.

[0025] Optionally, the pre-trained dereverberation model is trained according to the following steps:

[0026] Obtaining clean audio and its reverberation audio, using the reverberation audio spectrum of the reverberation audio as the input of the de-reverberation model, using the time-frequency masking matrix corresponding to the de-reverberation as the output of the de-reverberation model, and training the de-reverberation model; or,

[0027] Clean audio and reverberation audio thereof are obtained, a first target artificial reverberation audio is generated based on the clean audio, and a second target artificial reverberation audio is generated based on the reverberation audio; the reverberation audio is used as input of the de-reverberation model, a time-frequency masking matrix corresponding to the ratio of the first target artificial reverberation audio to the second target artificial reverberation audio is used as output of the de-reverberation model, and the de-reverberation model is trained.

[0028] Optionally, the audio content feature includes a music style; and determining a content-adaptive masking matrix based on the audio content feature includes:

[0029] Based on the music style detection probability at the current time point and the suppression coefficients at different frequencies, a reverberation weight associated with the music style at the current time point is determined.

[0030] Optionally, the audio content feature includes a density of drum beats; and determining a content-adaptive masking matrix based on the audio content feature includes:

[0031] Based on the density and corresponding frequency of the drum beats at the current time point, a monotonically decreasing reverberation weight calculation function is used to determine the reverberation weight associated with the drum beats at the current time point.

[0032] Optionally, the audio content feature includes a reverberation degree, and the reverberation degree is represented by a time-frequency masking matrix; and determining the content-adaptive masking matrix based on the audio content feature includes:

[0033] Based on the masking value in the time-frequency masking matrix corresponding to the current time-frequency point, a reverberation weight related to the reverberation degree of the current time-frequency point is determined using a monotonically increasing reverberation weight calculation function.

[0034] Optionally, when the input audio signal includes a plurality of the audio content features, determining the content-adaptive masking matrix based on the audio content features includes:

[0035] respectively determining a reverberation masking matrix corresponding to each of the audio content features;

[0036] The reverberation masking matrices corresponding to the respective audio content features are combined to obtain the content-adaptive masking matrix of the input audio signal.

[0037] Another aspect of the present disclosure provides an audio reverberation system, the audio reverberation system comprising:

[0038] A pre-processing module is used to pre-process the input audio signal to obtain a reverberation input signal;

[0039] a reverberation generation module, configured to perform reverberation processing on the reverberation input signal to generate an initial reverberation audio signal of a target scene;

[0040] An audio content analysis module, configured to perform audio content analysis on the input audio signal to obtain audio content features corresponding to the input audio signal;

[0041] a content adaptive masking module, configured to determine a content adaptive masking matrix based on the audio content features;

[0042] a content adaptive reverberation module, configured to perform weighted mixing of the content adaptive masking matrix and the initial reverberation audio signal to obtain a content adaptive reverberation signal;

[0043] A mixing module is used to perform weighted mixing on the content adaptive reverberation signal and the reverberation input signal according to a preset ratio to obtain a final reverberation audio signal.

[0044] Optionally, the audio content feature includes a music style; the audio content analysis module is configured to perform audio content analysis on the input audio signal to obtain the audio content feature of the input audio signal, including:

[0045] The audio content analysis module is specifically used to:

[0046] Obtaining the music style of the input audio signal based on the audio tag of the input audio signal; or,

[0047] Audio features are obtained based on the music spectrum of the input audio signal, and the audio features are input into a pre-trained music classification model to obtain the music style of the input audio signal; wherein the music classification model is trained using audio features and corresponding classification labels.

[0048] Optionally, the audio content feature includes a density of drum beats; and the audio content analysis module is configured to perform audio content analysis on the input audio signal to obtain the audio content feature of the input audio signal, including:

[0049] The audio content analysis module is specifically used to:

[0050] Based on a note starting point detection scheme, energy or spectrum change information is used to determine the mutation point of the input audio signal, the mutation point with a mutation degree greater than a preset mutation threshold is regarded as a drum beat, and the drum beat density of the input audio signal is obtained based on the number of drum beats within a preset duration; or

[0051] The multi-frame spectrum of the input audio signal is input into a pre-trained drum beat detection model to obtain the probability of each time point corresponding to a drum sound. The time point with the probability greater than a preset probability threshold is regarded as a drum beat. Based on the number of drum beats within a preset duration, the drum beat density of the input audio signal is obtained.

[0052] Optionally, the audio content feature includes a reverberation degree; and the audio content analysis module is configured to perform audio content analysis on the input audio signal to obtain the audio content feature of the input audio signal, including:

[0053] The audio content analysis module is specifically used to:

[0054] Inputting the signal spectrum of the input audio signal into a pre-trained de-reverberation model to obtain a time-frequency masking matrix corresponding to the de-reverberation; or,

[0055] performing dry sound and wet sound separation on a signal spectrum of the input audio signal, and using a ratio of the separated dry sound signal spectrum to the signal spectrum of the input audio signal as a time-frequency masking matrix;

[0056] The time-frequency masking matrix is ​​used to characterize the reverberation degree at each time-frequency point.

[0057] Optionally, the audio reverberation system further includes:

[0058] A model training module is used to train the dereverberation model according to the following steps:

[0059] Obtaining clean audio and its reverberation audio, using the reverberation audio spectrum of the reverberation audio as the input of the de-reverberation model, using the time-frequency masking matrix corresponding to the de-reverberation as the output of the de-reverberation model, and training the de-reverberation model; or,

[0060] Clean audio and reverberation audio thereof are obtained, a first target artificial reverberation audio is generated based on the clean audio, and a second target artificial reverberation audio is generated based on the reverberation audio; the reverberation audio is used as input of the de-reverberation model, a time-frequency masking matrix corresponding to the ratio of the first target artificial reverberation audio to the second target artificial reverberation audio is used as output of the de-reverberation model, and the de-reverberation model is trained.

[0061] Optionally, the audio content feature includes a music style; and the content adaptive masking module is configured to determine a content adaptive masking matrix based on the audio content feature, comprising:

[0062] The content adaptive masking module is specifically used to:

[0063] Based on the music style detection probability at the current time point and the suppression coefficients at different frequencies, a reverberation weight associated with the music style at the current time point is determined.

[0064] Optionally, the audio content feature includes a density of drum beats; and the content adaptive masking module is configured to determine a content adaptive masking matrix based on the audio content feature, comprising:

[0065] The content adaptive masking module is specifically used to:

[0066] Based on the density and corresponding frequency of the drum beats at the current time point, a monotonically decreasing reverberation weight calculation function is used to determine the reverberation weight associated with the drum beats at the current time point.

[0067] Optionally, the audio content feature includes a reverberation degree, and the reverberation degree is represented by a time-frequency masking matrix; the content adaptive masking module is configured to determine a content adaptive masking matrix based on the audio content feature, including:

[0068] The content adaptive masking module is specifically used to:

[0069] Based on the masking value in the time-frequency masking matrix corresponding to the current time-frequency point, a reverberation weight related to the reverberation degree of the current time-frequency point is determined using a monotonically increasing reverberation weight calculation function.

[0070] Optionally, when the input audio signal includes a plurality of the audio content features, the content adaptive masking module is configured to determine a content adaptive masking matrix based on the audio content features, including:

[0071] The content adaptive masking module is specifically used to:

[0072] respectively determining a reverberation masking matrix corresponding to each of the audio content features;

[0073] The reverberation masking matrices corresponding to the respective audio content features are combined to obtain the content-adaptive masking matrix of the input audio signal.

[0074] Compared with the prior art, the present disclosure performs pre-processing on an input audio signal to obtain a reverberation input signal, performs reverberation processing on the reverberation input signal to generate an initial reverberation audio signal of a target scene, performs audio content analysis on the input audio signal to obtain corresponding audio content features, determines a content-adaptive masking matrix based on the audio content features, performs weighted mixing of the content-adaptive masking matrix with the initial reverberation audio signal to obtain a content-adaptive reverberation signal, and performs weighted mixing of the content-adaptive reverberation signal and the reverberation input signal according to a preset ratio to obtain a final reverberation audio signal. This achieves adaptation of different reverberation effects to audio signals with different audio contents, thereby avoiding the distortion problem of the final reverberation audio signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings, and these exemplifications do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0076] FIG1 is a flow chart of an audio reverberation method provided by one embodiment of the present disclosure;

[0077] FIG2 is a schematic structural diagram of an audio reverberation system provided in another embodiment of the present disclosure. Implementation Method

[0078] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. However, it will be understood by those skilled in the art that in each embodiment of the present disclosure, many technical details are provided to enable readers to better understand the present disclosure. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present disclosure can be implemented. The division of the following embodiments is for the convenience of description and should not constitute any limitation on the specific implementation of the present disclosure. The various embodiments can be combined and referenced with each other under the premise that there is no contradiction.

[0079] One embodiment of the present disclosure relates to an audio reverberation method, the process of which is shown in FIG1 , including:

[0080] In step S110, the input audio signal is pre-processed to obtain a reverberation input signal. Specifically, pre-processing may include equalization (EQ) adjustment to adjust the loudness and timbre of each frequency band in the input audio signal. Pre-processing may also include delay control to ensure that audio signals in multiple frequency bands meet corresponding delay requirements simultaneously. Of course, pre-processing may also include other operations beyond equalization and delay control, and those skilled in the art may select and configure these operations based on actual needs.

[0081] Step S120 , performing reverberation processing on the reverberation input signal to generate an initial reverberation audio signal of the target scene.

[0082] Generally speaking, reverberation is typically divided into early reverberation (EARLY Reverb) and late reverberation (LATE Reverb) based on the time difference between the direct sound (i.e., the audio signal transmitted directly from the sound source to the sound collection device) and the reverberation transmitted to the sound collection device. For example, reverberation that reaches the sound collection device within 30 milliseconds after the direct sound can be considered early reverberation, while reverberation that reaches the sound collection device more than 30 milliseconds can be considered late reverberation. Therefore, reverberation processing can include early reverberation generation and late reverberation generation to respectively generate early reverberation and late reverberation analog signals of the target scene, fuse the early reverberation and late reverberation analog signals into the reverberation input signal, and perform decorrelation on the fused signal to obtain the initial reverberation audio signal of the target scene. The decorrelation operation can include, but is not limited to, adding / subtracting time delay and scaling of the source signal.

[0083] Step S130: Perform audio content analysis on the input audio signal to obtain audio content features of the input audio signal. Specifically, the audio content features may include, but are not limited to, the musical style of the audio, the density of drum beats, the degree of reverberation at each time-frequency point, and other types. Those skilled in the art may employ different analysis methods to analyze the input audio signal for different types of audio content features, and this embodiment does not limit this.

[0084] Step S140, based on the audio content features, determines a content-adaptive masking matrix. Specifically, the content-adaptive masking matrix can be a two-dimensional matrix for characterizing the weighted weights corresponding to different audio content features at each time-frequency point. The content-adaptive masking matrix can be a weighted weight for a certain type of audio content feature alone, such as a weighted weight for any one of the musical style, the drum beat density, and the reverberation level. It can also be a weighted weight for a combination of multiple types of audio content features, such as a weighted weight for a combination of musical style and drum beat density, or a weighted weight for a combination of musical style and reverberation level, or a weighted weight for a combination of drum beat density and reverberation level, or a weighted weight for a combination of musical style, drum beat density, and reverberation level, and so on.

[0085] Step S150 , performing weighted mixing of the content-adaptive masking matrix and the initial reverberation audio signal to obtain a content-adaptive reverberation signal. Specifically, step S150 may first perform a Short-Time Fourier Transform (STFT) operation on the initial reverberation audio signal to obtain an initial reverberation signal spectrum, then multiply the initial reverberation signal spectrum by the content-adaptive masking matrix to obtain a weighted mixed content-adaptive reverberation spectrum, and then perform an Inverse Short-Time Fourier Transform (ISTFT) operation on the content-adaptive reverberation spectrum to obtain the content-adaptive reverberation signal.

[0086] It should be noted that before multiplying the initial reverberation signal spectrum with the content-adaptive masking matrix, the content-adaptive masking matrix may be smoothed in the time and frequency dimensions in advance so as to multiply the initial reverberation signal spectrum with the smoothed content-adaptive masking matrix, thereby avoiding excessively rapid changes in the reverberation after weighted mixing.

[0087] Of course, this embodiment does not limit the specific weighted mixing method of the content-adaptive masking matrix and the initial reverberation audio signal. Those skilled in the art may also use other methods to weightedly mix the content-adaptive masking matrix and the initial reverberation audio signal, as long as the content-adaptive reverberation signal can be obtained by weighted mixing the content-adaptive masking matrix and the initial reverberation audio signal.

[0088] In step S160, the content adaptive reverberation signal and the reverberation input signal are weighted mixed according to a preset ratio to obtain a final reverberation audio signal. Specifically, the preset ratio can be set according to actual needs, for example, the preset ratio can be 1:1, 1.1:0.8, 0.7:1.2, etc.

[0089] It should be noted that after the final reverberation audio signal is obtained, adjustment operations such as delay control and gain control may be performed on the final reverberation audio signal to meet actual needs.

[0090] Compared with the prior art, the audio reverberation method provided by the embodiments of the present disclosure performs pre-processing on an input audio signal to obtain a reverberation input signal, performs reverberation processing on the reverberation input signal to generate an initial reverberation audio signal of a target scene, performs audio content analysis on the input audio signal to obtain corresponding audio content features, determines a content-adaptive masking matrix based on the audio content features, performs weighted mixing of the content-adaptive masking matrix with the initial reverberation audio signal to obtain a content-adaptive reverberation signal, and weighted mixing of the content-adaptive reverberation signal and the reverberation input signal according to a preset ratio to obtain a final reverberation audio signal. This achieves adaptation of different reverberation effects to audio signals with different audio contents, thereby avoiding distortion of the final reverberation audio signal.

[0091] For example, the audio content feature includes a music style. The music style is also called a music type, which may include but is not limited to pop, rock, folk, electronic music, etc.

[0092] When the input audio signal carries an audio tag indicating its musical style, step S130 may include obtaining the musical style of the input audio signal based on the audio tag of the input audio signal. In other words, step S130 may directly obtain the corresponding musical style from the audio tag carried by the input audio signal, thereby quickly and accurately obtaining the musical style of the input audio signal.

[0093] With respect to the audio content feature of music style, step S130 may also utilize a music classification model based on a neural network to automatically obtain the music style of the input audio signal.

[0094] For example, step S130 may include: obtaining audio features based on the music spectrum of the input audio signal, and inputting the audio features into a pre-trained music classification model to obtain the music style of the input audio signal. The music classification model is trained using the audio features and corresponding classification labels.

[0095] Specifically, step S130 can divide the input audio signal into several music segments of a specific duration (e.g., 3 seconds), obtain the music spectrum of each music segment through an STFT operation, and then extract corresponding audio features such as Mel Frequency Cepstrum Coefficients (MFCCs), chromaticity features, spectral contrast, etc. from the music spectrum of each music segment. The extracted audio features of each music segment are input into a pre-trained music classification model to obtain the musical style of each music segment, and thus the musical style of the input audio signal. Of course, when step S130 divides the input audio signal into several music segments of a specific duration, the duration of each music segment can be the same or different, and this embodiment does not limit this.

[0096] It should be noted that this embodiment does not limit the specific type of music classification model. For example, the network type of the music classification model can be a convolutional neural network (CNN), a long short-term memory (LSTM) network, etc. During the training phase of the music classification model, the audio features of a large number of music clips and the corresponding music classification labels can be used as training data, and the music classification model can be trained using this training data. During the testing phase of the music classification model, the audio features of the test audio clip can be used as the input of the music classification model, and the music style detection probability P corresponding to the test audio clip can be used as the output of the music classification model to test the music classification model. The music style detection probability P is expressed as P=[p1,p2,..,pK], p1+p2+...+pK=1, p1, p2,...,pK represent the detection probabilities corresponding to music styles 1, 2,...,K respectively, and K is the number of music styles.

[0097] By using a pre-trained music classification model to obtain the music style of the input audio signal, the accuracy of determining the music style of the input audio signal can be effectively improved.

[0098] Exemplarily, the audio content feature includes drum beat density. When performing audio content analysis on the input audio signal, step S130 can obtain the drum beat density of the input audio signal in two ways: one way is to obtain the drum beat density of the input audio signal using a note onset detection scheme, and the other way is to obtain the drum beat density of the input audio signal using a pre-trained neural network-based drum beat detection model.

[0099] When using the note starting point detection scheme to obtain the drum beat density of the input audio signal, step S130 may include: based on the note starting point detection scheme, using energy or spectrum change information to determine the mutation point of the input audio signal, taking the mutation point with a mutation degree greater than a preset mutation threshold as the drum beat, and based on the number of drum beats within a preset duration, obtaining the drum beat density of the input audio signal.

[0100] Specifically, a characteristic of a note onset is a sudden increase in energy or a change in spectral energy distribution. Based on this characteristic of the note onset, various mutation points in the input audio signal can be determined. It should be noted that this embodiment does not limit the specific steps of the note onset detection scheme; as long as the mutation points of the input audio signal can be determined using energy or spectral change information, it will suffice.

[0101] The intensity of drum beats in an input audio signal can be characterized by the drum beat density per unit time. Let T be the preset duration. Using a sliding window of duration T, if the number of drum beats within the sliding window is N, the drum beat density D per unit time of the input audio signal is calculated as D = N / T.

[0102] The specific size of the preset mutation threshold can be set according to actual needs. In this embodiment, the preset mutation threshold is used as a criterion for screening drum beats from mutation points. Only mutation points with a mutation degree greater than the preset mutation threshold are considered as drum beats, which improves the accuracy of drum beat detection and further improves the accuracy of drum beat density determination.

[0103] When using a pre-trained neural network-based drum beat detection model to obtain the drum beat density of the input audio signal, step S130 may include: inputting multiple frames of the input audio signal's spectrum into the pre-trained drum beat detection model, obtaining the probability of each time point corresponding to a drum sound, taking the time point whose probability is greater than a preset probability threshold as a drum beat, and obtaining the drum beat density of the input audio signal based on the number of drum beats within a preset time length.

[0104] Specifically, the drum beat detection model can be built based on a deep neural network. For example, the drum beat detection model can include a multi-stage temporal convolutional network and a classifier. Each temporal convolutional network can include a convolutional layer, a batch normalization layer, an activation layer, a residual convolutional layer, etc. Of course, this embodiment does not limit the specific network structure of the drum beat detection model. Those skilled in the art can customize it according to actual needs, as long as a pre-trained drum beat detection model can be used to determine the probability of a drum sound corresponding to each time point based on the multi-frame spectrum of the input audio signal.

[0105] It should be noted that the probability of each time point corresponding to a drum sound refers to the probability of the audio signal at that time point being a drum beat. The preset probability threshold can be set based on actual needs. By using the preset probability threshold as the criterion for filtering out drum beats from each time point, only those time points whose probability of corresponding to a drum beat is greater than the preset probability threshold are considered drum beats. This improves the accuracy of drum beat detection and, in turn, the accuracy of determining the density of drum beats.

[0106] Exemplarily, the audio content features include reverberation level. The reverberation level at each time-frequency point can be represented using a time-frequency masking matrix. When performing audio content analysis on the input audio signal, step S130 can determine the time-frequency masking matrix of the input audio signal in two ways: one using a pre-trained dereverberation model, and the other using dry and wet sound separation.

[0107] When a pre-trained de-reverberation model is used, step S130 may include: inputting a signal spectrum of the input audio signal into the pre-trained de-reverberation model to obtain a time-frequency masking matrix corresponding to the de-reverberation.

[0108] For example, the dreverberation model can be established based on a neural network. This embodiment does not limit the specific network structure of the dreverberation model, as long as a pre-trained dreverberation model can be used to obtain a time-frequency masking matrix corresponding to the dreverberation based on the signal spectrum of the input audio signal.

[0109] There are two ways to train the dereverberation model.

[0110] In the first training method, the pre-trained dereverberation model is trained according to the following steps:

[0111] The clean audio and its reverberation audio are obtained, the reverberation audio spectrum of the reverberation audio is used as the input of the de-reverberation model, the time-frequency masking matrix corresponding to the de-reverberation is used as the output of the de-reverberation model, and the de-reverberation model is trained.

[0112] Specifically, the reverberation audio here refers to the reverberation audio obtained by superimposing reverberation data on clean audio. A large number of clean audio and its reverberation audio constitute data pairs as training data. In the model training stage, the reverberation audio spectrum Y of the reverberation audio is used as the input of the de-reverberation model, and the time-frequency masking matrix M corresponding to the de-reverberation of the reverberation audio spectrum Y is used as the output of the de-reverberation model. Then, the de-reverberation audio spectrum X′ corresponding to the de-reverberation audio obtained by de-reverberating the reverberation audio spectrum Y is X′=Y*M. By minimizing the error between the clean audio spectrum X and the de-reverberation audio spectrum X′ of the clean audio, the training of the de-reverberation model can be completed by continuous iteration using the training data. In the model testing stage, the test audio signal spectrum is used as the input of the de-reverberation model to obtain the time-frequency masking matrix corresponding to the de-reverberation of the test audio signal spectrum. The time-frequency masking matrix output by the de-reverberation model trained based on this training method can adjust the weighting weight of the artificial reverberation.

[0113] In the second training method, the pre-trained dereverberation model is trained according to the following steps:

[0114] Clean audio and its reverberation audio are obtained, and a first target artificial reverberation audio is generated based on the clean audio, and a second target artificial reverberation audio is generated based on the reverberation audio. The reverberation audio is used as the input of a de-reverberation model, and the time-frequency masking matrix corresponding to the ratio of the first target artificial reverberation audio to the second target artificial reverberation audio is used as the output of the de-reverberation model. The de-reverberation model trained using this method outputs a time-frequency masking matrix that is the weighted value required for the ideal artificial reverberation.

[0115] By using the de-reverberation model to obtain the time-frequency masking matrix that characterizes the reverberation degree of each time-frequency point, the accuracy of determining the time-frequency masking matrix can be effectively improved.

[0116] When dry and wet sound separation is adopted, step S130 may include: separating the dry sound and the wet sound on the signal spectrum of the input audio signal, and using the ratio of the separated dry sound signal spectrum to the signal spectrum of the input audio signal as the time-frequency masking matrix.

[0117] Specifically, for a multi-channel input audio signal Y, the principal component analysis (PCA) method can be used to separate the dry sound Y_dry and the wet sound Y_wet. Then Y_dry / Y is the time-frequency masking matrix corresponding to the de-reverberation of the input audio signal Y.

[0118] By adopting the dry-wet separation method to obtain the time-frequency masking matrix corresponding to the de-reverberation of the input audio signal, the accuracy of determining the time-frequency masking matrix corresponding to the multi-channel input audio signal can be effectively improved.

[0119] Exemplarily, when the audio content feature includes a music style, step S140 may include: determining a reverberation weight associated with the music style at the current time point based on the music style detection probability at the current time point and the suppression coefficients at different frequencies.

[0120] Specifically, different reverberation weighting coefficients can be set in advance for different musical styles. For example, for musical style k, its corresponding reverberation weighting coefficient can be denoted as ak, and the value range of ak is [0, 1]. When the number of musical styles is K, the value range of k is 1, 2, ..., K. The reverberation weighting coefficients corresponding to each musical style form the reverberation weighting coefficient set A = [a1, a2, ..., aK], where a1, a2, ..., aK are the reverberation weighting coefficients corresponding to musical styles 1, 2, ..., K, respectively.

[0121] For a specific time point t, assuming its current music style detection probability is P(t), the reverberation weight W1(t) associated with the music style at time point t can be expressed as W1(t)=A*P(t)=a1*p1(t)+a2*p2(t)+...+aK*pK(t). Where p1(t), p2(t), ..., pK(t) represent the detection probabilities of music styles 1, 2, ..., K at time point t, respectively.

[0122] Different suppression coefficients can be set for different musical styles at different frequencies. For example, the suppression coefficients a1(f), a2(f), …, aK(f) at frequency f corresponding to musical styles 1, 2, …, K can form a suppression coefficient set A(f), which is expressed as A(f) = [a1(f), a2(f), …, aK(f)]. Based on this, the musical style-dependent reverberation weight W1(t,f) at a time-frequency point (t,f) can be expressed as W1(t,f) = A(f) * P(t) = a1(f) * p1(t) + a2(f) * p2(t) + … + aK(f) * pK(t).

[0123] In this embodiment, the reverberation weighting weights related to the music style are determined by the suppression coefficients at different frequencies. The reverberation weighting weights can be used to further improve the reverberation adaptation effect of audio signals of different music styles.

[0124] Exemplarily, the audio content feature includes the density of drum beats. Step S140 includes: determining a reverberation weight associated with the drum beats at the current time point using a monotonically decreasing reverberation weight calculation function based on the density of the drum beats at the current time point and the corresponding frequency.

[0125] Specifically, for a specific time point t, let the current drum beat density be D(t). The reverberation weight associated with the drum beats at that time point t, W2(t), can be expressed as W2(t) = Fd(D(t)). Fd() represents a monotonically decreasing reverberation weight calculation function based on the drum beat density. The higher the drum beat density at time point t, the lower the corresponding artificial reverberation weight associated with the drum beats, and the less reverberation associated with the audio.

[0126] The drum beat-related reverberation weight can also be different at different frequencies. For a time-frequency point (t, f), the drum beat-related reverberation weight W2(t, f) can be expressed as W2(t, f) = Fd(D(t), f).

[0127] It should be noted that this embodiment does not limit the specific form of the monotonically decreasing reverberation weight calculation function, as long as the reverberation weight calculation function satisfies the monotonically decreasing condition.

[0128] In this embodiment, a monotonically decreasing reverberation weight calculation function is used to determine the reverberation weight associated with the drum beats. The reverberation weight can be used to further improve the reverberation adaptation effect of audio signals with different drum beat densities.

[0129] Exemplarily, the audio content feature includes a reverberation level, which is represented by a time-frequency masking matrix. Step S140 includes determining a reverberation weight associated with the reverberation level at the current time-frequency point using a monotonically increasing reverberation weight calculation function based on a masking value in the time-frequency masking matrix corresponding to the current time-frequency point.

[0130] Specifically, for a specific time-frequency point (t, f), when determining the reverberation weight associated with its reverberation level in step S140, a masking value M(t, f) in the time-frequency masking matrix corresponding to the time-frequency point (t, f) can be obtained through a preset neural network. Then, the artificial reverberation weight W3(t, f) associated with the reverberation level corresponding to the time-frequency point (t, f) can be expressed as W3(t, f) = Fr(M(t, f)), where Fr() represents a monotonically increasing reverberation weight calculation function based on the reverberation level. The lower the masking value in the time-frequency masking matrix corresponding to the time-frequency point (t, f), the higher the reverberation level of the corresponding audio, and the lower the artificial reverberation weight that needs to be superimposed. In particular, when the time-frequency masking matrix corresponds to the weight required for ideal artificial reverberation, Fr(M(t, f)) = M(t, f).

[0131] It should be noted that this embodiment does not limit the specific method for obtaining the masking value in the time-frequency masking matrix corresponding to the current time-frequency point; as long as the masking value in the time-frequency masking matrix corresponding to the current time-frequency point can be obtained, it will suffice. This embodiment also does not limit the specific form of the monotonically increasing reverberation weight calculation function; as long as the reverberation weight calculation function satisfies the requirement of monotonically increasing reverberation weight calculation function, it will suffice.

[0132] In this embodiment, a monotonically increasing reverberation weight calculation function is used to determine a reverberation weight related to the reverberation degree. The reverberation weight can be used to further improve the reverberation adaptation effect of audio signals with different reverberation degrees.

[0133] Exemplarily, when the input audio signal includes multiple audio content features, step S140 includes: determining a reverberation masking matrix corresponding to each audio content feature; and combining the reverberation masking matrices corresponding to the respective audio content features to obtain a content-adaptive masking matrix of the input audio signal.

[0134] For example, when the input audio signal includes multiple audio content features, namely, music style, drum beat density, and reverberation degree, step S140 can first obtain the reverberation masking matrix related to the music style of the current time-frequency point (t, f), the reverberation masking matrix related to the drum beat, namely, the reverberation weighted weight W1(t, f), and the reverberation masking matrix related to the reverberation degree, namely, the reverberation weighted weight W3(t, f), and then combine the reverberation weights to obtain the content-adaptive masking matrix W(t, f) of the current time-frequency point (t, f). W(t, f) can be expressed as W(t, f)=W1(t, f)*W2(t, f)*W3(t, f).

[0135] By combining and using the reverberation masking matrices corresponding to the characteristics of various audio contents, the reverberation adaptation effect of audio signals of different music contents can be further improved.

[0136] Another embodiment of the present disclosure relates to an audio reverberation system, as shown in FIG2 , comprising:

[0137] The pre-processing module 210 is used to pre-process the input audio signal to obtain a reverberation input signal;

[0138] The reverberation generation module 220 is used to perform reverberation processing on the reverberation input signal to generate an initial reverberation audio signal of the target scene;

[0139] The audio content analysis module 230 is used to perform audio content analysis on the input audio signal to obtain audio content features corresponding to the input audio signal;

[0140] A content adaptive masking module 240 is configured to determine a content adaptive masking matrix based on audio content features;

[0141] a content-adaptive reverberation module 250 for performing weighted mixing of the content-adaptive masking matrix and the initial reverberation audio signal to obtain a content-adaptive reverberation signal;

[0142] The mixing module 260 is configured to perform weighted mixing on the content adaptive reverberation signal and the reverberation input signal according to a preset ratio to obtain a final reverberation audio signal.

[0143] Specifically, as shown in FIG2 , the pre-processing module 210 may further include an equalization adjustment unit 211, which performs equalization adjustment to adjust the loudness and timbre of each frequency band in the input audio signal. The pre-processing module 210 may further include a delay control unit 212, which performs delay control to enable audio signals in multiple frequency bands to simultaneously meet corresponding delay requirements. Of course, the pre-processing module 210 may further include other units in addition to the equalization adjustment unit 211 and the delay control unit 212, and those skilled in the art may select and set them according to actual needs.

[0144] As shown in Figure 2, the reverberation generation module 220 may include an early reverberation generation unit 221, a late reverberation generation unit 222, and a decorrelation operation unit 223. The early reverberation generation unit 221 and the late reverberation generation unit 222 are respectively used to generate early reverberation and late reverberation, so as to respectively generate an early reverberation simulation signal and a late reverberation simulation signal of the target scene. The early reverberation simulation signal and the late reverberation simulation signal are fused into the reverberation input signal, and the decorrelation operation unit 223 is used to perform decorrelation operations such as adding / subtracting the delay and scaling of the source signal on the fused signal, so as to obtain the initial reverberation audio signal of the target scene.

[0145] Compared with the prior art, the audio reverberation system provided by the embodiments of the present disclosure uses a pre-processing module to pre-process an input audio signal to obtain a reverberation input signal, uses a reverberation generation module to perform reverberation processing on the reverberation input signal to generate an initial reverberation audio signal of a target scene, uses an audio content analysis module to perform audio content analysis on the input audio signal to obtain corresponding audio content features, uses a content adaptive masking module to determine a content adaptive masking matrix based on the audio content features, uses the content adaptive reverberation module to weightedly mix the content adaptive masking matrix with the initial reverberation audio signal to obtain a content adaptive reverberation signal, and uses a mixing module to weightedly mix the content adaptive reverberation signal and the reverberation input signal according to a preset ratio to obtain a final reverberation audio signal. This achieves the adaptation of different reverberation effects to audio signals with different audio contents, thereby avoiding the distortion problem of the final reverberation audio signal.

[0146] Illustratively, the audio content feature includes music style.

[0147] The audio content analysis module 230 is used to perform audio content analysis on the input audio signal to obtain audio content features of the input audio signal, including:

[0148] The audio content analysis module 230 is specifically used to: obtain the music style of the input audio signal based on the audio label of the input audio signal; or, obtain audio features based on the music spectrum of the input audio signal, input the audio features into a pre-trained music classification model, and obtain the music style of the input audio signal; wherein the music classification model is trained using audio features and corresponding classification labels.

[0149] Exemplarily, the audio content feature includes the density of drum beats.

[0150] The audio content analysis module 230 is used to perform audio content analysis on the input audio signal to obtain audio content features of the input audio signal, including:

[0151] The audio content analysis module 230 is specifically used to: based on the note starting point detection scheme, use energy or spectrum change information to determine the mutation point of the input audio signal, take the mutation point with a mutation degree greater than a preset mutation threshold as the drum beat, and obtain the drum beat density of the input audio signal based on the number of drum beats within a preset duration; or input the multi-frame spectrum of the input audio signal into a pre-trained drum beat detection model to obtain the probability of the drum sound corresponding to each time point, take the time point with a probability greater than a preset probability threshold as the drum beat, and obtain the drum beat density of the input audio signal based on the number of drum beats within a preset duration.

[0152] Exemplarily, the audio content feature includes the degree of reverberation.

[0153] The audio content analysis module 230 is used to perform audio content analysis on the input audio signal to obtain audio content features of the input audio signal, including:

[0154] The audio content analysis module 230 is specifically used to: input the signal spectrum of the input audio signal into a pre-trained de-reverberation model to obtain a time-frequency masking matrix corresponding to the de-reverberation; or, separate the signal spectrum of the input audio signal into dry sound and wet sound, and use the ratio of the separated signal spectrum of the dry sound to the signal spectrum of the input audio signal as the time-frequency masking matrix; wherein the time-frequency masking matrix is ​​used to characterize the degree of reverberation at each time-frequency point.

[0155] Exemplarily, the audio reverberation system further includes:

[0156] The model training module is used to train the de-reverberation model according to the following steps:

[0157] Obtaining clean audio and its reverberation audio, using the reverberation audio spectrum of the reverberation audio as the input of the de-reverberation model, using the time-frequency masking matrix corresponding to the de-reverberation as the output of the de-reverberation model, and training the de-reverberation model; or,

[0158] Clean audio and its reverberation audio are obtained, a first target artificial reverberation audio is generated based on the clean audio, and a second target artificial reverberation audio is generated based on the reverberation audio; the reverberation audio is used as the input of a de-reverberation model, a time-frequency masking matrix corresponding to the ratio of the first target artificial reverberation audio to the second target artificial reverberation audio is used as the output of the de-reverberation model, and the de-reverberation model is trained.

[0159] Illustratively, the audio content feature includes music style.

[0160] The content-adaptive masking module 240 is configured to determine a content-adaptive masking matrix based on audio content features, including:

[0161] The content adaptive masking module 240 is specifically configured to determine a reverberation weight associated with the music style at the current time point based on the music style detection probability at the current time point and the suppression coefficients at different frequencies.

[0162] Exemplarily, the audio content feature includes the density of drum beats.

[0163] The content-adaptive masking module 240 is configured to determine a content-adaptive masking matrix based on audio content features, including:

[0164] The content adaptive masking module 240 is specifically configured to determine the reverberation weight associated with the drum beats at the current time point based on the density of the drum beats at the current time point and the corresponding frequency using a monotonically decreasing reverberation weight calculation function.

[0165] Exemplarily, the audio content feature includes a reverberation degree, and the reverberation degree is represented by a time-frequency masking matrix.

[0166] The content-adaptive masking module 240 is configured to determine a content-adaptive masking matrix based on audio content features, including:

[0167] The content adaptive masking module 240 is specifically configured to determine a reverberation weight related to the reverberation degree of the current time-frequency point based on the masking value in the time-frequency masking matrix corresponding to the current time-frequency point and using a monotonically increasing reverberation weight calculation function.

[0168] Exemplarily, when the input audio signal includes multiple audio content features, the content adaptive masking module 240 is configured to determine a content adaptive masking matrix based on the audio content features, including:

[0169] The content adaptive masking module 240 is specifically configured to: determine a reverberation masking matrix corresponding to each audio content feature; and combine the reverberation masking matrices corresponding to the respective audio content features to obtain a content adaptive masking matrix of the input audio signal.

[0170] 2 , the audio reverberation system further includes a system tuning module 270. The system tuning module 270 is configured to perform adjustment operations such as delay control and gain control on the final reverberation audio signal obtained by the mixing module, so as to output an adjusted final reverberation audio signal that meets actual requirements.

[0171] The specific implementation method of the audio reverberation system provided by the embodiment of the present disclosure can be found in the description of the audio reverberation method provided by the embodiment of the present disclosure, and will not be repeated here.

[0172] Those skilled in the art will appreciate that the above-mentioned embodiments are specific embodiments for implementing the present disclosure, and that in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present disclosure.

Claims

1. An audio reverberation method, characterized in that: The audio reverberation method comprises: Pre-processing the input audio signal to obtain a reverberation input signal; Performing reverberation processing on the reverberation input signal to generate an initial reverberation audio signal of a target scene; Performing audio content analysis on the input audio signal to obtain audio content features of the input audio signal; Determining a content adaptive masking matrix based on the audio content characteristics; Performing weighted mixing of the content adaptive masking matrix and the initial reverberation audio signal to obtain a content adaptive reverberation signal; The content adaptive reverberation signal and the reverberation input signal are weighted mixed according to a preset ratio to obtain a final reverberation audio signal.

2. The audio reverberation method according to claim 1, characterized in that: The audio content feature includes a music style; the audio content analysis of the input audio signal to obtain the audio content feature of the input audio signal includes: Based on the audio tag of the input audio signal, obtaining the music style of the input audio signal; or, An audio feature is obtained based on the music spectrum of the input audio signal, and the audio feature is input into a pre-trained music classification model to obtain the music style of the input audio signal; wherein the music classification model is trained using audio features and corresponding classification labels.

3. The audio reverberation method according to claim 1, characterized in that: The audio content feature includes the density of drum beats; the audio content analysis of the input audio signal to obtain the audio content feature of the input audio signal includes: Based on the note starting point detection scheme, the mutation point of the input audio signal is determined by using energy or spectrum change information, the mutation point with a mutation degree greater than a preset mutation threshold is taken as a drum point, and the density of the drum points of the input audio signal is obtained based on the number of the drum points within a preset duration; or, The multi-frame frequency spectrum of the input audio signal is input into a pre-trained drum beat detection model to obtain the probability of each time point corresponding to the drum sound, and the time point with the probability greater than a preset probability threshold is taken as the drum beat. Based on the number of the drum beats within a preset time length, the drum beat density of the input audio signal is obtained.

4. The audio reverberation method according to claim 1, characterized in that: The audio content feature includes a reverberation degree; the performing audio content analysis on the input audio signal to obtain the audio content feature of the input audio signal includes: Inputting the signal spectrum of the input audio signal into a pre-trained de-reverberation model to obtain a time-frequency masking matrix corresponding to de-reverberation; or, Separating the signal spectrum of the input audio signal into dry sound and wet sound, and using the ratio of the signal spectrum of the dry sound obtained by separation to the signal spectrum of the input audio signal as a time-frequency masking matrix; The time-frequency masking matrix is ​​used to characterize the reverberation degree at each time-frequency point.

5. The audio reverberation method according to claim 4, characterized in that: The pre-trained de-reverberation model is trained according to the following steps: Acquire clean audio and reverberation audio thereof, use the reverberation audio spectrum of the reverberation audio as the input of the de-reverberation model, use the time-frequency masking matrix corresponding to the de-reverberation as the output of the de-reverberation model, and train the de-reverberation model; or, Acquire clean audio and its reverberation audio, generate a first target artificial reverberation audio based on the clean audio, and generate a second target artificial reverberation audio based on the reverberation audio; use the reverberation audio as the input of the de-reverberation model, use the time-frequency masking matrix corresponding to the ratio of the first target artificial reverberation audio to the second target artificial reverberation audio as the output of the de-reverberation model, and train the de-reverberation model.

6. The audio reverberation method according to claim 1, characterized in that: The audio content feature includes a music style; and determining a content adaptive masking matrix based on the audio content feature includes: Based on the music style detection probability at the current time point and the suppression coefficients at different frequencies, a reverberation weight associated with the music style at the current time point is determined.

7. The audio reverberation method according to claim 1, characterized in that: The audio content feature includes a density of drum beats; and determining a content adaptive masking matrix based on the audio content feature includes: Based on the density of the drum beats and the corresponding frequency at the current time point, a reverberation weight associated with the drum beats at the current time point is determined using a monotonically decreasing reverberation weight calculation function.

8. The audio reverberation method according to claim 1, characterized in that: The audio content feature includes a reverberation degree, and the reverberation degree is represented by a time-frequency masking matrix; and determining a content adaptive masking matrix based on the audio content feature includes: Based on the masking value in the time-frequency masking matrix corresponding to the current time-frequency point, a reverberation weighting weight related to the reverberation degree of the current time-frequency point is determined using a monotonically increasing reverberation weight calculation function.

9. The audio reverberation method according to any one of claims 1 to 8, characterized in that: When the input audio signal includes a plurality of the audio content features, determining a content adaptive masking matrix based on the audio content features comprises: Determine the reverberation masking matrix corresponding to each of the audio content features respectively; The reverberation masking matrices corresponding to the respective audio content features are combined to obtain the content adaptive masking matrix of the input audio signal.

10. An audio reverberation system, characterized in that: The audio reverberation system comprises: A pre-processing module, used for pre-processing the input audio signal to obtain a reverberation input signal; A reverberation generation module, used to perform reverberation processing on the reverberation input signal to generate an initial reverberation audio signal of a target scene; An audio content analysis module, used to perform audio content analysis on the input audio signal to obtain audio content features of the input audio signal; A content adaptive masking module, configured to determine a content adaptive masking matrix based on the audio content features; A content adaptive reverberation module, configured to perform weighted mixing of the content adaptive masking matrix and the initial reverberation audio signal to obtain a content adaptive reverberation signal; The mixing module is used to perform weighted mixing of the content adaptive reverberation signal and the reverberation input signal according to a preset ratio to obtain a final reverberation audio signal.

Images