Sound signal separation method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of sound processing, and provides a sound signal separation method and device, electronic equipment and a storage medium, and the method comprises the steps: determining the short-time energy of environment sound based on the short-time energy of each channel signal and the cross correlation coefficient between the channel signals, and enabling the environment sound of each channel to have the same short-time energy; determining an environmental sound mask of each channel based on the short-time energy of the environmental sound and the short-time energy of each channel signal; and determining the environment sound and the coherent sound of each channel based on the environment sound mask of each channel and the signal of each channel. According to the invention, the environment sound and the coherent sound can be effectively separated by using the environment sound mask of each channel, the separation effect is improved, and the method can be suitable for various stereo scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of sound processing, and in particular to a sound signal separation method, device, electronic equipment and storage medium. Background Art

[0002] Spatial sound reproduction technology is a technology that reproduces the distribution and propagation of sound in three-dimensional space through technical means. Its main purpose is to create a realistic sound field around the listener so that the listener can feel the direction, distance and depth of the sound, thereby obtaining a more immersive auditory experience.

[0003] In order to achieve better spatial sound reproduction, it is necessary to separate the coherent sound from the ambient sound in stereo, so that the coherent sound and the ambient sound can be processed separately during later tuning. However, due to the limitations of existing stereo audio formats, coherent sound and ambient sound are often recorded and transmitted together, making it difficult to effectively separate the two. Therefore, how to effectively separate coherent sound from ambient sound has become the focus of current research. Summary of the invention

[0004] The present invention provides a sound signal separation method, device, electronic device and storage medium, which are used to solve the defect of difficulty in separating coherent sound from ambient sound in the related art.

[0005] The present invention provides a sound signal separation method, comprising: Determining the short-time energy of the ambient sound based on the short-time energy of each channel signal and the mutual correlation coefficient between the channel signals, wherein the ambient sound of each channel has the same short-time energy; Determining the ambient sound mask of each channel based on the short-time energy of the ambient sound and the short-time energy of each channel signal; Based on the ambient sound masks of the channels and the channel signals, the ambient sound and the coherent sound of the channels are determined.

[0006] According to a sound signal separation method provided by the present invention, the short-time energy of the ambient sound is determined based on the short-time energy of each channel signal and the mutual correlation coefficient between the channel signals, including: Performing short-time Fourier transform on the time domain representation of each channel signal to obtain the time-frequency domain representation of each channel signal; Determine a linear combination energy representation of each channel signal based on the time-frequency domain representation of each channel signal; Determining an energy representation of a cross-correlation coefficient between the channel signals based on a time-frequency domain representation of the channel signals; The short-time energy of the ambient sound is determined based on the linear combination energy representation, the energy representation of the cross-correlation coefficient, the short-time energy of each channel signal and the cross-correlation coefficient between each channel signal.

[0007] According to a sound signal separation method provided by the present invention, the step of determining the linear combination energy representation of each channel signal based on the time-frequency domain representation of each channel signal comprises: Based on the time-frequency domain representations of the channel signals and the equivalent relationship between the short-time energies of the ambient sounds of the channels, a linear combination energy representation of the channel signals is determined.

[0008] According to a sound signal separation method provided by the present invention, each channel signal includes a first channel signal and a second channel signal, the time domain representation of the first channel signal is obtained by superimposing the time domain representation of the coherent sound and the time domain representation of the first channel ambient sound, the time domain representation of the second channel signal is obtained by multiplying the time domain representation of the coherent sound by a difference factor and superimposing the time domain representation of the second channel ambient sound, and the difference factor is an amplitude difference factor between the first channel coherent sound and the second channel coherent sound.

[0009] According to a sound signal separation method provided by the present invention, the determining of the ambient sound mask of each channel based on the short-time energy of the ambient sound and the short-time energy of each channel signal includes: Constructing a time-frequency domain representation relationship between the ambient sound of each channel and the signal of each channel; Based on the time-frequency domain representation relationship, the linear combination energy representation of the channel signals, the short-time energy of the ambient sound and the short-time energy of the channel signals, the ambient sound mask of each channel is determined.

[0010] According to a sound signal separation method provided by the present invention, the determining of the ambient sound and the coherent sound of each channel based on the ambient sound mask of each channel and the signal of each channel includes: Multiplying the ambient sound mask of any channel by the signal of any channel to obtain the ambient sound of any channel; The ambient sound of any channel is subtracted from the signal of any channel to obtain the coherent sound of any channel.

[0011] The present invention also provides a sound signal separation device, comprising: An energy determination unit, configured to determine the short-time energy of the ambient sound based on the short-time energy of each channel signal and the mutual correlation coefficient between the channel signals, wherein the ambient sound of each channel has the same short-time energy; a mask determination unit, configured to determine the ambient sound mask of each channel based on the short-time energy of the ambient sound and the short-time energy of each channel signal; The sound separation unit is used to determine the ambient sound and the coherent sound of each channel based on the ambient sound mask of each channel and the channel signal.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, any of the above-mentioned sound signal separation methods is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the sound signal separation method as described above is implemented.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the sound signal separation method described above is implemented.

[0015] The sound signal separation method, device, electronic device and storage medium provided by the present invention can determine the short-time energy of the ambient sound based on the short-time energy of each channel signal and the mutual correlation coefficient between the channel signals. According to the short-time energy of the ambient sound and the short-time energy of each channel signal, the ambient sound mask of each channel can be further determined, that is, the mask of the ambient sound in each channel. Therefore, according to the ambient sound mask of each channel, the ambient sound and the coherent sound can be separated from the signal of each channel, thereby ensuring the effectiveness and reliability of the separation of the coherent sound and the ambient sound. The whole separation process is simple to calculate and easy to implement through programming, and can be applicable to a variety of stereo scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flow chart of the sound signal separation method provided by the present invention; Figure 2 It is a flow chart of the method for separating coherent sound from ambient sound based on time-frequency domain mask provided by the present invention; Figure 3 It is a structural schematic diagram of the sound signal separation device provided by the present invention; Figure 4It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] In order to allow users to have a more realistic sound experience in immersive systems such as virtual reality, spatial sound playback technology is particularly important. In order to achieve better spatial sound playback effects, it is necessary to separate the coherent sound and ambient sound in stereo so that the coherent sound and ambient sound can be processed separately during post-tuning. Among them, coherent sound refers to sound signals that can be located, such as sound signals generated by character dialogues, and ambient sound refers to sound signals that cannot be located, such as background sound effects.

[0020] However, current spatial sound reproduction technology faces a major challenge, namely the mixing of coherent sound and ambient sound. In existing stereo audio formats, coherent sound and ambient sound are usually recorded or stored in a mixed manner. Due to the mutual interweaving and interference between the two in the audio signal, it is difficult to completely distinguish them through simple signal processing methods. This mixing not only affects the spatial positioning accuracy of the sound, but also limits the implementation effect of spatial sound reproduction technology.

[0021] In this regard, the present invention provides a sound signal separation method based on time-frequency domain masking, which can separate the mixed sound in stereo into localizable coherent sound (such as character dialogue) and non-localizable environmental sound (such as background sound effects). The two separated signals can be processed again in the later stage to create a more enveloping and immersive auditory atmosphere, which helps to achieve flexible spatial sound playback, thereby overcoming the above-mentioned defects.

[0022] It should be noted that the significance of separating coherent sound from ambient sound in the present invention is: (1) enhancing the sense of immersion, by accurately locating the coherent sound source and finely controlling the ambient sound, so that the user feels as if they are in the scene; (2) improving the sound quality, clarity and fidelity of the sound; (3) adapting to personalized needs and providing personalized customization according to the preferences of different users.

[0023] Figure 1 is a flow chart of the sound signal separation method provided by the present invention, such as Figure 1 As shown, the method includes: Step 110: Determine the short-time energy of the ambient sound based on the short-time energy of each channel signal and the mutual correlation coefficient between the channel signals. The ambient sound of each channel has the same short-time energy.

[0024] It should be noted that each channel signal refers to a sound signal collected by different channels. For example, each channel signal can be a two-channel signal or a multi-channel signal, which is not specifically limited in the embodiment of the present invention. For ease of understanding, the embodiment of the present invention is mainly introduced by taking a two-channel signal as an example.

[0025] Specifically, when each channel signal is a two-channel signal, each channel signal may include a first channel signal and a second channel signal, where the first channel signal and the second channel signal refer to sound signals collected by the first channel and the second channel, respectively. The first channel and the second channel may be dual channels required to construct a stereo effect, i.e., two independent left and right audio channels. In other words, the first channel signal may be obtained by collecting the left channel, and the second channel signal may be obtained by collecting the right channel.

[0026] It can be understood that the sound signal collected in each channel can be understood as a sound signal obtained by superimposing coherent sound and ambient sound, that is, the first channel signal and the second channel signal are both mixed sound signals of coherent sound and ambient sound.

[0027] Furthermore, according to the characteristics of coherent sound and ambient sound, the coherent sound between different channels is completely correlated, the ambient sound between different channels is uncorrelated, and the coherent sound is uncorrelated with the ambient sound in each channel. Here, complete correlation means that two or more sound signals have a high degree of similarity and consistency in terms of waveform, frequency, phase, etc., which usually means that these sound signals come from the same sound source; uncorrelated means that two or more sound signals have no significant similarity and consistency in terms of waveform, frequency, phase, etc., which usually means that these sound signals come from different sound sources.

[0028] For the first channel signal and the second channel signal collected from different channels, a time domain representation of the first channel signal and the second channel signal can be established. Here, since the sound signal under each channel can be represented as the superposition of coherent sound and ambient sound, the time domain representation of the first channel signal can be understood as the linear superposition of the time domain representation of the coherent sound under the first channel and the time domain representation of the ambient sound; the time domain representation of the second channel signal is the linear superposition of the time domain representation of the coherent sound under the second channel and the time domain representation of the ambient sound.

[0029] Since the coherent sounds between different channels are completely correlated, the difference factor between the coherent sounds in the first channel and the coherent sounds in the second channel can be defined, that is, the coherent sounds in the second channel can be represented in the form of multiplying the difference factor and the coherent sounds in the first channel. Based on this, the time domain representation of the second channel signal can be determined by multiplying the time domain representation of the coherent sounds in the first channel by the difference factor, and then linearly superimposing it with the time domain representation of the ambient sound in the second channel.

[0030] After establishing the time domain representation of each channel signal, a short-time Fourier transform can be performed on each of them to convert the representation of each channel signal in the Fourier transform domain, that is, the time-frequency domain representation of each channel signal. Here, the short-time Fourier transform is a mathematical transformation related to the Fourier transform, which is used to determine the frequency and phase of the sine wave in the local area of ​​the time-varying signal. Unlike the Fourier transform, the short-time Fourier transform takes into account the time variability of the signal. By applying a time window to the signal, the signal is divided into multiple short time periods (or frames), and then the Fourier transform is performed on each short time period. In this way, the spectrum information of the signal in different time periods can be obtained, so as to better understand the time-frequency characteristics of the signal.

[0031] According to the time-frequency domain representation of each channel signal, the relationship between the short-time energy of each channel signal and the short-time energy of the coherent sound and the short-time energy of the ambient sound in each channel can be established. Here, the short-time energy represents the sum of the energy of the sound signal in a time period, which can be used to reflect the strength of the sound signal in the time period. For example, in the case where each channel signal is a two-channel signal, according to the time-frequency domain representation of the first channel signal and the second channel signal, the relationship between the short-time energy of the first channel signal and the short-time energy of the ambient sound and the short-time energy of the coherent sound in the first channel, and the relationship between the short-time energy of the second channel signal and the short-time energy of the ambient sound and the short-time energy of the coherent sound in the second channel can be established.

[0032] Since the first channel signal and the second channel signal are both known sound signals obtained by collection, the short-time energy value of the first channel signal and the short-time energy value of the second channel signal can be directly calculated, and the short-time energy value of the first channel signal and the short-time energy value of the second channel signal are substituted into the relationship between the short-time energies established above to solve them, so as to obtain the short-time energy values ​​of the ambient sound and the coherent sound respectively, as well as the difference factors of the coherent sound in different channels.

[0033] It should be understood that, according to the characteristics of ambient sound, the short-time energy of ambient sound collected in different channels is the same, therefore, the short-time energy value of ambient sound obtained by solution is unique, that is, the short-time energy value of ambient sound in each channel is consistent. In addition, according to the characteristics of coherent sound, the short-time energy of coherent sound collected in different channels is different, and the coherent sound between different channels is completely correlated, therefore, the short-time energy value of coherent sound obtained by solution can represent the short-time energy value of coherent sound in one channel, and the short-time energy value of coherent sound multiplied by the difference factor between channels can obtain the short-time energy value of coherent sound in another channel.

[0034] Step 120: Determine the ambient sound mask of each channel based on the short-time energy of the ambient sound and the short-time energy of each channel signal.

[0035] Specifically, based on the calculated short-time energy of the ambient sound (i.e., the short-time energy value obtained by solving) and the short-time energy of each channel signal (i.e., the known short-time energy value), the ambient sound mask of each channel can be calculated, that is, the mask of the ambient sound in each channel. For example, since the short-time energy of the ambient sound in each channel is the same, the ambient sound mask in the first channel can be calculated based on the short-time energy of the ambient sound and the short-time energy of the first channel signal; the ambient sound mask in the second channel can be calculated based on the short-time energy of the ambient sound and the short-time energy of the second channel signal.

[0036] Here, the ambient sound mask of each channel refers to the mask used to extract or separate the ambient sound from the original signal in the sound signal separation task. It can be regarded as a filter or weight matrix, which acts on the original signal to extract the ambient sound component of interest. It should be understood that the ambient sound mask of each channel is a quantity related to the autocorrelation and cross-correlation of each channel signal. In order to achieve extraction, the ambient sound mask of each channel can be limited to a positive real number.

[0037] Step 130: Determine the ambient sound and coherent sound of each channel based on the ambient sound mask of each channel and the channel signal.

[0038] Specifically, after calculating the ambient sound mask of each channel, the separated ambient sound can be obtained by multiplying the mask with the channel signal. Subsequently, the separated ambient sound is subtracted from the original channel signal to obtain the coherent sound.

[0039] Exemplarily, after calculating the ambient sound mask of the first channel and the ambient sound mask of the second channel, the ambient sound mask of the first channel is multiplied by the first channel signal to obtain the ambient sound of the first channel, and the estimated ambient sound of the first channel is subtracted from the first channel signal to obtain the coherent sound of the first channel. Similarly, the ambient sound mask of the second channel is multiplied by the second channel signal to obtain the ambient sound of the second channel, and the estimated ambient sound of the second channel is subtracted from the second channel signal to obtain the coherent sound of the second channel.

[0040] The method provided by the embodiment of the present invention can determine the short-time energy of the ambient sound based on the short-time energy of each channel signal and the mutual correlation coefficient between the channel signals. According to the short-time energy of the ambient sound and the short-time energy of each channel signal, the ambient sound mask of each channel can be further determined, that is, the mask of the ambient sound in each channel. Therefore, according to the ambient sound mask of each channel, the ambient sound and the coherent sound can be separated from the signal of each channel, thereby ensuring the effectiveness and reliability of the separation of the coherent sound and the ambient sound. The whole separation process is simple to calculate and easy to implement through programming, and can be applicable to a variety of stereo scenes.

[0041] Based on any of the above embodiments, each channel signal includes a first channel signal and a second channel signal, the time domain representation of the first channel signal is obtained by superimposing the time domain representation of the coherent sound and the time domain representation of the first channel ambient sound, the time domain representation of the second channel signal is obtained by multiplying the time domain representation of the coherent sound by a difference factor and superimposing the time domain representation of the second channel ambient sound, and the difference factor is an amplitude difference factor between the first channel coherent sound and the second channel coherent sound.

[0042] Specifically, assuming that the first channel is the left channel and the second channel is the right channel, the time domain representation of the first channel signal is And the time domain representation of the second channel signal They are defined as follows: In the formula, is the coherent sound component of the left channel, is the ambient sound component of the left channel. is the amplitude difference factor between the coherent sound components in the left channel and the right channel, is the coherent sound of the right channel, This is the ambient sound in the right channel. Represents the time domain sampling point.

[0043] Understandably, and They are the coherent sound and the ambient sound in the first channel respectively, that is, the coherent sound and the ambient sound in the first channel can be added to obtain the first channel signal. and The coherent sound and the ambient sound in the second channel are added together, that is, the second channel signal can be obtained by adding the coherent sound and the ambient sound in the second channel.

[0044] Based on the above embodiment, step 110 specifically includes: Step 111, performing short-time Fourier transform on the time domain representation of each channel signal to obtain the time-frequency domain representation of each channel signal.

[0045] It should be noted that in practical applications, most sound signals are non-stationary signals, that is, their frequency components change over time. Traditional sound signals in the time domain cannot provide time and frequency information at the same time, so they cannot effectively process non-stationary signals. Short-time Fourier transform captures the frequency characteristics of sound signals that change over time by dividing the sound signal into multiple short time periods (windows) in the time domain and performing Fourier transform on each time period.

[0046] Specifically, in order to better apply it to non-stationary states and situations where multiple sound sources exist simultaneously, the signal model can be converted to the short-time Fourier transform domain for processing. That is, the time domain representation of the left and right channel signals in the above formula (1) is subjected to a short-time Fourier transform, and the representation of the left and right channel signals in the Fourier transform domain (that is, the time-frequency domain representation) can be obtained: in, Represents the time-frequency domain representation of the left channel signal (i.e. the first channel signal), Represents the time-frequency domain representation of the right channel signal (i.e., the second channel signal), represents the coherent acoustic signal in the time-frequency domain, Represents the ambient sound signal in the time-frequency domain of the left channel, Represents the ambient sound signal in the time-frequency domain of the right channel, represents the timeframe index, Indicates the frequency index, Indicates the difference factor between the coherent sound of the left and right channels in the time-frequency domain. In order to keep the formula simple, the According to the above formula (2), we can estimate the parameter , , and The extraction of coherent sound and ambient sound can be completed.

[0047] Step 112: Determine a linear combination energy representation of each channel signal based on the time-frequency domain representation of each channel signal.

[0048] Specifically, based on the time-frequency domain representation of each channel signal, it can be determined that the short-time energy of the first channel signal is the sum of the short-time energy of the coherent sound of the first channel and the short-time energy of the ambient sound of the first channel, and the short-time energy of the second channel signal is the sum of the short-time energy of the coherent sound of the second channel and the short-time energy of the ambient sound of the second channel. Thus, the linear combination energy representation of the first channel signal and the linear combination energy representation of the second channel signal can be obtained.

[0049] Furthermore, step 112 specifically includes: Based on the time-frequency domain representations of the channel signals and the equivalent relationship between the short-time energies of the ambient sounds of the channels, a linear combination energy representation of the channel signals is determined.

[0050] Specifically, the linear combination energy representation includes the short-time energy representation of the first channel signal and the short-time energy representation of the second channel signal. Taking the first channel signal as the left channel signal as an example, the short-time energy of the first channel signal can be represented as follows: In the formula, is the short-time energy representation of the first channel signal, is the time-frequency domain representation of the first channel signal, Represents short-time average. Similarly, the short-time energy of the second channel signal (i.e., the right channel signal) can be expressed as: Assume that the ambient sound of the first channel signal and the second channel signal in formula (2) have the same short-term energy, which can be expressed as , the short-time energy of coherent sound is recorded as According to the short-time energy representation of the first channel signal and the second channel signal, the formula , the above formula (2) can be expressed as: In the formula, and are the short-time energies of the first channel signal and the second channel signal, is the short-time energy of the coherent sound in the first channel, is the short-time energy of the coherent sound in the second channel, is the short-time energy of the ambient sound. Formula (3) here is the linear combination energy representation of each channel signal. Considering that the short-time energy of the ambient sound in different channels is the same, in the linear combination energy representation of the first channel signal and the linear combination energy representation of the second channel signal, the short-time energy of the ambient sound can be expressed as .

[0051] Step 113: Determine the energy representation of the cross-correlation coefficients between the channel signals based on the time-frequency domain representation of the channel signals.

[0052] Specifically, after obtaining the time-frequency domain representation of the first channel signal and the second channel signal, the mutual correlation coefficient between the first channel signal and the second channel signal can be calculated, and in the process of calculating the mutual correlation coefficient, the time-frequency domain representation of the first channel signal and the time-frequency domain representation of the second channel signal can be substituted into the calculation formula of the mutual correlation coefficient. The mutual correlation coefficient thus obtained is the mutual correlation coefficient expressed in the form of short-time energy of the sound signal. In the embodiment of the present invention, the mutual correlation coefficient expressed in the form of short-time energy is recorded as the energy representation of the mutual correlation coefficient.

[0053] Here, the calculation formula of the mutual correlation coefficient between the first channel signal and the second channel signal may be: In the formula, is the mutual correlation coefficient, and are the time-frequency domain representations of the first channel signal and the second channel signal respectively.

[0054] Substituting the above formula (2) into formula (4), we can get the following formula: Since the coherent sound is unrelated to the ambient sound in each channel, , In addition, since the ambient sounds in different channels are unrelated, .

[0055] Therefore, the calculation formula of the mutual correlation coefficient can be expressed as follows, which is the energy representation of the mutual correlation coefficient: In the formula, the mutual correlation coefficient The short-term energy of the first channel signal , the short-time energy of the second channel signal , the short-term energy of coherent sound And the time-frequency domain representation of the difference factor express.

[0056] Step 114: determine the short-time energy of the ambient sound based on the linear combination energy representation, the energy representation of the cross-correlation coefficient, the short-time energy of each channel signal, and the cross-correlation coefficient between each channel signal.

[0057] Specifically, since the first channel signal and the second channel signal are both known sound signals, the short-time energy value of the first channel signal and the short-time energy value of the second channel signal can be directly calculated, and the value of the mutual correlation coefficient between the first channel signal and the second channel signal can be calculated. Therefore, the short-time energy value of the first channel signal, the short-time energy value of the second channel signal, and the value of the mutual correlation coefficient between the two can be substituted into the above-mentioned linear combination energy representation and the energy representation of the mutual correlation coefficient, that is, substituted into the above-mentioned formula (3) and formula (5), and the short-time energy value and difference factor of the ambient sound and the coherent sound in the above-mentioned linear combination energy representation and the energy representation of the mutual correlation coefficient are solved, so as to obtain the short-time energy value of the ambient sound and the coherent sound, and the value of the difference factor of the coherent sound in different channels.

[0058] Specifically, in the above formulas (3) and (5), and About and function, so the short-time energy of ambient sound can be solved as follows and the short-term energy of coherent sound , and the difference factor : in, It can be understood that since the first channel signal and the second channel signal are both known sound signals, the short-time energy of the first channel signal , the short-time energy of the second channel signal and the correlation coefficient is known. Based on this, the parameter can be calculated and The value of , from which the short-time energy of the ambient sound can be further calculated , the short-term energy of coherent sound and the difference factor .

[0059] The method provided by the embodiment of the present invention realizes the short-time energy solution of the ambient sound and the coherent sound based on the linear combination relationship between the ambient sound and the coherent sound contained in the first channel signal and the second channel signal, thereby providing conditions for realizing sound separation.

[0060] Based on any of the above embodiments, step 120 specifically includes: Step 121: construct a time-frequency domain representation relationship between the ambient sound of each channel and the signal of each channel.

[0061] Specifically, since the short-term energy of the ambient sound of the first channel and the second channel is the same, the ambient sound of each channel can be obtained by multiplying the ambient sound mask of each channel with the sound signal of the channel. Based on this, the time-frequency domain representation relationship between the ambient sound of the first channel and the first channel signal, and the time-frequency domain representation relationship between the ambient sound of the second channel and the second channel signal can be constructed: In the formula, is the time-frequency domain representation of the ambient sound of the first channel, is the ambient sound mask of the first channel, is the time-frequency domain representation of the first channel signal, is the time-frequency domain representation of the ambient sound of the second channel, is the ambient sound mask of the second channel, is the time-frequency domain representation of the second channel signal. Among them, the ambient sound mask , It is a quantity related to the autocorrelation and cross-correlation of the channel signal. In order to achieve extraction, , Restricted to positive real numbers.

[0062] Step 122: determine the ambient sound mask of each channel based on the time-frequency domain representation relationship, the linear combination energy representation of each channel signal, the short-time energy of the ambient sound and the short-time energy of each channel signal.

[0063] Specifically, in the above formula (3), since the ambient sound of each channel has the same short-term energy, it is recorded as , therefore, formula (7) can be transformed into: in, and are the short-time energies of the first channel signal and the second channel signal, respectively, both of which are known, and is the short-time energy of the ambient sound, which has been solved in formula (6). Therefore, according to formula (8), the ambient sound masks of the first channel and the second channel can be calculated as and .

[0064] Based on any of the above embodiments, step 130 specifically includes: Step 131, multiplying the ambient sound mask of any channel by the signal of any channel to obtain the ambient sound of any channel; Step 132: subtract the ambient sound of any channel from the signal of any channel to obtain the coherent sound of any channel.

[0065] Specifically, for the first channel, the ambient sound of the first channel can be obtained by multiplying the ambient sound mask of the first channel with the first channel signal, thereby completing the extraction of the ambient sound component from the first channel signal. The residual component of the first channel signal can be obtained by subtracting the ambient sound of the first channel from the first channel signal, and the residual component is used as the coherent sound component of the first channel, thereby completing the separation and extraction of the ambient sound and the coherent sound of the first channel.

[0066] Similarly, for the second channel, by multiplying the ambient sound mask of the second channel with the second channel signal, the ambient sound of the second channel can be obtained, and the ambient sound component can be extracted from the second channel signal. Subtracting the ambient sound of the second channel from the second channel signal can obtain the remaining components of the second channel signal, and these remaining components are the coherent sound components of the second channel, thereby completing the separation and extraction of the ambient sound and coherent sound of the second channel.

[0067] Based on any of the above embodiments, Figure 2 is a flow chart of the method for separating coherent sound from ambient sound based on time-frequency domain masking provided by the present invention, such as Figure 2 As shown, the core idea of ​​the present invention is to assume that the short-time energy of the ambient sound in the left and right channels is the same. By solving three sets of equations simultaneously, the short-time energy of the ambient sound can be obtained, and then the mask of the ambient sound in each channel is further obtained, and the mask is multiplied with the original channel signal to obtain the ambient sound. The original channel signal minus the estimated ambient sound can obtain the coherent sound, which has strong theoretical interpretability. The method includes the following steps: Step S1, obtaining the time domain input signal of the stereo left channel and the time domain input signal of the right channel .

[0068] Step S2, respectively perform short-time Fourier transform on the time domain input signals of the left and right channels to obtain the representation of the left and right channel signals in the Fourier transform domain. and .

[0069] Step S3, calculating the short-time energy of the left and right channel signals according to the representation of the left and right channel signals in the Fourier transform domain , and the correlation coefficient .

[0070] Step S4: according to the short-time energy of the left and right channel signals , and the correlation coefficient , calculate the short-time energy of ambient sound .

[0071] Step S5: according to the short-term energy of the ambient sound And the short-term energy of the left and right channel signals , , calculate the ambient sound mask of the left and right channels in the time-frequency domain , .

[0072] Step S6, multiplying the ambient sound mask of each channel by the signal of the channel to obtain the ambient sound signal of each channel, and the remaining signal is the coherent sound signal.

[0073] In the method provided in the embodiment of the present invention, the calculation process of sound separation is relatively simple and easy to implement by programming. The method has good adaptability to different types of sound signals and can adapt to various stereo scenes. In addition, the method can achieve a better sound separation effect and can effectively separate the coherent sound and the ambient sound.

[0074] The sound separation device provided by the present invention is described below. The sound separation device described below and the sound separation method described above can be referenced to each other.

[0075] Based on any of the above embodiments, Figure 3 Schematic diagram of the structure of the sound signal separation device provided by the present invention. Figure 3 As shown, the device comprises: An energy determination unit 310, configured to determine the short-time energy of the ambient sound based on the short-time energy of each channel signal and the mutual correlation coefficient between the channel signals, wherein the ambient sound of each channel has the same short-time energy; A mask determination unit 320, configured to determine the ambient sound mask of each channel based on the short-time energy of the ambient sound and the short-time energy of each channel signal; The sound separation unit 330 is configured to determine the ambient sound and the coherent sound of each channel based on the ambient sound mask of each channel and the channel signal.

[0076] The device provided by the embodiment of the present invention can determine the short-time energy of the ambient sound based on the short-time energy of each channel signal and the mutual correlation coefficient between the channel signals. According to the short-time energy of the ambient sound and the short-time energy of each channel signal, the ambient sound mask of each channel can be further determined, that is, the mask of the ambient sound in each channel. Therefore, according to the ambient sound mask of each channel, the ambient sound and the coherent sound can be separated from the channel signal, thereby ensuring the effectiveness and reliability of the separation of the coherent sound and the ambient sound. The whole separation process is simple to calculate and easy to implement through programming, and can be applicable to a variety of stereo scenes.

[0077] Based on any of the above embodiments, the energy determination unit 310 includes: A transform subunit, configured to perform short-time Fourier transform on the time domain representation of each channel signal to obtain a time-frequency domain representation of each channel signal; A first energy representation subunit, configured to determine a linear combination energy representation of each channel signal based on the time-frequency domain representation of each channel signal; A second energy representation subunit, configured to determine an energy representation of a cross-correlation coefficient between the channel signals based on a time-frequency domain representation of the channel signals; The short-time energy determination subunit is used to determine the short-time energy of the ambient sound based on the linear combination energy representation, the energy representation of the cross-correlation coefficient, the short-time energy of each channel signal and the cross-correlation coefficient between the channel signals.

[0078] Based on any of the above embodiments, the first energy representation subunit is specifically used for: Based on the time-frequency domain representations of the channel signals and the equivalent relationship between the short-time energies of the ambient sounds of the channels, a linear combination energy representation of the channel signals is determined.

[0079] Based on any of the above embodiments, each channel signal includes a first channel signal and a second channel signal, the time domain representation of the first channel signal is obtained by superimposing the time domain representation of the coherent sound and the time domain representation of the first channel ambient sound, the time domain representation of the second channel signal is obtained by multiplying the time domain representation of the coherent sound by a difference factor and superimposing the time domain representation of the second channel ambient sound, and the difference factor is an amplitude difference factor between the first channel coherent sound and the second channel coherent sound.

[0080] Based on any of the above embodiments, the mask determination unit 320 is specifically configured to: Constructing a time-frequency domain representation relationship between the ambient sound of each channel and the signal of each channel; Based on the time-frequency domain representation relationship, the linear combination energy representation of the channel signals, the short-time energy of the ambient sound and the short-time energy of the channel signals, the ambient sound mask of each channel is determined.

[0081] Based on any of the above embodiments, the sound separation unit 330 is specifically used for: Multiplying the ambient sound mask of any channel by the signal of any channel to obtain the ambient sound of any channel; The ambient sound of any channel is subtracted from the signal of any channel to obtain the coherent sound of any channel.

[0082] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the sound signal separation method, which includes: determining the short-time energy of the ambient sound based on the short-time energy of each channel signal and the mutual correlation coefficient between the channel signals, wherein the ambient sound of each channel has the same short-time energy; determining the ambient sound mask of each channel based on the short-time energy of the ambient sound and the short-time energy of the channel signals; determining the ambient sound and the coherent sound of each channel based on the ambient sound mask of each channel and the channel signals.

[0083] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0084] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the sound signal separation method provided by the above-mentioned methods. The method includes: determining the short-time energy of the ambient sound based on the short-time energy of each channel signal and the mutual correlation coefficient between the channel signals, and the ambient sound of each channel has the same short-time energy; determining the ambient sound mask of each channel based on the short-time energy of the ambient sound and the short-time energy of the channel signals; determining the ambient sound and coherent sound of each channel based on the ambient sound mask of each channel and the channel signals.

[0085] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the sound signal separation method provided by the above-mentioned methods, the method comprising: determining the short-time energy of the ambient sound based on the short-time energy of each channel signal and the mutual correlation coefficient between the channel signals, the ambient sound of each channel having the same short-time energy; determining the ambient sound mask of each channel based on the short-time energy of the ambient sound and the short-time energy of the channel signals; determining the ambient sound and coherent sound of each channel based on the ambient sound mask of each channel and the channel signals.

[0086] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0087] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A sound signal separation method, characterized in that: include: Determining the short-time energy of the ambient sound based on the short-time energy of each channel signal and the mutual correlation coefficient between the channel signals, wherein the ambient sound of each channel has the same short-time energy; Determining the ambient sound mask of each channel based on the short-time energy of the ambient sound and the short-time energy of each channel signal; Based on the ambient sound masks of the channels and the channel signals, the ambient sound and the coherent sound of the channels are determined.

2. The sound signal separation method according to claim 1, characterized in that: The determining of the short-time energy of the ambient sound based on the short-time energy of each channel signal and the mutual correlation coefficient between the channel signals comprises: Performing short-time Fourier transform on the time domain representation of each channel signal to obtain the time-frequency domain representation of each channel signal; Determine a linear combination energy representation of each channel signal based on the time-frequency domain representation of each channel signal; Determining an energy representation of a cross-correlation coefficient between the channel signals based on a time-frequency domain representation of the channel signals; The short-time energy of the ambient sound is determined based on the linear combination energy representation, the energy representation of the cross-correlation coefficient, the short-time energy of each channel signal and the cross-correlation coefficient between each channel signal.

3. The sound signal separation method according to claim 2, characterized in that: The step of determining the linear combination energy representation of each channel signal based on the time-frequency domain representation of each channel signal comprises: Based on the time-frequency domain representations of the channel signals and the equivalent relationship between the short-time energies of the ambient sounds of the channels, a linear combination energy representation of the channel signals is determined.

4. The sound signal separation method according to claim 2, characterized in that: Each channel signal includes a first channel signal and a second channel signal, the time domain representation of the first channel signal is obtained by superimposing the time domain representation of the coherent sound and the time domain representation of the first channel ambient sound, the time domain representation of the second channel signal is obtained by multiplying the time domain representation of the coherent sound by a difference factor and superimposing the time domain representation of the second channel ambient sound, and the difference factor is an amplitude difference factor between the first channel coherent sound and the second channel coherent sound.

5. The sound signal separation method according to claim 2, characterized in that: The determining of the ambient sound mask of each channel based on the short-time energy of the ambient sound and the short-time energy of each channel signal comprises: Constructing a time-frequency domain representation relationship between the ambient sound of each channel and the signal of each channel; Based on the time-frequency domain representation relationship, the linear combination energy representation of the channel signals, the short-time energy of the ambient sound and the short-time energy of the channel signals, the ambient sound mask of each channel is determined.

6. The method for extracting coherent sound and ambient sound according to any one of claims 1 to 5, characterized in that: The determining of the ambient sound and the coherent sound of each channel based on the ambient sound mask of each channel and the signal of each channel includes: Multiplying the ambient sound mask of any channel by the signal of any channel to obtain the ambient sound of any channel; The ambient sound of any channel is subtracted from the signal of any channel to obtain the coherent sound of any channel.

7. A sound signal separation device, characterized in that: include: An energy determination unit, configured to determine the short-time energy of the ambient sound based on the short-time energy of each channel signal and the mutual correlation coefficient between the channel signals, wherein the ambient sound of each channel has the same short-time energy; a mask determination unit, configured to determine the ambient sound mask of each channel based on the short-time energy of the ambient sound and the short-time energy of each channel signal; The sound separation unit is used to determine the ambient sound and the coherent sound of each channel based on the ambient sound mask of each channel and the channel signal.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the sound signal separation method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the sound signal separation method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the sound signal separation method according to any one of claims 1 to 6 is implemented.

Images