Sound separation method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of sound processing, and provides a sound separation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a first sound signal and a second sound signal; based on the short-time energy values of the first sound signal and the second sound signal and the linear combination relation between the environment sound and the coherent sound contained in the first sound signal and the second sound signal, determining the short-time energy values of the environment sound and the coherent sound and difference factors of the coherent sound on different channels; based on a weight mapping relation, the short-time energy values of the environment sound and the coherent sound and the difference factors are mapped into sound separation weights, and the weight mapping relation is obtained through linear fitting based on a linear combination relation; the ambient sound and the coherent sound are separated from the first sound signal and the second sound signal based on the sound separation weight. According to the method, the device, the electronic equipment and the storage medium provided by the invention, the effectiveness and the reliability of separation of the environment sound and the coherent sound are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of sound processing, and in particular to a sound separation method, device, electronic equipment and storage medium. Background Art

[0002] In order to optimize the user's auditory experience, spatial sound playback technology came into being.

[0003] A core link of spatial sound playback technology is to separate the coherent sound and ambient sound in the mixed sound, so that the coherent sound and ambient sound can be processed separately during post-tuning, thus creating a more enveloping and immersive auditory atmosphere.

[0004] However, due to the limitations of current stereo audio formats, coherent sound and ambient sound are often recorded and transmitted mixed together, making it difficult to completely distinguish them during post-separation, which to some extent limits the implementation effect of spatial sound playback technology. Summary of the invention

[0005] The present invention provides a sound separation method, device, electronic device and storage medium, which are used to solve the defect of difficulty in separating coherent sound and ambient sound in the related art.

[0006] The present invention provides a sound separation method, comprising: Acquire a first sound signal and a second sound signal, where the first sound signal and the second sound signal are acquired through different channels; Determine the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factors of the coherent sound on the different channels based on the short-time energy values ​​of the first sound signal and the second sound signal, and the linear combination relationship between the ambient sound and the coherent sound contained in the first sound signal and the second sound signal; Based on a weight mapping relationship, the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factor, are mapped into sound separation weights, wherein the weight mapping relationship is obtained by linear fitting based on the linear combination relationship; The ambient sound and the coherent sound are separated from the first sound signal and the second sound signal based on the sound separation weight.

[0007] According to a sound separation method provided by the present invention, based on the short-time energy values ​​of the first sound signal and the second sound signal, and the linear combination relationship between the ambient sound and the coherent sound respectively contained in the first sound signal and the second sound signal, the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factors of the coherent sound on the different channels are determined, including: Based on the linear combination relationship, determining a linear combination energy representation of each of the first sound signal and the second sound signal; determining an energy representation of a mutual correlation coefficient between the first sound signal and the second sound signal based on the linear combination relationship; Based on the linear combination energy representation and the energy representation of the mutual correlation coefficient, the short-time energy values ​​of the first sound signal and the second sound signal, and the mutual correlation coefficient between the first sound signal and the second sound signal, the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factors of the coherent sound on the different channels are determined.

[0008] According to a sound separation method provided by the present invention, determining the linear combination energy representation of each of the first sound signal and the second sound signal based on the linear combination relationship includes: Based on the linear combination relationship and the equivalent relationship between the short-time energy values ​​of the ambient sounds in different channels, a linear combination energy representation of each of the first sound signal and the second sound signal is determined.

[0009] According to a sound separation method provided by the present invention, the determination of the weight mapping relationship includes: Constructing a linear composition relationship of an estimated sound component, wherein the estimated sound component is an estimated ambient sound or an estimated coherent sound, and the linear composition relationship includes initial weights of the first sound signal and the second sound signal; determining an estimation error of the estimated sound component based on the linear composition relationship and the linear combination relationship; With the goal of making the estimation error uncorrelated with the first sound signal and the second sound signal, linear fitting is performed on the initial weights in the linear constituent relationship to obtain the weight mapping relationship.

[0010] According to a sound separation method provided by the present invention, the estimation error is respectively uncorrelated with the first sound signal and the second sound signal, and linear fitting is performed on the initial weights in the linear constituent relationship to obtain the weight mapping relationship, including: constructing a first cross-correlation function between the estimated error and the first sound signal, and a second cross-correlation function between the estimated error and the second sound signal; With the goal of setting the values ​​of the first cross-correlation function and the second cross-correlation function to zero, the estimation error and the linear combination relationship are substituted into the first cross-correlation function and the second cross-correlation function to obtain the weight mapping relationship.

[0011] According to a sound separation method provided by the present invention, the linear combination relationship includes adding the coherent sound to the ambient sound of the first channel to obtain the first sound signal, and multiplying the coherent sound by the difference factor and adding the resultant to the ambient sound of the second channel to obtain the second sound signal.

[0012] The present invention also provides a sound separation device, comprising: A signal acquisition unit, configured to acquire a first sound signal and a second sound signal, wherein the first sound signal and the second sound signal are acquired through different channels; a parameter acquisition unit, configured to determine the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factors of the coherent sound on the different channels based on the short-time energy values ​​of the first sound signal and the second sound signal, and the linear combination relationship between the ambient sound and the coherent sound contained in the first sound signal and the second sound signal; A weight mapping unit, configured to map the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factor, to sound separation weights based on a weight mapping relationship, wherein the weight mapping relationship is obtained by linear fitting based on the linear combination relationship; A sound separation unit is used to separate the ambient sound and the coherent sound from the first sound signal and the second sound signal based on the sound separation weight.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned sound separation methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the sound separation method as described above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the sound separation method described above is implemented.

[0016] The sound separation method, device, electronic device and storage medium provided by the present invention establish a linear combination relationship between the ambient sound and the coherent sound contained in the first sound signal and the second sound signal respectively, and perform linear fitting on the weight mapping relationship required for sound separation based on the linear combination relationship. In practical applications, the sound separation weights required for sound separation of the first sound signal and the second sound signal can be determined based on the weight mapping relationship, thereby achieving separation of ambient sound and coherent sound, thereby ensuring the effectiveness and reliability of separation of ambient sound and coherent sound, and the sound separation process is simple in calculation and easy to implement in programming, and can adapt to various stereo scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a schematic flow chart of the sound separation method provided by the present invention.

[0019] Figure 2 It is a flow chart of the method for determining the short-time energy value provided by the present invention.

[0020] Figure 3 It is a flow chart of the method for determining the weight mapping relationship provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of the sound separation device provided by the present invention.

[0022] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Spatial sound reproduction technology, also known as three-dimensional audio technology, is a technology that can construct a virtual auditory space, which enables listeners to have a specific spatial perception. With the development of artificial intelligence audio-visual technology, the construction of immersive virtual auditory space has gradually become a key component for achieving realistic virtual reality experience. Spatial sound reproduction technology has also been applied in audio-visual entertainment, voice communication and other fields.

[0025] In spatial sound playback technology, the coherent sound and ambient sound in the mixed sound are separated and extracted, and then the two are rendered separately, which can effectively optimize the effect of spatial sound playback. However, due to the limitations of the current stereo audio format, coherent sound and ambient sound are often mixed together, making it difficult to completely distinguish them during later separation. Therefore, how to effectively separate coherent sound and ambient sound has become a problem that needs to be solved urgently.

[0026] Figure 1 Schematic diagram of the sound separation method provided by the present invention. Figure 1 As shown, the sound separation method provided by the present invention can be used for separation of coherent sound and ambient sound, and the method comprises: Step 110: Acquire a first sound signal and a second sound signal, wherein the first sound signal and the second sound signal are acquired through different channels.

[0027] Specifically, the first sound signal and the second sound signal are two sound signals collected by different channels. The different channels referred to here can also be understood as the dual channels required to build a stereo effect, that is, two independent left and right audio channels. For example, the first sound signal can be collected by the left channel, and the second sound signal can be collected by the right channel.

[0028] In the embodiment of the present invention, no matter which channel is targeted, the sound signal collected in the channel can be understood as a sound signal obtained by superimposing coherent sound and ambient sound, that is, the first sound signal and the second sound signal are mixed sound signals of coherent sound and ambient sound. Here, coherent sound refers to a localizable sound signal, such as a sound signal generated by a character's dialogue, and ambient sound refers to a sound signal that cannot be localized, such as a background sound effect.

[0029] Furthermore, according to the characteristics of coherent sound and ambient sound, the coherent sound between different channels is completely correlated, and the ambient sound between different channels is uncorrelated; and the coherent sound is uncorrelated with the ambient sound in each channel. Here, complete correlation means that two or more sound signals have a high degree of similarity and consistency in terms of waveform, frequency, phase, etc., which usually means that these sound signals come from the same sound source; uncorrelated means that two or more sound signals have no significant similarity and consistency in terms of waveform, frequency, phase, etc., which usually means that these sound signals come from different sound sources.

[0030] Step 120, based on the short-time energy values ​​of the first sound signal and the second sound signal, and the linear combination relationship between the ambient sound and the coherent sound respectively contained in the first sound signal and the second sound signal, determine the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factors of the coherent sound on the different channels.

[0031] Specifically, for the first sound signal and the second sound signal collected through different channels, a linear combination relationship between the first sound signal and the second sound signal may be established.

[0032] The linear combination relationship here can be understood as follows: the first sound signal is a linear superposition of the coherent sound and the ambient sound in the channel corresponding to the first sound signal, and the second sound signal is a linear superposition of the coherent sound and the ambient sound in the channel corresponding to the second sound signal. For ease of explanation, in the embodiment of the present invention, the channel corresponding to the first sound signal is recorded as the first channel, and the channel corresponding to the second sound signal is recorded as the second channel.

[0033] Furthermore, since the coherent sounds between different channels are completely correlated, a difference factor between the coherent sounds in the first channel and the coherent sounds in the second channel can be defined, that is, the coherent sounds in the second channel can be represented in the form of multiplication of the difference factor and the coherent sounds in the first channel.

[0034] Therefore, in the linear combination relationship, the first sound signal can be expressed as a linear addition of coherent sound and ambient sound in the first channel, and the second sound signal can be expressed as a linear addition of coherent sound and ambient sound in the second channel.

[0035] After determining the linear combination relationship, the linear combination relationship can be converted to the frequency domain to calculate the relationship between the short-time energy of the first sound signal and the short-time energy of the ambient sound and the short-time energy of the coherent sound in the first channel under the linear combination relationship, and the relationship between the short-time energy of the second sound signal and the short-time energy of the ambient sound and the short-time energy of the coherent sound in the second channel. Here, the conversion to the frequency domain can be achieved through Fourier transformation; the short-time energy represents the total energy of the sound signal in a time period, which can be used to reflect the strength of the sound signal in the time period.

[0036] Since the first sound signal and the second sound signal are both known sound signals, the short-time energy value of the first sound signal and the short-time energy value of the second sound signal can be directly calculated, and the short-time energy values ​​of the first sound signal and the second sound signal are substituted into the relationship between the above-mentioned short-time energies to solve them, thereby obtaining the short-time energy values ​​of the ambient sound and the coherent sound, as well as the difference factors of the coherent sound in different channels.

[0037] It can be understood that, according to the characteristics of the ambient sound, the short-time energy of the ambient sound collected under different channels is consistent. Therefore, in step 120, the short-time energy of the ambient sound under the first channel and the second channel is consistent. The short-time energy value of the ambient sound thus solved is unique and can represent the short-time energy value of the ambient sound in the first sound signal and the short-time energy value of the ambient sound in the second sound signal.

[0038] In addition, according to the characteristics of coherent sound, the short-time energy of coherent sound collected in different channels is different, and the coherent sound between different channels is completely correlated. Therefore, the short-time energy value of the coherent sound solved in step 120 can represent the short-time energy value of the coherent sound in one of the channels, and the short-time energy value of the coherent sound combined with the difference factor between channels can represent the short-time energy value of the coherent sound in another channel.

[0039] Step 130, based on a weight mapping relationship, the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factor are mapped into sound separation weights, wherein the weight mapping relationship is obtained by linear fitting based on the linear combination relationship.

[0040] Specifically, to achieve sound separation for the first sound signal and the second sound signal, a linear composition relationship for coherent sound or for ambient sound can be established. Taking coherent sound as an example, the linear composition relationship for coherent sound can represent weighted summation of the first sound signal and the second sound signal to obtain coherent sound, and the weight used for weighted summation here is the sum required to separate coherent sound from ambient sound, which is recorded as sound separation weight in the embodiment of the present invention.

[0041] After determining the linear combination relationship, the sound separation weights required to separate the coherent sound and the ambient sound from the first sound signal and the second sound signal can be linearly fitted based on the linear combination relationship, thereby obtaining a weight mapping relationship that can reflect the relationship between the short-time energy and weight of the coherent sound and the ambient sound respectively.

[0042] It can be understood that the weight mapping relationship is a function with the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factor as independent variables, and the sound separation weight as the dependent variable. Therefore, after obtaining the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factor, respectively, in step 120, the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factor, respectively, can be substituted into the weight mapping relationship to obtain the value of the sound separation weight.

[0043] Step 140: Separate the ambient sound and the coherent sound from the first sound signal and the second sound signal based on the sound separation weight.

[0044] Specifically, after obtaining the sound separation weight, the first sound signal and the second sound signal can be sound separated according to the sound separation weight. For example, when the sound separation weight is a sound separation weight for coherent sound, the first sound signal and the second sound signal can be weighted and summed based on the sound separation weight, and the result of the weighted summation is used as the separated coherent sound, and the part of the first sound signal and the second sound signal minus the coherent sound is used as the ambient sound.

[0045] In the method provided in the embodiment of the present invention, a linear combination relationship between the ambient sound and the coherent sound contained in the first sound signal and the second sound signal is established, and a linear fitting is performed on the weight mapping relationship required for sound separation based on the linear combination relationship. In practical applications, the sound separation weights required for sound separation of the first sound signal and the second sound signal can be determined based on the weight mapping relationship, thereby achieving separation of ambient sound and coherent sound, thereby ensuring the effectiveness and reliability of separation of ambient sound and coherent sound. In addition, the sound separation process is simple in calculation and easy to implement in programming, and can adapt to various stereo scenes.

[0046] Based on the above embodiment, the linear combination relationship includes adding the coherent sound to the ambient sound of the first channel to obtain the first sound signal, and multiplying the coherent sound by the difference factor and adding the resultant to the ambient sound of the second channel to obtain the second sound signal.

[0047] Specifically, assuming that the first channel is the left channel and the second channel is the right channel, the linear combination relationship can be expressed as follows: In the formula, and They are the time domain input signals for the left channel and the right channel respectively. is the coherent sound of the left channel, This is the ambient sound in the left channel. is the amplitude difference factor between the coherent sounds in the left and right channels, is the coherent sound of the right channel, This is the ambient sound of the right channel.

[0048] in, may represent a first sound signal, A second sound signal may be indicated. and The first sound signal is obtained by adding the coherent sound and the ambient sound in the first channel. and The coherent sound and the ambient sound in the second channel are added together, that is, the second sound signal can be obtained by adding the coherent sound and the ambient sound in the second channel.

[0049] It can be seen from the above formula that the first sound signal and the second sound signal can both be expressed as a linear combination of coherent sound and ambient sound, that is, the first sound signal and the second sound signal can each be expressed as a linear combination of ambient sound and coherent sound.

[0050] Based on any of the above embodiments, Figure 2 is a flow chart of the method for determining the short-time energy value provided by the present invention, such as Figure 2 As shown, in step 120, based on the short-time energy values ​​of the first sound signal and the second sound signal, and the linear combination relationship between the ambient sound and the coherent sound respectively contained in the first sound signal and the second sound signal, the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factors of the coherent sound on the different channels are determined, including: Step 121: Determine linear combination energy representations of the first sound signal and the second sound signal based on the linear combination relationship.

[0051] Specifically, when a linear combination relationship is obtained, the linear combination relationship can be converted to the frequency domain, and the relationship between the short-time energy of the first sound signal and the short-time energy of the ambient sound and the short-time energy of the coherent sound in the first channel under the linear combination relationship, as well as the relationship between the short-time energy of the second sound signal and the short-time energy of the ambient sound and the short-time energy of the coherent sound in the second channel under the linear combination relationship are calculated in the frequency domain. In an embodiment of the present invention, the relationship between the short-time energy of the first sound signal and the short-time energy of the ambient sound and the short-time energy of the coherent sound in the first channel is recorded as the linear combination energy representation of the first sound signal. In addition, the relationship between the short-time energy of the second sound signal and the short-time energy of the ambient sound and the short-time energy of the coherent sound in the second channel is recorded as the linear combination energy representation of the second sound signal.

[0052] The linear combination relationship is expressed as as well as Take the linear combination relationship as an example, and convert it to the frequency domain, the linear combination relationship in the frequency domain can be obtained as follows: In the formula, and The first sound signal and second sound signal The frequency domain representation of For coherent sound The frequency domain representation of and They are respectively the ambient sound of the first channel and ambient sound in the second channel Frequency domain representation of . The difference factor Frequency domain representation of .

[0053] In the formula, represents the timeframe index, Indicates the frequency index. In order to keep the formula concise, the Therefore, the linear combination relationship in the frequency domain can be expressed as follows in a simplified form: That is, in the frequency domain, the frequency domain representation of the first sound signal is The frequency domain representation of the coherent sound of the first channel And the frequency domain representation of the ambient sound of the first channel The result of linear addition; the frequency domain representation of the second sound signal The frequency domain representation of the coherent sound of the second channel And the frequency domain representation of the ambient sound of the second channel The result of linear addition. The frequency domain representation of the coherent sound of the second channel is is the frequency domain representation of the difference factor Frequency domain representation of the coherent sound with the first channel The product of .

[0054] Based on the linear combination relationship in the frequency domain, it can be determined that the short-time energy of the first sound signal is the sum of the short-time energy of the coherent sound of the first channel and the short-time energy of the ambient sound of the first channel, and the short-time energy of the second sound signal is the sum of the short-time energy of the coherent sound of the second channel and the short-time energy of the ambient sound of the second channel. Thus, the linear combination energy representation of the first sound signal and the linear combination energy representation of the second sound signal can be obtained.

[0055] Further, in some embodiments, step 121 includes: Based on the linear combination relationship and the equivalent relationship between the short-time energy values ​​of the ambient sounds in different channels, a linear combination energy representation of each of the first sound signal and the second sound signal is determined.

[0056] Specifically, in the process of determining the linear combination energy representation of the first sound signal and the second sound signal, in addition to the linear combination relationship in the frequency domain, the characteristics of the ambient sound in different channels may also be considered. That is, the ambient sound in different channels has the same short-time energy value, that is, the short-time energy values ​​of the ambient sound in different channels are the same.

[0057] Taking the first sound signal as an example, the short-time energy of the first sound signal can be calculated by the following formula: In the formula, is the short-time energy of the first sound signal, is the frequency domain representation of the first sound signal, Based on the above formula for calculating short-time energy, the short-time energy of the second sound signal can also be calculated as .

[0058] Based on these two, the linear combination energy representation of the first sound signal and the second sound signal can be obtained, which can be specifically expressed as the following formula: In the formula, and are the short-time energies of the first sound signal and the second sound signal respectively, is the short-time energy of the coherent sound in the first channel, is the short-time energy of the coherent sound in the second channel, Here, considering that the short-time energy of the ambient sound in different channels is equal, in the linear combination energy representation of the first sound signal and the linear combination energy representation of the second sound signal, the short-time energy of the ambient sound can be expressed as .

[0059] Step 122: Determine an energy representation of a correlation coefficient between the first sound signal and the second sound signal based on the linear combination relationship.

[0060] Specifically, after obtaining the linear combination relationship of the first sound signal and the second sound signal, the mutual correlation coefficient between the first sound signal and the second sound signal can be calculated, and in the process of calculating the mutual correlation coefficient, the linear combination relationship of the first sound signal and the linear combination relationship of the second sound signal can be substituted into the calculation formula of the mutual correlation coefficient. The mutual correlation coefficient thus obtained is the mutual correlation coefficient expressed in the form of short-time energy of coherent sound, ambient sound, etc. In the embodiment of the present invention, the mutual correlation coefficient expressed in the form of short-time energy is recorded as the energy representation of the mutual interference coefficient.

[0061] Here, the calculation formula of the mutual correlation coefficient between the first sound signal and the second sound signal may be: In the formula, is the mutual correlation coefficient, and are frequency domain representations of the first sound signal and the second sound signal respectively.

[0062] Substituting the above linear combination relationship in the frequency domain into the calculation formula of the mutual correlation coefficient, we can get the following formula: Since the coherent sound is unrelated to the ambient sound in each channel, , In addition, since the ambient sounds in different channels are unrelated, .

[0063] Therefore, the calculation formula of the mutual correlation coefficient can be expressed as follows, which is the energy representation of the mutual correlation coefficient: In the formula, the mutual correlation coefficient The short-term energy of the first sound signal , the short-term energy of the second sound signal , the short-term energy of coherent sound And the frequency domain representation of the difference factor express.

[0064] Step 123, based on the linear combination energy representation and the energy representation of the cross-correlation coefficient, the short-time energy values ​​of the first sound signal and the second sound signal, and the cross-correlation coefficient between the first sound signal and the second sound signal, determine the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factors of the coherent sound on the different channels.

[0065] Specifically, since the first sound signal and the second sound signal are both known sound signals, the short-time energy value of the first sound signal and the short-time energy value of the second sound signal can be directly calculated, and the value of the correlation coefficient between the first sound signal and the second sound signal can be calculated.

[0066] Therefore, the short-time energy value of the first sound signal and the short-time energy value of the second sound signal, as well as the value of the correlation coefficient between the two, can be substituted into the above-mentioned linear combination energy representation and the energy representation of the correlation coefficient, and the short-time energy values ​​and difference factors of the ambient sound and the coherent sound in the above-mentioned linear combination energy representation and the energy representation of the correlation coefficient are solved, thereby obtaining the short-time energy values ​​of the ambient sound and the coherent sound, as well as the values ​​of the difference factors of the coherent sound on the different channels.

[0067] Furthermore, based on the above linear combination energy representation and the energy representation of the cross-correlation coefficient, the short-time energy of the ambient sound can be solved in the following formula: and the short-term energy of coherent sound , and the difference factor : in, The method provided in the embodiment of the present invention realizes the short-time energy solution of the ambient sound and the coherent sound based on the linear combination relationship between the ambient sound and the coherent sound respectively contained in the first sound signal and the second sound signal, thereby providing conditions for realizing sound separation.

[0068] Based on any of the above embodiments, Figure 3 FIG. 1 is a flow chart of a method for determining a weight mapping relationship provided by the present invention. Figure 3 As shown, the determination of the weight mapping relationship includes: Step 310: construct a linear composition relationship of estimated sound components, where the estimated sound components are estimated ambient sound or estimated coherent sound, and the linear composition relationship includes initial weights of the first sound signal and the second sound signal, respectively.

[0069] Specifically, to achieve sound separation for the first sound signal and the second sound signal, a linear composition relationship for coherent sound or for ambient sound can be established. It can be understood that the linear composition relationship here is used to estimate the ambient sound or coherent sound in the form of a weighted sum of the first sound signal and the second sound signal.

[0070] In the embodiment of the present invention, the estimated value for the ambient sound can be recorded as the estimated ambient sound, and the estimated value for the coherent sound can be recorded as the estimated coherent sound. On this basis, a linear constituent relationship can be constructed for the estimated ambient sound, or a linear constituent relationship can be constructed for the estimated coherent sound.

[0071] For example, the linear formation relationship for estimating coherent sound can be expressed as follows: In the formula, To estimate the coherent sound, and For the first sound signal and second sound signal The initial weight of .

[0072] Step 320: Determine the estimation error of the estimated sound component based on the linear composition relationship and the linear combination relationship.

[0073] Specifically, after obtaining the linear relationship for the estimated sound component, the estimated sound component can be subtracted from the real sound component to obtain the estimated error. For example, the estimated coherent sound can be subtracted from the real coherent sound, or the estimated ambient sound can be subtracted from the real ambient sound.

[0074] In the process of calculating the estimated error, the linear combination relationship can be substituted into the calculation formula of the estimated error, and the formula expression of the estimated error can be as follows: In the formula, is the estimation error for coherent sound, and are the real coherent sound and the estimated coherent sound respectively. Substituting the linear combination relationship into the linear composition relationship of the estimated coherent sound, we can get and , Difference Factor , and true coherent sound and real ambient sound Represents the estimated error.

[0075] Step 330 , with the goal of making the estimation error uncorrelated with the first sound signal and the second sound signal, linear fitting is performed on the initial weights in the linear constituent relationship to obtain the weight mapping relationship.

[0076] Specifically, after obtaining the estimated error represented by the initial weights and each parameter in the linear combination relationship, a linear fit can be performed on the initial weights, and the optimal estimate obtained by the linear fit on the initial weights is used as the weight mapping relationship. Here, the optimal estimate of the initial weights is the sound separation weight, and the optimal estimate is represented as the weight mapping relationship, which can reflect the relationship between the short-time energy of the coherent sound and the ambient sound and the sound separation weight.

[0077] The linear fitting of the initial weights can be implemented based on the least squares method (LS). In the process of linear fitting based on the least squares method, the optimal estimate of the initial weights is when the estimated error is uncorrelated with the first sound signal and the estimated error is uncorrelated with the second sound signal.

[0078] In specific implementation, the objective function can be constructed by estimating that the error is uncorrelated with the first sound signal and that the error is uncorrelated with the second sound signal, and the formula of the error is substituted into the objective function for solving, thereby obtaining the formula of the initial weight, i.e., the weight mapping relationship.

[0079] Based on any of the above embodiments, in step 330, the step of performing linear fitting on the initial weights in the linear constituent relationship with the goal of making the estimated error uncorrelated with the first sound signal and the second sound signal to obtain the weight mapping relationship includes: constructing a first cross-correlation function between the estimated error and the first sound signal, and a second cross-correlation function between the estimated error and the second sound signal; With the goal of setting the values ​​of the first cross-correlation function and the second cross-correlation function to zero, the estimation error and the linear combination relationship are substituted into the first cross-correlation function and the second cross-correlation function to obtain the weight mapping relationship.

[0080] Specifically, for the linear fitting target that the estimated error is uncorrelated with the first sound signal and the estimated error is uncorrelated with the second sound signal, a first cross-correlation function and a second cross-correlation function may be established respectively.

[0081] The first cross-correlation function is a cross-correlation function between the estimation error and the first sound signal. For a target where the estimation error is uncorrelated with the first sound signal, the value of the first cross-correlation function can be set to 0, that is, it can be specifically expressed as follows: In the formula, and are the estimation error and the first sound signal respectively, Indicates short-time average.

[0082] Similarly, the second cross-correlation function is a cross-correlation function between the estimation error and the second sound signal. For a target where the estimation error is uncorrelated with the second sound signal, the value of the second cross-correlation function can be set to 0, that is, it can be specifically expressed as follows: In the formula, and are the estimated error and the second sound signal, Indicates short-time average.

[0083] After obtaining the first cross-correlation function and the second cross-correlation function, the formula of the estimated error and the linear combination relationship between the first sound signal and the second sound signal are substituted into the first cross-correlation function and the second cross-correlation function to obtain the formula of the initial weight, that is, the weight mapping relationship in the following form: In the formula, the sound separation weight and The short-term energy of ambient sound and coherent sound , and the difference factor express.

[0084] Based on any of the above embodiments, a sound separation method may include the following steps: First, obtain the first sound signal and second sound signal .

[0085] Secondly, calculate the short-time energy value of the first sound signal and the short-time energy value of the second sound signal , and calculating a correlation coefficient between the first sound signal and the second sound signal .

[0086] Subsequently, based on , and The value of is used to calculate the short-time energy of ambient sound. and the short-term energy of coherent sound , and the difference factor .

[0087] Next, , and Substitute the value of into the weight mapping relationship to obtain the sound separation weight and .

[0088] Finally, based on the sound separation weights and , weighted summing the first sound signal and the second sound signal (i.e., calculating ), and get the estimated coherent sound , and As the estimated ambient sound under the first channel, As the estimated ambient sound under the second channel, sound separation is achieved.

[0089] In the sound separation method provided in the embodiment of the present invention, only matrix operation and Fourier transform are required to achieve sound separation. The calculation in the sound separation process is simple and easy to program. Moreover, the method has good adaptability to various types of sound signals and can adapt to various stereo scenes. In addition, the method can achieve a better sound separation effect to achieve efficient and stable sound separation.

[0090] The sound separation device provided by the present invention is described below. The sound separation device described below and the sound separation method described above can be referenced to each other.

[0091] Figure 4 Schematic diagram of the structure of the sound separation device provided by the present invention. Figure 4 As shown, the device comprises: A signal acquisition unit 410 is used to acquire a first sound signal and a second sound signal, where the first sound signal and the second sound signal are acquired through different channels; a parameter acquisition unit 420, configured to determine the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factors of the coherent sound on the different channels based on the short-time energy values ​​of the first sound signal and the second sound signal, and the linear combination relationship between the ambient sound and the coherent sound contained in the first sound signal and the second sound signal; A weight mapping unit 430, configured to map the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factor, to sound separation weights based on a weight mapping relationship, wherein the weight mapping relationship is obtained by linear fitting based on the linear combination relationship; The sound separation unit 440 is configured to separate the ambient sound and the coherent sound from the first sound signal and the second sound signal based on the sound separation weight.

[0092] Based on any of the above embodiments, the parameter acquisition unit is specifically used for: Based on the linear combination relationship, determining a linear combination energy representation of each of the first sound signal and the second sound signal; determining an energy representation of a mutual correlation coefficient between the first sound signal and the second sound signal based on the linear combination relationship; Based on the linear combination energy representation and the energy representation of the mutual correlation coefficient, the short-time energy values ​​of the first sound signal and the second sound signal, and the mutual correlation coefficient between the first sound signal and the second sound signal, the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factors of the coherent sound on the different channels are determined.

[0093] Based on any of the above embodiments, the parameter acquisition unit is specifically used for: Based on the linear combination relationship and the equivalent relationship between the short-time energy values ​​of the ambient sounds in different channels, a linear combination energy representation of each of the first sound signal and the second sound signal is determined.

[0094] Based on any of the above embodiments, the weight mapping unit is further used for: Constructing a linear composition relationship of an estimated sound component, wherein the estimated sound component is an estimated ambient sound or an estimated coherent sound, and the linear composition relationship includes initial weights of the first sound signal and the second sound signal; determining an estimation error of the estimated sound component based on the linear composition relationship and the linear combination relationship; With the goal of making the estimation error uncorrelated with the first sound signal and the second sound signal, linear fitting is performed on the initial weights in the linear constituent relationship to obtain the weight mapping relationship.

[0095] Based on any of the above embodiments, the weight mapping unit is specifically used for: constructing a first cross-correlation function between the estimated error and the first sound signal, and a second cross-correlation function between the estimated error and the second sound signal; With the goal of setting the values ​​of the first cross-correlation function and the second cross-correlation function to zero, the estimation error and the linear combination relationship are substituted into the first cross-correlation function and the second cross-correlation function to obtain the weight mapping relationship.

[0096] Based on any of the above embodiments, the linear combination relationship includes adding the coherent sound to the ambient sound of the first channel to obtain the first sound signal, and multiplying the coherent sound by the difference factor and adding the resultant to the ambient sound of the second channel to obtain the second sound signal.

[0097] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the sound separation method, which includes: Acquire a first sound signal and a second sound signal, where the first sound signal and the second sound signal are acquired through different channels; Determine the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factors of the coherent sound on the different channels based on the short-time energy values ​​of the first sound signal and the second sound signal, and the linear combination relationship between the ambient sound and the coherent sound contained in the first sound signal and the second sound signal; Based on a weight mapping relationship, the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factor, are mapped into sound separation weights, wherein the weight mapping relationship is obtained by linear fitting based on the linear combination relationship; The ambient sound and the coherent sound are separated from the first sound signal and the second sound signal based on the sound separation weight.

[0098] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0099] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the sound separation method provided by the above methods, the method comprising: obtaining a first sound signal and a second sound signal, the first sound signal and the second sound signal are collected by different channels; Determine the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factors of the coherent sound on the different channels based on the short-time energy values ​​of the first sound signal and the second sound signal, and the linear combination relationship between the ambient sound and the coherent sound contained in the first sound signal and the second sound signal; Based on a weight mapping relationship, the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factor, are mapped into sound separation weights, wherein the weight mapping relationship is obtained by linear fitting based on the linear combination relationship; The ambient sound and the coherent sound are separated from the first sound signal and the second sound signal based on the sound separation weight.

[0100] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the sound separation method provided by the above methods, the method comprising: Acquire a first sound signal and a second sound signal, where the first sound signal and the second sound signal are acquired through different channels; Determine the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factors of the coherent sound on the different channels based on the short-time energy values ​​of the first sound signal and the second sound signal, and the linear combination relationship between the ambient sound and the coherent sound contained in the first sound signal and the second sound signal; Based on a weight mapping relationship, the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factor, are mapped into sound separation weights, wherein the weight mapping relationship is obtained by linear fitting based on the linear combination relationship; The ambient sound and the coherent sound are separated from the first sound signal and the second sound signal based on the sound separation weight.

[0101] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0102] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A sound separation method, characterized in that: include: Acquire a first sound signal and a second sound signal, where the first sound signal and the second sound signal are acquired through different channels; Determine the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factors of the coherent sound on the different channels based on the short-time energy values ​​of the first sound signal and the second sound signal, and the linear combination relationship between the ambient sound and the coherent sound contained in the first sound signal and the second sound signal; Based on a weight mapping relationship, the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factor, are mapped into sound separation weights, wherein the weight mapping relationship is obtained by linear fitting based on the linear combination relationship; The ambient sound and the coherent sound are separated from the first sound signal and the second sound signal based on the sound separation weight.

2. The sound separation method according to claim 1, characterized in that: The determining, based on the short-time energy values ​​of the first sound signal and the second sound signal, and the linear combination relationship between the ambient sound and the coherent sound respectively contained in the first sound signal and the second sound signal, the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factors of the coherent sound on the different channels, comprises: Based on the linear combination relationship, determining a linear combination energy representation of each of the first sound signal and the second sound signal; determining an energy representation of a mutual correlation coefficient between the first sound signal and the second sound signal based on the linear combination relationship; Based on the linear combination energy representation and the energy representation of the mutual correlation coefficient, the short-time energy values ​​of the first sound signal and the second sound signal, and the mutual correlation coefficient between the first sound signal and the second sound signal, the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factors of the coherent sound on the different channels are determined.

3. The sound separation method according to claim 2, characterized in that: The determining, based on the linear combination relationship, a linear combination energy representation of each of the first sound signal and the second sound signal comprises: Based on the linear combination relationship and the equivalent relationship between the short-time energy values ​​of the ambient sounds in different channels, a linear combination energy representation of each of the first sound signal and the second sound signal is determined.

4. The sound separation method according to claim 1, characterized in that: The determination of the weight mapping relationship includes: Constructing a linear composition relationship of an estimated sound component, wherein the estimated sound component is an estimated ambient sound or an estimated coherent sound, and the linear composition relationship includes initial weights of the first sound signal and the second sound signal; determining an estimation error of the estimated sound component based on the linear composition relationship and the linear combination relationship; With the goal of making the estimation error uncorrelated with the first sound signal and the second sound signal, linear fitting is performed on the initial weights in the linear constituent relationship to obtain the weight mapping relationship.

5. The sound separation method according to claim 4, characterized in that: The step of performing linear fitting on the initial weights in the linear constituent relationship with the goal of making the estimated error uncorrelated with the first sound signal and the second sound signal to obtain the weight mapping relationship includes: constructing a first cross-correlation function between the estimated error and the first sound signal, and a second cross-correlation function between the estimated error and the second sound signal; With the goal of setting the values ​​of the first cross-correlation function and the second cross-correlation function to zero, the estimation error and the linear combination relationship are substituted into the first cross-correlation function and the second cross-correlation function to obtain the weight mapping relationship.

6. The sound separation method according to any one of claims 1 to 5, characterized in that: The linear combination relationship includes the coherent sound and the ambient sound of the first channel being added to obtain the first sound signal, and the coherent sound and the difference factor being multiplied and then added to the ambient sound of the second channel to obtain the second sound signal.

7. A sound separation device, characterized in that: include: A signal acquisition unit, configured to acquire a first sound signal and a second sound signal, wherein the first sound signal and the second sound signal are acquired through different channels; a parameter acquisition unit, configured to determine the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factors of the coherent sound on the different channels based on the short-time energy values ​​of the first sound signal and the second sound signal, and the linear combination relationship between the ambient sound and the coherent sound contained in the first sound signal and the second sound signal; A weight mapping unit, configured to map the short-time energy values ​​of the ambient sound and the coherent sound, and the difference factor, to sound separation weights based on a weight mapping relationship, wherein the weight mapping relationship is obtained by linear fitting based on the linear combination relationship; A sound separation unit is used to separate the ambient sound and the coherent sound from the first sound signal and the second sound signal based on the sound separation weight.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the sound separation method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the sound separation method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the sound separation method according to any one of claims 1 to 6 is implemented.

Images