Audio signal separation method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of audio signal separation, and provides an audio signal separation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining frequency domain signals corresponding to audio signals of different channels of audio equipment; decomposing the frequency domain signal to obtain a coherent sound signal and an environment sound signal of each channel; under the condition that the energy corresponding to the environmental sound signals of the different channels is the same, determining a target environmental sound signal of each channel according to the phase difference between the environmental sound signals of the different channels; and separating the target environment sound signal of the corresponding channel from the audio signal of each channel, and obtaining the target coherent sound signal of the channel. According to the method, the device, the electronic equipment and the storage medium provided by the invention, secondary processing is carried out on the separated target coherent sound signal and the target environment sound signal by a sound mixer or a multi-channel algorithm, an auditory atmosphere with a surrounding sense and a presence sense is created, and the auditory experience of a user is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of audio signal separation, and in particular to an audio signal separation method, device, electronic equipment and storage medium. Background Art

[0002] In order to allow users to obtain a more realistic sound experience in immersive systems such as virtual reality, spatial sound reproduction technology is indispensable, and the extraction of coherent sound and ambient sound helps to achieve flexible spatial sound reproduction.

[0003] However, due to the limitations of existing stereo audio, multi-channel audio, three-dimensional audio and surround sound audio formats, coherent sound and ambient sound are often mixed together. Therefore, how to effectively separate the two has become a difficult problem in current research. Summary of the invention

[0004] The present invention provides an audio signal separation method, device, electronic device and storage medium, which are used to solve the defect in the prior art that due to the limitations of existing audio formats such as stereo, multi-channel audio, three-dimensional audio and surround sound audio, coherent sound and ambient sound are often mixed together and the two are difficult to separate.

[0005] The present invention provides an audio signal separation method, comprising the following steps: Obtain frequency domain signals corresponding to audio signals of different channels of the audio device; Decomposing the frequency domain signal to obtain a coherent sound signal and an ambient sound signal of each channel; When the energies corresponding to the ambient sound signals of the different channels are the same, determining the target ambient sound signal of each channel according to the phase difference between the ambient sound signals of the different channels; The target ambient sound signal of the corresponding channel is separated from the audio signal of each channel to obtain the target coherent sound signal of the channel.

[0006] According to an audio signal separation method provided by the present invention, the different channels are a first channel and a second channel; The determining, when the energies corresponding to the ambient sound signals of the different channels are the same, of the target ambient sound signal of each channel according to the phase difference between the ambient sound signals of the different channels comprises: When a first short-time energy of a first ambient sound signal in the first channel is the same as a second short-time energy of a second ambient sound signal in the second channel, a first target ambient sound signal and a second target ambient sound signal are determined according to a phase difference between the first ambient sound signal and the second ambient sound signal.

[0007] According to an audio signal separation method provided by the present invention, when a first short-time energy of a first ambient sound signal in the first channel is the same as a second short-time energy of a second ambient sound signal in the second channel, determining a first target ambient sound signal and a second target ambient sound signal according to a phase difference between the first ambient sound signal and the second ambient sound signal, comprising: In a case where the amplitudes of the first ambient sound signal and the second ambient sound signal are the same, the first target ambient sound signal and the second target ambient sound signal are determined according to a phase angle difference between the first ambient sound signal and the second ambient sound signal.

[0008] According to an audio signal separation method provided by the present invention, the step of determining the phase angle difference comprises: Obtaining a mutual correlation coefficient between a first frequency domain signal and a second frequency domain signal; the first frequency domain signal is a frequency domain signal corresponding to the audio signal of the first channel, and the second frequency domain signal is a frequency domain signal corresponding to the audio signal of the second channel; determining an amplitude difference factor based on the cross-correlation coefficient, a first short-time energy of the first frequency domain signal, and a second short-time energy of the second frequency domain signal; Based on the amplitude difference factor, the phase angle difference is determined.

[0009] According to an audio signal separation method provided by the present invention, determining the first target ambient sound signal and the second target ambient sound signal according to the phase angle difference between the first ambient sound signal and the second ambient sound signal includes: Determine the first target ambient sound signal according to a first target phase angle of the first ambient sound signal, determine a second target phase angle of the second ambient sound signal based on the first target phase angle and the phase angle difference, and determine the second target ambient sound signal based on the second target phase angle; Or, based on the second target phase angle of the second ambient sound signal, determine the second target ambient sound signal, and based on the second target phase angle and the phase angle difference, determine the first target phase angle of the first ambient sound signal, and based on the first target phase angle, determine the first target ambient sound signal.

[0010] According to an audio signal separation method provided by the present invention, the step of determining the first target phase angle includes: Taking minimizing the first short-time energy of the first ambient sound signal as a goal, constructing a first target constraint between the first short-time energy and the first phase angle of the first ambient sound signal; The first target phase angle is determined based on the discrete value set of the first phase angle and the first target constraint; the first target phase angle minimizes the component amplitude of the first coherent acoustic signal in the first frequency domain signal.

[0011] According to an audio signal separation method provided by the present invention, the step of determining the second target phase angle includes: Taking minimizing the second short-time energy of the second ambient sound signal as a goal, constructing a second target constraint between the second short-time energy and a second phase angle of the second ambient sound signal; The second target phase angle is determined based on the discrete value set of the second phase angle and the second target constraint; the second target phase angle minimizes the component amplitude of the second coherent acoustic signal in the second frequency domain signal.

[0012] The present invention also provides an audio signal separation device, comprising the following units: A first acquisition unit, used to acquire frequency domain signals corresponding to audio signals of different channels of an audio device; A second acquisition unit, used to decompose the frequency domain signal to acquire a coherent sound signal and an ambient sound signal of each channel; A determination unit, configured to determine a target ambient sound signal of each channel according to a phase difference between the ambient sound signals of the different channels when the energies corresponding to the ambient sound signals of the different channels are the same; The separation unit is used to separate the target ambient sound signal of the corresponding channel from the audio signal of each channel to obtain the target coherent sound signal of the channel.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned audio signal separation methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the audio signal separation method described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the audio signal separation method described above is implemented.

[0016] The audio signal separation method, device, electronic device and storage medium provided by the present invention obtain frequency domain signals corresponding to the audio signals of different channels of the audio device, decompose the frequency domain signals, obtain the coherent sound signal and the ambient sound signal of each channel, and then, when the energy corresponding to the ambient sound signals of different channels is the same, determine the target ambient sound signal of each channel according to the phase difference between the ambient sound signals of different channels. Finally, separate the target ambient sound signal of the corresponding channel from the audio signal of each channel to obtain the target coherent sound signal of the channel. Then, the separated target coherent sound signal and target ambient sound signal are processed again by a mixer or a multi-channel algorithm to create a more enveloping and immersive auditory atmosphere and improve the user's auditory experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is one of the flow charts of the audio signal separation method provided by the present invention.

[0019] Figure 2 It is a flow chart of the steps of determining the phase angle difference provided by the present invention.

[0020] Figure 3 It is a flowchart diagram of the steps of determining the first target phase angle provided by the present invention.

[0021] Figure 4 It is a flowchart diagram of the steps of determining the second target phase angle provided by the present invention.

[0022] Figure 5 This is the second flow chart of the audio signal separation method provided by the present invention.

[0023] Figure 6 It is a structural schematic diagram of the audio signal separation device provided by the present invention.

[0024] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, rather than to describe a specific order or precedence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type.

[0027] Figure 1 is one of the flow charts of the audio signal separation method provided by the present invention, such as Figure 1 As shown, the method includes step 110 , step 120 , step 130 and step 140 .

[0028] Step 110: Obtain frequency domain signals corresponding to audio signals of different channels of the audio device.

[0029] Specifically, the frequency domain signals corresponding to the audio signals of different channels of the audio device can be obtained. Here, the audio device can be a smart phone, a tablet computer, a regional microphone, a surround microphone, a three-dimensional sound microphone, etc., which is not specifically limited in the embodiment of the present invention.

[0030] Accordingly, the audio signals of different channels of the audio device may be stereo audio, multi-channel audio, three-dimensional audio and surround sound audio, audio signals in virtual reality (VR) audio and augmented reality (AR) audio, etc., which is not specifically limited in the embodiments of the present invention.

[0031] It can be understood that, when the audio corresponding to the audio device is stereo audio, the different channels of the audio device include two channels, namely a first channel and a second channel, for example, a left channel and a right channel; when the audio corresponding to the audio device is multi-channel audio, the different channels of the audio device include multiple channels, and the embodiment of the present invention does not specifically limit this.

[0032] It should be noted that most audio signals in daily life are non-stationary signals, that is, their frequency components change over time. Traditional audio signals in the time domain cannot provide time and frequency information at the same time, so they cannot effectively process non-stationary signals. Short-time Fourier transform captures the frequency characteristics of audio signals that change over time by dividing the audio signal into multiple short time periods (windows) in the time domain and performing Fourier transform on each time period.

[0033] In the frequency domain, coherent sound signals and ambient sound signals usually have different spectral characteristics. For example, coherent sound signals may be concentrated in certain frequency ranges, while ambient sound signals may be distributed in a wider frequency range. By analyzing frequency domain signals, these spectral differences can be used to separate coherent sound signals from ambient sound signals.

[0034] In summary, in order to better apply to non-stationary states and situations where multiple sound sources exist simultaneously, the audio signal is processed by short-time Fourier transform (STFT) to obtain a frequency domain signal.

[0035] In the process of extracting coherent sound signals and ambient sound signals, the audio signal of each channel is usually represented as the superposition of the coherent sound signal and the ambient sound signal. According to the characteristics of the coherent sound signal and the ambient sound signal, it is assumed that the coherent sound signals between channels are completely correlated, and the coherent sound signals are uncorrelated with the ambient sound signals of each channel and the ambient sound signals between channels. Here, the audio corresponding to the audio device is stereo audio. The different channels of the audio device include the first channel and the second channel, that is, the left channel and the right channel. Therefore, the audio signal in the time domain , Defined as: (1) in, Represents the audio signal of the first channel of the audio device, Represents the audio signal of the second channel of the audio device. represents the coherent acoustic signal, represents the ambient sound signal of the first channel, Represents the ambient sound signal of the second channel, The amplitude difference factor representing the coherent acoustic signal of the first channel and the second channel.

[0036] Step 120: decompose the frequency domain signal to obtain the coherent sound signal and the ambient sound signal of each channel.

[0037] Specifically, after obtaining the frequency domain signal, the frequency domain signal may be decomposed to obtain the coherent sound signal and the ambient sound signal of each channel.

[0038] Here, the audio corresponding to the audio device is a stereo audio. The different channels of the audio device include a first channel and a second channel. The audio signal is represented as a frequency domain signal in the Fourier transform domain. The formula of the frequency domain signal is: (2) in, represents the frequency domain signal of the first channel, represents the frequency domain signal of the second channel, represents the coherent acoustic signal in the frequency domain, represents the ambient sound signal in the frequency domain of the first channel, represents the ambient sound signal in the frequency domain of the second channel, represents the timeframe index, Indicates the frequency index, Indicates the amplitude difference factor of the coherent sound signal of the first channel and the second channel. For the sake of simplicity, it can be omitted if necessary. .

[0039] It is understandable that stereo sound mainly includes two components with different properties. One is a directional sound component, called coherent sound; the other is a diffuse sound component whose direction cannot be distinguished, called ambient sound. Among them, coherent sound can be a dialogue between people, a solo of an instrument, etc., and ambient sound can be a background sound effect, such as traffic sound, wind sound, rain sound, etc. in the distance, which is not specifically limited in the embodiment of the present invention.

[0040] It should be noted that when the audio corresponding to the audio device is multi-channel audio, the ambient sound signal is mostly generated in the rear left channel and rear right channel in a 5.1-channel audio system, or the ambient sound signal is mostly generated in the side left channel and side right channel in a 7.1-channel audio system, and the coherent sound signal is mostly generated in the front left channel, front right channel and center channel. The formulas for the corresponding audio signals and frequency domain signals are similar and will not be repeated here.

[0041] In addition, when the audio corresponding to the audio device is three-dimensional audio, surround sound audio, virtual reality audio and augmented reality audio, the formulas of the corresponding audio signals and frequency domain signals are similar and will not be repeated here.

[0042] Step 130 : When the energies corresponding to the ambient sound signals of different channels are the same, determine the target ambient sound signal of each channel according to the phase difference between the ambient sound signals of different channels.

[0043] Specifically, the embodiment of the present invention first assumes that different channels are composed of coherent sound signals and ambient sound signals, respectively, and the energy of the ambient sound signals in different channels is consistent. Therefore, there is only a phase difference between the ambient sound signals of different channels. Based on this, the amplitude expression of the ambient sound signal can be obtained. The key to this method is to solve the phase of the ambient sound signal. In this process, the optimal phase of the ambient sound signal is solved in combination with the sparsity restriction of the audio, so as to separate the coherent sound signal and the ambient sound signal. This method has a small amount of calculation and a good separation effect.

[0044] Accordingly, when the audio corresponding to the audio device is stereo audio, it is first assumed that the first channel and the second channel are respectively composed of a coherent sound signal and an ambient sound signal, and the energy of the ambient sound signal in the first channel and the second channel is consistent. Therefore, there is only a phase difference between the ambient sound signals of the first channel and the second channel, so that the amplitude expression of the ambient sound signal can be obtained.

[0045] In summary, the general principle is to determine the target ambient sound signal of each channel according to the phase difference between the ambient sound signals of different channels when the energy corresponding to the ambient sound signals of different channels is the same. Here, the target ambient sound signal refers to the ambient sound signal that ultimately needs to be separated into audio signals.

[0046] Step 140 : Separate the target ambient sound signal of the corresponding channel from the audio signal of each channel to obtain the target coherent sound signal of the channel.

[0047] Specifically, after obtaining the target ambient sound signal of each channel, the target ambient sound signal of the corresponding channel can be separated from the audio signal of each channel to obtain the target coherent sound signal of the channel. For example, the target ambient sound signal of the corresponding channel can be subtracted from the audio signal of each channel to obtain the target coherent sound signal of the channel. Here, the target coherent sound signal is the coherent sound signal finally obtained.

[0048] It is understandable that the separated target coherent sound signal and target ambient sound signal can be processed again by a mixer or a multi-channel algorithm to create a more enveloping and immersive auditory atmosphere and achieve better auditory effects.

[0049] Furthermore, the significance of separating coherent sound signals from ambient sound signals is: 1. Enhance immersion: By accurately locating coherent sound sources and finely controlling ambient sound, users can feel as if they are in the scene. 2. Improve sound quality: Improve the clarity and fidelity of sound. 3. Adapt to personalized needs: Customize according to the preferences of different users.

[0050] It can be understood that the method provided in the embodiment of the present invention can be applied to multi-channel audio. The target coherent sound signal and the target ambient sound signal can be processed again by a mixer or a multi-channel algorithm, so as to provide a more immersive audio experience in a home theater or a movie theater. When applied to surround sound audio, this audio signal separation method can help improve the surround sound playback effect. When applied to three-dimensional audio, it can also further provide more accurate sound positioning and spatial sense. When applied to virtual reality and augmented reality audio, these immersive technologies require highly realistic spatial audio effects. This audio signal separation method can help achieve a more dynamic and realistic audio experience.

[0051] The method provided by the embodiment of the present invention obtains frequency domain signals corresponding to the audio signals of different channels of the audio device, decomposes the frequency domain signals, obtains the coherent sound signal and the ambient sound signal of each channel, and then, when the energy corresponding to the ambient sound signals of different channels is the same, determines the target ambient sound signal of each channel according to the phase difference between the ambient sound signals of different channels. Finally, separates the target ambient sound signal of the corresponding channel from the audio signal of each channel to obtain the target coherent sound signal of the channel, and then the separated target coherent sound signal and the target ambient sound signal are processed again by a mixer or a multi-channel algorithm to create a more enveloping and immersive auditory atmosphere and improve the user's auditory experience.

[0052] Based on the above embodiment, the different channels are the first channel and the second channel; Step 130 includes: Step 131: When a first short-time energy of a first ambient sound signal in the first channel is the same as a second short-time energy of a second ambient sound signal in the second channel, a first target ambient sound signal and a second target ambient sound signal are determined according to a phase difference between the first ambient sound signal and the second ambient sound signal.

[0053] Specifically, when the audio corresponding to the audio device is stereo audio, the different channels of the audio device include a first channel and a second channel. Here, the first channel is a left channel, and the second channel is a right channel.

[0054] When the first short-time energy of the first ambient sound signal in the first channel and the second short-time energy of the second ambient sound signal in the second channel are the same, the first target ambient sound signal and the second target ambient sound signal are determined according to the phase difference between the first ambient sound signal and the second ambient sound signal.

[0055] Here, the first target ambient sound signal is the ambient sound signal finally determined in the first channel, and the second target ambient sound signal is the ambient sound signal finally determined in the second channel.

[0056] Assume that the first short-time energy of the first ambient sound signal in the first channel and the second short-time energy of the second ambient sound signal in the second channel in formula (2) are the same, which can be expressed as , combined with relevant assumptions, formula (2) can be expressed as: (3) in, represents the short-time energy of the frequency domain signal of the first channel, Represents the short-time energy of the frequency domain signal of the second channel. The first short-time energy and the second short-time energy are both expressed as express, represents the short-time energy of the coherent acoustic signal, Represents the amplitude difference factor.

[0057] in, , The frequency domain signal of the second channel uses the same method to obtain its short-time energy .

[0058] It can be understood that, given the frequency domain signal of the first channel , the frequency domain signal of the second channel In the case of , , and The extraction of coherent sound signals and ambient sound signals can be completed.

[0059] In the embodiment of the present invention, the signal model and assumption of formula (1) are used to explore the constraint relationship of the phase of the ambient sound signal, and the sparsity of the coherent sound signal is used to estimate the phase of the ambient sound signal to complete the extraction of the coherent sound component.

[0060] Based on the above embodiment, in step 131, when the first short-time energy of the first ambient sound signal in the first channel and the second short-time energy of the second ambient sound signal in the second channel are the same, determining the first target ambient sound signal and the second target ambient sound signal according to the phase difference between the first ambient sound signal and the second ambient sound signal includes: Step 1311: when the amplitudes of the first ambient sound signal and the second ambient sound signal are the same, determine the first target ambient sound signal and the second target ambient sound signal according to the phase angle difference between the first ambient sound signal and the second ambient sound signal.

[0061] Specifically, when the amplitudes of the first ambient sound signal and the second ambient sound signal are the same, the first target ambient sound signal and the second target ambient sound signal are determined according to the phase angle difference between the first ambient sound signal and the second ambient sound signal.

[0062] Specifically, assuming that the ambient sound signal has the same energy in each channel, it can be assumed that the amplitude of the ambient sound signal same: (4) in, (5) In the formula, represents a first phase angle of the first ambient sound signal in the first channel, The second phase angle represents the second ambient sound signal in the second channel.

[0063] Substituting formula (4) into formula (2), we get: (6) because is a non-negative number, so we can get the following constraint relationship between the first phase angle and the second phase angle: (7) in, express The corresponding phase angle, Represents the amplitude difference factor.

[0064] It can be understood that, when the amplitude of the first ambient sound signal is the same as that of the second ambient sound signal, if the first target phase angle is determined first , the second target phase angle can be determined according to the phase angle difference between the first ambient sound signal and the second ambient sound signal. , and the first target phase angle and the second target phase angle are determined, then the first target ambient sound signal and the second target ambient sound signal can be determined respectively based on the first target phase angle and the second target phase angle.

[0065] That is, in formula (2), the frequency domain signal of the first channel is , the frequency domain signal of the second channel and amplitude difference factor All have been determined, when the and The extraction of coherent sound signals and ambient sound signals can be completed by just one of them.

[0066] Based on the above embodiments, Figure 2 is a flow chart of the steps of determining the phase angle difference provided by the present invention, such as Figure 2 As shown, the step of determining the phase angle difference includes: Step 210, obtaining a mutual correlation coefficient between a first frequency domain signal and a second frequency domain signal; the first frequency domain signal is a frequency domain signal corresponding to the audio signal of the first channel, and the second frequency domain signal is a frequency domain signal corresponding to the audio signal of the second channel; Step 220, determining an amplitude difference factor based on the mutual correlation coefficient, the first short-time energy of the first frequency domain signal, and the second short-time energy of the second frequency domain signal; Step 230: determining the phase angle difference based on the amplitude difference factor.

[0067] Specifically, first, a cross-correlation coefficient between a first frequency domain signal and a second frequency domain signal is obtained, wherein the first frequency domain signal is a frequency domain signal corresponding to an audio signal of a first channel, and the second frequency domain signal is a frequency domain signal corresponding to an audio signal of a second channel.

[0068] The formula for the mutual correlation coefficient is as follows: (8) Substituting formula (2) into formula (8) and combining the assumptions about the signal model, we obtain: (9) Since in formula (3) and formula (9), , and About , and Therefore, we can solve , and : in, It can be seen that the amplitude difference factor can be determined based on the mutual correlation coefficient, the first short-time energy of the first frequency domain signal, and the second short-time energy of the second frequency domain signal. .

[0069] Based on formula (7), it can be seen that based on the amplitude difference factor, the phase angle difference can be determined.

[0070] Based on the above embodiment, step 1311 includes: Determine the first target ambient sound signal according to a first target phase angle of the first ambient sound signal, determine a second target phase angle of the second ambient sound signal based on the first target phase angle and the phase angle difference, and determine the second target ambient sound signal based on the second target phase angle; Or, based on the second target phase angle of the second ambient sound signal, determine the second target ambient sound signal, and based on the second target phase angle and the phase angle difference, determine the first target phase angle of the first ambient sound signal, and based on the first target phase angle, determine the first target ambient sound signal.

[0071] Specifically, since in formula (2), the frequency domain signal of the first channel is , the frequency domain signal of the second channel and amplitude difference factor All have been determined, when the and The extraction of coherent sound signals and ambient sound signals can be completed by just one of them.

[0072] That is, the first target phase angle of the first ambient sound signal can be , determine the first target ambient sound signal , and based on the first target phase angle , determine the second target phase angle of the second ambient sound signal , based on the second target phase angle , determine the second target ambient sound signal .

[0073] Or, based on the second target phase angle of the second ambient sound signal , determine the second target ambient sound signal , and based on the second target phase angle , determine the first target phase angle of the first ambient sound signal , based on the first target phase angle , determine the first target ambient sound signal .

[0074] Based on the above embodiments, Figure 3 is a flow chart of the steps of determining the first target phase angle provided by the present invention, such as Figure 3 As shown, the step of determining the first target phase angle includes: Step 310, taking minimizing the first short-time energy of the first ambient sound signal as a target, constructing a first target constraint between the first short-time energy and the first phase angle of the first ambient sound signal; Step 320: determine the first target phase angle based on the discrete value set of the first phase angle and the first target constraint; the first target phase angle minimizes the component amplitude of the first coherent acoustic signal in the first frequency domain signal.

[0075] Specifically, since coherent sound signals are sparse, this feature has been widely used in many audio and music signals. Therefore, the sparsity of coherent sound signals can be used to determine the phase angle of an ambient sound signal. That is, with the first short-time energy of the first ambient sound signal minimized as the goal, the first target constraint between the first short-time energy and the first phase angle of the first ambient sound signal is constructed, and the formula is as follows: (10) in, represents the first phase angle, Represents the first short-time energy.

[0076] The objective function in formula (10) is non-convex. The embodiment of the present invention uses a discrete optimization method to determine .because The range is in the interval The set of discrete values ​​of the first phase angle is defined as: in, , is the total number of discrete phase values. Among the determined number of discrete phase values, a first phase angle that minimizes the amplitude of a component of a first coherent acoustic signal in a first frequency domain signal is selected as a first target phase angle.

[0077] Based on the above embodiments, Figure 4 is a flow chart of the steps of determining the second target phase angle provided by the present invention, such as Figure 4 As shown, the step of determining the second target phase angle includes: Step 410, constructing a second target constraint between the second short-time energy and the second phase angle of the second ambient sound signal with the goal of minimizing the second short-time energy of the second ambient sound signal; Step 420: determine the second target phase angle based on the discrete value set of the second phase angle and the second target constraint; the second target phase angle minimizes the component amplitude of the second coherent acoustic signal in the second frequency domain signal.

[0078] Specifically, based on the above description, the sparsity of the coherent sound signal can be used to determine the phase angle of an ambient sound signal. That is, with the second short-time energy of the second ambient sound signal minimized as the goal, a second target constraint between the second short-time energy and the second phase angle of the second ambient sound signal is constructed, and the formula is as follows: (11) in, represents the second phase angle, Represents the second short-time energy.

[0079] The objective function in formula (11) is non-convex. The embodiment of the present invention uses a discrete optimization method to determine .because The range is in the interval The set of discrete values ​​of the second phase angle is defined as: in, , is the total number of discrete phase values. Among the determined number of discrete phase values, a second phase angle that minimizes the amplitude of the component of the second coherent acoustic signal in the second frequency domain signal is selected as the second target phase angle.

[0080] Based on any of the above embodiments, Figure 5 FIG. 2 is a flow chart of the audio signal separation method provided by the present invention. Figure 5 As shown, the method includes: The first step is to obtain audio signals of different channels of the audio device, and perform short-time Fourier transform on the audio signals of different channels to obtain frequency domain signals corresponding to the audio signals of different channels.

[0081] The second step is to decompose the frequency domain signal to obtain the coherent sound signal and ambient sound signal of each channel.

[0082] In the third step, the above-mentioned different channels are the first channel and the second channel, and then the mutual correlation coefficient of the first frequency domain signal and the second frequency domain signal is obtained; the first frequency domain signal is the frequency domain signal corresponding to the audio signal of the first channel, and the second frequency domain signal is the frequency domain signal corresponding to the audio signal of the second channel.

[0083] In the fourth step, an amplitude difference factor is determined based on the mutual correlation coefficient, the first short-time energy of the first frequency domain signal, and the second short-time energy of the second frequency domain signal.

[0084] In the fifth step, the phase angle difference is determined based on the amplitude difference factor.

[0085] In the sixth step, when the amplitudes of the first ambient sound signal and the second ambient sound signal are the same, with the goal of minimizing the first short-time energy of the first ambient sound signal, a first target constraint between the first short-time energy and the first phase angle of the first ambient sound signal is constructed, and then based on the discrete value set of the first phase angle and the first target constraint, the first target phase angle is determined, wherein the first target phase angle minimizes the component amplitude of the first coherent sound signal in the first frequency domain signal.

[0086] Alternatively, when the amplitudes of the first ambient sound signal and the second ambient sound signal are the same, a second target constraint between the second short-time energy and the second phase angle of the second ambient sound signal is constructed with the goal of minimizing the second short-time energy of the second ambient sound signal, and then the second target phase angle is determined based on the discrete value set of the second phase angle and the second target constraint, wherein the second target phase angle minimizes the component amplitude of the second coherent sound signal in the second frequency domain signal.

[0087] In summary, the above method uses the sparsity of the coherent sound signal to determine the phase of the ambient sound signal, and selects a phase value that minimizes the component amplitude of the coherent sound signal.

[0088] Step 7: determining the first target ambient sound signal according to the first target phase angle of the first ambient sound signal, determining the second target phase angle of the second ambient sound signal based on the first target phase angle and the phase angle difference, and determining the second target ambient sound signal based on the second target phase angle; Or, based on the second target phase angle of the second ambient sound signal, determine the second target ambient sound signal, and based on the second target phase angle and the phase angle difference, determine the first target phase angle of the first ambient sound signal, and based on the first target phase angle, determine the first target ambient sound signal.

[0089] In step 8, the first target ambient sound signal is separated from the audio signal of the first channel to obtain the target coherent sound signal of the first channel, and the second target ambient sound signal is separated from the audio signal of the second channel to obtain the target coherent sound signal of the second channel.

[0090] The method provided by the embodiment of the present invention first assumes that the first channel and the second channel are respectively composed of a coherent sound signal and an ambient sound signal, and the energy of the ambient sound signals of the first channel and the second channel is consistent, so there is only a phase difference between the first ambient sound signal and the second ambient sound signal. Therefore, the key to this method is to solve the phase of the ambient sound signal, and it is proposed to use the sparsity restriction of the audio to solve the optimal phase of the ambient sound signal. There is a constraint relationship between the phases of the ambient sound signals of the first channel and the second channel, and only one of them needs to be solved, and the other can be determined. The method has a small amount of calculation and a good separation effect.

[0091] The audio signal separation device provided by the present invention is described below. The audio signal separation device described below and the audio signal separation method described above can be referred to each other.

[0092] Based on any of the above embodiments, the present invention provides an audio signal separation device, Figure 6 Schematic diagram of the structure of the audio signal separation device provided by the present invention. Figure 6 As shown, the device comprises: A first acquisition unit 610 is used to acquire frequency domain signals corresponding to audio signals of different channels of an audio device; A second acquisition unit 620, configured to decompose the frequency domain signal to acquire a coherent sound signal and an ambient sound signal of each channel; A determination unit 630 is configured to determine a target ambient sound signal of each channel according to a phase difference between the ambient sound signals of the different channels when the energies corresponding to the ambient sound signals of the different channels are the same; The separation unit 640 is used to separate the target ambient sound signal of the corresponding channel from the audio signal of each channel to obtain the target coherent sound signal of the channel.

[0093] The device provided by the embodiment of the present invention obtains frequency domain signals corresponding to the audio signals of different channels of the audio device, decomposes the frequency domain signals, obtains the coherent sound signal and the ambient sound signal of each channel, and then, when the energy corresponding to the ambient sound signals of different channels is the same, determines the target ambient sound signal of each channel according to the phase difference between the ambient sound signals of different channels. Finally, separates the target ambient sound signal of the corresponding channel from the audio signal of each channel to obtain the target coherent sound signal of the channel. The separated target coherent sound signal and the target ambient sound signal are then processed by a mixer or a multi-channel algorithm to create a more enveloping and immersive auditory atmosphere and improve the user's auditory experience.

[0094] Based on any of the above embodiments, the different channels are a first channel and a second channel; The determining unit 630 is specifically configured to: A determination subunit is used to determine a first target ambient sound signal and a second target ambient sound signal according to a phase difference between the first ambient sound signal and the second ambient sound signal when a first short-time energy of the first ambient sound signal in the first channel and a second short-time energy of the second ambient sound signal in the second channel are the same.

[0095] Based on any of the above embodiments, the determining subunit is specifically used to: In a case where the amplitudes of the first ambient sound signal and the second ambient sound signal are the same, the first target ambient sound signal and the second target ambient sound signal are determined according to a phase angle difference between the first ambient sound signal and the second ambient sound signal.

[0096] Based on any of the above embodiments, it further includes a phase angle difference determining unit, and the phase angle difference determining unit is specifically used to: Obtaining a mutual correlation coefficient between a first frequency domain signal and a second frequency domain signal; the first frequency domain signal is a frequency domain signal corresponding to the audio signal of the first channel, and the second frequency domain signal is a frequency domain signal corresponding to the audio signal of the second channel; determining an amplitude difference factor based on the cross-correlation coefficient, a first short-time energy of the first frequency domain signal, and a second short-time energy of the second frequency domain signal; Based on the amplitude difference factor, the phase angle difference is determined.

[0097] Based on any of the above embodiments, the determining subunit is specifically used to: Determine the first target ambient sound signal according to a first target phase angle of the first ambient sound signal, determine a second target phase angle of the second ambient sound signal based on the first target phase angle and the phase angle difference, and determine the second target ambient sound signal based on the second target phase angle; Or, based on the second target phase angle of the second ambient sound signal, determine the second target ambient sound signal, and based on the second target phase angle and the phase angle difference, determine the first target phase angle of the first ambient sound signal, and based on the first target phase angle, determine the first target ambient sound signal.

[0098] Based on any of the above embodiments, the invention further includes a first target phase angle determination unit, wherein the first target phase angle determination unit is specifically configured to: Taking minimizing the first short-time energy of the first ambient sound signal as a goal, constructing a first target constraint between the first short-time energy and the first phase angle of the first ambient sound signal; The first target phase angle is determined based on the discrete value set of the first phase angle and the first target constraint; the first target phase angle minimizes the component amplitude of the first coherent acoustic signal in the first frequency domain signal.

[0099] Based on any of the above embodiments, the invention further includes a second target phase angle determination unit, wherein the second target phase angle determination unit is specifically configured to: Taking minimizing the second short-time energy of the second ambient sound signal as a goal, constructing a second target constraint between the second short-time energy and a second phase angle of the second ambient sound signal; The second target phase angle is determined based on the discrete value set of the second phase angle and the second target constraint; the second target phase angle minimizes the component amplitude of the second coherent acoustic signal in the second frequency domain signal.

[0100] Figure 7 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 7 As shown, the electronic device may include: a processor (processor 710, a communication interface (Communications Interface) 720, a memory (memory) 730 and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call the logic instructions in the memory 730 to execute the audio signal separation method, which includes: obtaining frequency domain signals corresponding to audio signals of different channels of the audio device; decomposing the frequency domain signals to obtain a coherent sound signal and an ambient sound signal of each channel; when the energies corresponding to the ambient sound signals of the different channels are the same, determining the target ambient sound signal of each channel according to the phase difference between the ambient sound signals of the different channels; separating the target ambient sound signal of the corresponding channel from the audio signal of each channel, and obtaining the target coherent sound signal of the channel.

[0101] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0102] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the audio signal separation method provided by the above-mentioned methods, which includes: obtaining frequency domain signals corresponding to the audio signals of different channels of an audio device; decomposing the frequency domain signals to obtain a coherent sound signal and an ambient sound signal of each channel; when the energies corresponding to the ambient sound signals of different channels are the same, determining the target ambient sound signal of each channel based on the phase difference between the ambient sound signals of different channels; separating the target ambient sound signal of the corresponding channel from the audio signal of each channel to obtain the target coherent sound signal of the channel.

[0103] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the audio signal separation method provided by the above-mentioned methods, the method comprising: obtaining frequency domain signals corresponding to the audio signals of different channels of an audio device; decomposing the frequency domain signals to obtain a coherent sound signal and an ambient sound signal of each channel; when the energies corresponding to the ambient sound signals of different channels are the same, determining the target ambient sound signal of each channel based on the phase difference between the ambient sound signals of different channels; separating the target ambient sound signal of the corresponding channel from the audio signal of each channel to obtain the target coherent sound signal of the channel.

[0104] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0105] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for separating an audio signal, characterized in that: include: Obtain frequency domain signals corresponding to audio signals of different channels of the audio device; Decomposing the frequency domain signal to obtain a coherent sound signal and an ambient sound signal of each channel; When the energies corresponding to the ambient sound signals of the different channels are the same, determining the target ambient sound signal of each channel according to the phase difference between the ambient sound signals of the different channels; The target ambient sound signal of the corresponding channel is separated from the audio signal of each channel to obtain the target coherent sound signal of the channel.

2. The audio signal separation method according to claim 1, characterized in that: The different channels are a first channel and a second channel; The determining, when the energies corresponding to the ambient sound signals of the different channels are the same, of the target ambient sound signal of each channel according to the phase difference between the ambient sound signals of the different channels comprises: When a first short-time energy of a first ambient sound signal in the first channel is the same as a second short-time energy of a second ambient sound signal in the second channel, a first target ambient sound signal and a second target ambient sound signal are determined according to a phase difference between the first ambient sound signal and the second ambient sound signal.

3. The audio signal separation method according to claim 2, characterized in that: The method of determining a first target ambient sound signal and a second target ambient sound signal according to a phase difference between the first ambient sound signal and the second ambient sound signal when the first short-time energy of the first ambient sound signal in the first channel and the second short-time energy of the second ambient sound signal in the second channel are the same includes: In a case where the amplitudes of the first ambient sound signal and the second ambient sound signal are the same, the first target ambient sound signal and the second target ambient sound signal are determined according to a phase angle difference between the first ambient sound signal and the second ambient sound signal.

4. The audio signal separation method according to claim 3, characterized in that: The step of determining the phase angle difference comprises: Obtaining a mutual correlation coefficient between a first frequency domain signal and a second frequency domain signal; the first frequency domain signal is a frequency domain signal corresponding to the audio signal of the first channel, and the second frequency domain signal is a frequency domain signal corresponding to the audio signal of the second channel; determining an amplitude difference factor based on the cross-correlation coefficient, a first short-time energy of the first frequency domain signal, and a second short-time energy of the second frequency domain signal; Based on the amplitude difference factor, the phase angle difference is determined.

5. The audio signal separation method according to claim 2, characterized in that: The determining the first target ambient sound signal and the second target ambient sound signal according to the phase angle difference between the first ambient sound signal and the second ambient sound signal includes: Determine the first target ambient sound signal according to a first target phase angle of the first ambient sound signal, determine a second target phase angle of the second ambient sound signal based on the first target phase angle and the phase angle difference, and determine the second target ambient sound signal based on the second target phase angle; Or, based on the second target phase angle of the second ambient sound signal, determine the second target ambient sound signal, and based on the second target phase angle and the phase angle difference, determine the first target phase angle of the first ambient sound signal, and based on the first target phase angle, determine the first target ambient sound signal.

6. The audio signal separation method according to claim 5, characterized in that: The step of determining the first target phase angle comprises: Taking minimizing the first short-time energy of the first ambient sound signal as a goal, constructing a first target constraint between the first short-time energy and the first phase angle of the first ambient sound signal; The first target phase angle is determined based on the discrete value set of the first phase angle and the first target constraint; the first target phase angle minimizes the component amplitude of the first coherent acoustic signal in the first frequency domain signal.

7. The audio signal separation method according to claim 5, characterized in that: The step of determining the second target phase angle comprises: Taking minimizing the second short-time energy of the second ambient sound signal as a goal, constructing a second target constraint between the second short-time energy and a second phase angle of the second ambient sound signal; The second target phase angle is determined based on the discrete value set of the second phase angle and the second target constraint; the second target phase angle minimizes the component amplitude of the second coherent acoustic signal in the second frequency domain signal.

8. An audio signal separation device, characterized in that: include: A first acquisition unit, used to acquire frequency domain signals corresponding to audio signals of different channels of an audio device; A second acquisition unit, used to decompose the frequency domain signal to acquire a coherent sound signal and an ambient sound signal of each channel; A determination unit, configured to determine a target ambient sound signal of each channel according to a phase difference between the ambient sound signals of the different channels when the energies corresponding to the ambient sound signals of the different channels are the same; The separation unit is used to separate the target ambient sound signal of the corresponding channel from the audio signal of each channel to obtain the target coherent sound signal of the channel.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the audio signal separation method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the audio signal separation method according to any one of claims 1 to 7 is implemented.

Images