Audio main and vocal accompaniment separation method and device, storage medium and program product

Abstract

The invention discloses an audio main and accompaniment separation method and device, a storage medium and a program product, and relates to the technical field of signal processing. The method comprises the following steps: acquiring song audio, and extracting a monaural related signal and a monaural unrelated signal from the song audio; obtaining an energy envelope difference signal corresponding to the song audio, and determining a polar left signal interval and a polar right signal interval; obtaining a singing left channel signal and a singing right channel signal based on the single channel related signal, and obtaining an accompaniment left channel signal and an accompaniment right channel signal based on the single channel unrelated signal; respectively adding sound image information to the singing left channel signal in the polar left signal interval and the singing right channel signal in the polar right signal interval by using the single-channel irrelevant signal to obtain a dual-channel singing signal; and adjusting the vocal accompaniment right channel signal in the polar left signal interval and the vocal accompaniment left channel signal in the polar right signal interval to obtain a dual-channel vocal accompaniment signal. The refining degree of separation of the main singing and the accompaniment can be improved at low cost.

Description

Technical Field

[0001] The present invention relates to the technical field of signal processing, and in particular to an audio main and accompaniment vocal separation method, device, storage medium and program product. Background Art

[0002] In the prior art, the separation of main and accompaniment vocals is performed by extracting the monophonic main vocal signal and the monophonic accompaniment vocal signal from the two-channel vocal signal, and then simply copying the monophonic signal into two tracks to obtain the two-channel main vocal signal and the two-channel accompaniment vocal signal; however, the accuracy of separation in this way is not high. In the prior art, the separation of main and accompaniment vocals based on deep learning is also used, specifically by training a large amount of audio data of the main vocal and accompaniment vocals, so that the neural network has the ability to separate the main and accompaniment vocal signals; however, a large amount of data is required for model training, and the technical cost is high. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide an audio main and accompaniment vocal separation method, device, storage medium and program product, which can improve the refinement of the separation of main and accompaniment vocals and achieve low-cost main and accompaniment vocal separation that restores sound and image information. The specific scheme is as follows:

[0004] In a first aspect, the present application discloses a method for separating main and accompaniment vocals from an audio source, comprising:

[0005] Acquire song audio, and extract a mono-channel related signal and a mono-channel unrelated signal from the song audio;

[0006] Acquire an energy envelope difference signal of the song audio, and determine a far left signal interval and a far right signal interval according to the energy envelope difference signal;

[0007] Based on the mono-channel related signal, a lead vocal left channel signal and a lead vocal right channel signal are obtained; based on the mono-channel unrelated signal, a backing vocal left channel signal and a backing vocal right channel signal are obtained;

[0008] Using the monophonic unrelated signal, respectively add the sound and image information to the left channel signal of the lead singer in the extreme left signal interval and the right channel signal of the lead singer in the extreme right signal interval to obtain a dual-channel lead singer signal;

[0009] The accompaniment right channel signal in the extreme left signal interval is adjusted, and the accompaniment left channel signal in the extreme right signal interval is adjusted to obtain a dual-channel accompaniment signal.

[0010] Optionally, obtaining the energy envelope difference signal of the song audio includes:

[0011] Respectively dividing the left channel signal and the right channel signal of the song audio into frames, and calculating the energy envelope of each frame signal to obtain the energy envelope of the left channel and the energy envelope of the right channel;

[0012] According to the difference between the energy envelope of the left channel and the energy envelope of the right channel, an energy envelope difference signal corresponding to the song audio is obtained.

[0013] Optionally, determining a far left signal interval and a far right signal interval according to the energy envelope difference signal includes:

[0014] Obtain a first energy threshold and a second energy threshold; the first energy threshold is greater than the second energy threshold; the first energy threshold is an absolute value, and the first energy threshold corresponds to a first positive energy threshold and a first negative energy threshold; the second energy threshold is an absolute value, and the second energy threshold corresponds to a second positive energy threshold and a second negative energy threshold;

[0015] Locate a first region whose energy is greater than a first positive energy threshold value from the energy envelope difference signal, search along the energy envelope difference signal to both sides with the first region as the center, locate two time points whose energy is equal to the second positive energy threshold value, and use the time period between the two time points as the extreme left signal interval;

[0016] A second area whose energy is less than the first energy negative threshold is located from the energy envelope difference signal, and the energy envelope difference signal is searched to both sides with the second area as the center to locate two time points whose energy is equal to the second energy negative threshold, and the time period between the two time points is used as the extreme right signal interval.

[0017] Optionally, calculating the energy envelope of each frame signal to obtain the energy envelope of the left channel and the energy envelope of the right channel includes:

[0018] Calculating the energy envelope of each frame signal, performing smoothing and normalization processing on the energy envelope to obtain the energy envelope of the left channel and the energy envelope of the right channel;

[0019] The energy envelope is smoothed, comprising:

[0020] The energy envelope is upsampled to obtain an upsampled signal sequence, the upsampled signal sequence is time-domain convolved to obtain a convolved sequence, and the convolved sequence is downsampled to obtain a downsampled signal sequence.

[0021] Optionally, the method of using the monophonic unrelated signal to add sound and image information to the lead singer left channel signal in the extreme left signal interval and the lead singer right channel signal in the extreme right signal interval to obtain a dual-channel lead singer signal includes:

[0022] Obtaining an updated lead vocal left channel signal according to a difference between a lead vocal left channel signal in a far left signal interval and an unrelated signal in the far left signal interval;

[0023] Obtaining an updated lead vocal right channel signal according to a difference between a lead vocal right channel signal in a far right signal interval and an unrelated signal in the far right signal interval;

[0024] Based on the updated left-channel lead vocal signal and the updated right-channel lead vocal signal, a dual-channel lead vocal signal is obtained.

[0025] Optionally, the adjusting the accompaniment right channel signal in the extreme left signal interval and the adjusting the accompaniment left channel signal in the extreme right signal interval to obtain a dual-channel accompaniment signal includes:

[0026] The right channel signal of the accompaniment in the extreme left signal interval is adjusted to 0 to obtain an updated right channel signal of the accompaniment;

[0027] The left channel signal of the accompaniment vocals in the extreme right signal interval is adjusted to 0 to obtain an updated left channel signal of the accompaniment vocals;

[0028] Based on the updated accompaniment right channel signal and the updated accompaniment left channel signal, a two-channel accompaniment signal is obtained.

[0029] Optionally, after adjusting the right channel signal of the accompaniment in the extreme left signal interval and adjusting the left channel signal of the accompaniment in the extreme right signal interval to obtain a dual-channel accompaniment signal, the method further includes:

[0030] The two-channel accompaniment singing signal is subjected to decorrelation processing to obtain a two-channel accompaniment singing signal after decorrelation processing.

[0031] Optionally, the decorrelation processing of the two-channel accompaniment signal includes:

[0032] Decorrelation processing is performed by delaying sampling of the two-channel accompaniment signal;

[0033] Or, decorrelation processing is performed by changing the phase of the two-channel accompaniment signal.

[0034] In a second aspect, the present application discloses an electronic device, comprising:

[0035] Memory, used to store computer programs;

[0036] The processor is used to execute the computer program to implement the aforementioned audio main and accompaniment vocal separation method.

[0037] In a third aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned audio main and accompaniment vocal separation method.

[0038] In the present application, song audio is obtained, and a monophonic correlated signal and a monophonic uncorrelated signal are extracted from the song audio; an energy envelope difference signal of the song audio is obtained, and a far left signal interval and a far right signal interval are determined according to the energy envelope difference signal; a lead vocal left channel signal and a lead vocal right channel signal are obtained based on the monophonic correlated signal, and a accompaniment vocal left channel signal and a accompaniment vocal right channel signal are obtained based on the monophonic uncorrelated signal; the monophonic uncorrelated signal is used to add sound and image information to the lead vocal left channel signal in the far left signal interval and the lead vocal right channel signal in the far right signal interval, respectively, to obtain a dual-channel lead vocal signal; the accompaniment vocal right channel signal in the far left signal interval is adjusted, and the accompaniment vocal left channel signal in the far right signal interval is adjusted to obtain a dual-channel accompaniment vocal signal. It can be seen that by detecting the energy envelope difference between the left and right channels of the song audio, the intervals of the possible extreme left signal and the extreme right signal are determined, and the extreme left and extreme right signals are used to give sound and image information to the lead singer and the accompaniment vocals, and finally a two-channel lead singer signal and a two-channel accompaniment singer signal are obtained, thereby improving the degree of refinement of the separation of the lead singer and the accompaniment vocals, and achieving low-cost separation of the lead and accompaniment vocals and restoring sound and image information. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0040] FIG1 is a flow chart of an audio main and accompaniment vocal separation method provided by the present application;

[0041] FIG2 is a schematic diagram of a specific energy envelope provided in the present application;

[0042] FIG3 is a flow chart of a specific method for separating main and accompaniment vocals of an audio program provided by the present application;

[0043] FIG4 is a schematic diagram of a specific audio main and accompaniment vocal separation system framework provided by the present application;

[0044] FIG5 is a flow chart of a specific method for separating main and accompaniment vocals of an audio program provided by the present application;

[0045] FIG6 is a structural diagram of an electronic device provided in this application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] In the prior art, the separation of main and accompaniment vocals is performed by extracting a monophonic main vocal signal and a monophonic accompaniment vocal signal from a two-channel vocal signal, and then simply copying the monophonic signal into two tracks to obtain a two-channel main vocal signal and a two-channel accompaniment vocal signal; however, the accuracy of separation in this way is not high. In the prior art, the separation of main and accompaniment vocals based on deep learning is also used, specifically by training a large amount of lead vocal and accompaniment vocal audio data so that the neural network has the ability to separate the main and accompaniment vocal signals; however, a large amount of data is required for model training, and the technical cost is high. In order to overcome the above technical problems, the present application proposes an audio main and accompaniment vocal separation method, which can improve the degree of refinement of the separation of lead vocals and accompaniment vocals, and achieve low-cost separation of main and accompaniment vocals and restore sound and image information.

[0048] The present application embodiment discloses a method for separating main and accompaniment vocals from an audio source. Referring to FIG1 , the method may include the following steps:

[0049] Step S11: Acquire song audio, and extract a mono-channel related signal and a mono-channel unrelated signal from the song audio.

[0050] First, the song audio is obtained. It should be noted that the song audio obtained at this time is the song audio containing the human voice signal, and the human voice signal specifically includes the signal of the lead singer and the backing singer, that is, the song audio at this time is a song that needs to be separated from the lead singer and the backing singer. The lead singer refers to the singer who plays the main role in the band or music group; the backing singer refers to singing beside and cooperating with the lead singer's performance as a modification of the lead singer.

[0051] Specifically, the mono-channel related signal is extracted from the song audio, that is, the lead vocal signal is extracted; at the same time, the binaural related signal is extracted from the song audio, that is, the accompaniment vocal signal is extracted. The extraction of the above-mentioned related and unrelated signals can be specifically based on digital signal processing algorithms, such as M / S (Mid / Side) decomposition, adaptive filtering, PCA-based signal decomposition, etc., to separate the two-channel related signals and unrelated signals from the human voice binaural signal.

[0052] Taking M / S decomposition as an example, the formula is as follows:

[0053] ; ;

[0054] in, is the left channel signal, The right channel signal. In the Mid / Side stereo system, the Mid channel is located in the center of the stereo image, and the Side channel is located on both sides of the stereo image. In general song mixing, the lead vocal is usually mixed in the center of the stereo image, and the accompaniment is mixed to both sides of the stereo image. Thus, the lead vocal mono signal Mid (n) and the accompaniment mono signal Side (n) are obtained.

[0055] Step S12: Obtain an energy envelope difference signal of the song audio, and determine a far left signal interval and a far right signal interval according to the energy envelope difference signal.

[0056] According to the energy envelope difference signal corresponding to the song audio, the extreme left signal interval and the extreme right signal interval are determined. The extreme left signal means that for a stereo audio file, only the left channel has a signal; similarly, the extreme right signal means that for a stereo audio file, only the right channel has a signal. It can be understood that since the extreme left signal means that in a stereo audio file, only the left channel has a signal, the signal energy of the left channel in the extreme left signal interval must be much greater than the signal energy of the right channel; the same applies to the right channel. Therefore, the occurrence time and end time of the extreme left and extreme right signals, that is, the starting and ending points of the extreme left signal interval and the extreme right signal interval, can be located according to the energy envelope difference between the left and right channels.

[0057] Wherein, the energy envelope difference signal of the song audio can be obtained by: dividing the left channel signal and the right channel signal of the song audio into frames respectively, and calculating the energy envelope of each frame signal to obtain the energy envelope of the left channel and the energy envelope of the right channel; according to the difference between the energy envelope of the left channel and the energy envelope of the right channel, obtaining the energy envelope difference signal corresponding to the song audio. Wherein, calculating the energy envelope of each frame signal to obtain the energy envelope of the left channel and the energy envelope of the right channel specifically includes: calculating the energy envelope of each frame signal, smoothing and normalizing the energy envelope to obtain the energy envelope of the left channel and the energy envelope of the right channel; smoothing the energy envelope specifically includes: up-sampling the energy envelope to obtain an up-sampled signal sequence, performing time domain convolution on the up-sampled signal sequence to obtain a convolved sequence, and down-sampling the convolved sequence to obtain a down-sampled signal sequence.

[0058] To calculate the signal energy envelope, first, the input signal is divided into frames, and the energy envelope of each frame is calculated. The energy envelope calculation formula can use the short-time average energy calculation formula, as follows:

[0059] ;

[0060] Among them, x (n) represents the nth frame of speech signal, N represents the window length, and this formula is used to calculate the short-time average energy of the speech signal at a certain moment. After obtaining the energy envelope of each frame, the energy envelope is then smoothed and normalized; the smoothing process includes: upsampling the energy envelope, then convolution window function in the time domain, and then downsampling; finally, normalization is performed to obtain the smoothed energy envelope, so that the energy envelope difference signal can be obtained based on the energy envelope difference.

[0061] In some embodiments, determining the extreme left signal interval and the extreme right signal interval according to the energy envelope difference signal specifically includes the following steps: obtaining a first energy threshold and a second energy threshold; the first energy threshold is greater than the second energy threshold; the first energy threshold is an absolute value, and the first energy threshold corresponds to a first positive energy threshold and a first negative energy threshold; the second energy threshold is an absolute value, and the second energy threshold corresponds to a second positive energy threshold and a second negative energy threshold; locating a first area with energy greater than the first positive energy threshold from the energy envelope difference signal, searching along the energy envelope difference signal on both sides with the first area as the center, locating two time points with energy equal to the second positive energy threshold, and taking the time period between the two time points as the extreme left signal interval; locating a second area with energy less than the first negative energy threshold from the energy envelope difference signal, searching along the energy envelope difference signal on both sides with the second area as the center, locating two time points with energy equal to the second negative energy threshold, and taking the time period between the two time points as the extreme right signal interval.

[0062] That is to say, in this embodiment, the double threshold detection method can be used to extract the extreme left and extreme right intervals, that is, to determine the start and end points of the extreme left and extreme right signals. The extraction of the start and end points of the extreme left and extreme right signals can be considered as an endpoint detection task based on the smoothed energy envelope difference signal of the left and right channels. The double threshold detection method is used for extraction, that is, a higher first energy threshold T1 is selected, and the energy envelope absolute value higher than the T1 threshold is considered to be the extreme left and extreme right signals, and then the search is extended to both sides from the intersection to find the two points where the energy absolute value intersects with the lower second energy threshold T2, which are considered to be the start and end points of the extreme left and extreme right signals. For example, as shown in Figure 2, the shaded part is the waveform of the left channel of the human voice of the song audio, and the curve (energy diff) is the smoothed energy envelope difference signal. In this embodiment, the energy envelope after the left channel is smoothed minus the energy envelope after the right channel is smoothed. Of course, the energy envelope after the right channel is smoothed minus the energy envelope after the left channel is smoothed. The signal in Figure 2 contains an extreme left signal interval and an extreme right signal interval.

[0063] Step S13: obtaining a left channel signal of lead vocal and a right channel signal of lead vocal based on the mono-channel correlated signal, and obtaining a left channel signal of accompaniment vocal and a right channel signal of accompaniment vocal based on the mono-channel uncorrelated signal.

[0064] Since the extracted lead vocals and accompaniment are mono signals, they need to be copied into dual tracks first to become dual channels. Specifically, the left channel signal of the lead vocal and the right channel signal of the lead vocal are obtained by copying the mono related signal, and the left channel signal of the accompaniment and the right channel signal of the accompaniment are obtained by copying the mono unrelated signal. That is, the dual channel contains two independent audio signals, the left channel and the right channel, which are played through the left and right speakers respectively. However, the copied left channel signal of the lead vocal and the right channel signal of the lead vocal have no sound and image information, and the correlation is too strong, and the phase information of the original human voice is lost; the left channel signal of the accompaniment and the right channel signal of the accompaniment are the same. Therefore, it is necessary to use the extreme left signal and the extreme right signal to give the lead vocal and the accompaniment information. The spatial position of each part displayed in the listener's hearing, and the sound picture formed thereby is called sound image.

[0065] In addition, there is no set order for the above steps S12 and S13, and step S13 may be performed first and then step S12. Of course, they may also be performed simultaneously.

[0066] Step S14: using the mono-channel unrelated signal, respectively adding sound and image information to the lead singer left channel signal in the extreme left signal interval and the lead singer right channel signal in the extreme right signal interval to obtain a dual-channel lead singer signal.

[0067] That is, for the sound image assignment of the lead vocal signal, the extreme left and extreme right signals are weighted on the dual channels obtained by copying the mono channel to obtain a dual-channel lead vocal signal containing sound image information.

[0068] In some embodiments, the use of the monophonic unrelated signal to add sound and image information to the lead singer left channel signal in the extreme left signal interval and the lead singer right channel signal in the extreme right signal interval, respectively, to obtain a dual-channel lead singer signal, may specifically include: using the unrelated signal in the extreme left signal interval to add sound and image information to the lead singer left channel signal in the extreme left signal interval to obtain an updated lead singer left channel signal; using the unrelated signal in the extreme right signal interval to add sound and image information to the lead singer right channel signal in the extreme right signal interval to obtain an updated lead singer right channel signal; that is, according to the difference between the lead singer left channel signal in the extreme left signal interval and the unrelated signal in the extreme left signal interval, to obtain the updated lead singer left channel signal; according to the difference between the lead singer right channel signal in the extreme right signal interval and the unrelated signal in the extreme right signal interval, to obtain the updated lead singer right channel signal; finally, based on the updated lead singer left channel signal and the updated lead singer right channel signal, to obtain a dual-channel lead singer signal.

[0069] It is understandable that when the sound image of the main accompaniment is assigned to the extreme left or extreme right signal, the separation of the main accompaniment based on the relevant signal and the irrelevant signal is not completely accurate. Therefore, after determining the starting and ending points of several extreme left and extreme right signals, further sound image assignment is required to improve the refinement of the separation. The irrelevant signals obtained by the previous separation are considered to be the waveform data of the extreme left and extreme right signals. For the lead singer, the lead singer stereo without sound image information obtained by copying the relevant signal into two tracks includes: the lead singer left channel LV_L and the lead singer right channel LV_R; if the starting and ending points of the extreme left signal are detected to be [t1, t2], then LV_L[t1 :t2] = LV_L[t1 : t2] - irrelevant signal [t1 : t2]; the same is true for the extreme right signal, the starting and ending points of the extreme right signal are [t3,t4], LV_R[t3 : t4] = LV_R[t3 : t4] - irrelevant signal [t3 : t4].

[0070] Step S15: adjusting the accompaniment right channel signal within the extreme left signal interval, and adjusting the accompaniment left channel signal within the extreme right signal interval to obtain a dual-channel accompaniment signal.

[0071] It can be seen that the relevant and unrelated signals of the monophonic signal are first extracted from the stereo vocal sound, and then the possible extreme left and extreme right sound and image information is extracted by detecting the energy envelope difference of the left and right channels in the stereo vocal sound, and then added to the separated main and accompaniment stereo sound to obtain the two-channel lead vocal and two-channel accompaniment vocal containing sound and image information.

[0072] The execution order of the above step S14 and step S15 is not limited, and step S15 may be executed first, or may be executed simultaneously.

[0073] The above-mentioned adjustment of the accompaniment right channel signal within the extreme left signal interval, and adjustment of the accompaniment left channel signal within the extreme right signal interval, to obtain a dual-channel accompaniment signal, may include: adjusting the accompaniment right channel signal within the extreme left signal interval to 0, to obtain an updated accompaniment right channel signal; adjusting the accompaniment left channel signal within the extreme right signal interval to 0, to obtain an updated accompaniment left channel signal; based on the updated accompaniment right channel signal and the updated accompaniment left channel signal, to obtain a dual-channel accompaniment signal.

[0074] It can be understood that in the extreme left signal interval, it can be considered that there is no signal in the right channel, and the same applies to the extreme right signal interval. Therefore, for the accompaniment, the accompaniment stereo without sound and image information is generated by copying the unrelated signal into two tracks: the accompaniment left channel BV_L and the accompaniment right channel BV_R; if the start and end points of the extreme left signal are detected as [t1, t2], then BV_R[t1: t2] = 0; the same applies to the extreme right signal, BV_L[t3: t4] = 0.

[0075] As can be seen from the above, in this embodiment, song audio is obtained, and a monophonic correlated signal and a monophonic uncorrelated signal are extracted from the song audio; an energy envelope difference signal of the song audio is obtained, and a far left signal interval and a far right signal interval are determined according to the energy envelope difference signal; a lead vocal left channel signal and a lead vocal right channel signal are obtained based on the monophonic correlated signal, and a accompaniment left channel signal and a accompaniment right channel signal are obtained based on the monophonic uncorrelated signal; the monophonic uncorrelated signal is used to add sound and image information to the lead vocal left channel signal in the far left signal interval and the lead vocal right channel signal in the far right signal interval, respectively, to obtain a dual-channel lead vocal signal; the accompaniment right channel signal in the far left signal interval is adjusted, and the accompaniment left channel signal in the far right signal interval is adjusted to obtain a dual-channel accompaniment signal. It can be seen that by detecting the energy envelope difference between the left and right channels of the song audio, the intervals of the possible extreme left signal and the extreme right signal are determined, and the extreme left and extreme right signals are used to give sound and image information to the lead singer and the accompaniment vocals, and finally a two-channel lead singer signal and a two-channel accompaniment singer signal are obtained, thereby improving the degree of refinement of the separation of the lead singer and the accompaniment vocals, and achieving low-cost separation of the lead and accompaniment vocals and restoring sound and image information.

[0076] The present application embodiment discloses a specific method for separating the main and accompaniment vocals of an audio, as shown in FIG3 , the method may include the following steps:

[0077] Step S21: Acquire song audio, and extract a mono-channel related signal and a mono-channel unrelated signal from the song audio.

[0078] Step S22: Obtain an energy envelope difference signal of the song audio, and determine a far left signal interval and a far right signal interval according to the energy envelope difference signal.

[0079] Step S23: obtaining a left channel signal of lead vocal and a right channel signal of lead vocal based on the mono-channel correlated signal, and obtaining a left channel signal of accompaniment vocal and a right channel signal of accompaniment vocal based on the mono-channel uncorrelated signal.

[0080] Step S24: using the monophonic unrelated signal, respectively adding the sound and image information to the left channel signal of the lead singer in the extreme left signal interval and the right channel signal of the lead singer in the extreme right signal interval to obtain a dual-channel lead singer signal.

[0081] Step S25: adjusting the accompaniment right channel signal within the extreme left signal interval, and adjusting the accompaniment left channel signal within the extreme right signal interval to obtain a dual-channel accompaniment signal.

[0082] Step S26: performing decorrelation processing on the two-channel accompaniment signal to obtain a two-channel accompaniment signal after decorrelation processing.

[0083] Since the phases of the lead vocal and accompaniment vocal signals are similar, it will feel like the lead vocal and accompaniment vocals are both coming from directly in front of the listener, resulting in a low user experience. In order to enhance the immersive user experience, the main and accompaniment vocals need to be placed in different spatial positions. Specifically, the two-channel accompaniment vocal signal is decorrelated so that the user feels that the accompaniment vocals are coming from both sides. The main and accompaniment vocal signals thus separated can be used in panoramic sound production.

[0084] The above-mentioned decorrelation processing of the two-channel accompaniment signal may include: decorrelation processing by delaying sampling of the two-channel accompaniment signal; or decorrelation processing by changing the phase of the two-channel accompaniment signal. That is, decorrelation can be performed by delaying sampling or phase adjustment, and of course, the method that can decorrelate is also acceptable.

[0085] The above decorrelation processing specifically includes but is not limited to: time domain sampling point delay, passing through an all-pass filter, phase shift of 90 degrees, etc. Among them, the implementation principle of time domain sampling point delay is to delay the signal data by several sampling points. The formula is: y(n)=x(n - t); y(n) is the delayed signal, x(n) is the input signal, and t is the number of delayed sampling points.

[0086] Among them, an all-pass filter is a filter that allows all frequencies to pass; unlike other common filter types (such as low-pass filters or high-pass filters), its main function is to change the phase response of the signal while maintaining its amplitude. Taking a first-order all-pass filter as an example, it is expressed as:

[0087] ;

[0088] H(z) is the z-transform transfer function of the all-pass filter in the frequency domain, and c is the filter coefficient;

[0089] By multiplying the frequency domain representation X(z) of the input signal by the transfer function, we can get the frequency domain representation Y(z) of the output signal, Y(n) =X(n)H(z).

[0090] Among them, the phase shift of 90 degrees is also a way to achieve the decorrelation effect by changing the phase of the signal; its advantage is to give each frequency component in the signal a phase shift of plus or minus 90°.

[0091] Among them, the specific process of the above steps S21 to S25 can refer to the corresponding content disclosed in the above embodiments, and will not be repeated here.

[0092] As can be seen from the above, in this embodiment, the song audio is taken, and a mono-channel related signal and a mono-channel unrelated signal are extracted from the song audio; the energy envelope difference signal of the song audio is obtained, and the extreme left signal interval and the extreme right signal interval are determined according to the energy envelope difference signal; the lead singer left channel signal and the lead singer right channel signal are obtained based on the mono-channel related signal, and the accompaniment left channel signal and the accompaniment right channel signal are obtained based on the mono-channel unrelated signal; the mono-channel unrelated signal is used to add sound and image information to the lead singer left channel signal in the extreme left signal interval and the lead singer right channel signal in the extreme right signal interval, respectively, to obtain a dual-channel lead singer signal; the accompaniment right channel signal in the extreme left signal interval is adjusted, and the accompaniment left channel signal in the extreme right signal interval is adjusted to obtain a dual-channel accompaniment signal; the dual-channel accompaniment signal is decorrelated to obtain a dual-channel accompaniment signal after decorrelation processing. It can be seen that by decorrelating the accompaniment, different spatial positioning of the main accompaniment can be achieved, which enhances the user's experience and immersion.

[0093] In the audio main and accompaniment separation solution of the present application, the system framework used can be specifically shown in FIG4, and can specifically include: a backend server and a number of user terminals that establish communication connections with the backend server. The user terminals include but are not limited to tablet computers, laptop computers, smart phones, and personal computers (PCs), which are not limited here.

[0094] For example, as shown in FIG5, in the present application, the user end is used to send song audio (vocals) to be separated to the background server, and the audio is a stereo (stereo), and the background server executes the steps of the audio main and accompaniment separation method, including extracting a mono correlated signal (mono) and a mono uncorrelated signal (un-correlated signal) from the song audio; obtaining an energy envelope difference signal of the song audio, and determining a far left signal interval and a far right signal interval according to the energy envelope difference signal; obtaining a lead vocal left channel signal and a lead vocal right channel signal based on the mono correlated signal, and obtaining an accompaniment left channel signal and an accompaniment right channel signal based on the mono uncorrelated signal, thereby obtaining stereo from mono; using the mono uncorrelated signal, adding sound and image information to the lead vocal left channel signal in the far left signal interval and the lead vocal right channel signal in the far right signal interval, respectively, to obtain a dual-channel lead vocal signal (lead vocal); adjust the right channel signal of the accompaniment in the extreme left signal interval, and adjust the left channel signal of the accompaniment in the extreme right signal interval to obtain a two-channel accompaniment signal (background vocal). The backend server pushes the two-channel lead vocal signal and the two-channel accompaniment signal corresponding to the song audio to the user end.

[0095] Below, taking a certain music APP as an example, the technical solution of this application is explained.

[0096] Assume that the user has installed this music app on the user side, and sends the song audio that he wants to separate the main and accompaniment vocals to the background server of this music app, and sends a separation request. The background server obtains the song audio, extracts the mono related signal from the song audio, obtains the mono main vocal signal, extracts the mono unrelated signal from the song audio, obtains the mono accompaniment vocal signal; calculates the energy envelope difference signal corresponding to the song audio, and determines all the extreme left signal intervals and extreme right signal intervals in the song audio according to the energy envelope difference signal. The lead vocal left channel signal and the lead vocal right channel signal are obtained by copying the mono related signal, and the accompaniment left channel signal and the accompaniment right channel signal are obtained by copying the mono unrelated signal; the unrelated signal in the extreme left signal interval is used to add the sound and image information to the lead vocal left channel signal in the extreme left signal interval to obtain the updated lead vocal left channel signal; the unrelated signal in the extreme right signal interval is used to add the sound and image information to the lead vocal right channel in the extreme right signal interval to obtain the updated lead vocal right channel signal; based on the updated lead vocal left channel signal and the updated lead vocal right channel signal, a dual-channel lead vocal signal is obtained. The accompaniment right channel signal in the extreme left signal interval is adjusted, and the accompaniment left channel signal in the extreme right signal interval is adjusted to obtain a dual-channel accompaniment vocal signal.

[0097] Correspondingly, the embodiment of the present application also discloses an audio main and accompaniment vocal separation device, the device comprising:

[0098] The signal extraction module 11 is used to obtain song audio and extract a mono-channel related signal and a mono-channel unrelated signal from the song audio;

[0099] A signal interval determination module 12 is used to obtain an energy envelope difference signal of the song audio, and determine a far left signal interval and a far right signal interval according to the energy envelope difference signal;

[0100] A dual-channel generation module 13, used to obtain a lead vocal left channel signal and a lead vocal right channel signal based on the mono-channel related signal, and to obtain a backing vocal left channel signal and a backing vocal right channel signal based on the mono-channel unrelated signal;

[0101] The lead singer sound image adjustment module 14 is used to use the monophonic unrelated signal to add sound image information to the lead singer left channel signal in the extreme left signal interval and the lead singer right channel signal in the extreme right signal interval, respectively, to obtain a dual-channel lead singer signal;

[0102] The accompaniment sound image adjustment module 15 is used to adjust the accompaniment right channel signal in the extreme left signal interval, and adjust the accompaniment left channel signal in the extreme right signal interval to obtain a dual-channel accompaniment signal.

[0103] As can be seen from the above, in this embodiment, song audio is obtained, and a monophonic correlated signal and a monophonic uncorrelated signal are extracted from the song audio; an energy envelope difference signal of the song audio is obtained, and a far left signal interval and a far right signal interval are determined according to the energy envelope difference signal; a lead vocal left channel signal and a lead vocal right channel signal are obtained based on the monophonic correlated signal, and a accompaniment left channel signal and a accompaniment right channel signal are obtained based on the monophonic uncorrelated signal; the monophonic uncorrelated signal is used to add sound and image information to the lead vocal left channel signal in the far left signal interval and the lead vocal right channel signal in the far right signal interval, respectively, to obtain a dual-channel lead vocal signal; the accompaniment right channel signal in the far left signal interval is adjusted, and the accompaniment left channel signal in the far right signal interval is adjusted to obtain a dual-channel accompaniment signal. It can be seen that by detecting the energy envelope difference between the left and right channels of the song audio, the intervals of the possible extreme left signal and the extreme right signal are determined, and the extreme left and extreme right signals are used to give sound and image information to the lead singer and the accompaniment vocals, and finally a two-channel lead singer signal and a two-channel accompaniment singer signal are obtained, thereby improving the degree of refinement of the separation of the lead singer and the accompaniment vocals, and achieving low-cost separation of the lead and accompaniment vocals and restoring sound and image information.

[0104] In some specific embodiments, the signal interval determination module 12 may specifically include:

[0105] An energy envelope acquisition unit is used to frame the left channel signal and the right channel signal of the song audio respectively, and calculate the energy envelope of each frame signal to obtain the energy envelope of the left channel and the energy envelope of the right channel;

[0106] The energy envelope difference signal determination unit is used to obtain the energy envelope difference signal corresponding to the song audio according to the difference between the energy envelope of the left channel and the energy envelope of the right channel.

[0107] In some specific embodiments, the signal interval determination module 12 may specifically include:

[0108] A threshold acquisition unit, used to acquire a first energy threshold and a second energy threshold; the first energy threshold is greater than the second energy threshold; the first energy threshold is an absolute value, and the first energy threshold corresponds to a first positive energy threshold and a first negative energy threshold; the second energy threshold is an absolute value, and the second energy threshold corresponds to a second positive energy threshold and a second negative energy threshold;

[0109] A far left signal interval determination unit is used to locate a first area whose energy is greater than a first energy positive threshold from the energy envelope difference signal, search along the energy envelope difference signal to both sides with the first area as the center, locate two time points whose energy is equal to the second energy positive threshold, and use the time period between the two time points as the far left signal interval;

[0110] A far-right signal interval determination unit is used to locate a second area whose energy is less than a first energy negative threshold from the energy envelope difference signal, search along the energy envelope difference signal to both sides with the second area as the center, locate two time points whose energy is equal to the second energy negative threshold, and use the time period between the two time points as the far-right signal interval.

[0111] In some specific embodiments, the energy envelope acquisition unit may specifically include:

[0112] A smoothing and normalization processing unit, used for performing smoothing and normalization processing on the energy envelope;

[0113] The smoothing normalization process may specifically include:

[0114] The smoothing processing unit is used to upsample the energy envelope to obtain an upsampled signal sequence, perform time domain convolution on the upsampled signal sequence to obtain a convolved sequence, and downsample the convolved sequence to obtain a downsampled signal sequence.

[0115] In some specific embodiments, the lead singer sound image adjustment module 14 may specifically include:

[0116] A lead vocal left channel signal adjustment unit, used for obtaining an updated lead vocal left channel signal according to a difference between the lead vocal left channel signal in the extreme left signal interval and an unrelated signal in the extreme left signal interval;

[0117] A lead singer right channel signal adjustment unit, used for obtaining an updated lead singer right channel signal according to a difference between the lead singer right channel signal in the extreme right signal interval and an unrelated signal in the extreme right signal interval;

[0118] A dual-channel lead vocal signal determination unit is used to obtain an updated lead vocal right channel signal based on the difference between the lead vocal right channel signal in the extreme right signal interval and the unrelated signal in the extreme right signal interval; and obtain a dual-channel lead vocal signal based on the updated lead vocal left channel signal and the updated lead vocal right channel signal.

[0119] In some specific embodiments, the accompaniment sound image adjustment module 15 may specifically include:

[0120] The accompaniment right channel signal adjustment unit is used to adjust the accompaniment right channel signal in the extreme left signal interval to 0 to obtain an updated accompaniment right channel signal;

[0121] The accompaniment left channel signal adjustment unit is used to adjust the accompaniment left channel signal in the extreme right signal interval to 0 to obtain an updated accompaniment left channel signal;

[0122] The two-channel accompaniment signal determining unit is used to obtain the two-channel accompaniment signal based on the updated accompaniment right channel signal and the updated accompaniment left channel signal.

[0123] In some specific embodiments, the audio main and accompaniment singing separation device may specifically include:

[0124] The decorrelation unit is used to adjust the accompaniment right channel signal in the extreme left signal interval and the accompaniment left channel signal in the extreme right signal interval to obtain a dual-channel accompaniment signal, and then perform decorrelation processing on the dual-channel accompaniment signal to obtain a dual-channel accompaniment signal after decorrelation processing.

[0125] In some specific embodiments, the decorrelation unit may specifically include:

[0126] A first decorrelation unit, configured to perform decorrelation processing on the two-channel accompaniment signal by delaying sampling;

[0127] Or, a second decorrelation unit is used to perform decorrelation processing by changing the phase of the two-channel accompaniment signal.

[0128] Furthermore, an embodiment of the present application also discloses an electronic device, as shown in FIG6 . The content in the figure cannot be regarded as any limitation on the scope of use of the present application.

[0129] FIG6 is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the audio main and accompaniment separation method disclosed in any of the aforementioned embodiments.

[0130] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0131] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 221, a computer program 222, and data 223 including song audio, etc. The storage method can be temporary storage or permanent storage.

[0132] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, so as to realize the operation and processing of the massive data 223 in the memory 22 by the processor 21, and it can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the audio main and accompaniment separation method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0133] Furthermore, an embodiment of the present application also discloses a computer program product, including a computer program, which, when executed by a processor, implements the steps of the audio main and accompaniment vocal separation method disclosed in any of the aforementioned embodiments.

[0134] Furthermore, an embodiment of the present application also discloses a computer storage medium, in which computer executable instructions are stored. When the computer executable instructions are loaded and executed by a processor, the steps of the audio main and accompaniment separation method disclosed in any of the aforementioned embodiments are implemented.

[0135] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0136] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0137] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0138] The above is a detailed introduction to an audio main and accompaniment separation method, device, storage medium and program product provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for separating main and accompaniment vocals from audio, characterized in that: include: Acquire song audio, and extract a mono-channel related signal and a mono-channel unrelated signal from the song audio; Acquire an energy envelope difference signal of the song audio, and determine a far left signal interval and a far right signal interval according to the energy envelope difference signal; Based on the mono-channel related signal, a lead vocal left channel signal and a lead vocal right channel signal are obtained; based on the mono-channel unrelated signal, a backing vocal left channel signal and a backing vocal right channel signal are obtained; Using the monophonic unrelated signal, respectively add the sound and image information to the left channel signal of the lead singer in the extreme left signal interval and the right channel signal of the lead singer in the extreme right signal interval to obtain a dual-channel lead singer signal; The accompaniment right channel signal in the extreme left signal interval is adjusted, and the accompaniment left channel signal in the extreme right signal interval is adjusted to obtain a dual-channel accompaniment signal.

2. The method for separating the main and accompaniment vocals of the audio according to claim 1, characterized in that: The step of obtaining the energy envelope difference signal of the song audio comprises: Respectively dividing the left channel signal and the right channel signal of the song audio into frames, and calculating the energy envelope of each frame signal to obtain the energy envelope of the left channel and the energy envelope of the right channel; According to the difference between the energy envelope of the left channel and the energy envelope of the right channel, an energy envelope difference signal corresponding to the song audio is obtained.

3. The method for separating the main and accompaniment vocals of the audio according to claim 2, characterized in that: The determining of the extreme left signal interval and the extreme right signal interval according to the energy envelope difference signal comprises: Obtain a first energy threshold and a second energy threshold; the first energy threshold is greater than the second energy threshold; the first energy threshold is an absolute value, and the first energy threshold corresponds to a first positive energy threshold and a first negative energy threshold; the second energy threshold is an absolute value, and the second energy threshold corresponds to a second positive energy threshold and a second negative energy threshold; Locate a first region whose energy is greater than a first positive energy threshold value from the energy envelope difference signal, search along the energy envelope difference signal to both sides with the first region as the center, locate two time points whose energy is equal to the second positive energy threshold value, and use the time period between the two time points as the extreme left signal interval; A second area whose energy is less than the first energy negative threshold is located from the energy envelope difference signal, and the energy envelope difference signal is searched to both sides with the second area as the center to locate two time points whose energy is equal to the second energy negative threshold, and the time period between the two time points is used as the extreme right signal interval.

4. The method for separating the main and accompaniment vocals of the audio according to claim 2, characterized in that: The step of calculating the energy envelope of each frame signal to obtain the energy envelope of the left channel and the energy envelope of the right channel includes: Calculating the energy envelope of each frame signal, performing smoothing and normalization processing on the energy envelope to obtain the energy envelope of the left channel and the energy envelope of the right channel; The energy envelope is smoothed, comprising: The energy envelope is upsampled to obtain an upsampled signal sequence, the upsampled signal sequence is time-domain convolved to obtain a convolved sequence, and the convolved sequence is downsampled to obtain a downsampled signal sequence.

5. The method for separating the main and accompaniment vocals of the audio according to claim 1, characterized in that: The method of using the monophonic unrelated signal to add sound and image information to the left channel signal of the lead singer in the extreme left signal interval and the right channel signal of the lead singer in the extreme right signal interval to obtain a dual-channel lead singer signal includes: Obtaining an updated lead vocal left channel signal according to a difference between a lead vocal left channel signal in a far left signal interval and an unrelated signal in the far left signal interval; Obtaining an updated lead vocal right channel signal according to a difference between a lead vocal right channel signal in a far right signal interval and an unrelated signal in the far right signal interval; Based on the updated left-channel lead vocal signal and the updated right-channel lead vocal signal, a dual-channel lead vocal signal is obtained.

6. The method for separating the main and accompaniment vocals of the audio according to claim 1, characterized in that: The step of adjusting the right channel signal of the accompaniment within the extreme left signal interval and adjusting the left channel signal of the accompaniment within the extreme right signal interval to obtain a dual-channel accompaniment signal includes: The right channel signal of the accompaniment in the extreme left signal interval is adjusted to 0 to obtain an updated right channel signal of the accompaniment; The left channel signal of the accompaniment vocals in the extreme right signal interval is adjusted to 0 to obtain an updated left channel signal of the accompaniment vocals; Based on the updated accompaniment right channel signal and the updated accompaniment left channel signal, a two-channel accompaniment signal is obtained.

7. The method for separating main and accompaniment vocals according to any one of claims 1 to 6, characterized in that: After adjusting the right channel signal of the accompaniment in the extreme left signal interval and adjusting the left channel signal of the accompaniment in the extreme right signal interval to obtain a dual-channel accompaniment signal, the method further includes: The two-channel accompaniment singing signal is subjected to decorrelation processing to obtain a two-channel accompaniment singing signal after decorrelation processing.

8. The method for separating main and accompaniment vocals of audio according to claim 7, characterized in that: The decorrelation processing of the two-channel accompaniment signal comprises: Decorrelation processing is performed by delaying sampling of the two-channel accompaniment signal; Or, decorrelation processing is performed by changing the phase of the two-channel accompaniment signal.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor is used to execute the computer program to implement the audio main and accompaniment vocal separation method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: Used to store computer programs; wherein when the computer programs are executed by a processor, the audio main and accompaniment vocal separation method as described in any one of claims 1 to 8 is implemented.

11. A computer program product, characterized in that It includes a computer program, which, when executed by a processor, implements the audio main and accompaniment vocal separation method described in any one of claims 1 to 8.

Images