Song main melody extraction method, song processing method, computer equipment and product

Abstract

The present application relates to the field of audio technology, and provides a method for extracting the main melody of a song, a song processing method, a computer device, and a product. The present application can effectively remove the accompaniment containing the accompaniment and non-accompaniment parts of a song, eliminate the interference signals existing in the main melody extraction process, and increase the accuracy of the main melody estimated based on the dry sound signal. The method includes: obtaining a song signal of a target song and an accompaniment signal of the original accompaniment corresponding to the target song; the original accompaniment contains the accompaniment part and the non-accompaniment part; the accompaniment signal of the original accompaniment and the song signal of the target song are input into an adaptive filter, and the adaptive filter identifies the preliminary dry sound signal in the target song; the song signal of the target song is input into a vocal extraction model, and the vocal signal in the target song output by the vocal extraction model is obtained; according to the preliminary dry sound signal and the vocal signal, the target dry sound signal of the target song is determined; according to the target dry sound signal, the main melody of the target song is obtained.

Description

Technical Field

[0001] The present application relates to the field of audio technology, and in particular to a method for extracting the main melody of a song, a song processing method, a computer device, and a product. Background Art

[0002] With the development of internet technology, users are increasingly interacting with applications. In practice, users can sing songs and receive feedback from applications. Applications can also search for relevant songs based on user-provided song clips. When performing evaluations or searches, applications often first identify the song's main melody before performing any subsequent processing.

[0003] In related technologies, a source separation model can be used to separate and extract the vocal signal from a song, and then the main melody can be identified directly based on the current vocal signal. However, this approach often mistakenly identifies interference signals in the song as the main melody signal, affecting the correct identification of the main melody and resulting in low main melody recognition accuracy. Summary of the Invention

[0004] Based on this, it is necessary to provide a song main melody extraction method, song processing method, computer equipment and product that can improve the accuracy of main melody recognition in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for extracting the main melody of a song. The method comprises:

[0006] Obtain a song signal of a target song and an accompaniment signal of an original accompaniment corresponding to the target song; wherein the original accompaniment includes an accompaniment part and a non-accompaniment part;

[0007] inputting the accompaniment signal of the original accompaniment and the song signal of the target song into an adaptive filter, and allowing the adaptive filter to identify a preliminary dry sound signal in the target song;

[0008] Inputting the song signal of the target song into a vocal extraction model to obtain the vocal signal of the target song output by the vocal extraction model;

[0009] Determining a target dry sound signal of the target song according to the preliminary dry sound signal and the vocal signal;

[0010] The main melody of the target song is obtained according to the target dry sound signal.

[0011] In a second aspect, the present application also provides a song processing method. The method comprises:

[0012] Get the song hummed by the user;

[0013] Obtaining the main melody of the song hummed by the user according to any one of the above methods for extracting the main melody of the song;

[0014] According to the main melody, user-required songs that meet preset requirements are retrieved and the user-required songs are displayed; or, according to the main melody, humming evaluation information is generated and the humming evaluation information is displayed.

[0015] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are performed:

[0016] Obtain a song signal of a target song and an accompaniment signal of an original accompaniment corresponding to the target song; wherein the original accompaniment includes an accompaniment part and a non-accompaniment part;

[0017] inputting the accompaniment signal of the original accompaniment and the song signal of the target song into an adaptive filter, and allowing the adaptive filter to identify a preliminary dry sound signal in the target song;

[0018] Inputting the song signal of the target song into a vocal extraction model to obtain the vocal signal of the target song output by the vocal extraction model;

[0019] Determining a target dry sound signal of the target song according to the preliminary dry sound signal and the vocal signal;

[0020] The main melody of the target song is obtained according to the target dry sound signal.

[0021] In a fourth aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0022] Get the song hummed by the user;

[0023] Obtaining the main melody of the song hummed by the user according to any one of the above methods for extracting the main melody of the song;

[0024] According to the main melody, user-required songs that meet preset requirements are retrieved and the user-required songs are displayed; or, according to the main melody, humming evaluation information is generated and the humming evaluation information is displayed.

[0025] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

[0026] Obtain a song signal of a target song and an accompaniment signal of an original accompaniment corresponding to the target song; wherein the original accompaniment includes an accompaniment part and a non-accompaniment part;

[0027] inputting the accompaniment signal of the original accompaniment and the song signal of the target song into an adaptive filter, and allowing the adaptive filter to identify a preliminary dry sound signal in the target song;

[0028] Inputting the song signal of the target song into a vocal extraction model to obtain the vocal signal of the target song output by the vocal extraction model;

[0029] Determining a target dry sound signal of the target song according to the preliminary dry sound signal and the vocal signal;

[0030] The main melody of the target song is obtained according to the target dry sound signal.

[0031] In a sixth aspect, the present application further provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

[0032] Get the song hummed by the user;

[0033] Obtaining the main melody of the song hummed by the user according to any one of the above methods for extracting the main melody of the song;

[0034] According to the main melody, user-required songs that meet preset requirements are retrieved and the user-required songs are displayed; or, according to the main melody, humming evaluation information is generated and the humming evaluation information is displayed.

[0035] The above-mentioned song main melody extraction method, song processing method, computer device and computer program product can obtain the song signal of the target song and the accompaniment signal of the original accompaniment corresponding to the target song, wherein the original accompaniment includes the accompaniment part and the non-accompaniment part; then the accompaniment signal of the original accompaniment and the song signal of the target song can be input into the adaptive filter, and the adaptive filter identifies the preliminary dry sound signal in the target song, and the song signal of the target song is input into the vocal extraction model to obtain the vocal signal in the target song output by the vocal extraction model, and the target dry sound signal of the target song is determined according to the preliminary dry sound signal and the vocal signal, and the main melody of the target song is obtained according to the target dry sound signal. By combining the recognition results of the adaptive filter and the vocal extraction model, the present application can effectively remove the accompaniment containing the accompaniment and the non-accompaniment part in the song, eliminate the interference signals existing in the main melody extraction process, improve the reliability of the dry sound signal, and increase the accuracy of the main melody estimated based on the dry sound signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 1 is a flow chart of a method for extracting the main melody of a song in one embodiment;

[0037] Figure 2 is a schematic diagram of a song signal in one embodiment;

[0038] Figure 3 is a schematic diagram of a preliminary dry sound signal in one embodiment;

[0039] Figure 4 is a schematic diagram of a human voice signal recognized by a human voice extraction model in one embodiment;

[0040] Figure 5 A schematic diagram of a flow chart of a step of obtaining an accompaniment in one embodiment;

[0041] Figure 6 A schematic flow chart of a step of obtaining a dry sound signal in one embodiment;

[0042] Figure 7 is a schematic diagram of a dry sound signal in one embodiment;

[0043] Figure 8 A schematic diagram of an application environment of a song processing method according to an embodiment;

[0044] Figure 9 1 is a flow chart of a song processing method according to an embodiment;

[0045] Figure 10 is a diagram of the internal structure of a computer device in one embodiment;

[0046] Figure 11 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0048] The method for extracting the main melody of a song provided in the embodiment of the present application can be executed by a computer device such as a terminal or a server. For example, it can be applied to a terminal or a server alone, or it can be applied to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. Among them, the terminal can be, but is not limited to, various personal computers, laptops, and tablet computers; the server can be implemented as an independent server or a server cluster composed of multiple servers. In terms of application scenarios, the method for extracting the main melody of a song provided in the present application can be specifically applied in the scenario of identifying the main melody of a song. By analyzing the target song, the accompaniment signal of the song accompaniment in the target song (including the accompaniment and non-accompaniment parts) is removed as an interference signal to obtain a target dry sound signal, and then the main melody of the song can be determined based on the identified target dry sound signal.

[0049] The audio processing method provided in this application is described below in conjunction with various embodiments and corresponding drawings.

[0050] In one embodiment, Figure 1 As shown, a method for extracting the main melody of a song is provided, which may include the following steps:

[0051] Step S110, obtaining a song signal of a target song and an accompaniment signal of an original accompaniment corresponding to the target song; the original accompaniment includes a vocal part and a non-vocalic part.

[0052] As an example, the target song can be a song with a main melody to be identified, which can be monotonous or polyphonic. In polyphonic music, the dominant vocals often have associated polyphonic accompaniments, which can highly couple and superimpose sounds from different sources according to the harmonic structure. For example, the target song can be audio recorded while playing the song accompaniment and singing. In addition to the singer's voice, the target song can also include the sound of the song accompaniment or ambient noise.

[0053] In practical applications, the song signal of the target song and the accompaniment signal of the original accompaniment corresponding to the target song can be obtained. For example, after obtaining the target song, the original accompaniment corresponding to the target song can be determined from the pre-stored original accompaniment library, and the accompaniment signal of the original accompaniment can be obtained.

[0054] The original accompaniment library may pre-store accompaniments corresponding to multiple songs, i.e., original accompaniments. In addition to the non-accompaniment part composed of non-human vocal content such as pure music or instrument sounds, the original accompaniment may also include an accompaniment part, i.e., accompaniment content performed by human voice. When the user plays the original accompaniment and sings, the user's singing voice and the accompaniment part in the original accompaniment can form a harmony, thereby optimizing the singing effect. In one example, if the target song obtained is a song to be evaluated, the song identifier corresponding to the target song may also be obtained when obtaining the target song, and the original accompaniment corresponding to the target song may be obtained from the pre-stored original accompaniment library based on the song identifier.

[0055] In step S120 , the accompaniment signal of the original accompaniment and the song signal of the target song are input into an adaptive filter, and the adaptive filter recognizes a preliminary dry sound signal in the target song.

[0056] Among them, the adaptive filter is a digital filter that can automatically adjust its performance according to the input signal to perform digital signal processing. It can be applied to signal processing systems that do not have a priori statistical characteristics of the signal and are difficult to process using fixed parameter filters. In practical applications, adaptive filtering can use the filter parameter results obtained at the previous moment to automatically adjust the filter parameters at the current moment to adapt to the statistical characteristics of the signal that change over time, thereby achieving optimal filtering.

[0057] The preliminary dry sound signal can be a dry sound signal estimated by an adaptive filter, wherein the dry sound can also be called naked sound, which is pure human voice without music. In this embodiment, it can refer to pure human voice without song accompaniment, that is, the singer's dry sound signal.

[0058] In specific implementation, due to various reasons such as the singer's voice characteristics and singing style, the dry sound signal in the target song often lacks prior statistical information, and it is difficult to directly extract the dry sound signal from the song signal of the target song. The accompaniment signal of the original accompaniment and the song signal of the target song can be input into the adaptive filter. The adaptive filter can filter out the accompaniment signal existing in the song signal in a time-varying manner according to the accompaniment signal of the original accompaniment, and identify the preliminary dry sound signal in the target song.

[0059] In some main melody extraction methods, the spectral characteristics and pitch continuity of the music signal are used to estimate the main melody. However, in polyphonic music, the leading voice usually has at least one polyphonic accompaniment, which results in the spectrum of polyphonic music including multiple sound signals from different sound sources that are highly coupled and superimposed according to the harmonic structure. For example, Figure 2 As shown, it is the song signal corresponding to the target song. Figure 2The song signal displayed is the result of blending the dry sound signal and the accompaniment signal. The dry sound signal and the accompaniment signal are superimposed on each other, and the difficulty of separating the spectrum into the corresponding notes increases significantly. To address this situation, the saliency-based method uses the spectral information of the music signal to define a pitch saliency function, and uses the maximum position in the pitch saliency function as the candidate melody pitch. However, the candidate melody pitches are often distributed on harmonics, and during the recognition process, it is easy to mistakenly identify the fundamental frequency as a multiple frequency, resulting in an octave error in the final main melody. In addition, when there is a backing vocal part in the song, if the dry sound sung by the user does not appear when the backing vocals appear, but since both the backing vocals and the dry sound are human voices, it is easy to mistakenly identify the backing vocal part as the dry main melody sung by the user, reducing the accuracy of main melody recognition.

[0060] In this embodiment, the adaptive filter can adaptively adjust the filter parameters according to the accompaniment signal of the input original accompaniment and filter out the accompaniment signal in the song signal, adapting to the statistical characteristics of the accompaniment signal and the song signal changing over time, thereby effectively identifying the preliminary dry sound signal.

[0061] Step S130: inputting the song signal of the target song into the vocal extraction model to obtain the vocal signal of the target song output by the vocal extraction model.

[0062] Specifically, after obtaining the song signal of the target song, the song signal of the target song can be input into the vocal extraction model, and the vocals in the target song can be recognized by the vocal extraction model to obtain the vocal signal of the target song output by the model.

[0063] In a specific implementation, the accompaniment in a song often includes non-accompaniment content, such as accompaniment composed of pure music or instrument sounds. Therefore, when identifying the main melody of a song, a simple classification can be performed first to identify the human voice in the song and obtain the vocal signal of the human voice in the song. In other words, when the target song includes both the singer's voice and the accompaniment part of the song accompaniment, the dry voice signal corresponding to the singer and the vocal signal of the accompaniment part can be obtained through the vocal extraction model and output as the recognition result. For example, when classifying signals, classification can be performed through source separation methods or machine learning methods, such as a dry voice accompaniment separation model based on a deep neural network. Those skilled in the art can select a vocal extraction model based on actual conditions.

[0064] Step S140 , determining a target dry sound signal of the target song according to the preliminary dry sound signal and the vocal signal.

[0065] As an example, the target dry voice signal may be the dry voice signal of the singer in the target song that is finally identified.

[0066] After obtaining the preliminary dry sound signal and the vocal signal, the preliminary dry sound signal and the vocal signal may be combined to determine a target dry sound signal for the target song.

[0067] Specifically, the adaptive filter does not need to rely on model training, but can timely estimate the accompaniment in the song based on the original accompaniment. The preliminary dry sound signal estimated by the adaptive filter can be Figure 3 As shown, the accompaniment signal for the accompaniment part in the solid box at the end of the song signal can be effectively removed, but due to the influence of the convergence speed of the adaptive filter (for example, the convergence speed is slow), some accompaniment signals for the non-accompaniment part still remain in the dotted box in the middle of the song.

[0068] When using the vocal extraction model to extract the vocal signal, although the vocal extraction model can better eliminate the parts other than the vocal signal in the song signal by classifying the signal, such as the accompaniment signal of the non-accompaniment part, the accompaniment part of the accompaniment is also composed of the vocal signal. The vocal extraction model is likely to mistakenly identify the accompaniment signal of the accompaniment part as the dry voice signal of the singer and retain it, such as Figure 4 As shown, this is the vocal signal obtained based on the vocal extraction model. It can be seen that there is still a residual signal in the dotted box at the end of the song signal. The residual signal is the vocal signal of the accompaniment part, while the accompaniment in the solid box in the middle of the song signal (the accompaniment signal of the non-accompaniment part) is effectively eliminated.

[0069] In this embodiment, the accompaniment signal that is not filtered out by the adaptive filter (for example, the accompaniment signal of the chorus part in the song signal) can be further identified with the help of the recognition results of the human voice extraction model. Alternatively, the accompaniment including the chorus part that the human voice extraction model fails to identify can be further identified through the recognition results of the adaptive filter, thereby obtaining the target dry sound signal of the target song.

[0070] Step S150: obtaining the main melody of the target song according to the target dry sound signal.

[0071] After the target dry sound signal is obtained, the main melody of the song can be determined based on the target dry sound signal.

[0072] In this embodiment, the song signal of the target song and the accompaniment signal of the original accompaniment corresponding to the target song can be obtained, wherein the original accompaniment includes the accompaniment part and the non-accompaniment part; and then the accompaniment signal of the original accompaniment and the song signal of the target song can be input into the adaptive filter, and the adaptive filter identifies the preliminary dry sound signal in the target song, and the song signal of the target song is input into the vocal extraction model to obtain the vocal signal in the target song output by the vocal extraction model, and the target dry sound signal of the target song is determined according to the preliminary dry sound signal and the vocal signal, and the main melody of the target song is obtained according to the target dry sound signal. By combining the recognition results of the adaptive filter and the vocal extraction model, the present application can effectively remove the accompaniment containing the accompaniment and the non-accompaniment part of the song, eliminate the interference signals existing in the main melody extraction process, improve the reliability of the dry sound signal, and increase the accuracy of the main melody estimated based on the dry sound signal.

[0073] Moreover, the existing model can be used directly when extracting the human voice model, and the adaptive filter does not require model training. Therefore, the above-mentioned song main melody extraction method does not rely on the deep model based on labeled data, which significantly reduces the computational complexity.

[0074] In an exemplary embodiment, step S120 inputs the accompaniment signal of the original accompaniment and the song signal of the target song into an adaptive filter, and the adaptive filter identifies a preliminary dry sound signal in the target song, including:

[0075] The accompaniment signal of the original accompaniment and the song signal of the target song are input into the adaptive filter, and the adaptive filter recognizes the accompaniment signal of the song accompaniment in the target song; based on the accompaniment signal of the target song and the song signal of the target song, a preliminary dry sound signal in the target song is obtained.

[0076] Among them, the song accompaniment includes the accompaniment part and the non-accompaniment part.

[0077] Specifically, the target song consists of the accompaniment (including the accompaniment part and the non-accompaniment part) and the dry voice signal of the singer. By removing the accompaniment signal from the target song signal, the dry voice signal of the target song can be obtained. In order to facilitate the distinction from the dry voice signal obtained by subsequent further processing, the dry voice signal obtained here can be called the preliminary dry voice signal.

[0078] In a specific implementation, the target song may be composed of the accompaniment played by the user and the dry sound recorded when the user sings. Although the original accompaniment corresponding to the target song can be obtained at present, the composition of the accompaniment and the dry sound in the target song is not yet clear. In other words, the accompaniment signal corresponding to the accompaniment part in the song signal of the target song is not completely consistent with the accompaniment signal of the original accompaniment. It is difficult to determine the dry sound signal of the dry sound in the target song by directly removing the accompaniment signal of the original accompaniment from the song signal of the target song.

[0079] But the accompaniment signal of the song accompaniment in the target song can be predicted and obtained by the accompaniment signal of the original accompaniment, and the song signal of the target song can be understood as being composed of two parts, one part is the dry sound signal corresponding to the dry sound in the target song, and the other part is the accompaniment signal corresponding to the song accompaniment, when the dry sound signal corresponding to the dry sound in the target song is fixed, accordingly, the accompaniment signal of the original accompaniment and the song signal of the target song can constitute a causal system. Based on this, the accompaniment signal of the original accompaniment and the song signal of the target song can be input into an adaptive filter, and the adaptive filter dynamically predicts the mapping relationship between the accompaniment signal of the original accompaniment and the accompaniment signal of the song accompaniment based on the accompaniment signal of the input and the song signal of the target song as the signal changes, thereby the accompaniment signal included in the song signal of the target song can be identified by the adaptive filter according to the accompaniment signal of the original accompaniment, and the accompaniment signal includes the signal of the accompaniment part and the non-accompaniment part. After predicting the accompaniment signal of the song accompaniment in the target song, then according to this accompaniment signal and the song signal, the preliminary dry sound signal in the song can be determined.

[0080] The present application can use a system identification method to estimate the mapping relationship between the song accompaniment containing the accompaniment part and the non-accompaniment part of the target song and the original accompaniment by using an adaptive filter to obtain a system model. At the same time, the adaptive filter can automatically and continuously adapt to the accompaniment signal of the currently input original accompaniment and the song signal of the target song to adjust the parameters in time, which greatly improves the accuracy of the estimation of the song accompaniment in the target song. In addition to removing the accompaniment of the non-accompaniment part of the target song, it can also effectively eliminate the accompaniment of the accompaniment part of the target song, which can improve the accuracy and reliability of the preliminary dry sound signal finally identified.

[0081] In an exemplary embodiment, Figure 5 As shown, in step S120, the accompaniment signal of the original accompaniment and the song signal of the target song are input into the adaptive filter, and the accompaniment signal of the song accompaniment in the target song is identified by the adaptive filter, which may include the following steps:

[0082] In step S121, the accompaniment signal of the original accompaniment at each frequency point and the song signal of the target song are input into the adaptive filter. The adaptive filter composed of the filter parameters at the current frequency point determines the predicted accompaniment signal at the current frequency point based on the accompaniment signal at the current frequency point and the song signal at the current frequency point.

[0083] Among them, the current frequency point is a frequency point among all frequency points, and when the current frequency point is not the first frequency point, the filter parameters at the current frequency point are calculated based on the filter parameters at the previous frequency point of the current frequency point, the accompaniment signal at the previous frequency point, and the song signal at the previous frequency point.

[0084] In practical applications, the accompaniment signal of the original accompaniment and the song signal of the target song can be signals in the frequency domain, that is, the accompaniment signal and the song signal can be the signal amplitude corresponding to each frequency point. Specifically, for the obtained original accompaniment and target song, the corresponding signal frames can be obtained by frame division and windowing respectively. For example, the song and the original accompaniment can be divided into signal frames of fixed time length, and there are overlapping areas between adjacent signal frames. Then, a Hanning window is added to each signal frame for smoothing to prevent spectrum leakage. After obtaining multiple audio frames of the original accompaniment and multiple audio frames of the song, the time domain signal in the song signal frame y(t) and the time domain signal in the accompaniment signal frame x(t) can be Fourier transformed to obtain the song signal Y in the frequency domain. n (k) and accompaniment signal X n (k), where k and n are the frequency index and frame index of the signal frame in the frequency domain, respectively.

[0085] After obtaining the accompaniment signal and song signal at each frequency point, they can be input into the adaptive filter, and one of the frequency points is used as the current frequency point to be analyzed. When the current frequency point is not the first frequency point among multiple frequency points, the filter parameters at the current frequency point can be estimated based on the filter parameters at the previous frequency point of the current frequency point, the accompaniment signal at the previous frequency point, and the song signal at the previous frequency point. Then, the adaptive filter composed of the filter parameters at the current frequency point can determine the predicted accompaniment signal at the current frequency point based on the corresponding filter parameters at the current frequency point, the accompaniment signal at the current frequency point, and the song signal at the current frequency point. It can be understood that as the accompaniment signal of the original accompaniment at each frequency point and the song signal of the target song change, the adaptive filter can adjust the filter parameters accordingly.

[0086] For example, the adaptive filter can predict the accompaniment signal in the song signal according to the following formula to obtain a predicted accompaniment signal, and determine the preliminary dry sound signal based on the predicted accompaniment signal:

[0087]

[0088] S n (k) = Y n (k)-G n (k)

[0089] in, is the filter parameter, which can also be called the system model, G n (k) is the predicted accompaniment signal, S n (k) is the preliminary dry sound signal.

[0090] When obtaining the predicted accompaniment signal, since the original accompaniment signal and the song signal form a causal system, that is, the accompaniment signal of the original accompaniment has a mapping relationship with the accompaniment signal in the target song, and the dry sound signal in the song signal has nothing to do with the accompaniment signal, it is possible to predict the accompaniment signal based on the original accompaniment signal X. n (k) and song signal Y n (k) Construct system identification and estimate the original accompaniment signal X by adaptive filtering method n (k) and song signal Y n (k) System model H between n (k), and then simultaneously estimate the preliminary dry sound signal S n (k); Among them, system identification is a mathematical model that describes the system behavior based on the system's input and output time functions, and is one of the applications of adaptive filters.

[0091] In one example, when determining the filter parameters, the normalized minimum mean square error algorithm can be used. The idea is to adjust the filter parameters of the adaptive filter based on the error between the output signal currently output by the adaptive filter and the expected signal, and process the next signal according to the adjusted filter parameters. The above process is repeated. During the processing, the adaptive filter parameters gradually converge.

[0092] After the original accompaniment signal and song signal at the corresponding frequency point are input into the adaptive filter, the adaptive filter can obtain a predicted accompaniment signal based on the filter parameters and the original accompaniment signal, and compare the predicted accompaniment signal with the expected signal to determine the corresponding error. When the filter parameters are adjusted multiple times according to the error and the adaptive filter is converged, the original accompaniment signal can be correctly mapped to the accompaniment signal in the song signal. Specifically, the song signal can be understood as consisting of two parts, including the true accompaniment signal M1 and the true dry sound signal M2, that is, the song signal is the dry sound signal M2 in addition to the accompaniment signal M1, and the dry sound signal M2 is not triggered by the accompaniment signal M1, that is, the accompaniment signal M1 and the dry sound signal M2 are independent and unrelated to each other. On this basis, when determining the error based on the expected signal (i.e., the expected accompaniment signal) and the predicted accompaniment signal, the only signal in the song signal that can cancel the expected signal and reduce the error is the signal related to the accompaniment part, i.e., the predicted accompaniment signal determined based on the accompaniment signal M1. Since the dry sound signal M2 is not related to the accompaniment signal M1, the signal M2' output by the adaptive filter is also unrelated to the accompaniment signal. Therefore, if the dry sound signal M2 in the song signal is mistakenly identified as the accompaniment signal M1, the unrelated output signal M2' will also be reflected in the error. Therefore, in order to reduce the error of the adaptive filter, the filter parameters will be adjusted to more accurately identify the accompaniment signal M1 in the song signal and improve the correlation between the predicted accompaniment signal and the expected signal, so that the accompaniment signal in the song signal can be identified when the adaptive filter converges. In actual applications, the true accompaniment signal in the song signal cannot be obtained in advance as the expected signal. The expected signal can use the accompaniment signal of the original accompaniment, such as the accompaniment signal at the corresponding moment or the accompaniment signal with a delay.

[0093] Step S122: obtaining the accompaniment signal of the song accompaniment in the target song according to the predicted accompaniment signal of each current frequency point.

[0094] After obtaining the predicted accompaniment signal of each current frequency point, the predicted accompaniment signals of multiple current frequency points can be combined to obtain information about the predicted accompaniment signal at different frequency points in the frequency domain, such as the signal amplitude of the predicted accompaniment signal at each frequency point in the frequency domain. In this way, the accompaniment signal of the song accompaniment in the target song can be determined based on the predicted accompaniment signals of multiple current frequency points, and the song accompaniment in the target song can also be obtained based on the accompaniment signal of the song accompaniment.

[0095] In this embodiment, the accompaniment signal of the original accompaniment at each frequency point and the song signal of the target song can be input into the adaptive filter. The adaptive filter composed of the filter parameters at the current frequency point determines the predicted accompaniment signal of the current frequency point based on the accompaniment signal at the current frequency point and the song signal at the current frequency point. The filter parameters at the current frequency point can be calculated based on the filter parameters at the previous frequency point of the current frequency point, the accompaniment signal at the previous frequency point and the song signal at the previous frequency point. Then, the accompaniment signal of the song accompaniment in the target song can be obtained based on the predicted accompaniment signals of each current frequency point. Based on the input accompaniment signal, the accompaniment signal associated with the accompaniment and non-accompaniment parts in the song signal can be adaptively predicted accurately, providing a basis for subsequent acquisition of a reliable preliminary dry sound signal.

[0096] In an exemplary embodiment, the filter parameters of the adaptive filter may be obtained by:

[0097] If the current frequency point is the first frequency point, the initialized filter parameters are obtained, and the filter parameters of the next frequency point are determined according to the initialized filter parameters, the accompaniment signal of the original accompaniment, and the song signal of the target song.

[0098] In practical applications, if the current frequency point is the first frequency point among multiple frequency points, the filter parameters can be initialized. For example, a pre-set value can be obtained as the initialized filter parameter, and then the filter parameters corresponding to the next frequency point can be determined based on the initialized filter parameters, the original accompaniment signal and the song signal.

[0099] Exemplarily, the filter parameters for the next frequency point can be determined by the following formula:

[0100]

[0101] P n (k)=ρP n (k-1)+(1-ρ)|X n (k)| 2

[0102] Among them, H n (k+1) is the filter parameter of the next frequency point, H n (k) is the filter parameter of the current frequency point, μ is the update step size, which can be a fixed step size or a variable step size, P n (k) is the intermediate factor used to update the parameters during the filter parameter adjustment process, and ρ is a preset parameter that can be used to adjust the convergence speed of the adaptive filter. When initializing the filter parameters, P n (k-1) and H n(k) Initialization is performed to provide a basis for calculating the filter parameters of the next frequency point.

[0103] In an exemplary embodiment, step S130 of obtaining a preliminary dry sound signal of the target song based on the accompaniment signal of the target song and the song signal of the target song can be specifically implemented by the following steps:

[0104] Acquire the accompaniment signal of the target song and the song signal of the target song at each frequency point; remove the accompaniment signal from the song signal at each frequency point to obtain a preliminary dry sound signal of the target song.

[0105] In practical applications, after obtaining the predicted accompaniment signal at each frequency point, and the song signal Y n After (k), for each frequency point, the accompaniment signal in the song signal can be removed to obtain the preliminary dry sound signal of the song. For example, the preliminary dry sound signal S at each frequency point can be determined by the following formula n (k):

[0106]

[0107] In this embodiment, for each frequency point, a preliminary dry sound signal is obtained by removing the accompaniment signal from the song signal, where the removed accompaniment signal includes the accompaniment part and the non-accompaniment part.

[0108] In an exemplary embodiment, Figure 6 As shown, step S150 can be implemented by following the steps S151-S152 to determine the target dry sound signal of the target song based on the preliminary dry sound signal and the vocal signal:

[0109] Step S151 , obtaining the probability of dry sound existence of the target song at each frequency point based on the preliminary dry sound signal and the human voice signal at each frequency point.

[0110] In the specific implementation, the preliminary dry sound signal and human voice signal in the frequency domain can be obtained. For each frequency point, the probability of the dry sound of the target song at this frequency point can be obtained based on the preliminary dry sound signal and human voice signal at this frequency point. In other words, based on the preliminary dry sound signal and human voice signal, it is possible to estimate whether there is a dry sound signal and the size of the dry sound signal at each frequency point, and obtain the probability of the dry sound at this frequency point.

[0111] Step S152 : Processing the song signal of the target song at each frequency point according to the probability of the dry sound of the target song at each frequency point to obtain the target dry sound signal of the target song.

[0112] After obtaining the dry sound presence probability, for each frequency point in the song signal, the song signal can be processed according to the dry sound presence probability of the song signal of the target song at that frequency point to obtain the dry sound signal at that frequency point. By combining the dry sound signals at multiple frequency points, the target dry sound signal of the target song can be obtained. For example, the target dry sound signal of the target song at each frequency point can be obtained by the following formula:

[0113] G n (k) = β n (k)×Y n (k)

[0114] Among them, β n (k) is the probability of dry sound at the frequency point, G n (k) is the dry sound signal of the target song at this frequency point. For example, the final dry sound signal can be as follows Figure 7 shown.

[0115] In this embodiment, the probability of dry sound existence of the target song at each frequency point can be obtained based on the preliminary dry sound signal and the vocal signal at each frequency point, and the song signal of the target song at each frequency point can be processed based on the probability of dry sound existence of the target song at each frequency point to obtain the target dry sound signal of the target song, so that the dry sound signal finally estimated can effectively eliminate both the non-singing accompaniment and the accompaniment of the accompaniment part.

[0116] In an exemplary embodiment, obtaining the probability of dry sound existence of the target song at each frequency point based on the preliminary dry sound signal and the human voice signal at each frequency point includes:

[0117] Based on the ratio of the preliminary dry sound signal and the song signal at each frequency point, the probability of the first dry sound at each frequency point is determined; based on the ratio of the human voice signal and the song signal at each frequency point, the probability of the second dry sound at each frequency point is determined; based on the product of the probability of the first dry sound at each frequency point and the probability of the second dry sound at each frequency point, the probability of the dry sound at each frequency point of the target song is obtained.

[0118] In a specific implementation, the preliminary dry sound signal, human voice signal and song signal at each frequency point can be obtained. For each frequency point, the probability β of the first dry sound at that frequency point can be determined based on the ratio of the preliminary dry sound signal to the song signal at that frequency point. n1 (k), for example, can be as follows:

[0119]

[0120] Moreover, based on the human voice signal Z at this frequency point, n The ratio of (k) to the song signal determines the probability of the second dry sound at this frequency point β n2(k), for example, can be as follows:

[0121]

[0122] Then, based on the product of the probability of the first dry sound at each frequency point and the probability of the second dry sound at each frequency point, the probability of the dry sound at each frequency point corresponding to the song can be obtained. n (k):

[0123]

[0124] In this embodiment, based on the ratio of the preliminary dry sound signal and the song signal at each frequency point, the probability of the first dry sound at each frequency point is determined, based on the ratio of the human voice signal and the song signal at each frequency point, the probability of the second dry sound at each frequency point is determined, and based on the product of the first dry sound probability and the second dry sound probability at each frequency point, the probability of the dry sound at each frequency point of the target song is obtained. The probability of the dry sound can be estimated by combining the human voice extraction model and the adaptive filter at the same time, so that the accompaniment signal in the song signal, especially the accompaniment part, can be effectively removed, thereby improving the accuracy of the main melody extraction.

[0125] In an exemplary embodiment, obtaining the main melody of the target song according to the target dry sound signal in step S160 may include the following steps:

[0126] A fundamental frequency sequence of a target dry sound signal is obtained, and the main melody of a target song is obtained based on the fundamental frequency sequence of the target dry sound signal.

[0127] In practical applications, the sound composed of the target dry sound signal is monotonous music. Accordingly, the main melody of the target song can be identified based on the target dry sound signal. In this embodiment, the fundamental frequency sequence of the target dry sound signal can be obtained, and the main melody of the target song can be determined by the fundamental frequency sequence of the target dry sound signal.

[0128] For example, the main melody of a song can be identified by the autocorrelation function, which is a measure of the similarity between a signal and its delay. After obtaining the target dry sound signal, the target dry sound signal can be processed in the time domain to obtain the target dry sound signal in the time domain, and the period of the dry sound signal in the time domain can be determined using the autocorrelation function, wherein the autocorrelation function can obtain a maximum value at the repetition period of the target dry sound signal. After determining the period of the target dry sound signal, the fundamental frequency of the target dry sound signal can be obtained based on the inverse of the period, and based on multiple continuously smoothed fundamental frequencies, the fundamental frequency sequence of the target dry sound signal can be obtained, and the fundamental frequency sequence constitutes the main melody of the song.

[0129] In this embodiment, the fundamental frequency sequence of the target dry sound signal can be obtained, and an accurate and reliable main melody of the song can be obtained based on the fundamental frequency sequence.

[0130] In one embodiment, a song processing method is provided, which can be applied to Figure 8 The application environment shown includes a terminal and a server.

[0131] The terminal communicates with the server over the network. The data storage system can store data that the server needs to process, such as the original accompaniment library. The data storage system can be integrated with the server or located in the cloud or other network servers. Terminals include, but are not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart car devices. The server can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0132] In one embodiment, Figure 9 As shown, a song processing method is provided, which is applied to Figure 8 The server in the example is used for illustration. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server.

[0133] In this embodiment, the following steps may be included:

[0134] Step S210: obtaining the song hummed by the user.

[0135] As an example, the song hummed by the user may be a song containing the user's singing voice.

[0136] In a specific implementation, the user can sing a song when playing a song, and record the song through the terminal to obtain the user humming the song. After obtaining the user humming the song, the terminal can send the user humming the song to the server so that the server obtains the user humming the song.

[0137] Step S220: Obtain the main melody of the song hummed by the user according to the song main melody extraction method.

[0138] After obtaining the song hummed by the user, the server may analyze the song hummed by the user using any of the above methods for extracting the main melody of the song to obtain the main melody of the song hummed by the user.

[0139] Step S230: Search for user songs that meet preset requirements based on the main melody and display the user songs.

[0140] After obtaining the main melody of the song hummed by the user, the server can search the songs in the preset music library according to the main melody, and determine the songs that meet the preset requirements as the user's required songs, wherein the preset requirements can be determined according to actual conditions.

[0141] For example, if a user needs to retrieve a song corresponding to a main melody, the user can execute a preset search trigger operation, such as clicking a button for cover song recognition or humming recognition. In response to the search trigger operation, after obtaining the song hummed by the user, the terminal can send a song recognition request carrying the song hummed by the user to the server, instructing the server to obtain the corresponding user-required song based on the current user-hummed song. The server can then obtain the candidate song main melodies of the candidate songs in the preset music library, and determine the candidate song whose main melody matches the main melody of the user-hummed song as the user-required song.

[0142] For another example, if it is necessary to recommend songs based on the main melody of the song hummed by the user, the melody features of the main melody of the song hummed by the user can be extracted, and candidate songs with the melody features can be retrieved from the preset music library as the songs required by the user.

[0143] After the user's required songs are retrieved, the user's required songs can be returned to the terminal for display.

[0144] By using the above-mentioned song main melody extraction method to extract the main melody of the song hummed by the user, and retrieving the user's required songs that meet the preset requirements based on the main melody, the accuracy and recall rate of the music retrieval service can be improved.

[0145] Step S240: Generate humming evaluation information based on the main melody and display the humming evaluation information.

[0146] As an example, the humming evaluation information may be evaluation information for a song hummed by a user. Exemplarily, the humming evaluation information may be an evaluation level or an evaluation score.

[0147] After obtaining the main melody of the song hummed by the user, the server can compare the main melody with a preset reference melody, generate humming evaluation information of the song hummed by the user according to the comparison result, and return the humming evaluation information to the terminal for display.

[0148] For example, when recording a user humming a song, if a song evaluation is desired, the user can trigger an evaluation by, for example, clicking a button for a singing evaluation. In response to the detected evaluation trigger, the terminal, after acquiring the user's humming song, can send a song evaluation request containing the user's humming song to the server, instructing the server to generate humming evaluation information based on the current user's humming song. By using the aforementioned song main melody extraction method to extract the main melody of the user's humming song and performing a song evaluation based on this main melody, the reliability and accuracy of the evaluation results can be effectively improved.

[0149] In this embodiment, after obtaining the song hummed by the user, the main melody of the song hummed by the user can be obtained according to the above-mentioned song main melody extraction method, and then the user's required songs that meet the preset requirements can be retrieved according to the main melody, and the user's required songs can be displayed; or, humming evaluation information can be generated and displayed according to the main melody, which can improve the accuracy and recall rate of the music retrieval service, and can also improve the reliability and accuracy of the singing evaluation results, thereby enhancing the user experience.

[0150] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0151] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 10 As shown. The computer device includes a processor, a memory and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store multiple songs and original accompaniments. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a song main melody extraction method or a song processing method is implemented.

[0152] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 11As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for extracting the main melody of a song or a method for processing a song is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0153] Those skilled in the art will understand that Figure 10 、 11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0154] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0155] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0156] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0157] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0158] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0159] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for extracting the main melody of a song, characterized in that: The method comprises: Obtain a song signal of a target song and an accompaniment signal of an original accompaniment corresponding to the target song; wherein the original accompaniment includes an accompaniment part and a non-accompaniment part; inputting the accompaniment signal of the original accompaniment and the song signal of the target song into an adaptive filter, and allowing the adaptive filter to identify a preliminary dry sound signal in the target song; Inputting the song signal of the target song into a vocal extraction model to obtain the vocal signal of the target song output by the vocal extraction model; Determine the probability of a first dry sound at each frequency point based on the ratio of the preliminary dry sound signal at each frequency point to the song signal, and determine the probability of a second dry sound at each frequency point based on the ratio of the human voice signal at each frequency point to the song signal; Determine the probability of the dry sound existing at each frequency point of the target song based on the first dry sound existence probability and the second dry sound existence probability at each frequency point; Processing the song signal of the target song at each frequency point according to the dry sound existence probability of the target song at each frequency point to obtain the target dry sound signal of the target song; The main melody of the target song is obtained according to the target dry sound signal.

2. The method according to claim 1, characterized in that The step of inputting the accompaniment signal of the original accompaniment and the song signal of the target song into an adaptive filter, and allowing the adaptive filter to identify a preliminary dry sound signal in the target song, comprises: Inputting the accompaniment signal of the original accompaniment and the song signal of the target song into an adaptive filter, and using the adaptive filter to identify the accompaniment signal of the song accompaniment in the target song; wherein the song accompaniment includes a vocal part and a non-singing part; A preliminary dry sound signal of the target song is obtained according to the accompaniment signal of the target song and the song signal of the target song.

3. The method according to claim 2, characterized in that The step of inputting the accompaniment signal of the original accompaniment and the song signal of the target song into an adaptive filter, and allowing the adaptive filter to identify the accompaniment signal of the song accompaniment in the target song, comprises: The accompaniment signal of the original accompaniment at each frequency point and the song signal of the target song are input into the adaptive filter, and the adaptive filter composed of the filter parameters at the current frequency point determines the predicted accompaniment signal of the current frequency point according to the accompaniment signal at the current frequency point and the song signal at the current frequency point; wherein the current frequency point is a frequency point among the frequency points, and when the current frequency point is not the first frequency point, the filter parameters at the current frequency point are calculated according to the filter parameters at a previous frequency point of the current frequency point, the accompaniment signal at the previous frequency point, and the song signal at the previous frequency point; According to the predicted accompaniment signals of each of the current frequency points, an accompaniment signal of the song accompaniment in the target song is obtained.

4. The method according to claim 2, characterized in that The step of obtaining a preliminary dry sound signal of the target song according to the accompaniment signal of the target song and the song signal of the target song comprises: Acquire the accompaniment signal of the target song and the song signal of the target song at each frequency point; The accompaniment signal in the song signal is removed at each of the frequency points to obtain a preliminary dry sound signal in the target song.

5. The method according to claim 1, characterized in that The target song includes audio recorded while playing the song accompaniment and singing.

6. The method according to claim 1, characterized in that The determining, based on the first dry sound existence probability and the second dry sound existence probability at each frequency point, the dry sound existence probability of the target song at each frequency point includes: Based on the product of the first dry sound existence probability and the second dry sound existence probability at each frequency point, the dry sound existence probability of the target song at each frequency point is obtained.

7. The method according to any one of claims 1 to 6, characterized in that The step of obtaining the main melody of the target song according to the target dry sound signal includes: A fundamental frequency sequence of the target dry sound signal is obtained, and the main melody of the target song is obtained based on the fundamental frequency sequence of the target dry sound signal.

8. A song processing method, characterized in that: The method comprises: Get the song hummed by the user; The method according to any one of claims 1 to 7, obtaining the main melody of the song hummed by the user; According to the main melody, user-required songs that meet preset requirements are retrieved and the user-required songs are displayed; or, according to the main melody, humming evaluation information is generated and the humming evaluation information is displayed.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images