Main melody extraction model training method and components, singing detection method and components

Abstract

The present application discloses a main melody extraction model training method and component, a singing voice detection method and component in the field of computer technology. The present application uses labeled song audio frames marked with main melody and unlabeled song audio frames without main melody to perform supervised training and semi-supervised training, which not only increases the model training volume, but also enables the model to learn the processing capabilities of labeled data and unlabeled data at the same time, and no additional data labeling work is required. Therefore, the model training efficiency is high and the model generalization ability is also improved. In addition, the scheme can directly process the song audio without the need to separate the background sound and the human voice, thereby improving the training efficiency and the main melody extraction efficiency. Correspondingly, the main melody extraction model training component and the singing voice detection method and component provided by the present application also have the above-mentioned technical effects.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a main melody extraction model training method and component, and a singing voice detection method and component. Background Art

[0002] At present, when extracting the main melody of a song, it is necessary to perform source separation on the song signal and then extract the main melody from the dry sound signal. Therefore, the melody extraction performance is limited by the source separation model and the extraction complexity is also very high. Of course, machine learning methods can also be used to extract the main melody of a song, but the existing machine learning methods require annotators who understand music theory to listen and play on special equipment to record the melody label, so the cost of annotating data is high and the efficiency is low. Summary of the invention

[0003] In view of this, the purpose of this application is to provide a main melody extraction model training method and component, a singing detection method and component, so as to improve the efficiency of song main melody extraction. The specific scheme is as follows:

[0004] To achieve the above objectives, on the one hand, the present application provides a main melody extraction model training method, comprising:

[0005] Obtain labeled song audio frames and unlabeled song audio frames from the training set;

[0006] Extracting labeled audio features from the labeled song audio frames, and extracting unlabeled audio features from the unlabeled song audio frames; the labeled audio features and the unlabeled audio features both include: spectral features and cepstrum features;

[0007] Inputting the labeled audio features into a supervised learning module to perform supervised training on a target model in the supervised training module and determine a corresponding supervised loss;

[0008] Inputting the labeled audio features and the unlabeled audio features into a semi-supervised learning module to perform semi-supervised training on the target model using the semi-supervised learning module and determine a corresponding semi-supervised loss;

[0009] Determine a comprehensive loss based on the supervised loss and the semi-supervised loss;

[0010] If the comprehensive loss converges, the current target model is determined as the main melody extraction model; otherwise, the model parameters of the current target model are updated based on the comprehensive loss, and labeled song audio frames and unlabeled song audio frames are selected from the training set to iteratively train the updated target model until the comprehensive loss converges.

[0011] Optionally, the extracting the labeled audio features from the labeled song audio frames and extracting the unlabeled audio features from the unlabeled song audio frames includes:

[0012] The labeled song audio frame and the unlabeled song audio frame are respectively used as target objects;

[0013] Converting the target object into a frequency spectrum feature through Fourier transform;

[0014] Logarithmic operation and inverse Fourier transform are performed on the frequency spectrum features to obtain cepstrum features.

[0015] Optionally, inputting the labeled audio features into a supervised learning module to perform supervised training on a target model in the supervised training module and determining a corresponding supervised loss includes:

[0016] The labeled audio features are input into the supervised learning module, so that the supervised learning module uses the target model to process the labeled audio features, obtain a binary classification prediction result of whether there is a main melody and a main melody prediction result, and determine the loss between the binary classification prediction result, the main melody prediction result and the label of the labeled song audio frame as the supervised loss.

[0017] Optionally, inputting the labeled audio features and the unlabeled audio features into a semi-supervised learning module to perform semi-supervised training on the target model using the semi-supervised learning module and determining a corresponding semi-supervised loss includes:

[0018] The labeled audio features and the unlabeled audio features are input into the semi-supervised learning module, so that the semi-supervised learning module performs the following steps:

[0019] Processing the labeled audio features using the target model to obtain a first labeled prediction result;

[0020] Processing the unlabeled audio feature using the target model to obtain a first unlabeled prediction result;

[0021] Processing the labeled audio features and the disturbance vector using the target model to obtain a second labeled prediction result;

[0022] Processing the unlabeled audio features and the disturbance vector using the target model to obtain a second unlabeled prediction result;

[0023] Calculate a first loss between the first labeled prediction result and the second labeled prediction result, and a second loss between the first unlabeled prediction result and the second unlabeled prediction result;

[0024] Half of the sum of the first loss and the second loss is determined as the semi-supervised loss.

[0025] Optionally, determining a comprehensive loss based on the supervised loss and the semi-supervised loss comprises:

[0026] The comprehensive loss is calculated using a target formula, which is:

[0027]

[0028] Among them, L total is the comprehensive loss, L d is the loss between the binary classification prediction result and the label of the labeled song audio frame, η is the ratio of the number of labeled song audio frames in the training set to the total number of song audio frames in the training set, and L s is the loss between the main melody prediction result and the label of the labeled song audio frame, L * u is the semi-supervised loss.

[0029] Optionally, it also includes:

[0030] After dividing the songs on the music platform into frames, the audio frames obtained by dividing the frames are windowed to obtain new unlabeled song audio frames;

[0031] Adding the new unlabeled song audio frame to the training set to obtain a new training set;

[0032] The main melody extraction model is further trained using the new training set, and the new main melody extraction model obtained through training is used for singing detection.

[0033] On the other hand, the present application also provides a singing voice detection method, comprising:

[0034] Each audio frame in the target song is used as an audio frame to be detected;

[0035] Extracting the audio feature to be detected from the audio frame to be detected;

[0036] Inputting the audio feature to be detected into a main melody extraction model so that the main melody extraction model outputs the main melody of the target song; the main melody extraction model is obtained based on any one of the main melody extraction model training methods described above;

[0037] The main melody is compared with a reference main melody of the target song, and a singing detection result of the target song is determined based on the comparison result.

[0038] Optionally, it also includes:

[0039] The singing detection result is used to perform cover recognition and / or singing evaluation.

[0040] On the other hand, the present application also provides an electronic device, comprising a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement any of the aforementioned methods.

[0041] On the other hand, the present application further provides a storage medium, in which computer executable instructions are stored. When the computer executable instructions are loaded and executed by a processor, any of the aforementioned methods is implemented.

[0042] This application uses labeled song audio frames with main melodies and unlabeled song audio frames without main melodies for supervised training and semi-supervised training, which not only increases the amount of model training, but also enables the model to learn the processing capabilities of labeled and unlabeled data at the same time, without the need for additional data labeling. Therefore, the model training efficiency is high and the model generalization ability is also improved. In addition, this solution can directly process song audio without the need to separate background sound and human voice, thereby improving the training efficiency and main melody extraction efficiency.

[0043] Correspondingly, the present application provides a main melody extraction model training component, a singing voice detection method and component, which also have the above technical effects. The component includes a device and a storage medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0045] Figure 1 A schematic diagram of a physical architecture applicable to the present application provided for the present application;

[0046] Figure 2 A flow chart of a main melody extraction model training method provided in this application;

[0047] Figure 3 A flow chart of a singing voice detection method provided in this application;

[0048] Figure 4 A flow chart of a music retrieval method provided for this application;

[0049] Figure 5 A schematic diagram of online updating logic of a main melody extraction model provided in this application;

[0050] Figure 6 A server structure diagram provided for this application;

[0051] Figure 7 A terminal structure diagram provided for this application. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In addition, in the embodiments of the present application, "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0053] The existing method of extracting the main melody of a song requires source separation of the song signal, and then extracting the main melody from the dry sound signal. Therefore, the melody extraction performance is limited by the source separation model, and the extraction complexity is also very high. Of course, machine learning methods can also be used to extract the main melody of a song, but the existing machine learning methods require annotators who understand music theory to listen and play on special equipment to record melody labels, so the cost of annotating data is high and the efficiency is low.

[0054] In view of the above-mentioned problems, this application proposes a main melody extraction model training scheme, which uses labeled song audio frames with main melody and unlabeled song audio frames without main melody for supervised training and semi-supervised training, which not only increases the model training volume, but also enables the model to learn the processing capabilities of labeled data and unlabeled data at the same time, and no additional data labeling is required. Therefore, the model training efficiency is high and the model generalization ability is also improved. In addition, the scheme can directly process the song audio without the need to separate the background sound and the human voice, thereby improving the training efficiency and the main melody extraction efficiency.

[0055] For ease of understanding, the physical framework to which this application applies is first introduced.

[0056] It should be understood that the main melody extraction model training method provided in the present application can be applied to a system or program with a main melody extraction model training function. Specifically, the system or program with a main melody extraction model training function can be run in a server, a personal computer, or other device. Correspondingly, the singing voice detection method provided in the present application can be applied to a system or program with a singing voice detection function. Specifically, the system or program with a singing voice detection function can be run in a server, a personal computer, or other device.

[0057] like Figure 1 As shown, Figure 1 This is a schematic diagram of the physical architecture applicable to this application. Figure 1 In the present invention, a system or program with a main melody extraction model training function can be run on a server, which obtains a training set from other terminal devices through a network, obtains labeled song audio frames and unlabeled song audio frames from the training set; extracts labeled audio features from labeled song audio frames, and extracts unlabeled audio features from unlabeled song audio frames; both labeled audio features and unlabeled audio features include: spectral features and cepstrum features; inputs the labeled audio features into a supervised learning module to perform supervised training on a target model in the supervised training module, and determines a corresponding supervised loss; inputs the labeled audio features and unlabeled audio features into a semi-supervised learning module to perform semi-supervised training on the target model using the semi-supervised learning module, and determines a corresponding semi-supervised loss; determines a comprehensive loss based on the supervised loss and the semi-supervised loss; if the comprehensive loss converges, the current target model is determined as the main melody extraction model; otherwise, updates the model parameters of the current target model based on the comprehensive loss, and selects labeled song audio frames and unlabeled song audio frames from the training set to iteratively train the updated target model until the comprehensive loss converges.

[0058] Correspondingly, the server obtains the target song from other terminal devices through the network, and uses each audio frame in the target song as the audio frame to be detected; extracts the audio features to be detected from the audio frames to be detected; inputs the audio features to be detected into the main melody extraction model, so that the main melody extraction model outputs the main melody of the target song; the main melody extraction model is obtained based on the main melody extraction model training method described in any of the above items; compares the main melody with the baseline main melody of the target song, and determines the singing detection result of the target song based on the comparison result.

[0059] Figure 1 A variety of terminal devices are shown in FIG. In actual scenarios, more or fewer types of terminal devices may participate in the main melody extraction model training or singing detection process. The specific number and type depend on the actual scenario and are not limited here. In addition, Figure 1 One server is shown in the figure, but in an actual scenario, multiple servers may be involved, and the specific number of servers depends on the actual scenario.

[0060] It should be noted that the main melody extraction model training method provided in this embodiment can be performed offline, that is, the server has a training set stored locally, which can directly use the solution provided in this application to train the main melody extraction model.

[0061] It is understandable that the above-mentioned system and program with the main melody extraction model training function can be used as a kind of cloud service program. The specific operation mode depends on the actual scenario and is not limited here.

[0062] Specifically, after the main melody extraction model is trained, the main melody extraction model can be used in scenarios such as cover song recognition and singing evaluation.

[0063] Combined with the above commonalities, please see Figure 2 , Figure 2 A flow chart of a main melody extraction model training method provided in an embodiment of the present application. Figure 2 As shown, the main melody extraction model training method may include the following steps:

[0064] S201, obtaining labeled song audio frames and unlabeled song audio frames from a training set.

[0065] The tags include: whether the corresponding song audio frame has a main melody, and the corresponding main melody sequence (i.e., baseband sequence) when there is a main melody. The song audio frame is: the audio frames obtained after framing and windowing the song.

[0066] S202. Extracting labeled audio features from labeled song audio frames, and extracting unlabeled audio features from unlabeled song audio frames.

[0067] The labeled audio features and the unlabeled audio features both include: spectrum features and cepstrum features.

[0068] In a specific implementation, extracting labeled audio features from labeled song audio frames and extracting unlabeled audio features from unlabeled song audio frames include: taking labeled song audio frames and unlabeled song audio frames as target objects respectively; converting the target objects into spectral features through Fourier transform; performing logarithmic operations and inverse Fourier transforms on the spectral features to obtain cepstrum features. It can be seen that the cepstrum features are obtained by further processing the spectral features.

[0069] S203: Input the labeled audio features into a supervised learning module to perform supervised training on a target model in the supervised training module, and determine a corresponding supervised loss.

[0070] In a specific implementation, the labeled audio features are input into a supervised learning module to perform supervised training on the target model in the supervised training module, and the corresponding supervised loss is determined, including: the labeled audio features are input into the supervised learning module so that the supervised learning module uses the target model to process the labeled audio features to obtain a binary classification prediction result of whether there is a main melody and a main melody prediction result, and the loss between the binary classification prediction result and the main melody prediction result and the label of the labeled song audio frame is determined as the supervised loss. It can be seen that the supervised loss includes two loss values, one is the loss between the "binary classification prediction result" and the "label of whether there is a main melody" in the label (that is, L in the target formula). d ), and the other is the loss between the “main melody prediction result” and the “main melody sequence” in the label (i.e., L s ).

[0071] S204: Input the labeled audio features and the unlabeled audio features into the semi-supervised learning module, so as to perform semi-supervised training on the target model using the semi-supervised learning module and determine the corresponding semi-supervised loss.

[0072] In this embodiment, the execution order of S203 and S204 can be interchanged or performed simultaneously. In this embodiment, the target model is only set in the supervised learning module, and the semi-supervised learning module performs semi-supervised training on the target model in the supervised learning module.

[0073] In a specific implementation, labeled audio features and unlabeled audio features are input into a semi-supervised learning module so as to perform semi-supervised training on a target model using the semi-supervised learning module and determine a corresponding semi-supervised loss, including: inputting labeled audio features and unlabeled audio features into the semi-supervised learning module so that the semi-supervised learning module performs the following steps: processing labeled audio features using the target model to obtain a first labeled prediction result; processing unlabeled audio features using the target model to obtain a first unlabeled prediction result; processing labeled audio features and a perturbation vector using the target model to obtain a second labeled prediction result; processing unlabeled audio features and a perturbation vector using the target model to obtain a second unlabeled prediction result; calculating a first loss between the first labeled prediction result and the second labeled prediction result, and a second loss between the first unlabeled prediction result and the second unlabeled prediction result; and determining half of the sum of the first loss and the second loss as the semi-supervised loss.

[0074] It can be seen that the semi-supervised learning module can make the target model's output after adding the perturbation vector close to the model output without adding the perturbation vector, so that the target model can resist the interference in the sample. vt It can be determined based on the following formula: Among them, ε is the coefficient of the control disturbance vector, represents differential operation, D represents non-complex metric function, r is a randomly initialized noise vector, Represents the gradient in the r direction. Indicates: Input data Y k (f) * The input parameters to the model are The corresponding output in the model is Z k (N) * . Accordingly, Indicates: Input data Y k (f) * +r input to the model parameter is The corresponding output in the model is Z vt (N) * Among them, Y k (f) * It can be either labeled data or unlabeled data. k (f) * When Y represents annotated data, the model output corresponds to the current annotated data. k (f) * When representing unlabeled data, the model output corresponds to the current unlabeled data.

[0075] Among them, the calculation formula of semi-supervised loss is: represents the first loss, represents the second loss, It can be seen that the semi-supervised loss also includes two loss values, one is the loss between the "first labeled prediction result obtained by processing the labeled audio features" and the "second labeled prediction result obtained by processing the labeled audio features and the perturbation vector" (that is, the loss in the formula ), and the other is the loss between the first unlabeled prediction result obtained by processing the unlabeled audio features and the second unlabeled prediction result obtained by processing the unlabeled audio features and the perturbation vector (i.e., ).

[0076] S205. Determine a comprehensive loss based on the supervised loss and the semi-supervised loss.

[0077] In a specific implementation, determining the comprehensive loss based on the supervised loss and the semi-supervised loss includes: calculating the comprehensive loss using a target formula, where the target formula is:

[0078]

[0079] Among them, L total is the comprehensive loss, L dis the loss between the binary classification prediction result and the label of the labeled song audio frame, η is the ratio of the number of labeled song audio frames in the training set to the total number of song audio frames in the training set, and L s is the loss between the main melody prediction result and the label of the labeled song audio frame, L * u is the semi-supervised loss.

[0080] Specifically, the semi-supervised loss L * u The calculation principle is:

[0081]

[0082] Among them, CE represents the cross entropy loss function, N* represents the number of candidate pitches, and Y k (f) * Represents labeled audio features or unlabeled audio features. When the input is labeled data, Z k (N) * is the model output of the labeled data, and the loss function is When the input is unlabeled data, Z k (N) * is the model output of unlabeled data, and the loss function is

[0083] S206, determine whether the comprehensive loss converges to a preset threshold; if so, execute S207; if not, execute S208.

[0084] The condition for whether the comprehensive loss converges may be: the comprehensive loss is small enough, or the comprehensive loss does not change or changes very little in several consecutive iterations. Specifically, a preset threshold may be set for judgment.

[0085] S207: Determine the current target model as the main melody extraction model.

[0086] S208. Update the model parameters of the current target model based on the comprehensive loss, and execute S201 to select labeled song audio frames and unlabeled song audio frames from the training set to iteratively train the updated target model until the comprehensive loss converges.

[0087] In order to further improve the performance of the main melody extraction model, the training set can be enhanced, and the enhanced training set can be used to continue training the main melody extraction model. In a specific implementation, it also includes: after framing the songs on the music platform, windowing the audio frames obtained by framing to obtain new unlabeled song audio frames; adding the new unlabeled song audio frames to the training set to obtain a new training set; using the new training set to continue training the main melody extraction model, and using the trained new main melody extraction model for singing detection. Among them, singing detection can be specifically used for cover recognition, singing evaluation, etc.

[0088] It can be seen that this embodiment uses labeled song audio frames with main melody and unlabeled song audio frames without main melody for supervised training and semi-supervised training, which not only increases the model training volume, but also enables the model to learn the processing capabilities of labeled data and unlabeled data at the same time, and no additional data labeling is required. Therefore, the model training efficiency is high and the model generalization ability is also improved. In addition, this solution can directly process the song audio without the need to separate the background sound and the human voice, thereby improving the training efficiency and the main melody extraction efficiency.

[0089] The following embodiments provide a specific use scheme of the main melody extraction model, which is introduced by taking singing voice detection as an example.

[0090] See also Figure 3 The singing voice detection method provided in this embodiment includes:

[0091] S301, obtaining a target song.

[0092] S302: taking each audio frame in the target song as an audio frame to be detected.

[0093] S303: extracting audio features to be detected from the audio frame to be detected.

[0094] S304, inputting the audio features to be detected into a main melody extraction model so that the main melody extraction model outputs the main melody of the target song; the main melody extraction model is obtained based on any of the main melody extraction model training methods described above.

[0095] S305: Compare the main melody with the reference main melody of the target song, and determine the singing detection result of the target song based on the comparison result.

[0096] The reference main melody of the target song can be considered as a relatively standard song main melody. If the main melody output by the model is similar to the reference main melody, the singing voice of the target song in step S301 can be considered relatively standard. Otherwise, the singing voice of the target song in step S301 is considered not to be standard. The target song can be a song recorded by the user himself, and the reference main melody is the main melody of the original song. Then, according to this embodiment, the difference between the song sung by the user himself and the original song can be compared.

[0097] In a specific implementation, the singing detection result can also be used for cover recognition and / or singing evaluation. If the singing detection result is used for cover recognition, it can be detected whether the target song is a cover. If the singing detection result is used for singing evaluation, the singing skills of the target song can be scored.

[0098] It can be seen that the main melody extraction model trained in the above embodiment can be widely used in scenarios such as cover song recognition and singing evaluation. It can be seen that the model has good versatility and a wide range of application scenarios.

[0099] The following describes a specific application scenario example to introduce the main melody extraction model training scheme provided by this application. That is, an application scheme for extracting the main melody of a song using the main melody extraction model. This scheme can be used in scenarios such as cover song recognition and singing evaluation.

[0100] This embodiment takes music retrieval as an example for specific introduction, and sets a main melody extraction model in the server. Figure 4 The specific process includes:

[0101] S401, the user submits the song audio to be retrieved at the terminal.

[0102] S402: The terminal sends the song audio to the server.

[0103] S403: The server divides the entire song audio into audio frames, and uses the obtained audio frames as audio frames to be detected.

[0104] S404: The server extracts the audio features to be detected from the audio frames to be detected.

[0105] S405: The server inputs the audio features to be detected into the main melody extraction model, so that the main melody extraction model outputs the main melody.

[0106] The main melody extraction model is obtained based on the main melody extraction model training method described in any of the aforementioned embodiments.

[0107] S406. The server compares the main melody output by the model with each reference main melody in the preset database. If the main melody output by the model has the greatest similarity with a reference main melody in the database, the song audio corresponding to the reference main melody is obtained.

[0108] S407: The server sends the song audio corresponding to the reference main melody to the terminal.

[0109] S408: The user plays the song audio corresponding to the reference main melody and the song audio submitted by the user on the terminal for comparison.

[0110] The terminal may be a device such as a smart phone, tablet computer, laptop computer or desktop computer. The song submitted by the user at the terminal may be a song sung by the user himself, and the benchmark main melody is the main melody of the original song. Then, the user plays the original song and the song audio submitted by the user at the terminal, and can compare the difference between the song sung by the user and the original song. Of course, the song evaluation tool can also be used to analyze the similarity between the main melody output by the model and the benchmark main melody with the greatest similarity, so as to score the song sung by the user.

[0111] The main melody extraction model used in this embodiment can be trained according to the following process. The basic processing flow includes: determining a training set, feature extraction, supervised learning, semi-supervised learning, etc.

[0112] The training set can be obtained by framing and windowing existing songs. Framing means dividing the song signal into signal frames of fixed time length, with overlapping adjacent frames. Windowing means adding a Hanning window to each signal frame for smoothing to prevent spectrum leakage. Some of these signal frames in the training set are labeled, while others are not. The label means whether the corresponding audio frame has a main melody mark, and the corresponding main melody sequence when there is a main melody.

[0113] Feature extraction is to extract the spectral features of the above signal frames and determine the cepstrum features based on the spectral features. Among them, the spectral feature Y can be obtained by Fourier transforming the song time domain signal frame y(t) k l (f), the time domain signal frame is transformed into the spectrum feature and is expressed as: Y k l (f) = F(y(t)). F represents Fourier transform. In order to enable the model to learn more audio information, the cepstrum feature Y is further determined based on the spectrum feature. k 2 (f) is used to characterize the time period information of the fundamental frequency. This feature is related to the period of the main melody. The cepstrum feature is further determined based on the spectrum feature and is expressed as: k 2 (f) = F -1 ((Y k 1 (f)) γ ). Among them, F -1represents the inverse Fourier transform, and γ is the nonlinear mapping coefficient. After that, the superimposed spectrum features and cepstrum features are used as the model input. The input features of the main melody extraction task can be enriched by accumulating multi-dimensional information. The final input feature is Y k (f), Y k (f) = Y k 1 (f)+Y k 2 (f). f is the frequency index of the signal frame in the frequency domain, and k is the frame index of the signal frame.

[0114] Supervised learning means using the labeled audio frames in the training set for supervised training. Specifically, the supervised model in the supervised learning module can use the SkipConvNet network, whose input data is the labeled Y k (f), the network output is the main melody line Z k (N) and M indicating whether the current input has a main melody line k Mark, the network can be expressed as p(Z k (N)|Y k (f), Θ), the loss function is L s =CE[Z k (N), Z k (N) label ]and Among them, CE is the cross entropy, Z k (N) label is the main melody sequence in the label, N represents the number of candidate pitches, and has a mapping relationship with f. k ∈[0, 1], It is the mark of whether there is a main melody in the label. It can be seen that the supervised model in the supervised learning module needs to handle two tasks, one is the main melody extraction task, and the other is the binary classification task. These two tasks correspond to a loss respectively. Therefore, after the supervised learning module iterates once, it outputs two losses: L d and L s .

[0115] Semi-supervised learning is to use virtual adversarial learning to perform semi-supervised training. Specifically, the semi-supervised learning module is also trained for the above-mentioned supervised model, and the input data is Y representing labeled data or unlabeled data. k (f) * , the corresponding output of the network is the main melody line. At the same time, the input data also includes: Y k (f) * With the disturbance vector r vt The loss function is It can be seen that the semi-supervised learning module can make the model output corresponding to the perturbed input data close to the model output corresponding to the unperturbed input data, and the input data covers both labeled data and unlabeled data. Therefore, the semi-supervised learning module can enable the model to learn the processing ability of unlabeled data. After each iteration, the output loss L * u .

[0116] After the supervised learning module and the semi-supervised learning module have each iterated once, the total loss is calculated to balance the weights of supervised learning and semi-supervised learning. The calculation formula is: Then based on the total loss L total At the same time, the supervised model is updated until the number of training iterations reaches a threshold or L total When it is lower than a certain threshold, the supervised model output is used as the main melody extraction model.

[0117] It should be noted that using supervised modules and semi-supervised modules to train the same model can increase the model loss without modifying the original structure and parameters of the model, thereby producing a regularization effect and improving the robustness of the model.

[0118] The final output of the main melody extraction model can be used for online cover recognition, singing evaluation, etc. In this way, the model will involve more new songs during application, and these new songs can be used to enhance the training set. Figure 5 As shown in the figure, when the main melody extraction model is used on the business side, the music library can be filled in time to obtain new unlabeled data. After the data is added, the supervised module and the semi-supervised module can be used to continue training the main melody extraction model obtained last time. After the training is completed, the model of the online application on the business side is updated based on the newly obtained main melody extraction model. Of course, the online system can periodically synchronize and pull the latest offline model.

[0119] It can be seen that in this embodiment, a basic main melody extraction model can be trained through the aforementioned learning framework. When there is new unlabeled data, further iterative training is performed on the basic main melody extraction model, thereby improving the generalization performance of the model for new data. While improving the performance of the model, the dependence on labeled data is reduced, and the accuracy of the main melody extraction model can be further improved, thereby improving the service experience such as music information retrieval.

[0120] An electronic device provided in an embodiment of the present application is introduced below. The electronic device described below and related implementation steps of the above embodiment can be referenced to each other.

[0121] Furthermore, the present application also provides an electronic device. The electronic device can be Figure 6 The server 50 shown may also be Figure 7 Terminal 60 is shown. Figure 6 and Figure 7 All of them are structural diagrams of electronic devices according to an exemplary embodiment, and the contents in the diagrams cannot be regarded as any limitation on the scope of use of the present application.

[0122] Figure 6 A schematic diagram of the structure of a server provided in an embodiment of the present application. The server 50 may specifically include: at least one processor 51, at least one memory 52, a power supply 53, a communication interface 54, an input / output interface 55, and a communication bus 56. The memory 52 is used to store a computer program, which is loaded and executed by the processor 51 to implement the relevant steps in the main melody extraction model training disclosed in any of the aforementioned embodiments.

[0123] In this embodiment, the power supply 53 is used to provide working voltage for each hardware device on the server 50; the communication interface 54 can create a data transmission channel between the server 50 and external devices, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 55 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0124] In addition, the memory 52, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 521, a computer program 522 and data 523, etc. The storage method can be temporary storage or permanent storage.

[0125] The operating system 521 is used to manage and control the hardware devices and computer programs 522 on the server 50, so as to realize the operation and processing of the data 523 in the memory 52 by the processor 51, and it can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the main melody extraction model training method disclosed in any of the aforementioned embodiments, the computer program 522 can further include a computer program that can be used to complete other specific tasks. In addition to including data such as application update information, the data 523 can also include data such as application developer information.

[0126] Figure 7 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. The terminal 60 may specifically include but is not limited to a smart phone, a tablet computer, a laptop computer or a desktop computer.

[0127] Generally, the terminal 60 in this embodiment includes: a processor 61 and a memory 62 .

[0128] Among them, the processor 61 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 61 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 61 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 61 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 61 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0129] The memory 62 may include one or more computer-readable storage media, which may be non-transitory. The memory 62 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 62 is at least used to store the following computer program 621, wherein, after the computer program is loaded and executed by the processor 61, it can implement the relevant steps in the main melody extraction model training method performed by the terminal side disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 62 may also include an operating system 622 and data 623, etc., and the storage method may be short-term storage or permanent storage. Among them, the operating system 622 may include Windows, Unix, Linux, etc. Data 623 may include, but is not limited to, update information of the application.

[0130] In some embodiments, the terminal 60 may further include a display screen 63 , an input / output interface 64 , a communication interface 65 , a sensor 66 , a power supply 67 , and a communication bus 68 .

[0131] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation on the terminal 60, and the terminal 60 may include more or less components than those shown in the figure.

[0132] A storage medium provided in an embodiment of the present application is introduced below. The storage medium described below and the related implementation steps of the above embodiment can be referenced to each other.

[0133] Furthermore, the embodiment of the present application also discloses a storage medium, in which computer executable instructions are stored. When the computer executable instructions are loaded and executed by a processor, the main melody extraction model training method disclosed in any of the above embodiments is implemented. For the specific steps of the method, reference can be made to the corresponding contents disclosed in the above embodiments, and no further description will be given here.

[0134] It should be pointed out that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0135] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0136] Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A main melody extraction model training method, characterized in that: include: Obtain labeled song audio frames and unlabeled song audio frames from the training set; Extracting labeled audio features from the labeled song audio frames, and extracting unlabeled audio features from the unlabeled song audio frames; The labeled audio features and the unlabeled audio features both include: spectral features and cepstrum features; Inputting the labeled audio features into a supervised learning module to perform supervised training on a target model in the supervised learning module and determine a corresponding supervised loss; Inputting the labeled audio features and the unlabeled audio features into a semi-supervised learning module to perform semi-supervised training on the target model using the semi-supervised learning module and determine a corresponding semi-supervised loss; A comprehensive loss is determined based on the supervised loss and the semi-supervised loss; the comprehensive loss is a loss determined based on the two losses included in the supervised loss and the semi-supervised loss; the two losses are the binary classification prediction result of whether there is a main melody, the loss between the main melody prediction result and the label of the labeled song audio frame, and the binary classification prediction result and the main melody prediction result are both prediction results obtained by the supervised learning module after processing the labeled audio features using the target model; If the comprehensive loss converges, the current target model is determined as the main melody extraction model; otherwise, the model parameters of the current target model are updated based on the comprehensive loss, and labeled song audio frames and unlabeled song audio frames are selected from the training set to iteratively train the updated target model until the comprehensive loss converges.

2. The method according to claim 1, characterized in that The step of extracting the labeled audio features from the labeled song audio frames and extracting the unlabeled audio features from the unlabeled song audio frames comprises: The labeled song audio frame and the unlabeled song audio frame are respectively used as target objects; Converting the target object into a frequency spectrum feature through Fourier transform; Logarithmic operation and inverse Fourier transform are performed on the frequency spectrum features to obtain cepstrum features.

3. The method according to claim 1, characterized in that The step of inputting the labeled audio features and the unlabeled audio features into a semi-supervised learning module, using the semi-supervised learning module to perform semi-supervised training on the target model, and determining a corresponding semi-supervised loss includes: The labeled audio features and the unlabeled audio features are input into the semi-supervised learning module, so that the semi-supervised learning module performs the following steps: Processing the labeled audio features using the target model to obtain a first labeled prediction result; Processing the unlabeled audio feature using the target model to obtain a first unlabeled prediction result; Processing the labeled audio features and the disturbance vector using the target model to obtain a second labeled prediction result; Processing the unlabeled audio features and the disturbance vector using the target model to obtain a second unlabeled prediction result; Calculate a first loss between the first labeled prediction result and the second labeled prediction result, and a second loss between the first unlabeled prediction result and the second unlabeled prediction result; Half of the sum of the first loss and the second loss is determined as the semi-supervised loss.

4. The method according to claim 3, characterized in that The determining of the comprehensive loss based on the supervised loss and the semi-supervised loss comprises: The comprehensive loss is calculated using a target formula, which is: Among them, L total is the comprehensive loss, L d is the loss between the binary classification prediction result and the label of the labeled song audio frame, η is the ratio of the number of labeled song audio frames in the training set to the total number of song audio frames in the training set, and L s is the loss between the main melody prediction result and the label of the labeled song audio frame, L * u is the semi-supervised loss.

5. The method according to any one of claims 1 to 4, characterized in that: Also includes: After dividing the songs on the music platform into frames, the audio frames obtained by dividing the frames are windowed to obtain new unlabeled song audio frames; Adding the new unlabeled song audio frame to the training set to obtain a new training set; The main melody extraction model is further trained using the new training set, and the new main melody extraction model obtained through training is used for singing detection.

6. A singing voice detection method, characterized in that: include: Each audio frame in the target song is used as an audio frame to be detected; Extracting the audio feature to be detected from the audio frame to be detected; Inputting the audio feature to be detected into a main melody extraction model so that the main melody extraction model outputs the main melody of the target song; The main melody extraction model is obtained based on the method described in any one of claims 1 to 5; The main melody is compared with a reference main melody of the target song, and a singing detection result of the target song is determined based on the comparison result.

7. The method according to claim 6, characterized in that Also includes: The singing detection result is used to perform cover recognition and / or singing evaluation.

8. An electronic device, characterized in that: The electronic device comprises a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

9. A storage medium, characterized in that: The storage medium stores computer executable instructions, and when the computer executable instructions are loaded and executed by the processor, the method according to any one of claims 1 to 7 is implemented.

Images