Song recognition model training method, song recognition method, and related device

Abstract

The application discloses a song recognition model training method, a song recognition method and related devices. The training method comprises the following steps: obtaining training audio and corresponding audio labels; extracting audio features of the training audio to obtain training features; inputting the training features into an initial model to obtain training recognition results; wherein the initial model comprises a first convolution layer and a second convolution layer, the first convolution layer and the second convolution layer have rectangular convolution kernels, the long side of the first rectangular convolution kernel is arranged along the frequency axis direction, and the long side of the second rectangular convolution kernel is arranged along the time axis direction; generating a loss value by using the training recognition results and the audio labels, and performing parameter adjustment processing on the initial model by using the loss value; if it is detected that a preset completion condition is met, the initial model after the parameter adjustment is determined as a song recognition model; and the song recognition model obtained by the method has strong anti-noise interference capability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio processing, in particular to a singing voice recognition model training method, a singing voice recognition method and related devices. BACKGROUND

[0002] With the development of the multimedia industry and the rise of short videos, music is no longer simply a simple listening song, but can also be consumed by the public in a more diverse and rich form. Music can appear as background music or background music in live streaming, short videos, outdoor and other scenarios. In such scenarios, singing voices in audio waveforms are mixed with some steady-state noise, environmental noise, transient noise, human voice noise and other sounds. For example, different recording devices will result in different audio loudness, sound field and other sound collection effects, thereby causing different spectral distributions of the audio. Singing voice detection (SVD) is a relatively basic task in the field of music information retrieval (MIR), but its robustness is low and its anti-interference ability is weak, and its performance will drop sharply in complex noise scenarios. SUMMARY

[0003] Therefore, the purpose of the present application is to provide a singing voice recognition model training method, a singing voice recognition method and related devices, so that the singing voice recognition model has strong anti-noise interference ability and can accurately distinguish singing voices.

[0004] To solve the above technical problems, in a first aspect, the present application provides a singing voice recognition model training method, comprising:

[0005] obtaining training audio and corresponding audio labels; wherein the training audio includes noisy audio disturbed by noise, and the audio labels are used to indicate whether the training audio is singing audio or non-singing audio;

[0006] extracting audio features of the training audio to obtain training features;

[0007] inputting the training features into an initial model to obtain training recognition results; wherein the initial model includes a first convolutional layer and a second convolutional layer, the first convolutional layer and the second convolutional layer have rectangular convolutional kernels, the long side of the first rectangular convolutional kernel is arranged along the frequency axis direction, and the long side of the second rectangular convolutional kernel is arranged along the time axis direction;

[0008] generating a loss value using the training recognition results and the audio labels, and performing parameter adjustment processing on the initial model using the loss value;

[0009] if it is detected that a preset completion condition is met, the initial model after parameter adjustment is determined as a singing voice recognition model.

[0010] Optionally, the obtaining the training audio comprises:

[0011] obtaining initial training audio;

[0012] performing dynamic range control processing on the initial training audio to obtain the training audio.

[0013] Optionally, the obtaining the training audio comprises:

[0014] obtaining initial training audio;

[0015] determining a preset audio length, and performing fragmentation processing and / or zero padding processing on the initial training audio based on the preset audio length to obtain the training audio.

[0016] Optionally, the generating process of the audio label comprises:

[0017] determining an audio category corresponding to the training audio;

[0018] generating the audio label based on the audio category.

[0019] Optionally, the extracting the audio feature of the training audio to obtain a training feature comprises:

[0020] performing Mel-spectrum extraction processing and / or Mel-frequency cepstrum coefficient extraction processing on the training audio in an audio frame granularity to obtain the training feature.

[0021] Optionally, after the extracting the audio feature of the training audio to obtain a training feature, the method further comprises:

[0022] dividing the training feature and the corresponding audio label into a training set and a validation set;

[0023] Correspondingly, the inputting the training feature into an initial model to obtain a training recognition result comprises:

[0024] inputting the training feature contained in the training set into an initial model to obtain a training recognition result;

[0025] The determining the initial model with the adjusted parameters as a song voice recognition model if it is detected that the preset completion condition is met comprises:

[0026] If it is detected that the preset training condition is met, performing recognition accuracy verification on the initial model with the adjusted parameters by using the validation set.

[0027] If the identification accuracy of the initial model after parameter adjustment does not meet the preset accuracy condition, the step of inputting the training features contained in the training set into the initial model to obtain a training identification result is returned; until the identification accuracy of the initial model after parameter adjustment meets the preset accuracy condition, the initial model after parameter adjustment is determined as the song voice identification model.

[0028] In a second aspect, the present application also provides a song voice identification method, comprising:

[0029] Obtaining a to-be-tested audio;

[0030] Extracting audio features of the to-be-tested audio to obtain to-be-tested features;

[0031] Inputting the to-be-tested features into a song voice identification model to obtain a song voice identification result; wherein the song voice identification model is obtained based on the above song voice identification model training method.

[0032] Optionally, the step of extracting audio features of the to-be-tested audio to obtain to-be-tested features comprises:

[0033] Performing feature extraction processing on the to-be-tested audio to obtain initial audio features;

[0034] Determining a preset audio length, and performing fragmentation processing and / or zero padding processing on the initial audio features based on the preset audio length to obtain the to-be-tested features.

[0035] Optionally, if the to-be-tested features are multiple, the step of inputting the to-be-tested features into a song voice identification model to obtain a song voice identification result comprises:

[0036] Respectively inputting each to-be-tested feature into the song voice identification model to obtain a fragmented identification result;

[0037] Performing fusion processing on the fragmented identification result to obtain the song voice identification result.

[0038] Optionally, it further comprises:

[0039] Determining the start and end time range of each to-be-tested feature relative to the to-be-tested audio;

[0040] Using the start and end time range to sort each fragmented identification result to obtain a first sequence;

[0041] Performing same classification boundary fusion processing on the first sequence to obtain a second sequence;

[0042] Determining the song voice endpoint corresponding to the to-be-tested audio based on the second sequence.

[0043] In a third aspect, the present application also provides an electronic device comprising a memory and a processor, wherein:

[0044] The memory is configured to store a computer program.

[0045] The processor is configured to execute the computer program to implement the song recognition model training method and / or the song recognition method.

[0046] In a fourth aspect, the present application also provides a computer readable storage medium configured to store a computer program, wherein the computer program is configured to be executed by a processor to implement the song recognition model training method and / or the song recognition method.

[0047] It can be seen that the method uses special training data and a special initial model to realize song recognition of noisy audio, and has strong robustness and anti-interference ability. Specifically, the training data includes noisy audio, so that the model can learn how to distinguish song audio under noise interference. The initial model includes a first convolutional layer and a second convolutional layer, and the convolutional kernels of the two convolutional layers are rectangular. The convolutional kernel arranged along the frequency axis direction can obtain frequency domain information such as pitch and vocal range in a larger frequency domain range, and the convolutional kernel arranged along the time axis direction can obtain time domain information such as rhythm and melody in a larger time domain range. Through the first convolutional layer and the second convolutional layer, the initial model can obtain more information, which helps to resist the interference of noise and realize accurate classification. The song recognition model obtained after training can have strong anti-noise interference ability and can accurately distinguish song.

[0048] In addition, the present application also provides a song recognition method and related devices, which also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0050] Figure 1 A hardware composition framework schematic diagram applicable to a song recognition model training method and / or a song recognition method provided by an embodiment of the present application;

[0051] Figure 2 A hardware composition framework schematic diagram applicable to another song recognition model training method and / or a song recognition method provided by an embodiment of the present application;

[0052] Figure 3 A flowchart of a song recognition model training method provided by an embodiment of the present application is shown in FIG. 1.

[0053] Figure 4 A dynamic range control effect diagram provided by an embodiment of the present application is shown in FIG. 2.

[0054] Figure 5 A schematic diagram of a first convolution kernel and a second convolution kernel provided by an embodiment of the present application is shown in FIG. 3.

[0055] Figure 6 A song starting point positioning effect diagram provided by an embodiment of the present application is shown in FIG. 4.

[0056] Figure 7 A song recognition flowchart provided by an embodiment of the present application is shown in FIG. 5. DETAILED DESCRIPTION

[0057] To make the objectives, technical solutions, and advantages of embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0058] To facilitate understanding, the hardware composition framework used by the scheme corresponding to the song recognition model training method and / or song recognition method provided by an embodiment of the present application is introduced first. Please refer to FIG. 6. Figure 1 , Figure 1 A hardware composition framework diagram applicable to the song recognition model training method and / or song recognition method provided by an embodiment of the present application is shown in FIG. 6. The electronic device 100 can include a processor 101 and a memory 102, and can further include one or more of a multimedia component 103, an information input / output (I / O) interface 104, and a communication component 105.

[0059] The processor 101 is configured to control the overall operation of the electronic device 100 to complete all or part of the steps in the song recognition model training method and / or the song recognition method. The memory 102 is configured to store various types of data to support the operation of the electronic device 100. The memory 102 can be implemented by any type of volatile or non-volatile storage devices or a combination thereof, such as one or more of static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. In the embodiment, the memory 102 at least stores programs and / or data for implementing the following functions:

[0060] obtaining training audio and corresponding audio labels; wherein the training audio includes noisy audio disturbed by noise, and the audio labels are used to represent whether the training audio is song audio or non-song audio;

[0061] extracting audio features of the training audio to obtain training features;

[0062] inputting the training features into an initial model to obtain training recognition results; wherein the initial model includes a first convolutional layer and a second convolutional layer, the first convolutional layer and the second convolutional layer have rectangular convolutional kernels, the long side of the first rectangular convolutional kernel is arranged along the frequency axis direction, and the long side of the second rectangular convolutional kernel is arranged along the time axis direction;

[0063] generating a loss value using the training recognition results and the audio labels, and performing parameter adjustment processing on the initial model using the loss value;

[0064] if it is detected that a preset completion condition is met, the initial model after parameter adjustment is determined as a song recognition model.

[0065] The multimedia component 103 can include a screen and an audio component. The screen can be a touch screen, for example. The audio component is configured to output and / or input audio signals. For example, the audio component can include a microphone configured to receive external audio signals. The received audio signals can be further stored in the memory 102 or transmitted through the communication component 105. The audio component also includes at least one speaker configured to output audio signals. The I / O interface 104 provides an interface between the processor 101 and other interface modules, which can be a keyboard, a mouse, a button, and the like. The buttons can be virtual buttons or physical buttons. The communication component 105 is configured to perform wired or wireless communication between the electronic device 100 and other devices. The wireless communication, for example, Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component 105 can include a Wi-Fi component, a Bluetooth component, and an NFC component.

[0066] The electronic device 100 can be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic elements, for performing the song recognition model training method and / or the song recognition method.

[0067] Of course, Figure 1 The structure of the electronic device 100 shown does not constitute a limitation on the electronic device in the embodiments of the present application. In actual applications, the electronic device 100 can include more or fewer components than those shown, or some components can be combined. Figure 1

[0068] It can be understood that the number of electronic devices in the embodiments of the present application is not limited, and a plurality of electronic devices can be cooperated to complete the song recognition model training method and / or the song recognition method. In a possible implementation, please refer to Figure 2 Figure 2 Another hardware component framework suitable for the song recognition model training method and / or the song recognition method provided in the embodiments of the present application is shown in the figure. The hardware component framework includes a processor 101, a memory 102, an I / O interface 104, and a communication component 105. Figure 2 ​​It can be known that the hardware composition framework can include the first electronic device 11 and the second electronic device 12, which are connected through the network 13.

[0069] In the embodiment of the present application, the hardware structure of the first electronic device 11 and the second electronic device 12 can refer to the electronic device 100. Figure 1 That is, it can be understood that there are two electronic devices 100 in the embodiment, which interact with each other. Further, the form of the network 13 is not limited in the embodiment of the present application, that is, the network 13 can be a wireless network (such as WIFI, Bluetooth, etc.), or a wired network.

[0070] Among them, the first electronic device 11 and the second electronic device 12 can be the same kind of electronic device, such as the first electronic device 11 and the second electronic device 12 are both servers; or different types of electronic devices, for example, the first electronic device 11 can be a smart phone or other smart terminal, and the second electronic device 12 can be a server. In a possible implementation manner, a server with strong computing power can be used as the second electronic device 12 to improve the data processing efficiency and reliability, and thus improve the processing efficiency of model training and / or song recognition. At the same time, a smart phone with low cost and wide application range is used as the first electronic device 11 to realize the interaction between the second electronic device 12 and the user. It can be understood that the interaction process can be that the smart phone obtains and plays the training audio from the server, and obtains the audio label, and sends the audio label to the server, and the server uses the obtained audio label to perform subsequent model training steps. After the server generates the song recognition model, it obtains the audio to be tested sent by the smart phone and performs song recognition.

[0071] Specifically, please refer to Figure 3 , Figure 3 a flowchart of a song recognition model training method provided in the embodiment of the present application. The method in the embodiment includes:

[0072] S101: Obtain training audio and corresponding audio label.

[0073] The training audio includes noisy audio interfered by noise, and in addition, can also include non-noisy audio not interfered by noise. Interference by noise means that in the training audio, in addition to singing, there are also noise such as steady noise, environmental noise, transient noise, and human voice noise, which makes the interference in the noisy audio more, and is not conducive to identifying whether it is singing audio. The audio content of the training audio is not limited, and considering the diversity of real noise scenes, audio data in corresponding scenes can be collected according to different categories of audio labels. For example, the singing audio label can include singing audio with music performance and a cappella singing audio, the sound data of the singing audio with music performance can be various soundtracks, various instrumental performances, concerts, ktv performances, etc., and the sound data of the a cappella singing audio is dry sound of vocal separation, singing teaching, etc. In addition, the types of non-singing audio are more diverse, for example, can include audio with music speaking, audio with white, pure music audio, pure noise audio, etc. Among them, the sound data of the audio with music speaking is audio book, film and television, short video, variety show, etc., the sound data of the audio with white is recitation, news, meeting, chat, sketch, cross talk, etc., the sound data of the pure music audio is instrumental solo, symphony, accompaniment, advertisement music, etc., and the sound data of the pure noise audio is white noise (water sound, rain sound, etc.), laughter, applause, eating sound, etc. The data format of the training audio is not limited, for example, can be ts code stream (Transport Stream code stream) obtained from streaming media, or an audio file can be obtained as training audio, and the file format can be mp3, m4a, wav, etc.

[0074] The audio label is used to represent that the training audio is singing audio or non-singing audio, and it needs to be noted that the audio label is not used to represent whether the training audio is noisy audio, that is, whether the training audio contains noise, and is not related to whether the training audio is singing audio. The noisy audio can be singing audio or non-singing audio.

[0075] The specific acquisition manner of the training audio is not limited in the present application. In an embodiment, the training audio and the audio label can be generated in advance and stored in a specified location, and can be read from the specified location when needed. In another embodiment, the training audio can be generated by itself when the singing voice recognition model is trained. Specifically, in an embodiment, since the length of the model input is usually fixed, the length of the training audio can be set. When the training audio is acquired, the initial training audio is first acquired. The preset audio length is determined, which is the length corresponding to each training audio. The initial training audio is processed by slicing and / or zero padding based on the preset audio length to obtain the training audio. The slicing processing refers to processing of dividing the initial training audio into multiple segments when the length of the initial training audio is greater than the preset audio length, and the length of each segment is equal to the preset audio length. There can be no overlap or overlap between the segments, that is, the slicing frame step length when slicing can be less than, equal to or greater than the preset audio length. The zero padding processing refers to processing of supplementing zero data after the initial training audio or the segment (usually the last segment) obtained after slicing the initial training audio to make the length reach the preset audio length. In a specific embodiment, the preset audio length can be represented by dur', and the unit can be millisecond ms.

[0076] In another embodiment, different recording devices and other factors can affect the loudness, sound field and various sound effects of the audio, thereby causing different spectral distributions. For example, when the recording device is far away from the sound source, the collected audio signal can have low loudness, far sound field and other problems, and some sound details are not easy to capture. On the contrary, when the recording device is too close to the sound source, the loudness can be too large, and problems such as burr and distortion can occur. In order to solve this problem, the present application can use the audio dynamic range control (Dynamic Range Control, DRC) technology to process the audio spectral distribution. Specifically, the initial training audio is first acquired, which is the audio not processed by DRC in the present embodiment. The initial training audio is processed by dynamic range control to obtain the training audio.

[0077] The specific process of DRC processing is not limited in the present application. In an embodiment, the signal x after digital sampling, that is, the initial training audio, is first acquired by using an audio tool (such as librosa audio tool). Then, the linear x signal is converted into a decibel (db) signal, x db = 20*log10(x). The dB signal is transmitted to a static feature equation (i.e., DRC static curve x sc ) to obtain a difference value, and then a gain curve g c = x sc -x dbThen, the inflection point of the gain curve is smoothly transitioned to obtain a curve g s , and then the curve g s is gain compensated to obtain a DRC gain control curve g m , and finally the curve is converted from the dB value state to the linear value state to obtain g lin = 10^(g m / 20). Finally, the original audio signal x is dynamically adjusted by applying the calculated gain control signal g lin to obtain the dynamically adjusted audio signal y = g lin x. Through DRC processing, different degrees of gain adjustment can be made for the signal amplitudes of different stages (i.e., the noise stage, the medium amplitude stage, the large amplitude stage, etc.), and ultimately the sound will be more stable and soft. Please refer to Figure 4 , Figure 4 for a dynamic range control effect diagram provided by an embodiment of the present application. The left side of the multiplication sign is the signal x, the right side is the signal g lin , and the processed signal y after the equal sign is the training audio in the present embodiment. As can be seen, after DRC processing, the parts with large amplitudes in the x signal are weakened, and the parts with small amplitudes are enhanced.

[0078] The present application does not limit the specific generation method of the audio label. In one embodiment, each training audio can be played and labeled by human beings. In another embodiment, the audio label can be generated according to the category of the training audio. As described above, the training audio can include six categories, i.e., singing with music, a cappella, speaking with music, speech, pure music, and pure noise. After determining the category, singing with music and a cappella can be determined as singing audio, and the other types can be determined as non-singing audio. The song type can be determined according to the category label of the training audio, i.e., the training data with the pre-existing category label can be used to complete the acquisition of the training label by mapping the category label to the audio label of the present application. It should be noted that the specific form of the audio label is not limited in the present application. In the first embodiment, the category labels such as singing with music and a cappella can be reconfigured as the same label, e.g., label 1, indicating singing audio. Alternatively, in the second embodiment, they can be configured as different labels, but the labels corresponding to the two types of audio belong to the same audio label, i.e., singing audio, e.g., the content of the category label is kept unchanged, and the corresponding relationship between the two category labels of singing with music and a cappella and the singing audio label is generated. It can be understood that if the second embodiment is adopted, the subsequent singing recognition model can classify the audio at the granularity of the audio category label, achieve fine detection, and can map the audio category to the label of whether it is singing audio in the post-processing process.

[0079] S102: Extract the audio features of the training audio to obtain training features.

[0080] After obtaining the training audio, the corresponding audio features, i.e., the training features, can be obtained through feature extraction. The specific manner of the audio features is not limited in the embodiment. In an implementation, the training audio is subjected to Mel-spectrum extraction processing and / or Mel-frequency cepstral coefficient extraction processing with audio frames as the granularity to obtain the training features. The Mel-spectrum (Mel Spectrogram, Mel) is a kind of spectrum diagram obtained by pre-emphasizing, framing and windowing the original audio signal, performing short-time Fourier transform on each frame of signal, and then passing through a Mel filter bank, wherein the abscissa represents time and the ordinate represents frequency, and the characteristic is close to the nonlinear frequency perception of the human ear, for example, the perception of low-frequency signals is more sensitive than that of high-frequency signals. The Mel-frequency cepstral coefficient (Mel Frequency Cepstral Coefficients, MFCC) is also a kind of spectrum diagram, which is exactly the cepstral coefficient of the Mel frequency, and the characteristic is beneficial to the representation of the sound timbre.

[0081] Specifically, the librosa tool can be used to extract the Mel features and the MFCC features of the training audio with audio frames as the granularity. The Mel features are close to the nonlinear hearing perception of the human ear, which is beneficial to the neural network to analyze the audio features from the perspective of the frequency correlation of the hearing perception, and the MFCC features contain the spectrum envelope representing the timbre-related information, so that the neural network can learn to distinguish different sound compositions such as human voice and noise from the timbre dimension. The specific forms of the two kinds of features are not limited in the embodiment. For example, the Mel features can be composed of a 128xN matrix F mel , and the MFCC features are composed of a 20xN matrix F mfcc , wherein all represent the frame length of the audio (rounded down), dur represents the time length of the audio in ms, sr represents the sampling rate of the audio reading in Hz, and hop represents the frame shift, i.e., the number of samples between consecutive frames. The above Mel features and MFCC features are spliced to obtain a feature F input of 148xN dimensions, which is the training feature. input .

[0082] The training feature can be saved in the form of a file, for example, the file format of “.npy”,

[0083] S103: Input the training feature into the initial model to obtain a training recognition result.

[0084] The initial model refers to a singing voice recognition model whose parameters have not been adjusted yet, and specifically can be a model built based on a convolutional neural network. The initial model includes a first convolution layer and a second convolution layer. The first convolution layer and the second convolution layer have rectangular convolution kernels. The long side of the first rectangular convolution kernel is set along the frequency axis, and the long side of the second rectangular convolution kernel is set along the time axis. That is, the initial model has two mutually orthogonal two-dimensional convolution layers of special scales. For example, they can be a time domain convolution layer of 32×1 dimensions and a frequency domain convolution layer of 7×64 dimensions. These two convolution layers can learn different degrees of time domain information (such as rhythm, melody) and frequency domain information (such as pitch, range) on the time axis and frequency axis of the audio features, respectively. In addition to the first and second convolutional layers, convolutional layers of different combinations are stacked in sequence to form an initial model, so as to better understand the contextual information and relevance of the audio content. Finally, the output is composed of C neural network nodes, where C is the category of the audio label. When the audio label is divided into two categories (singing audio and non-singing audio), C is 2. When the audio label is divided into six categories (singing with music, a cappella, speaking with music, narration, pure music, and pure noise), C is 6.

[0085] Please refer to Figure 5 , Figure 5 A schematic diagram of a first convolution kernel and a second convolution kernel provided in an embodiment of the present application, wherein the rectangle marked with "frequency domain convolution layer" represents the convolution kernel of the frequency domain convolution layer, i.e., the first rectangular convolution kernel, and the rectangle marked with "time domain convolution layer" represents the convolution kernel of the time domain convolution layer, i.e., the second rectangular convolution kernel.

[0086] In the process of forward propagation, the audio feature F input Input the initial model individually or in batches, and perform matrix operations on the parameters of the initial model to output the vector Z = [z 1, z2,…,z i ], where z i Represents the output value of the i-th node. The softmax activation function is used to convert the above output value into a probability distribution P=[p1,p2,…,p i ],in:

[0087]

[0088] The C in the activation function is the number of output nodes, that is, the number of category labels.

[0089] S104: Generate a loss value using the training recognition results and the audio labels, and use the loss value to adjust the parameters of the initial model.

[0090] The application does not limit the specific calculation method of the loss value. For example, the loss value can be calculated using a cross-entropy loss function. Specifically, the probability p of each class is calculated by the cross-entropy loss function i The labeled category information (i.e., the audio label, which can be the six category labels described above or can be two audio labels) l i The sum of the entropy values is Loss softmax The labeled category information can be represented as L = [l1, l2, …, l i The back propagation process is a chain derivation from the model output loss value Loss softmax to the input F input , updating the parameters of the initial model. Wherein:

[0091]

[0092] After M times (M is not limited in size) of forward propagation and back propagation, the initial model gradually learns the commonalities and differences between song audio and non-song audio labels from the audio features.

[0093] S105: If it is detected that the preset completion condition is met, the initial model after parameter adjustment is determined as the song voice recognition model.

[0094] The preset completion condition is used to indicate that the initial model has been trained enough, and the number and specific content are not limited, for example, it can be a condition for limiting the recognition accuracy of the initial model, or it can be a condition for limiting the training time of the initial model, or it can be a condition for limiting the training rounds of the initial model. When one, a specified number of, or all preset completion conditions are met, the initial model after parameter adjustment can be determined as the song voice recognition model, indicating that the model training process is complete.

[0095] In the embodiments of the application, in order to ensure the recognition accuracy of the initial model after parameter adjustment in actual application, the training of the model can be divided into two parts: model parameter adjustment and model recognition accuracy verification. Therefore, after obtaining the training audio and the corresponding audio label, the training audio with the audio label can be divided into a training set and a verification set. After obtaining the training features, the training features with the audio label can also be divided into a training set and a verification set.

[0096] For example, all labeled data can be used to form a data set C total , and the data set can be randomly divided into a training set of C train samples and a verification set of C valThe verification set (i.e., verification data) of the sample can be divided in the division process to ensure that the number of audio of the six class labels (i.e., singing with music, a cappella, speaking with music, oral white, pure music, and pure noise) is as equal as possible.

[0097] The labeled data can be labeled training audio, and after obtaining the training audio and the corresponding audio label, the labeled training audio is divided into a training set and a verification set, and then the audio features of the training audio included in the training set and the audio features of the training audio included in the verification set are extracted according to the operation of S102.

[0098] The labeled data can also be labeled training features, and after the audio features of the training audio are extracted to obtain the training features, the labeled training features are divided into a training set and a verification set.

[0099] Based on the division of the training set and the verification set, the preset completion condition can include a preset training condition and a preset accuracy condition.

[0100] In the model training stage, the training features included in the training set can be input into the initial model to obtain training recognition results; the loss value is generated by using the training recognition results and the audio labels included in the training set, and the initial model is adjusted in parameters by using the loss value.

[0101] If it is detected that the preset training condition is met, the initial model after parameter adjustment can be verified for recognition accuracy by using the verification set.

[0102] The preset training condition can include a condition for limiting the training time of the initial model, or a condition for limiting the training rounds of the initial model, which is not limited herein.

[0103] If the recognition accuracy of the initial model after parameter adjustment does not meet the preset accuracy condition, the step of inputting the training features included in the training set into the initial model to obtain the training recognition results is returned; until the recognition accuracy of the initial model after parameter adjustment meets the preset accuracy condition, the initial model after parameter adjustment is determined as the singing recognition model.

[0104] The prediction accuracy condition can be a condition for limiting the recognition accuracy of the initial model, for example, the accuracy of the audio label recognized by the initial model reaches more than 90%.

[0105] By dividing the training set and the verification set, the recognition accuracy of the singing recognition model can be effectively ensured.

[0106] After the song recognition model is trained based on the training method described above, the model can be used to process audio. Specifically, an audio to be tested is obtained, and the length and specific content of the audio to be tested are not limited. The audio to be tested can be audio with song or audio without song, of course, it can be audio with noise or audio without noise. The audio to be tested can be audio obtained directly without processing or audio processed by dynamic range control (DRC). The audio features of the audio to be tested are extracted to obtain test features. It should be understood that the extraction method of the test features should be the same as the extraction method of the training features described above. The test features are input into the song recognition model to obtain a song recognition result.

[0107] Specifically, since the length of the audio to be tested can be long, in this case, the audio to be tested needs to be processed by slicing. If the length of the audio to be tested is short, it needs to be processed by zero padding. During feature extraction, the audio to be tested is first processed by feature extraction to obtain initial audio features. It should be understood that the feature extraction method is the same as the training process. A preset audio length is determined, and the initial audio features are processed by slicing and / or zero padding based on the preset audio length to obtain test features. It should be understood that the number of test features can be one or more, and the specific slicing method of the initial audio features can refer to the slicing method when the training audio is generated. In an embodiment, an audio signal with a single time duration dur obtained as the audio to be tested is processed by audio feature extraction to obtain initial audio features F input with a dimension of 148×N. The initial audio features F input are sliced in the time axis according to the audio frame length to meet the input requirement of the minimum length dur' of the song recognition model. If the width of the last slice is less than the audio frame length N', zero padding is performed to ensure that the feature dimensions of all audio slices after slicing are consistent, and finally an audio feature matrix F input ' with a dimension of W×148×N' is obtained, where W test features are included. The features F input ' are input into the trained song recognition model described above, and W output results are output, each of which corresponds to a test feature obtained after slicing. The content of the output result is related to the label content used during training. For example, when the six types of labels described above are used as audio labels, an output result is a probability distribution vector p w =[p0, p1, …, p i ] of the corresponding audio features in the six types of labels singing with music, a cappella, speaking with music, spoken word, pure music, and pure noise, where 0≤p i ≤1, 0≤i≤5, and i represents the number of category labels.

[0108] The specific form and quantity of the song recognition result are not limited. When the number of audio features is multiple, in an embodiment, the model output result corresponding to each audio feature can be taken as a song recognition result, indicating whether the to-be-detected audio is a song in the time period corresponding to the audio feature. In another embodiment, a certain post-processing step can be performed to obtain a song recognition result capable of representing whether the entire to-be-detected audio is a song. Specifically, if the to-be-detected feature is multiple, each to-be-detected feature can be input into the song recognition model to obtain a piecewise recognition result, and the piecewise recognition result can be fused to obtain a song recognition result. The piecewise recognition result is the model output result p w . The specific manner of fusion processing is not limited. For example, in one manner, the average probability of the entire audio can be calculated, and the song recognition result can be obtained based on the average probability. Based on the generation process of the F input , the average probability vector p mean =[p0’, p1’, …, p i ’] is obtained by taking the average of the W probability vectors longitudinally, where p i ’=p wi / W, and p wi represents the probability value of the i-th category of the w-th vector. Since singing with music and a cappella both correspond to the audio label of the song audio, the probabilities p0’ and p1’ in the average probability vector both represent the song, and thus the average probability p vocal =p0’+p1’ of the song can be determined. When the proportion of the actual song in the to-be-detected audio is larger, the value of p vocal is higher. A preset threshold can be set. When p vocal is greater than the preset threshold, the song recognition result is determined to be a song, otherwise, it is determined to be non-song.

[0109] In addition, the start and end points of the song in the to-be-tested audio can be located. Specifically, the start and end time ranges of each to-be-tested feature relative to the to-be-tested audio are determined, that is, the time range corresponding to the to-be-tested feature in the to-be-tested audio; the start and end time ranges are used to sort the recognition results of each segment to obtain a first sequence. Adjacent segment recognition results in the first sequence can be the same or different. Through same classification boundary fusion processing on the first sequence, the boundaries between the two adjacent segment recognition results in time are removed, small time segments are fused into large time segments, the boundaries between the two adjacent segment recognition results in time are retained, a second sequence is obtained, and it can be determined that the two sides of each boundary in the second sequence correspond to different types. Therefore, each boundary in the second sequence is the start point or end point of the song, which can be collectively referred to as an endpoint. Therefore, the song endpoints corresponding to the to-be-tested audio can be determined based on the second sequence.

[0110] Specifically, for W probability vectors p w , the start and end time points of the wth probability vector are t_start w = (w-1) × dur' and t_end w = w × dur', respectively, each start and end time point is taken as a minimum unit interval, and the two together constitute the start and end time range corresponding to the wth to-be-tested audio. The maximum value index w i of each probability vector is taken, which represents the maximum probability class label of the vector. Adjacent and same index values index w i are fused into a large time interval, so that class label intervals with different time lengths are obtained, and the start and end time points of each class label interval are the endpoints of the interval. In this way, the song endpoints at the frame level can be located and identified, and the division of the song region and the non-song region can be realized. Alternatively, in another implementation, the song region with music, the a cappella region, the song region with music, the spoken region, the pure music region and the pure noise region can be further subdivided.

[0111] It can be understood that the smaller dur' is, the more accurate the endpoint positioning is, and the smaller the granularity is. Please refer to Figure 6 , Figure 6 for a song start point positioning effect diagram provided by an embodiment of the present application. The upper half is the first sequence arranged in time order, and the length between every two adjacent dashed lines is dur'. After the same classification boundary fusion processing, the second sequence in the lower half is obtained.

[0112] Please refer toFigure 7 , Figure 7 A song recognition flowchart is provided in the embodiment of the present application. After obtaining an audio signal (i.e. the audio to be tested), the DRC processing can be performed thereon, which is more convenient for subsequent song recognition. The audio signal processed by the DRC is input into a feature extraction module to obtain the to-be-tested feature, and the to-be-tested feature is input into a song detection model (i.e. a song recognition model), and then the output result obtained by the song recognition model is input into a post-processing module for post-processing. The specific content of the post-processing is described previously.

[0113] The song recognition method provided in the embodiment of the present application uses special training data and a special initial model to realize the song recognition of the noisy audio, and has strong robustness and anti-interference ability. Specifically, the training audio including the noisy audio is set, so that the model can learn how to distinguish the song audio under the noise interference by using the training audio. The initial model includes a first convolution layer and a second convolution layer, and the convolution kernels of the two convolution layers are rectangular. The convolution kernel arranged along the frequency axis direction can obtain the frequency domain information in a larger frequency domain range, such as pitch and vocal range. The convolution kernel arranged along the time axis direction can obtain the time domain information in a larger time domain range, such as rhythm and melody. Through the first convolution layer and the second convolution layer, the initial model can obtain more information, which is helpful to resist the interference of the noise and realize accurate classification. The song recognition model obtained after the training can have strong anti-noise interference ability and can accurately distinguish the song.

[0114] The computer readable storage medium provided in the embodiment of the present application is introduced below, and the computer readable storage medium described below can be correspondingly referred to the song recognition model training method and / or the song recognition method described above.

[0115] The present application also provides a computer readable storage medium, and the computer program is stored on the computer readable storage medium. When the computer program is executed by the processor, the steps of the song recognition model training method and / or the song recognition method described above are realized.

[0116] The computer readable storage medium can include a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0117] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the related parts can be referred to the method part.

[0118] Those skilled in the art can further understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description in general terms. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0119] The steps of the method or algorithm described in combination with the embodiments disclosed herein can be directly implemented by hardware, a software module executed by a processor, or a combination of both. The software module can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0120] Finally, it should be noted that, in this document, relationships such as first and second are intended to distinguish one entity or operation from another, and do not necessarily require or imply any actual such relationship or order between the entities or operations. Moreover, the term including, comprising or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or apparatus including a series of elements includes not only those elements, but also other elements not explicitly listed, or other elements inherent in such a process, method, article or apparatus.

[0121] The principles and implementation modes of the present application are described in specific examples in this document, and the above example descriptions are only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in specific implementation modes and application ranges; based on the above, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A singing voice recognition model training method, characterized in that: include: Obtaining training audio and corresponding audio tags; wherein the training audio includes noisy audio interfered with by noise, and the audio tags are used to indicate whether the training audio is singing audio or non-singing audio; Extracting audio features of the training audio to obtain training features; Inputting the training features into an initial model to obtain a training recognition result; wherein the initial model includes a first convolutional layer and a second convolutional layer, the first convolutional layer and the second convolutional layer have rectangular convolution kernels, the long side of the first rectangular convolution kernel is arranged along the frequency axis direction, and the long side of the second rectangular convolution kernel is arranged along the time axis direction; Generating a loss value using the training recognition result and the audio label, and performing parameter adjustment processing on the initial model using the loss value; If it is detected that the preset completion conditions are met, the initial model after parameter adjustment is determined as the singing voice recognition model.

2. The singing voice recognition model training method according to claim 1, wherein The obtaining of training audio includes: Get the initial training audio; Performing dynamic range control processing on the initial training audio to obtain the training audio.

3. The singing voice recognition model training method according to claim 1, wherein The obtaining of training audio includes: Get the initial training audio; A preset audio length is determined, and based on the preset audio length, the initial training audio is subjected to segmentation processing and / or zero-padding processing to obtain the training audio.

4. The singing voice recognition model training method according to claim 1, wherein The audio tag generation process includes: Determining the audio category corresponding to the training audio; The audio tag is generated based on the audio category.

5. The singing voice recognition model training method according to claim 1, wherein The extracting audio features of the training audio to obtain training features includes: The training audio is subjected to a Mel spectrum extraction process and / or a Mel cepstral coefficient extraction process with an audio frame as a granularity to obtain the training features.

6. The singing voice recognition model training method according to claim 1, wherein After extracting the audio features of the training audio and obtaining the training features, the method further includes: Dividing the training features and corresponding audio labels into a training set and a validation set; Accordingly, inputting the training features into the initial model to obtain the training recognition results includes: Inputting the training features contained in the training set into the initial model to obtain a training recognition result; If it is detected that the preset completion condition is met, determining the initial model after parameter adjustment as the singing voice recognition model includes: If it is detected that the preset training conditions are met, the recognition accuracy of the initial model after parameter adjustment is verified using the verification set; If the recognition accuracy of the initial model after parameter adjustment does not meet the preset accuracy condition, then return to the step of inputting the training features contained in the training set into the initial model to obtain the training recognition result; until the recognition accuracy of the initial model after parameter adjustment meets the preset accuracy condition, the initial model after parameter adjustment is determined as the singing recognition model.

7. A singing voice recognition method, characterized in that: include: Get the audio to be tested; Extracting audio features of the audio to be tested to obtain features to be tested; The feature to be measured is input into a singing voice recognition model to obtain a singing voice recognition result; wherein, the singing voice recognition model is obtained based on the singing voice recognition model training method according to any one of claims 1 to 6.

8. The singing voice recognition method according to claim 7, wherein: The extracting the audio features of the audio to be measured to obtain the features to be measured includes: Performing feature extraction processing on the audio to be tested to obtain initial audio features; A preset audio length is determined, and based on the preset audio length, the initial audio feature is subjected to slicing processing and / or zero-padding processing to obtain the feature to be measured.

9. The singing voice recognition method according to claim 7, wherein: If there are multiple features to be measured, the features to be measured are input into a singing voice recognition model to obtain a singing voice recognition result, including: Inputting each of the features to be tested into the singing voice recognition model respectively to obtain a segmented recognition result; The segmented recognition results are fused to obtain the singing voice recognition result.

10. The singing voice recognition method according to claim 9, wherein: Also includes: Determining a start and end time range of each of the features to be measured relative to the audio to be measured; Sort the fragment recognition results by using the start and end time range to obtain a first sequence; Performing same classification boundary fusion processing on the first sequence to obtain a second sequence; The singing endpoint corresponding to the audio to be measured is determined based on the second sequence.

11. An electronic device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to implement the singing voice recognition model training method according to any one of claims 1 to 6, and / or the singing voice recognition method according to any one of claims 7 to 10.

12. A computer-readable storage medium, characterized in that Used to store a computer program, wherein when the computer program is executed by a processor, it implements the singing voice recognition model training method according to any one of claims 1 to 6, and / or the singing voice recognition method according to any one of claims 7 to 10.

Images