Song detection model training method, song detection method, device, and medium
By employing sound source separation and audio fusion methods, the accuracy of singing detection in neural network models was improved, the problem of a lack of high-quality training samples in neural network models was solved, and the performance of singing detection was enhanced.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PING AN TECH (SHENZHEN) CO LTD
- Filing Date
- 2023-05-19
- Publication Date
- 2026-04-28
AI Technical Summary
Existing neural network models lack high-quality training samples for singing detection, resulting in low detection accuracy.
By acquiring the original singing data of the singer, the background music audio is removed using a sound source separation model, the sample human voice spectrum is obtained, and after being fused with the reference audio, it is segmented and input into a neural network model for singing detection. The loss function is then optimized to train the neural network model.
This improved the accuracy of the singing detection model, enhanced its ability to learn lyrics, and trained a more robust singing detection model.
Smart Images
Figure CN116564278B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a training method, singing detection method, device and medium for a singing detection model. Background Technology
[0002] In the process of singing detection, it is often necessary to detect the parts of a piece of music that contain lyrics. Most commonly used singing detection methods rely on neural network models. However, current neural network models lack a large number of high-quality training samples for training, resulting in insufficient accuracy in sound detection. Therefore, how to improve the accuracy of singing detection models has become an urgent technical problem to be solved. Summary of the Invention
[0003] The main objective of this application is to propose a training method, a singing detection method, a device, and a medium for a singing detection model, aiming to improve the accuracy of the singing detection model in detecting singing.
[0004] To achieve the above objectives, a first aspect of this application proposes a training method for a singing voice detection model, the training method comprising:
[0005] Obtain the original singing data of the singing object, wherein the original singing data includes background music audio without lyrics and sample vocal spectrograms containing lyrics;
[0006] Based on a preset sound source separation model, the original singing data is separated into sound sources, the background music audio is removed, and the sample human voice spectrum is obtained.
[0007] The sample human voice spectrum and the preset reference audio are fused to obtain sample audio data, wherein the reference audio is audio data without lyrics;
[0008] The sample audio data is segmented to obtain multiple sample audio segments;
[0009] The sample audio segments are input into a preset neural network model for singing detection to obtain a sample singing time series of the sample audio data, which includes the time periods in the sample audio data where lyrics are present.
[0010] The loss function of the neural network model is optimized based on the sample singing time series to train the neural network model and obtain a singing detection model.
[0011] In some embodiments, the step of separating the original singing data based on a preset sound source separation model, removing the background music audio, and obtaining the sample vocal spectrogram includes:
[0012] The original singing data is segmented based on the sound source separation model to obtain multiple original audio segments.
[0013] Based on the sound source separation model, the original audio segments are classified to obtain the sound source type of each original audio segment;
[0014] Based on the sound source type, the sample vocal spectra are separated from the original singing data.
[0015] In some embodiments, the step of audio fusion of the sample human voice spectrum and a preset reference audio to obtain sample audio data includes:
[0016] The sample human voice spectrum is extracted to obtain sample audio track data, and the reference audio is extracted to obtain reference audio track data.
[0017] The sample audio data is obtained by adding the sample audio track data and the reference audio track data together.
[0018] In some embodiments, the step of inputting the sample audio segment into a preset neural network model for singing detection to obtain a sample singing time series of the sample audio data includes:
[0019] The sample audio segment is input into the neural network model, which includes a spectrum transform layer, a convolutional layer, an LSTM layer, a flattening layer, and a fully connected layer.
[0020] The sample audio segment is subjected to spectral transformation by the spectral transformation layer to obtain the sample Mel-frequency cepstral features;
[0021] The sample audio features are obtained by extracting the Mel-Cepstral features of the sample through the convolutional layer;
[0022] The LSTM layer is used to extract contextual information from each sample audio feature to obtain intermediate audio features.
[0023] The intermediate audio features are smoothed by the flattening layer to obtain variable-dimensional audio features;
[0024] The variable-dimensional audio features are predicted by the fully connected layer to obtain the probability value of lyrics for each sample audio segment.
[0025] The sample audio segments are filtered according to the probability values to obtain sample singing time periods, and the sample singing time series is obtained according to the sample singing time periods.
[0026] In some embodiments, before performing sound source separation on the original singing data based on a preset sound source separation model, removing the background music audio, and obtaining the sample vocal spectrogram, the method further includes pre-training the sound source separation model, specifically including:
[0027] Obtain training audio files, wherein each training audio file includes a first audio track file and a second audio track file, the first audio track file includes first audio track data containing lyrics and second audio track data not containing lyrics, and the second audio track file includes only the first audio track data;
[0028] Input the first audio track file and the second audio track file into the sound source separation model;
[0029] The first audio track file is segmented based on the sound source separation model to obtain multiple audio track segments.
[0030] Based on the sound source separation model, the audio track segment is masked to obtain multiple masked spectrograms. The multiple masked spectrograms are then spliced together to obtain a predicted spectrogram.
[0031] Based on the sound source separation model, the second audio track file is mapped to a preset vector space to obtain the target spectrogram;
[0032] Based on the predicted spectrogram and the target spectrogram, the model loss value is calculated, and the parameters of the sound source separation model are optimized according to the model loss value to update the sound source separation model.
[0033] To achieve the above objectives, a second aspect of this application proposes a singing voice detection method, the method comprising:
[0034] Obtain the target singing data;
[0035] The target singing data is input into the singing detection model for singing detection to obtain the target singing time series of the target audio data, wherein the target singing time series includes the time periods in the target singing data where lyrics are present, and the singing detection model is trained according to the training method described in the first aspect.
[0036] To achieve the above objectives, a third aspect of this application provides a training apparatus for a singing voice detection model, the training apparatus comprising:
[0037] The original singing acquisition module is used to acquire original singing data, wherein the original singing data includes background music audio without lyrics and sample vocal spectrograms containing lyrics;
[0038] The sound source separation module is used to separate the sound sources of the original singing data based on a preset sound source separation model, remove the background music audio, and obtain the sample human voice spectrum.
[0039] The audio fusion module is used to fuse the sample human voice spectrum and the preset reference accompaniment audio to obtain sample audio data, wherein the reference audio is audio data without lyrics.
[0040] The audio segmentation module is used to segment the sample audio data to obtain multiple sample audio segments;
[0041] The sample singing detection module is used to input the sample audio segment into a preset neural network model to detect singing, and obtain a sample singing time series of the sample audio data, wherein the sample singing time series includes the time periods in the sample audio data where lyrics are present.
[0042] The model training module is used to optimize the loss function of the neural network model based on the sample singing time series in order to train the neural network model and obtain the singing detection model.
[0043] To achieve the above objectives, a fourth aspect of this application provides a singing voice detection device, the device comprising:
[0044] The target vocal performance acquisition module is used to acquire target vocal performance data;
[0045] The target singing voice detection module is used to input the target singing data into the singing voice detection model for singing voice detection, and obtain the target singing voice time series of the target audio data, wherein the target singing voice time series includes the time periods in the target singing data where lyrics are present, and the singing voice detection model is trained according to the training device as described in the third aspect.
[0046] To achieve the above objectives, a fifth aspect of the present application provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first aspect or the method described in the second aspect.
[0047] To achieve the above objectives, a sixth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect or the method described in the second aspect.
[0048] This application proposes a method for training a singing voice detection model, a method for detecting singing voices, a training device for a singing voice detection model, a singing voice detection device, an electronic device, and a computer-readable storage medium. It acquires the original singing data of the performer; based on a preset sound source separation model, it performs sound source separation on the original singing data, removes background music audio, and obtains a sample vocal spectrogram. This sample vocal spectrogram serves as the basis for singing voice detection, effectively training the model's performance in singing voice detection and improving the model's ability to learn from the parts of the sample vocal spectrogram containing lyrics. Furthermore, it fuses the sample vocal spectrogram with a preset reference audio to obtain sample audio data. The reference audio is audio data without lyrics, which can effectively increase the number of training samples (i.e., sample audio segments) to train a more robust singing voice detection model. Further, it segments the sample audio data to obtain multiple sample audio segments, and inputs these sample audio segments into a preset neural network model for singing voice detection, obtaining a sample singing voice time series. This sample singing voice time series includes the time periods in the sample audio data where lyrics are present, allowing for convenient prediction of whether lyrics exist in the sample audio segments using the neural network model. Finally, the loss function of the neural network model is optimized based on the time series of sample singing voices, and the neural network model is trained to obtain a singing voice detection model, which can significantly improve the accuracy of singing voice detection. Attached Figure Description
[0049] Figure 1 This is a flowchart of the training method for the singing detection model provided in the embodiments of this application;
[0050] Figure 2 This is another flowchart of the training method for the singing detection model provided in the embodiments of this application;
[0051] Figure 3 yes Figure 1 The flowchart of step S102 in the document;
[0052] Figure 4 yes Figure 1 The flowchart of step S103 in the process;
[0053] Figure 5 yes Figure 1 The flowchart of step S105 in the process;
[0054] Figure 6 This is a flowchart of the singing voice detection method provided in the embodiments of this application;
[0055] Figure 7 This is a schematic diagram of the structure of the training device for the singing detection model provided in the embodiments of this application;
[0056] Figure 8This is a schematic diagram of the structure of the singing detection device provided in the embodiments of this application;
[0057] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0058] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0059] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.
[0061] First, let's analyze some of the terms used in this application:
[0062] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.
[0063] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.
[0064] Information Extraction (NER) is a text processing technique that extracts factual information such as entities, relationships, and events from natural language text and outputs it as structured data. Information extraction is a technique for extracting specific information from text data. Text data is composed of specific units, such as sentences, paragraphs, and chapters. Text information is composed of smaller, specific units, such as characters, words, phrases, sentences, paragraphs, or combinations of these units. Extracting noun phrases, names of people, and place names from text data is an example of text information extraction. Of course, text information extraction techniques can extract information of various types.
[0065] In the process of singing detection, it is often necessary to detect the parts of a piece of music that contain lyrics. Most commonly used singing detection methods rely on neural network models. However, current neural network models lack a large number of high-quality training samples for training, resulting in insufficient accuracy in sound detection. Therefore, how to improve the accuracy of singing detection models has become an urgent technical problem to be solved.
[0066] Based on this, embodiments of this application provide a training method for a singing voice detection model, a singing voice detection method, a training device for a singing voice detection model, a singing voice detection device, an electronic device, and a computer-readable storage medium, aiming to improve the accuracy of the singing voice detection model in detecting singing voices.
[0067] The singing detection model training method, singing detection method, singing detection model training device, singing detection device, electronic device, and computer-readable storage medium provided in this application are specifically described through the following embodiments. First, the singing detection method in the embodiments of this application is described.
[0068] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.
[0069] Foundational technologies in artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.
[0070] The singing detection model training method or singing detection method provided in this application relates to the field of artificial intelligence technology. The singing detection model training method or singing detection method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the singing detection model training method or singing detection method, but is not limited to the above forms.
[0071] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0072] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user voice data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.
[0073] Figure 1 This is an optional flowchart of the training method for the singing voice detection model provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S106.
[0074] Step S101: Obtain the original singing data of the singing object, wherein the original singing data includes background music audio without lyrics and sample vocal spectrograms containing lyrics;
[0075] Step S102: Based on the preset sound source separation model, the original singing data is separated into sound sources, the background music audio is removed, and the sample human voice spectrum is obtained.
[0076] Step S103: Perform audio fusion on the sample human voice spectrum and the preset reference audio to obtain sample audio data. The reference audio is audio data that does not contain lyrics.
[0077] Step S104: The sample audio data is segmented to obtain multiple sample audio segments;
[0078] Step S105: Input the sample audio segment into the preset neural network model to detect the singing voice, and obtain the sample singing voice time series of the sample audio data. The sample singing voice time series includes the time period in the sample audio data where lyrics are present.
[0079] Step S106: Optimize the loss function of the neural network model based on the sample singing time series to train the neural network model and obtain the singing detection model.
[0080] Steps S101 to S106 of this embodiment involve acquiring the original singing data of the performer; performing sound source separation on the original singing data based on a preset sound source separation model, removing background music audio, and obtaining a sample vocal spectrogram. This sample vocal spectrogram serves as the basis for vocal detection, effectively training the model's performance in vocal detection and improving its ability to learn from segments containing lyrics in the sample vocal spectrogram. Further, audio fusion is performed on the sample vocal spectrogram and a preset reference audio to obtain sample audio data. The reference audio is audio data without lyrics, which effectively increases the number of training samples (i.e., sample audio segments) to train a more robust vocal detection model. Further, the sample audio data is segmented to obtain multiple sample audio segments, which are then input into a preset neural network model for vocal detection, resulting in a sample vocal time series. This sample vocal time series includes the time periods in the sample audio data where lyrics are present, allowing for convenient prediction of the presence of lyrics in sample audio segments using the neural network model. Finally, the loss function of the neural network model is optimized based on the time series of sample singing voices, and the neural network model is trained to obtain a singing voice detection model, which can significantly improve the accuracy of singing voice detection.
[0081] In step S101 of some embodiments, a web crawler can be written, and after setting up a data source, data can be crawled in a targeted manner to obtain the original singing data of the performer. The data source can be various types of online platforms, social media, or specific audio databases, etc., and the original singing data can be the musical material of the performer. Original singing data can also be obtained through other methods, not limited to these. The original singing data includes background music audio without lyrics and sample vocal spectrograms containing lyrics.
[0082] Please see Figure 2 In some embodiments, prior to step S102, the training method for the singing detection model further includes pre-training a sound source separation model, which may include, but is not limited to, steps S201 to S206:
[0083] Step S201: Obtain training audio files, wherein each training audio file includes a first audio track file and a second audio track file. The first audio track file includes first audio track data containing lyrics and second audio track data not containing lyrics. The second audio track file only includes the first audio track data.
[0084] Step S202: Input the first audio track file and the second audio track file into the sound source separation model;
[0085] Step S203: The first audio track file is segmented based on the sound source separation model to obtain multiple audio track segments;
[0086] Step S204: Based on the sound source separation model, the audio track segment is masked to obtain multiple masked spectrograms, and the multiple masked spectrograms are spliced together to obtain the predicted spectrogram.
[0087] Step S205: Based on the sound source separation model, the second audio track file is mapped to a preset vector space to obtain the target spectrogram;
[0088] Step S206: Based on the predicted spectrum and the target spectrum, calculate the model loss value, and optimize the parameters of the sound source separation model according to the model loss value to update the sound source separation model.
[0089] In step S201 of some embodiments, training audio files can be obtained by writing a web crawler, setting up a data source, and then selectively crawling data. Alternatively, training audio files can be obtained through database extraction or downloading from a web platform, and are not limited to these methods. For example, training audio files can be obtained using audio processing tools such as MUSDB, Bean, and Catalog. Each training audio file includes a first audio track file and a second audio track file. The first audio track file includes first audio track data containing lyrics and second audio track data without lyrics, while the second audio track file only includes the first audio track data.
[0090] Taking a training audio file from the MUSDB audio processing tool as an example, a training audio file includes 5 sub-files, namely a mixed audio file, a drum sound track file, a bass sound track file, a vocal sound track file, and other audio track files. The mixed audio file is used as the first audio track file. This first audio track file (i.e., the mixed audio file) contains multiple sound source data such as drum sounds, bass sounds, and vocals. The vocal sound track file is used as the second audio track file. This second audio track file only contains vocals, that is, the data of the singer singing the lyrics.
[0091] In step S202 of some embodiments, the first audio track file and the second audio track file are input into the sound source separation model using a computer program or script program. The sound source separation model includes a voice splitter, a segmentation layer, and a U-Net network.
[0092] In step S203 of some embodiments, the first audio track file is divided into multiple audio track segments of fixed duration by a segmentation layer, and these fixed duration audio track segments are used as training units. The fixed duration can be set according to actual needs and is not limited. For example, the first audio track file is divided into 11.88s audio track segments by a segmentation layer.
[0093] In step S204 of some embodiments, the U-Net network is used to perform source masking on each audio track segment. The part of each audio track segment where lyrics appear (i.e., the human voice part) is marked as the first mask, and the part of the audio track segment where no lyrics appear (i.e., the non-human voice part) is marked as the second mask. The audio track segments are then masked to filter out the non-human voice parts of the audio track segments, resulting in multiple mask spectrograms. The multiple mask spectrograms are then spliced together according to the chronological order to obtain the predicted spectrogram.
[0094] In step S205 of some embodiments, the second audio track file is mapped to a preset vector space through the U-Net network, realizing the transformation of the second audio track file from the spectrum space to the vector space, and obtaining the target spectrogram, wherein the target spectrogram and the predicted spectrogram are in the same vector dimension.
[0095] In step S206 of some embodiments, when calculating the model loss value based on the predicted spectrogram and the target spectrogram, the norm between the target spectrogram and the predicted spectrogram can be calculated and used as the model loss value. The source separation model is then optimized based on the model loss value, and the source separation model is updated so that the predicted spectrogram of the first audio track file (i.e., the masked first audio track file) is as close as possible to the target spectrogram of the second audio track file (i.e., the second audio track file). The source separation model that makes the masked first audio track file and the second audio track file the closest is taken as the final source separation model.
[0096] Through the above steps S201 to S206, the sound source separation model can be trained relatively easily, so that the trained sound source separation model has better sound source separation performance, can more accurately separate the human voice part in the mixed audio, and improve the sound source separation accuracy of the model.
[0097] Please see Figure 3 In some embodiments, the sound source separation model includes a voice separator, a segmentation layer, and a U-Net network. Step S102 may include, but is not limited to, steps S301 to S303:
[0098] Step S301: The original singing data is segmented based on the sound source separation model to obtain multiple original audio segments;
[0099] Step S302: Based on the sound source separation model, classify the original audio segments by sound source to obtain the sound source type of each original audio segment;
[0100] Step S303: Based on the sound source type, separate the sample vocal spectra from the original singing data.
[0101] In step S301 of some embodiments, the original singing data is segmented by a segmentation layer to obtain multiple original audio segments of fixed duration. The fixed duration can be set according to actual needs and is not limited. For example, the original singing data can be segmented into 10-second original audio segments by the segmentation layer.
[0102] In step S302 of some embodiments, the original audio segments are identified by a voice separator and a U-Net network to determine that each original audio segment includes a human voice part and various instrument sounds, thereby obtaining the sound source type of each original audio segment. If the original audio segment is identified as a human voice audio, it is considered to be a human voice singing type. If the original audio segment is identified as various instrument audio, it is considered to be an instrument accompaniment type.
[0103] In step S303 of some embodiments, the original audio segment belonging to the human voice singing type is used as the sample human voice spectrum according to the different sound source types.
[0104] Through the above steps S301 to S303, the vocal parts containing lyrics in the original audio data can be separated relatively easily and used as sample vocal spectra. Using the sample vocal spectra as the basis for singing detection, the model's performance in singing detection can be effectively trained, and the model's ability to learn the part of the sample vocal spectra containing lyrics can be improved, so that the model can better detect the lyrics in the singing and improve the accuracy of the model's singing detection.
[0105] Please see Figure 4 In some embodiments, step S103 may include, but is not limited to, steps S401 to S402:
[0106] Step S401: Extract the signal from the sample human voice spectrum to obtain sample audio track data, and extract the signal from the reference audio to obtain reference audio track data;
[0107] Step S402: Add the sample audio track data and the reference audio track data together to obtain the sample audio data.
[0108] In step S401 of some embodiments, signal extraction can be performed directly on the sample human voice spectrum to obtain the audio signal in the sample human voice spectrum, and this audio signal can be used as sample audio track data; similarly, signal extraction can be performed directly on the reference audio to obtain the reference audio signal in the reference audio, and this reference audio signal can be used as reference audio track data. It should be noted that the reference audio is audio data that does not contain lyrics. For example, the reference audio can be pure music, that is, the reference audio can be formed by playing natural instruments and / or playing electronic instruments. Natural instruments include piano, violin, guitar, etc., and the reference audio played by electronic instruments can be formed by heavy metal sound or other methods, and is not limited to these.
[0109] In step S402 of some embodiments, the sample audio track data and the reference audio track data are added together. This signal addition process can be to directly add the two different audio tracks from the sample audio track data and the reference audio track data to obtain multiple sample audio data.
[0110] Through the above steps S401 to S402, the audio fusion of sample human voice spectra and reference audio can be achieved relatively easily. A preset audio signal (e.g., reference audio including instrumental music) is added to the relatively clean human voice audio separated from the original audio data, which can better increase more training samples (i.e. sample audio segments) so as to train a singing voice detection model with better robustness.
[0111] In step S104 of some embodiments, the sample audio data is segmented according to a preset audio segment length to obtain multiple sample audio segments. The audio segments can be determined according to actual business needs and are not limited. For example, to improve detection efficiency, the preset audio segment length can be 600ms, that is, the sample audio data can be segmented into multiple 600ms long sample audio segments.
[0112] Please see Figure 5 In some embodiments, step S105 may include, but is not limited to, steps S501 to S507:
[0113] Step S501: Input the sample audio segment into the neural network model. The neural network model includes a spectrum transform layer, a convolutional layer, an LSTM layer, a flattening layer, and a fully connected layer.
[0114] Step S502: Perform spectral transformation on the sample audio segment through the spectral transformation layer to obtain the sample Mel-frequency cepstral features;
[0115] Step S503: Extract the sample audio features by performing Mel-Cepstral features on the sample through a convolutional layer;
[0116] Step S504: Extract contextual information from the audio features of each sample using an LSTM layer to obtain intermediate audio features;
[0117] Step S505: Smooth the intermediate audio features through a flattening layer to obtain variable-dimensional audio features;
[0118] Step S506: The variable-dimensional audio features are predicted and processed by a fully connected layer to obtain the probability value of lyrics in each audio segment.
[0119] Step S507: Filter the sample audio segments according to the probability values to obtain the sample singing time periods, and obtain the sample singing time series according to the sample singing time periods.
[0120] In step S501 of some embodiments, a sample audio segment is input into a neural network model using a computer program or script program. The neural network model includes a spectrum transform layer, a convolutional layer, an LSTM layer, a flattening layer, and a fully connected layer.
[0121] In step S502 of some embodiments, the sample audio segment is first transformed from the time domain to the frequency domain by a spectrum transform layer to obtain a sample spectrum. Then, the sample spectrum is filtered by a Mel-Cepstral Spectrum Filter of the spectrum transform layer. Specifically, the sample spectrum features are first logarithmically processed to obtain the sample logarithmic spectrum, and then an inverse Fourier transform is performed on the sample logarithmic spectrum to obtain the sample Mel-Cepstral Spectrum. Finally, the sample Mel-Cepstral Spectrum features are extracted to obtain the sample Mel-Cepstral Spectrum Features.
[0122] In step S503 of some embodiments, feature extraction is performed on the sample Mel-Cepstral features through a convolutional layer to capture the lyric feature information in the sample Mel-Cepstral features and obtain the sample audio features. The convolutional layer can be constructed according to a convolutional neural network model, and the convolutional kernel of the convolutional layer can be 3×3.
[0123] In step S504 of some embodiments, context information is extracted for each sample audio feature through an LSTM layer to obtain the spectral feature information of the adjacent sample audio segments of the sample audio segment corresponding to the current sample audio feature. Based on the obtained spectral feature information of the adjacent sample audio segments, information is added to the feature content of the current sample audio feature to obtain the intermediate audio feature.
[0124] In step S505 of some embodiments, the intermediate audio features are smoothed by a flattening layer, and the multidimensional intermediate audio features are converted into one-dimensional features to obtain variable-dimensional audio features.
[0125] In step S506 of some embodiments, the variable-dimensional audio features are predicted using the prediction function of the fully connected layer. The prediction function can be a softmax function or a sigmaod function, etc. Taking the softmax function as an example, a binary classification method is used to predict whether the variable-dimensional audio features are segments with lyrics. The probability value of whether the variable-dimensional audio features have lyrics is represented as a probability value, and the probability value of each sample audio segment having lyrics is obtained.
[0126] In step S507 of some embodiments, sample audio segments are filtered according to probability values, and sample audio segments with probability values greater than a preset threshold are selected as sample singing time periods. All sample singing time periods are included in the same set to obtain a sample singing time series.
[0127] Through the above steps S501 to S507, the neural network model can be used to predict whether there are lyrics in the sample audio segments, and the sample audio segments with lyrics are aggregated into a sample singing time series, realizing the accurate detection of the time when lyrics exist in the audio by the neural network model.
[0128] In step S106 of some embodiments, the loss function can be the cross-entropy loss function. The loss value between the sample singing time series and the time series of lyrics that actually exist in the original audio data is calculated by the cross-entropy loss function. The loss function of the neural network model is optimized based on the loss value, and the neural network model is trained so that the sample singing time series and the time series of lyrics that actually exist in the original audio data are as close as possible, thus obtaining the singing detection model.
[0129] The training method for the singing detection model in this application embodiment obtains the original singing data of the singer; based on a preset sound source separation model, it performs sound source separation on the original singing data, removes background music audio, and obtains sample vocal spectrograms. These sample vocal spectrograms serve as the basis for singing detection, effectively training the model's performance in singing detection and improving its ability to learn from samples containing lyrics. Furthermore, it performs audio fusion between the sample vocal spectrograms and a preset reference audio to obtain sample audio data. The reference audio is audio data without lyrics, which effectively increases the number of training samples (i.e., sample audio segments) to train a more robust singing detection model. Further, it segments the sample audio data to obtain multiple sample audio segments, and inputs these segments into a preset neural network model for singing detection, obtaining a sample singing time series. This sample singing time series includes the time periods in the sample audio data where lyrics are present, allowing for convenient prediction of whether lyrics exist in the sample audio segments using the neural network model. Finally, the loss function of the neural network model is optimized based on the time series of sample singing voices, and the neural network model is trained to obtain a singing voice detection model, which can significantly improve the accuracy of singing voice detection.
[0130] Figure 6 This is an optional flowchart of the singing voice detection method provided in the embodiments of this application. Figure 6 The method may include, but is not limited to, steps S601 to S602.
[0131] Step S601: Obtain target singing data;
[0132] Step S602: Input the target singing data into the singing detection model to detect the singing voice and obtain the target singing voice time series of the target audio data. The target singing voice time series includes the time periods in the target singing data where lyrics are present. The singing voice detection model is trained according to the training method described above.
[0133] In step S601 of some embodiments, the target singing data can be obtained by writing a web crawler, setting up a data source, and then crawling data in a targeted manner. Alternatively, the target singing data can be obtained through other means such as database extraction or downloading from a network platform.
[0134] In step S602 of some embodiments, the target singing data is input into the singing detection model. The singing detection model segments the target singing data into multiple target singing segments, converts each target singing segment into a corresponding target Mel-Cepstral feature, and extracts the lyrics content information from the target Mel-Cepstral feature. The target Mel-Cepstral feature after feature extraction is subjected to contextual information extraction and smoothing to obtain a one-dimensional audio feature. Finally, a fully connected layer is used to predict whether the target singing segment corresponding to the one-dimensional audio feature has lyrics. That is, if the predicted value of the target singing segment exceeds the threshold, it is determined that there are lyrics, i.e., there is human singing. If the predicted value of the target singing segment is less than the threshold, it is determined that there are no lyrics, i.e., there is no human singing. Finally, the discrimination results of the target singing segments are spliced together in chronological order to obtain the target singing time series. Thus, based on the target singing time series, it is determined which time periods in the entire target singing data have lyrics (i.e., there is human singing).
[0135] The singing detection method of this application uses a singing detection model to identify the time periods in the target singing data where lyrics (i.e., human voices) are present, thereby improving detection accuracy and efficiency.
[0136] Please see Figure 7 This application also provides a training device for a singing voice detection model, which can implement the above-described training method for the singing voice detection model. The training device includes:
[0137] The original singing acquisition module 701 is used to acquire original singing data, which includes background music audio without lyrics and sample vocal spectrograms containing lyrics.
[0138] The sound source separation module 702 is used to separate the sound sources of the original singing data based on a preset sound source separation model, remove the background music audio, and obtain the sample human voice spectrum.
[0139] The audio fusion module 703 is used to fuse the sample human voice spectrogram and the preset reference accompaniment audio to obtain sample audio data. The reference audio is audio data without lyrics.
[0140] The audio segmentation module 704 is used to segment the sample audio data to obtain multiple sample audio segments;
[0141] The sample singing detection module 705 is used to input sample audio segments into a preset neural network model for singing detection, and obtain a sample singing time series of sample audio data. The sample singing time series includes the time periods in the sample audio data where lyrics are present.
[0142] The model training module 706 is used to optimize the loss function of the neural network model based on the time series of sample singing voices in order to train the neural network model and obtain the singing voice detection model.
[0143] The specific implementation of the training device for the singing voice detection model is basically the same as the specific implementation of the training method for the singing voice detection model described above, and will not be repeated here.
[0144] Please see Figure 8 This application also provides a singing voice detection device that can implement the above-described singing voice detection method. The device includes:
[0145] The target singing acquisition module 801 is used to acquire target singing data;
[0146] The target singing voice detection module 802 is used to input the target singing data into the singing voice detection model for singing voice detection, and obtain the target singing voice time series of the target audio data. The target singing voice time series includes the time periods in the target singing data where lyrics are present. The singing voice detection model is trained according to the training device mentioned above.
[0147] The specific implementation method of the singing detection device is basically the same as the specific implementation method of the singing detection method described above, and will not be repeated here.
[0148] This application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the training method or the singing detection method of the aforementioned singing detection model. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.
[0149] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:
[0150] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.
[0151] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 to execute the training method or singing detection method of the singing detection model in the embodiments of this application.
[0152] The input / output interface 903 is used to implement information input and output;
[0153] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).
[0154] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);
[0155] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.
[0156] This application also provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the above-described training method or singing detection method for the singing detection model.
[0157] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0158] The singing detection model training method, singing detection method, singing detection model training device, singing detection device, electronic device, and computer-readable storage medium provided in this application embodiment acquire the original singing data of the singing object; perform sound source separation on the original singing data based on a preset sound source separation model, remove the background music audio, and obtain the sample human voice spectrogram. This sample human voice spectrogram can be used as the basis for singing detection, effectively training the model's performance in singing detection and improving the model's ability to learn the parts of the sample human voice spectrogram containing lyrics. Furthermore, audio fusion is performed on the sample human voice spectrogram and a preset reference audio to obtain sample audio data. The reference audio is audio data without lyrics, which can better increase the number of training samples (i.e., sample audio segments) to train a more robust singing detection model. Furthermore, the sample audio data is segmented to obtain multiple sample audio segments. These segments are then input into a pre-defined neural network model for singing detection, resulting in a sample singing time series. This time series includes the time periods in the sample audio data where lyrics are present, making it easier to predict the presence of lyrics in the sample audio segments using the neural network model. Finally, the loss function of the neural network model is optimized based on the sample singing time series, and the model is trained to obtain a singing detection model that significantly improves the accuracy of singing detection.
[0159] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.
[0160] It will be understood by those skilled in the art that Figure 1-6 The technical solutions shown do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.
[0161] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0162] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or appropriate combinations thereof.
[0163] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0164] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0165] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0166] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0167] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0168] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0169] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.
Claims
1. A training method for a singing voice detection model, characterized in that, The training method includes: Obtain the original singing data of the singing object, wherein the original singing data includes background music audio without lyrics and sample vocal spectrograms containing lyrics; Based on a preset sound source separation model, the original singing data is separated into sound sources, the background music audio is removed, and the sample human voice spectrum is obtained. The sample human voice spectrum and the preset reference audio are fused to obtain sample audio data, wherein the reference audio is audio data without lyrics; The sample audio data is segmented to obtain multiple sample audio segments; The sample audio segments are input into a preset neural network model for singing detection to obtain a sample singing time series of the sample audio data, which includes the time periods in the sample audio data where lyrics are present. The loss function of the neural network model is optimized based on the sample singing time series to train the neural network model and obtain a singing detection model.
2. The training method according to claim 1, characterized in that, The process of separating the original singing data based on a preset sound source separation model, removing the background music audio, and obtaining the sample vocal spectrogram includes: The original singing data is segmented based on the sound source separation model to obtain multiple original audio segments. Based on the sound source separation model, the original audio segments are classified to obtain the sound source type of each original audio segment; Based on the sound source type, the sample vocal spectra are separated from the original singing data.
3. The training method according to claim 1, characterized in that, The step of fusing the sample human voice spectrum with a preset reference audio to obtain sample audio data includes: The sample human voice spectrum is extracted to obtain sample audio track data, and the reference audio is extracted to obtain reference audio track data. The sample audio data is obtained by adding the sample audio track data and the reference audio track data together.
4. The training method according to claim 1, characterized in that, The step of inputting the sample audio segment into a preset neural network model for singing detection to obtain a sample singing time series of the sample audio data includes: The sample audio segment is input into the neural network model, which includes a spectrum transform layer, a convolutional layer, an LSTM layer, a flattening layer, and a fully connected layer. The sample audio segment is subjected to spectral transformation by the spectral transformation layer to obtain the sample Mel-frequency cepstral features; The sample audio features are obtained by extracting the Mel-Cepstral features of the sample through the convolutional layer; The LSTM layer is used to extract contextual information from each sample audio feature to obtain intermediate audio features. The intermediate audio features are smoothed by the flattening layer to obtain variable-dimensional audio features; The variable-dimensional audio features are predicted by the fully connected layer to obtain the probability value of lyrics for each sample audio segment. The sample audio segments are filtered according to the probability values to obtain sample singing time periods, and the sample singing time series is obtained according to the sample singing time periods.
5. The training method according to any one of claims 1 to 4, characterized in that, Before performing sound source separation on the original singing data based on the preset sound source separation model, removing the background music audio, and obtaining the sample vocal spectrogram, the method further includes pre-training the sound source separation model, specifically including: Obtain training audio files, wherein each training audio file includes a first audio track file and a second audio track file, the first audio track file includes first audio track data containing lyrics and second audio track data not containing lyrics, and the second audio track file includes only the first audio track data; Input the first audio track file and the second audio track file into the sound source separation model; The first audio track file is segmented based on the sound source separation model to obtain multiple audio track segments. Based on the sound source separation model, the audio track segment is masked to obtain multiple masked spectrograms. The multiple masked spectrograms are then spliced together to obtain a predicted spectrogram. Based on the sound source separation model, the second audio track file is mapped to a preset vector space to obtain the target spectrogram; Based on the predicted spectrogram and the target spectrogram, the model loss value is calculated, and the parameters of the sound source separation model are optimized according to the model loss value to update the sound source separation model.
6. A method for detecting singing voice, characterized in that, The method includes: Obtain the target singing data; The target singing data is input into the singing detection model for singing detection to obtain the target singing time series of the target singing data, wherein the target singing time series includes the time periods in the target singing data where lyrics are present, and the singing detection model is trained according to the training method described in any one of claims 1 to 5.
7. A training device for a singing voice detection model, characterized in that, The training device includes: The original singing acquisition module is used to acquire original singing data, wherein the original singing data includes background music audio without lyrics and sample vocal spectrograms containing lyrics; The sound source separation module is used to separate the sound sources of the original singing data based on a preset sound source separation model, remove the background music audio, and obtain the sample human voice spectrum. An audio fusion module is used to fuse the sample human voice spectrogram with a preset reference audio to obtain sample audio data, wherein the reference audio is audio data without lyrics. The audio segmentation module is used to segment the sample audio data to obtain multiple sample audio segments; The sample singing detection module is used to input the sample audio segment into a preset neural network model to detect singing, and obtain a sample singing time series of the sample audio data, wherein the sample singing time series includes the time periods in the sample audio data where lyrics are present. The model training module is used to optimize the loss function of the neural network model based on the sample singing time series in order to train the neural network model and obtain the singing detection model.
8. A singing voice detection device, characterized in that, The device includes: The target vocal performance acquisition module is used to acquire target vocal performance data; The target singing voice detection module is used to input the target singing data into the singing voice detection model for singing voice detection, and obtain the target singing voice time series of the target singing data, wherein the target singing voice time series includes the time periods in the target singing data where lyrics exist, and the singing voice detection model is trained according to the training device as described in claim 7.
9. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement: The training method for the singing voice detection model as described in any one of claims 1 to 5; or, The singing voice detection method as described in claim 6.
10. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to achieve the following: The training method for the singing voice detection model as described in any one of claims 1 to 5; or, The singing voice detection method as described in claim 6.
Citation Information
Patent Citations
Singing synthesis method and device, computer device and storage medium
CN113555001A
Singing detection method and device, equipment and storage medium
CN114550676A