Human voice main melody extraction method and device, electronic equipment and storage medium

Abstract

The embodiment of the application provides a kind of human voice main melody extraction method and device, electronic equipment and storage medium, belong to the field of financial technology.The method comprises: obtaining harmonic splicing data;Harmonic splicing data is input to the preset original melody extraction model to obtain audio saliency data by saliency calculation, and audio saliency data is discriminated to obtain human voice discrimination information;According to audio saliency data and human voice discrimination information, sample audio data is extracted to obtain human voice main melody sequence;Human voice main melody sequence, human voice discrimination information and preset main melody reference sequence, discrimination reference information are calculated to obtain target loss data;According to target loss data, the original melody extraction model is adjusted to obtain the target melody extraction model;Target splicing data is input to the target melody extraction model to extract the target main melody sequence.The embodiment of the application can improve the extraction effect of human voice main melody.

Description

Technical Field

[0001] This application relates to the field of financial technology (Fintech), and in particular to a method and apparatus for extracting the main melody of human voices, an electronic device, and a storage medium. Background Technology

[0002] With the rise of computer technology, extracting the main melody of vocals from audio files for subsequent vocal content recognition has become a common technique in speech recognition. For example, singing scoring systems in karaoke software or humming-based song recognition systems need to extract the main melody of vocals from recorded audio files to score singing or identify songs.

[0003] In related technologies, if there is no human voice in the audio file, the extraction effect of the main melody is poor, and audio frames without human voice are easily extracted as the main melody. Therefore, how to improve the extraction effect of the main melody has become an urgent technical problem to be solved. Summary of the Invention

[0004] The main objective of this application is to provide a method, apparatus, electronic device, and storage medium for extracting the main melody of human voices, aiming to improve the extraction effect of the main melody of human voices.

[0005] To achieve the above objectives, a first aspect of this application proposes a method for extracting the main melody of human voices, the method comprising:

[0006] Obtain sample audio data;

[0007] The sample audio data is subjected to harmonic splicing processing to obtain harmonic splicing data;

[0008] The harmonic splicing data is input into a preset original melody extraction model; wherein, the original melody extraction model includes: a convolutional neural network, a convolutional recurrent neural network, and a fully connected classifier;

[0009] The convolutional neural network is used to calculate the saliency of the harmonic splicing data to obtain audio saliency data, and the fully connected classifier is used to perform human voice discrimination on the audio saliency data to obtain human voice discrimination information.

[0010] The main melody of human voice is extracted from the sample audio data using the convolutional recurrent neural network, the audio saliency data, and the human voice discrimination information to obtain a sequence of main melody of human voice.

[0011] The target loss data is obtained by performing loss calculations on the human voice melody sequence, the preset melody reference sequence, the human voice discrimination information, and the preset discrimination reference information.

[0012] The original melody extraction model is adjusted according to the target loss data to obtain the target melody extraction model;

[0013] The acquired target audio data is processed by harmonic splicing to obtain target spliced ​​data, and the target spliced ​​data is input into the target melody extraction model to extract the main melody of human voice, thereby obtaining the target main melody sequence.

[0014] In some embodiments, the harmonic splicing processing of the sample audio data to obtain harmonic splicing data includes:

[0015] The harmonic spectrum of the sample audio data is calculated to obtain the audio spectrum;

[0016] The audio spectrum is classified to obtain spectrum categories;

[0017] The audio spectrum is spliced ​​according to the spectrum category to obtain the harmonic splicing data.

[0018] In some embodiments, the step of performing harmonic spectrum calculation on the sample audio data to obtain the audio spectrum includes:

[0019] The sample audio data is segmented into frames to obtain audio frame data;

[0020] The audio frame data is subjected to Fourier transform processing to obtain the audio frame spectrum;

[0021] The audio frame spectrum is spliced ​​together to obtain the audio spectrum.

[0022] In some embodiments, the convolutional neural network includes: a harmonic convolutional layer and an activation layer; the step of calculating the saliency of the harmonic splicing data using the convolutional neural network to obtain audio saliency data includes:

[0023] The audio spectrum is convolved by the harmonic convolution layer to obtain a spectrum sequence;

[0024] The saliency of the spectral sequence is calculated through the activation layer to obtain candidate saliency data.

[0025] The candidate saliency data is obtained by concatenating the activation layer and the spectrum category.

[0026] In some embodiments, the candidate saliency data is concatenated using the activation layer and the spectral category to obtain the audio saliency data, including:

[0027] Target weights are selected from preset weights based on the spectrum category;

[0028] The target weight and the candidate saliency data are weighted and summed through the activation layer to obtain the audio saliency data.

[0029] In some embodiments, the convolutional recurrent neural network includes: a recurrent convolutional layer, a sequence transformation layer, and a fully connected layer; the step of extracting the main melody of the human voice from the sample audio data using the convolutional recurrent neural network, the audio saliency data, and the human voice discrimination information to obtain a main melody sequence includes:

[0030] The audio saliency data is used to extract features through the recurrent convolutional layer to obtain saliency features;

[0031] The saliency features are serialized through the sequence transformation layer to obtain a saliency sequence;

[0032] The main melody of the human voice is extracted from the sample audio data using the fully connected layer, the saliency sequence, and the human voice discrimination information to obtain the main melody sequence of the human voice.

[0033] In some embodiments, the step of performing loss calculations on the vocal melody sequence, the preset melody reference sequence, the vocal discrimination information, and the preset discrimination reference information to obtain target loss data specifically includes:

[0034] Loss calculations are performed on the vocal melody sequence and the melody reference sequence to obtain melody loss data;

[0035] Loss calculation is performed on the human voice discrimination information and the discrimination reference information to obtain discrimination loss data;

[0036] The target loss data is obtained by weighting and summing the preset weight parameters, the melody loss data, and the discrimination loss data.

[0037] To achieve the above objectives, a second aspect of this application provides a device for extracting the main melody of a human voice, the device comprising:

[0038] The data acquisition module is used to acquire sample audio data;

[0039] The harmonic splicing module is used to perform harmonic splicing processing on the sample audio data to obtain harmonic splicing data;

[0040] The data input module is used to input the harmonic splicing data into a preset original melody extraction model; wherein, the original melody extraction model includes: a convolutional neural network, a convolutional recurrent neural network, and a fully connected classifier.

[0041] The data processing module is used to calculate the saliency of the harmonic splicing data through the convolutional neural network to obtain audio saliency data, and to perform human voice discrimination on the audio saliency data through the fully connected classifier to obtain human voice discrimination information.

[0042] The human voice melody extraction module is used to extract the human voice melody from the sample audio data through the convolutional recurrent neural network, the audio saliency data and the human voice discrimination information to obtain a human voice melody sequence.

[0043] The loss calculation module is used to perform loss calculation on the vocal melody sequence, the preset melody reference sequence, the vocal discrimination information, and the preset discrimination reference information to obtain the target loss data.

[0044] The parameter adjustment module is used to adjust the parameters of the original melody extraction model according to the target loss data to obtain the target melody extraction model.

[0045] The target melody extraction module is used to perform harmonic splicing processing on the acquired target audio data to obtain target spliced ​​data, and input the target spliced ​​data into the target melody extraction model to extract the main melody of human voice, thereby obtaining the target main melody sequence.

[0046] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0047] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0048] The method, apparatus, electronic device, and storage medium for extracting the main melody of human voices proposed in this application obtain harmonic splicing data by harmonic concatenation of sample audio data. This reduces interference information in the sample audio data, highlights harmonic relationships, and improves the extraction accuracy of the main melody. Simultaneously, the original melody extraction model extracts the main melody sequence of human voices more accurately based on audio saliency data and human voice discrimination information. By constructing target loss data, the original melody extraction model is adjusted according to the target loss data to obtain a target melody extraction model with even better main melody extraction performance. Therefore, the target melody extraction model can extract a more accurate main melody of human voices from audio data, making singing scoring or song identification based on the main melody of human voices more accurate. Attached Figure Description

[0049] Figure 1 This is a flowchart of the method for extracting the main melody of human voices provided in the embodiments of this application;

[0050] Figure 2 yes Figure 1 The flowchart of step S102 in the document;

[0051] Figure 3 yes Figure 2 The flowchart of step S201 in the text;

[0052] Figure 4 This is a flowchart of a method for extracting the main melody of human voices according to another embodiment of this application;

[0053] Figure 5 yes Figure 4 The flowchart of step S403 in the process;

[0054] Figure 6 yes Figure 1 The flowchart of step S105 in the process;

[0055] Figure 7 yes Figure 1 The flowchart of step S106 in the process;

[0056] Figure 8 This is a flowchart of a method for extracting the main melody of human voices according to another embodiment of this application;

[0057] Figure 9 This is a schematic diagram of the structure of the vocal melody extraction device provided in the embodiments of this application;

[0058] Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0060] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0062] First, let's analyze some of the terms used in this application:

[0063] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0064] Harmonics: Harmonics refer to the components that are greater than integer multiples of the fundamental frequency obtained by Fourier series decomposition of a periodic non-sinusoidal alternating current. They are usually called higher harmonics, while the fundamental frequency refers to the component whose frequency is the same as the power frequency (50Hz).

[0065] Fundamental frequency: The fundamental frequency refers to the frequency of the fundamental tone. The fundamental tone is the lowest pure tone in any musical note, and its intensity is the greatest, determining the pitch of the entire sound. In sound, the fundamental frequency refers to the frequency of the fundamental tone in a complex tone. Among the several tones that make up a complex tone, the fundamental tone has the lowest frequency and the greatest intensity. The level of the fundamental frequency determines the pitch of a sound. The frequency of speech, as commonly referred to, refers to the frequency of the fundamental tone. The fundamental frequency is represented by the symbol FO.

[0066] Convolutional Neural Network (CNN): A CNN is a feedforward neural network whose artificial neurons can respond to a portion of the surrounding units within their coverage area, making it excellent for large-scale image processing. It consists of alternating convolutional layers and pooling layers.

[0067] Convolutional Recurrent Neural Network (CRNN): CRNN is a popular image-to-text recognition model capable of recognizing long text sequences. It includes a CNN feature extraction layer and a BLSTM sequence feature extraction layer, enabling end-to-end joint training. It utilizes BLSTM and CTC components to learn the contextual relationships within character images, effectively improving text recognition accuracy and making the model more robust. During prediction, the front-end uses a standard CNN network to extract features from the text image, then uses BLSTM to fuse the feature vectors to extract contextual features of the character sequence, obtaining the probability distribution of each feature column. Finally, the text sequence is predicted using a transcription layer (CTC rule). The CRNN network structure consists of three parts: convolutional layers (CNN), recurrent layers (RNN), and transcriptional layers (CTC loss). The CNN uses a deep CNN to extract features from the input image and obtain a feature map. The RNN uses a bidirectional RNN (BLSTM) to predict the feature sequence, learns each feature vector in the sequence, and outputs the predicted label (true value) distribution. The CTC loss uses CTC loss to transform a series of label distributions obtained from the recurrent layers into the final label sequence.

[0068] Classifiers: Classification is a crucial method in data mining. The concept of classification involves learning a classification function or constructing a classification model (i.e., a classifier) ​​based on existing data. This function or model maps data records in a database to a given category, thus enabling data prediction. In short, classifiers are a general term for methods used in data mining to classify samples, encompassing algorithms such as decision trees, logistic regression, Naive Bayes, and neural networks.

[0069] With the development of computer technology, extracting the main melody of human voices from music or speech data, primarily through a main melody extraction model to extract the pitch sequence of human voices from the original audio data, has become a common technique in speech recognition. For example, in karaoke software, a scoring system is implemented by extracting the main melody of the human voice for scoring. Similarly, in identifying song information, it's necessary to recognize the main melody of the human voice in the audio file for song information recognition. However, the main melody extraction techniques in these technologies perform poorly in detecting frames without human voices, misinterpreting parts where no human voice is present, resulting in poor overall extraction performance.

[0070] Based on this, embodiments of this application provide a method, apparatus, electronic device, and storage medium for extracting the main melody of human voices. Harmonic splicing is performed on sample audio data to obtain harmonic spliced ​​data. This harmonic spliced ​​data is then input into an original melody extraction model, which includes a convolutional neural network, a convolutional recurrent neural network, and a fully connected classifier. The convolutional neural network calculates the saliency of the sample audio data to obtain audio saliency data. The fully connected classifier performs human voice discrimination on the audio saliency data to obtain human voice discrimination information. The main melody of human voices is then extracted from the sample audio data using the convolutional recurrent neural network, the audio saliency data, and the human voice discrimination information to obtain a main melody sequence. Therefore, by calculating the loss of the main melody sequence, the main melody reference sequence, the human voice discrimination information, and the discrimination reference information to obtain target loss data, the parameters of the original melody extraction model are adjusted based on the target loss data to obtain a target melody extraction model. The main melody is then extracted from the spliced ​​target data using the target melody extraction model to obtain a target main melody sequence, thereby improving the extraction effect and accuracy of the main melody of human voices. Therefore, by constructing a target melody extraction model that can accurately identify the main melody of the human voice, the extraction effect of the main melody of the human voice is improved. Then, singing scoring or song recognition based on the accurate main melody of the human voice will also be more accurate, thereby improving the user experience of karaoke software or humming song recognition system and helping to increase the frequency of use of karaoke software or humming song recognition system.

[0071] The method, apparatus, electronic device, and storage medium for extracting the main melody of human voices provided in this application are specifically described through the following embodiments. First, the method for extracting the main melody of human voices in this application is described.

[0072] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0073] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0074] The method for extracting the main melody of human voices provided in this application relates to the fields of artificial intelligence and financial technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the method for extracting the main melody of human voices, but is not limited to the above forms.

[0075] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0076] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of the embodiments of this application obtained.

[0077] Figure 1 This is an optional flowchart of the method for extracting the main melody of human voices provided in the embodiments of this application. Figure 1The method may include, but is not limited to, steps S101 to S108.

[0078] Step S101: Obtain sample audio data;

[0079] Step S102: Perform harmonic splicing processing on the sample audio data to obtain harmonic splicing data;

[0080] Step S103: Input the harmonic splicing data into the preset original melody extraction model; wherein, the original melody extraction model includes: a convolutional neural network, a convolutional recurrent neural network, and a fully connected classifier.

[0081] Step S104: The audio saliency data is obtained by calculating the saliency of the harmonic splicing data through a convolutional neural network, and the human voice discrimination information is obtained by performing human voice discrimination on the audio saliency data through a fully connected classifier.

[0082] Step S105: Extract the main melody of human voice from the sample audio data using a convolutional recurrent neural network, audio saliency data, and human voice discrimination information to obtain a sequence of main melody of human voice.

[0083] Step S106: Perform loss calculation on the vocal melody sequence, the preset melody reference sequence, the vocal discrimination information, and the preset discrimination reference information to obtain the target loss data;

[0084] Step S107: Adjust the parameters of the original melody extraction model based on the target loss data to obtain the target melody extraction model;

[0085] Step S108: The acquired target audio data is processed by harmonic splicing to obtain target spliced ​​data, and the target spliced ​​data is input into the target melody extraction model to extract the main melody of human voice, thereby obtaining the target main melody sequence.

[0086] Steps S101 to S108 of this embodiment involve harmonic splicing of sample audio data to obtain harmonic splicing data, and inputting the harmonic splicing data into a preset original melody extraction model. The original melody extraction model includes a convolutional neural network, a convolutional recurrent neural network, and a fully connected classifier. The convolutional neural network calculates the saliency of the harmonic splicing data to obtain audio saliency data, and the fully connected classifier performs human voice discrimination on the audio saliency data to obtain human voice discrimination information. The convolutional recurrent neural network, audio saliency data, and human voice discrimination information are used to extract the main melody of the sample audio data to obtain a main melody sequence. The main melody sequence, a preset main melody reference sequence, human voice discrimination information, and preset discrimination reference information are used to calculate a loss to obtain target loss data. The parameters of the original melody extraction model are adjusted using the target loss data to obtain a target melody extraction model. The target splicing data obtained from harmonic splicing is then input into the target melody extraction model to extract the target main melody sequence. Therefore, harmonic splicing is performed on the sample audio data to obtain harmonic spliced ​​data, reducing interference information in the sample audio data, highlighting harmonic relationships, and improving the accuracy of main melody extraction. Simultaneously, the original melody extraction model extracts the main melody sequence of the human voice more accurately based on audio saliency data and human voice discrimination information. By constructing target loss data and adjusting the original melody extraction model based on the target loss data, a target melody extraction model with even better human voice main melody extraction performance is obtained.

[0087] In step S101 of some embodiments, sample audio data can be directly extracted from the sample database. However, other methods can also be used to collect the current audio data in real time, and this is not limited to these. For example, if the application scenario is a scoring system for a karaoke app, and the sample audio data is song audio data with vocals, it is necessary to extract the vocal melody sequence from the song audio data to compare the vocal melody sequence with the original song sequence to obtain a song score. Alternatively, if the application scenario is a humming song recognition system, and the sample audio data is humming audio data with vocals, it is necessary to extract the vocal melody sequence based on the humming audio data to perform song recognition based on the vocal melody sequence. Therefore, it is necessary to collect song audio data with vocals as sample audio data, and each sample audio data is matched with a corresponding melody reference sequence, where the melody reference sequence is the actual vocal melody sequence used as a reference in the sample audio data.

[0088] Please see Figure 2 In some embodiments, step S102 may include, but is not limited to, steps S201 to S203:

[0089] Step S201: Calculate the harmonic spectrum of the sample audio data to obtain the audio spectrum;

[0090] Step S202: Classify the audio spectrum to obtain spectrum categories;

[0091] Step S203: Perform spectrum splicing processing on the audio spectrum according to the spectrum category to obtain harmonic splicing data.

[0092] In step S201 of some embodiments, the harmonic spectrum of the sample audio data is first calculated to obtain the audio spectrum, so as to splice the audio spectrum into harmonic splicing data.

[0093] In step S202 of some embodiments, the audio spectrum is classified, that is, the audio spectrum is classified according to a preset number of categories, so that the number of audio spectrum splits is the number of categories. Therefore, the audio spectrum is classified to obtain the spectrum category, that is, to determine the spectrum category of the candidate fundamental frequency in each audio spectrum.

[0094] It should be noted that in this embodiment, the audio spectrum is divided into 320 spectrum categories, and each spectrum category corresponds to a candidate fundamental frequency, so as to obtain 320 candidate fundamental frequencies.

[0095] In step S203 of some embodiments, the audio spectrum is spliced ​​according to the spectrum category. This involves splicing the harmonic positions corresponding to each candidate fundamental frequency, and splicing together the candidate fundamental frequency and the spectrum whose frequencies are integer multiples of the candidate fundamental frequency to obtain harmonic splicing data. Therefore, splicing the spectrum of frequencies that are multiples of the candidate fundamental frequency to construct harmonic splicing data reduces interference information in the spectrum, highlights harmonic relationships, and improves the accuracy of the melody extraction model in extracting the main melody based on the harmonic splicing data. This, in turn, improves the scoring accuracy of karaoke software or the accuracy of song recognition systems.

[0096] It's important to note that the candidate fundamental frequency is the frequency of the fundamental tone in a polyphony. Among the several tones that make up a polyphony, the fundamental tone has the lowest frequency and the highest intensity, and its level determines the pitch of a note. Therefore, in extracting the main melody of a vocal performance, it's necessary to extract the main melody for each audio frame. If there is interference noise in an audio frame, it will affect the extraction effect. Since the interference noise in the audio data is not an integer multiple of the fundamental frequency, while the harmonic frequencies of continuously recorded vocals are integer multiples of the candidate fundamental frequency, harmonic splicing data is obtained by concatenating the spectra that are integer multiples of each candidate fundamental frequency. That is, a new spectrogram is constructed based on the original spectrogram to utilize the harmonic characteristics of vocals, reduce interference noise in the audio spectrum before extracting the vocal melody, highlight the harmonic relationships in the audio spectrum, and improve the extraction effect of the main melody. For example, the interference noise in the audio data recorded in karaoke software includes background music and environmental noise, so background music and environmental noise will affect the extraction effect of the main melody in the audio data. In a humming-based music recognition system, the main source of interference noise in the audio data is environmental noise, and it is necessary to reduce the impact of environmental noise.

[0097] In steps S201 to S203 of the embodiments of this application, the audio spectrum is obtained by calculating the harmonic spectrum of the sample audio data, and the audio spectrum is classified to obtain the spectrum category of each audio spectrum. The audio spectrum is then spliced ​​according to the spectrum category, that is, the spectrum of each candidate fundamental frequency and harmonic frequency is spliced ​​to obtain harmonic splicing data, so as to reduce interference information in the spectrum, highlight the harmonic relationship, and thus improve the effect of extracting the main melody of the human voice.

[0098] Please see Figure 3 In some embodiments, step S201 may include, but is not limited to, steps S301 to S303:

[0099] Step S301: Perform frame segmentation on the sample audio data to obtain audio frame data;

[0100] Step S302: Perform Fourier transform processing on the audio frame data to obtain the audio frame spectrum;

[0101] Step S303: The audio frame spectrum is spliced ​​to obtain the audio spectrum.

[0102] In step S301 of some embodiments, since the main melody of the human voice needs to be extracted from audio frames and human voice discrimination needs to be performed on each audio frame, the sample audio data needs to be processed into audio frame data by frame segmentation. The constructed audio spectrum consists of multiple audio frame spectra to facilitate the extraction of the main melody of the human voice and human voice discrimination.

[0103] In step S302 of some embodiments, a Fourier transform is performed on each audio frame data. Specifically, the audio frame data is transformed using the Short-Time Fourier Transform (SFT) method to obtain the audio frame spectrum. Since the SFT method has a better analysis effect on audio signals, performing a SFT on the audio frame data yields a more accurate audio frame spectrum.

[0104] In step S303 of some embodiments, the audio spectrum is obtained by splicing all the audio frame spectra in chronological order. Therefore, by constructing an audio spectrum that combines the spectra of multiple audio frames, it is easier to perform human voice discrimination on each audio frame spectrum. This allows for more accurate extraction of the main melody of the human voice from the audio data based on the human voice discrimination information, thereby improving the accuracy of karaoke software scoring or humming song recognition systems in identifying songs.

[0105] In steps S301 to S303 of the embodiments of this application, audio frame data is obtained by performing frame-segmentation processing on the sample audio data, and audio frame spectrum is obtained by performing short-time Fourier transform on the audio frame data. All audio frame spectra are then spliced ​​together to construct an audio spectrum including multiple audio frame spectra. This facilitates the identification of human voices in each audio frame spectrum to obtain human voice identification information. Based on the human voice identification information, the extraction of the main melody of human voices from the audio data is more accurate, thereby improving the accuracy of karaoke software scoring or humming song recognition system in identifying song information, thus enhancing the user experience of using karaoke software or humming song recognition system.

[0106] In step S103 of some embodiments, by inputting the spliced ​​harmonic data into a preset original melody extraction model, the original melody extraction model extracts the vocal melody more accurately from the harmonic splicing data because the harmonic splicing data reduces interference information in the spectrum. The original melody extraction model includes a convolutional neural network (CNN) and a convolutional recurrent neural network (RNN). The CNN not only utilizes its feature extraction capabilities but also leverages the sequence modeling capabilities of the RNN. It can predict the vocal melody from the harmonic splicing data by learning the distribution characteristics of the melody, thereby improving the extraction effect of the vocal melody and enhancing the accuracy of song scoring in karaoke software or the recognition of song information in humming systems.

[0107] Please see Figure 4 In some embodiments, the convolutional neural network includes a harmonic convolutional layer and an activation layer. The audio saliency data is obtained by calculating the saliency of the harmonic splicing data using the convolutional neural network, which may include, but is not limited to, steps S401 to S403.

[0108] Step S401: The audio spectrum is convolved using a harmonic convolution layer to obtain a spectrum sequence;

[0109] Step S402: The saliency of the spectral sequence is calculated through the activation layer to obtain candidate saliency data;

[0110] Step S403: The candidate saliency data is concatenated by the activation layer and spectrum category to obtain audio saliency data.

[0111] In step S401 of some embodiments, since the harmonic splicing data includes the audio spectrum of each spectrum category, and the number of convolutional neural networks is consistent with the spectrum category, the audio spectrum of each spectrum category is input into the convolutional neural network respectively, and the audio spectrum of each spectrum category is input into the harmonic convolutional layer for convolution processing to obtain the spectrum sequence.

[0112] Specifically, the audio spectrum is input to the harmonic convolutional layer in the form of harmonic convolution kernels, with a kernel size of 5N*W, where N is the harmonic data, W is the receptive field in the time dimension, and the stride of the harmonic convolutional layer is (1, 1). Therefore, each audio spectrum is convolved by each harmonic convolutional layer to obtain a spectrum sequence with a size of (1, L), where L is the number of frames in the audio frame data.

[0113] In step S402 of some embodiments, the saliency of the spectrum sequence is calculated by activating the layer, that is, the saliency of the audio spectrum of each spectrum category is calculated, and the saliency corresponding to each candidate fundamental frequency is calculated to obtain candidate saliency data.

[0114] In step S403 of some embodiments, after calculating the candidate saliency data for each audio spectrum, the candidate saliency data for all spectrum categories are concatenated through an activation layer. This concatenation of the saliency data for all candidate fundamental frequencies yields the audio saliency data, which represents the saliency of each candidate fundamental frequency in the entire sample audio data. Therefore, by calculating the audio saliency data for the entire sample audio data, it is possible to extract the main melody of the vocals based on the audio saliency data, or to more accurately identify vocals, thereby improving the accuracy of karaoke software scoring or humming-based song recognition systems in identifying song information.

[0115] In steps S401 to S403 of the embodiments of this application, the audio spectrum of each spectrum category is convolved by a harmonic convolution layer to obtain a spectrum sequence, and the saliency of the spectrum sequence is calculated by an activation layer to obtain candidate saliency data. Then, the candidate saliency data of each spectrum category is concatenated by the activation layer to obtain audio saliency data, so as to calculate the audio saliency data of the entire sample audio data. The main melody of the human voice can be extracted based on the audio saliency data to improve the extraction effect of the main melody of the human voice, thereby improving the accuracy of karaoke software scoring or humming song recognition system in identifying song information, and thus increasing the frequency of users using karaoke software or humming song recognition system.

[0116] Please see Figure 5 In some embodiments, step S403 may include, but is not limited to, steps S501 to S502:

[0117] Step S501: Select the target weight from the preset weights according to the spectrum category;

[0118] Step S502: The target weight and candidate saliency data are weighted and summed through the activation layer to obtain the audio saliency data.

[0119] In step S501 of some embodiments, since the candidate saliency data is the saliency data of each spectrum category, concatenating the saliencies of all spectrum categories to form the audio spectrum is not a simple concatenation. Instead, it is necessary to select candidate saliency data based on the importance of each spectrum category to obtain more accurate audio saliency data. The audio saliency data can accurately characterize the saliency of the sample audio data. Therefore, a preset weight is set for each spectrum category, and a target weight is selected from the preset weights according to the spectrum category to determine the target weight corresponding to each spectrum category.

[0120] In step S502 of some embodiments, the target weights and candidate saliency data are weighted and calculated through an activation layer to obtain more accurate audio saliency data. In this embodiment, 320 spectrum categories are set, so the audio saliency data is obtained by weighting and summing the 320 target weights and 320 candidate saliency data, making the audio saliency data calculation simpler.

[0121] In steps S501 to S502 of the embodiments of this application, the target weight corresponding to each spectrum category is selected from the preset weight according to the spectrum category, and then the target weight and candidate saliency data are weighted and summed through the activation layer to obtain the audio saliency data, which makes the calculation of audio saliency data simple.

[0122] In some embodiments, audio saliency data is input into a fully connected classifier for voice discrimination processing. The audio saliency data includes candidate saliency data for each audio frame. The fully connected classifier performs voice melody discrimination on the candidate saliency data for each audio frame to obtain discrimination information for each audio frame. All discrimination information is then concatenated to obtain voice discrimination information. Therefore, by performing voice discrimination based on audio saliency data, it is possible to determine whether a voice melody exists in each audio frame, leading to better extraction of the main melody based on the voice discrimination information. For example, in karaoke software, after a user finishes singing a song, they output audio data. Based on this audio data, audio saliency data is determined. A fully connected layer analyzes the audio saliency data to determine whether a voice exists. If no voice is present, the main melody is not extracted, and a no-voice prompt is output. Only if a voice is present is the main melody extracted. Therefore, by determining whether to extract the main melody based on the voice discrimination information, accurate extraction of the main melody is achieved, ensuring accurate scoring in karaoke software.

[0123] Specifically, the audio saliency data is input into a fully connected classifier to perform human voice discrimination, outputting human voice discrimination information of size (1, L). (1, L) represents that the sample audio data includes L audio frames, and that each audio frame contains human voices. If the nth human voice discrimination information is 0, it means that the nth audio frame does not contain human voices; if the nth human voice discrimination information is 1, it means that the nth audio frame contains human voices. Therefore, obtaining human voice discrimination information as reference information for human voice melody extraction is beneficial to improving the extraction effect of the human voice melody.

[0124] Please see Figure 6 In some embodiments, the convolutional recurrent neural network includes: a recurrent convolutional layer, a sequence transformation layer, and a fully connected layer. Step S105 includes, but is not limited to, steps S601 to S603:

[0125] Step S601: Extract features from the audio saliency data using a recurrent convolutional layer to obtain saliency features;

[0126] Step S602: The saliency features are serialized through a sequence transformation layer to obtain a saliency sequence;

[0127] Step S603: Extract the main melody of the human voice from the sample audio data through a fully connected layer, saliency sequence, and human voice discrimination information to obtain the main melody sequence of the human voice.

[0128] In step S601 of some embodiments, after the convolutional neural network outputs audio saliency data, the audio saliency data is input into a recurrent convolutional layer for feature extraction to obtain saliency features. The recurrent convolutional network has eight recurrent convolutional layers, meaning that saliency features are extracted one by one through these eight recurrent convolutional layers, so that the extracted saliency features can better represent the audio saliency data.

[0129] It should be noted that the kernel size in the recurrent convolutional layer is 7*3, and after feature extraction through the recurrent convolutional layer, it goes through a batch regularization layer for regularization to reduce the error in feature extraction.

[0130] In step S602 of some embodiments, the saliency features output by the recurrent convolutional layer are input to the sequence transformation layer for sequence transformation to obtain a saliency sequence, so as to represent the saliency features in the form of a sequence. The sequence transformation layer is a Bi-LSTM model, and the Bi-LSTM model is a sequence labeling model. Therefore, the Bi-LSTM model can capture hidden features from the saliency features to construct a saliency sequence, so that the saliency features can be represented more accurately through the saliency sequence.

[0131] In step S603 of some embodiments, the main melody of the human voice is extracted from the sample audio data through a fully connected layer, human voice discrimination information and saliency sequence. The main melody of the human voice is extracted from the sample audio data according to the saliency sequence to obtain the main melody sequence of the human voice, so that the extraction effect of the main melody of the human voice is more accurate, thereby improving the accuracy of karaoke software scoring or humming song recognition system in identifying song information.

[0132] It should be noted that the initial melody sequence is extracted from the sample audio data by a fully connected layer based on the saliency sequence, and then the human voice discrimination information is converted into a human voice discrimination sequence. The human voice discrimination sequence and the initial melody sequence are then spliced ​​together to obtain the human voice melody sequence, so as to construct an accurate human voice melody sequence.

[0133] Specifically, in the model application process, if the voice discrimination information of the audio frame data indicates the presence of voices, then a corresponding main melody sequence exists; if the voice discrimination information of the audio frame data indicates the absence of voices, then a corresponding main melody sequence does not exist. Therefore, the initial main melody sequence and the voice discrimination information are concatenated to obtain the main melody sequence, thereby improving the extraction effect of the main melody and thus enhancing the accuracy of karaoke software scoring or humming-based song recognition systems in identifying song information.

[0134] In steps S601 to S603 of the embodiments of this application, saliency features are obtained by extracting features from audio saliency data through a recurrent convolutional layer, and saliency sequences are obtained by serializing the saliency features through a sequence transformation layer. Then, the main melody of the human voice is extracted from the sample audio data through a fully connected layer, human voice discrimination information and saliency sequences, so that the effect of extracting the main melody of the human voice is better.

[0135] See Figure 7 In some embodiments, step S106 may include, but is not limited to, steps S701 to S703:

[0136] Step S701: Perform loss calculation on the vocal melody sequence and the melody reference sequence to obtain melody loss data;

[0137] Step S702: Perform loss calculation on the human voice discrimination information and discrimination reference information to obtain discrimination loss data;

[0138] Step S703: The preset weight parameters, melody loss data and discrimination loss data are weighted and summed to obtain the target loss data.

[0139] In step S701 of some embodiments, in order to determine the extraction effect of the vocal melody of the original melody extraction model, the melody loss data between the vocal melody sequence and the preset melody reference sequence is calculated, so as to determine the extraction effect of the original melody extraction model by training the loss data.

[0140] In step S702 of some embodiments, in order to determine the human voice discrimination effect of the fully connected classifier, discrimination loss data is obtained by performing loss calculation on the human voice discrimination information and discrimination reference information. Then, the human voice discrimination information can be determined by the discrimination loss data.

[0141] In step S703 of some embodiments, the accuracy of the target loss data calculation is improved by weighting and summing preset weight parameters, melody loss data, and discriminant loss data. The weight parameters include a first weight and a second weight, where the first weight represents the importance of the training loss data, and the second weight represents the importance of the discriminant loss data. Therefore, obtaining the target loss data by weighting and summing the first weight, melody loss data, second weight, and discriminant loss data makes the target loss data calculation more accurate. The formula for calculating the target loss data is Z = A*X + B*Y, where A is the first weight, B is the second weight, X is the melody loss data, and Y is the discriminant loss data, simplifying the calculation of the target loss data.

[0142] In steps S701 to S703 of the embodiments of this application, melody loss data is obtained by performing loss calculation on the vocal melody sequence and the melody reference sequence, then discrimination loss data is obtained by performing loss calculation on the vocal discrimination information and discrimination reference information, and finally the target loss data is obtained by weighted summation of the first weight, the melody loss data, the second weight and the discrimination loss function, making the calculation of the target loss data simple and accurate.

[0143] In step S106 of some embodiments, the parameters of the original melody extraction model are adjusted according to the target loss data, that is, the parameters of the original melody extraction model are adjusted according to the melody loss data, and the parameters of the fully connected classifier are adjusted according to the discriminant loss data until the target loss data converges, so as to obtain the target melody extraction model and construct a target melody extraction model that can accurately extract the main melody of human voice.

[0144] In step S107 of some embodiments, harmonic splicing is performed on the target audio data to obtain target spliced ​​data, thereby reducing interference information in the spectrum and highlighting harmonic relationships. Then, the target spliced ​​data is input into the target melody extraction model to extract the main vocal melody, resulting in the target main melody sequence. The target melody extraction model has the same structure as the original melody extraction model, and the operation of the target melody extraction model in extracting the main vocal melody is the same as that of the original melody extraction model; therefore, it will not be described further here.

[0145] It should be noted that the target melody extraction model includes a convolutional neural network, a convolutional recurrent neural network, and a fully connected classifier. The convolutional neural network outputs target saliency data based on the target concatenation data. Then, the convolutional neural network extracts the primary melody sequence from the target saliency data. Next, it performs voice discrimination based on the target saliency data to obtain target discrimination information. Finally, the primary melody sequence and the target discrimination information are concatenated to obtain the target melody sequence, thus improving the effectiveness of voice melody extraction.

[0146] Please refer to Figure 8This application embodiment acquires sample audio data, performs frame-by-frame processing on the sample audio data to obtain audio frame data, performs short-time Fourier transform on the audio frame data to obtain audio frame spectrum, and concatenates all audio frame spectra to form an audio spectrum. The audio spectrum is then classified to obtain the spectrum category of each audio spectrum, and the audio spectrum is concatenated according to the spectrum category to obtain harmonic concatenation data. Since harmonic concatenation data reduces interference information in the spectrum, the original melody extraction model extracts the main vocal melody more accurately from the harmonic concatenation data. Therefore, the concatenated harmonic concatenation data is input into a preset original melody extraction model, which includes a convolutional neural network, a convolutional recurrent neural network, and a fully connected classifier. Convolutional Neural Networks (CNNs) consist of harmonic convolutional layers and activation layers. The harmonic convolutional layers process the audio spectrum of each spectral category to obtain a spectral sequence. The activation layers then calculate the saliency of this spectral sequence to obtain candidate saliency data. Based on the spectral category, target weights are selected from preset weights for each spectral category. Finally, the activation layers sum the target weights and candidate saliency data using weighted averages to obtain audio saliency data. This audio saliency data is then input into a fully connected classifier for human voice discrimination to obtain human voice discrimination information. Convolutional Recurrent Neural Networks (RNNs) consist of recurrent convolutional layers, sequence transformation layers, and fully connected layers. The recurrent convolutional layers extract saliency features from the audio saliency data. The sequence transformation layers serialize these saliency features to obtain a saliency sequence. Finally, the fully connected layers, human voice discrimination information, and the saliency sequence are used to extract the main melody from the sample audio data, resulting in a more effective main melody extraction. Melody loss data is obtained by calculating the loss of the vocal melody sequence and the melody reference sequence. Then, discriminative loss data is obtained by calculating the loss of the vocal discrimination information and the preset discrimination reference information. Based on the discriminative loss data, the parameters of the fully connected classifier are adjusted, and the parameters of the original melody extraction model are adjusted based on the melody loss data until the target loss data converges to obtain the target melody extraction model. The target concatenated data is then input into the target melody extraction model to extract the vocal melody sequence, thus improving the extraction effect of the vocal melody. Therefore, for karaoke software or humming recognition systems, determining whether vocals are present in the recorded audio data before extracting the vocal melody reduces the extraction operation of audio data without vocals and improves the efficiency of vocal melody extraction. This improves the accuracy of karaoke software scoring or humming recognition systems in identifying song information, enhancing the user experience and increasing the frequency of use.

[0147] Please see Figure 9This application also provides a device for extracting the main melody of human voices, which can implement the above-mentioned method for extracting the main melody of human voices. The device includes:

[0148] Data acquisition module 901 is used to acquire sample audio data;

[0149] The harmonic splicing module 902 is used to perform harmonic splicing processing on the sample audio data to obtain harmonic splicing data;

[0150] Data input module 903 is used to input harmonic splicing data into a preset original melody extraction model; wherein, the original melody extraction model includes: a convolutional neural network, a convolutional recurrent neural network, and a fully connected classifier.

[0151] The data processing module 904 is used to calculate the saliency of the harmonic splicing data through a convolutional neural network to obtain audio saliency data, and to perform human voice discrimination on the audio saliency data through a fully connected classifier to obtain human voice discrimination information.

[0152] The human voice melody extraction module 905 is used to extract the human voice melody from the sample audio data through a convolutional recurrent neural network, audio saliency data and human voice discrimination information to obtain a human voice melody sequence.

[0153] The loss calculation module 906 is used to perform loss calculation on the human voice main melody sequence, the preset main melody reference sequence, the human voice discrimination information and the preset discrimination reference information to obtain the target loss data;

[0154] The parameter adjustment module 907 is used to adjust the parameters of the original melody extraction model according to the target loss data to obtain the target melody extraction model;

[0155] The target melody extraction module 908 is used to perform harmonic splicing processing on the acquired target audio data to obtain target spliced ​​data, and input the target spliced ​​data into the target melody extraction model to extract the main melody of human voice, thereby obtaining the target main melody sequence.

[0156] The specific implementation of the human voice melody extraction device is basically the same as the specific implementation of the human voice melody extraction method described above, and will not be repeated here.

[0157] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method for extracting the main melody of human voices. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0158] Please see Figure 10 , Figure 10 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0159] The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0160] The memory 1002 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001 using the human voice melody extraction method of the embodiments of this application.

[0161] Input / output interface 1003 is used to implement information input and output;

[0162] The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0163] Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004);

[0164] The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0165] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for extracting the main melody of human voices.

[0166] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0167] The method, apparatus, electronic device, and storage medium for extracting the main melody of human voices provided in this application embodiment acquire sample audio data and input it into an original melody extraction model. The original melody extraction model includes a convolutional neural network, a convolutional recurrent neural network, and a fully connected classifier. The convolutional neural network calculates the saliency of the sample audio data to obtain audio saliency data. The fully connected classifier performs human voice discrimination on the audio saliency data to obtain human voice discrimination information. The convolutional recurrent neural network, audio saliency data, and human voice discrimination information are then used to extract the main melody of human voices from the sample audio data to obtain a main melody sequence. A target loss data is obtained by calculating the loss using the main melody sequence, a preset main melody reference sequence, human voice discrimination information, and preset discrimination reference information. The parameters of the original melody extraction model are adjusted using the target loss data to obtain a target melody extraction model. The target melody extraction model is then used to extract the main melody of human voices from the target spliced ​​data to obtain a target main melody sequence. Therefore, by performing harmonic splicing preprocessing on the audio data before extracting the main melody of human voices, interference information in the spectrum is reduced, and harmonic relationships are highlighted. Then, a convolutional recurrent network is used to extract the main melody of the vocals, which improves the extraction effect. A fully connected classifier is used to discriminate the vocals, and the vocal melody sequence is further evaluated based on the vocal discrimination information to improve the extraction effect. This improves the accuracy of karaoke software scoring or humming recognition systems in identifying song information.

[0168] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0169] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0170] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0171] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0172] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0173] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0174] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0175] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0176] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0177] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0178] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for extracting the main melody of human voices, characterized in that, The method includes: Obtain sample audio data; The sample audio data is subjected to harmonic splicing processing to obtain harmonic splicing data; The harmonic splicing data is input into a preset original melody extraction model; wherein, the original melody extraction model includes: a convolutional neural network, a convolutional recurrent neural network, and a fully connected classifier; The convolutional neural network is used to calculate the saliency of the harmonic splicing data to obtain audio saliency data, and the fully connected classifier is used to perform human voice discrimination on the audio saliency data to obtain human voice discrimination information. The main melody of human voice is extracted from the sample audio data using the convolutional recurrent neural network, the audio saliency data, and the human voice discrimination information to obtain a sequence of main melody of human voice. The target loss data is obtained by performing loss calculations on the human voice melody sequence, the preset melody reference sequence, the human voice discrimination information, and the preset discrimination reference information. The original melody extraction model is adjusted according to the target loss data to obtain the target melody extraction model; The acquired target audio data is processed by harmonic splicing to obtain target spliced ​​data, and the target spliced ​​data is input into the target melody extraction model to extract the main melody of human voice, thereby obtaining the target main melody sequence.

2. The method according to claim 1, characterized in that, The harmonic splicing processing of the sample audio data to obtain harmonic splicing data includes: The harmonic spectrum of the sample audio data is calculated to obtain the audio spectrum; The audio spectrum is classified to obtain spectrum categories; The audio spectrum is spliced ​​according to the spectrum category to obtain the harmonic splicing data.

3. The method according to claim 2, characterized in that, The process of calculating the harmonic spectrum of the sample audio data to obtain the audio spectrum includes: The sample audio data is segmented into frames to obtain audio frame data; The audio frame data is subjected to Fourier transform processing to obtain the audio frame spectrum; The audio frame spectrum is spliced ​​together to obtain the audio spectrum.

4. The method according to claim 2, characterized in that, The convolutional neural network includes: a harmonic convolutional layer and an activation layer; the step of calculating the saliency of the harmonic splicing data using the convolutional neural network to obtain audio saliency data includes: The audio spectrum is convolved by the harmonic convolution layer to obtain a spectrum sequence; The saliency of the spectral sequence is calculated through the activation layer to obtain candidate saliency data. The candidate saliency data is obtained by concatenating the activation layer and the spectrum category.

5. The method according to claim 4, characterized in that, The candidate saliency data is obtained by concatenating the activation layer and the spectral category to obtain the audio saliency data, including: Target weights are selected from preset weights based on the spectrum category; The target weight and the candidate saliency data are weighted and summed through the activation layer to obtain the audio saliency data.

6. The method according to any one of claims 1 to 5, characterized in that, The convolutional recurrent neural network includes: a recurrent convolutional layer, a sequence transformation layer, and a fully connected layer; the step of extracting the main melody of the human voice from the sample audio data using the convolutional recurrent neural network, the audio saliency data, and the human voice discrimination information to obtain the main melody sequence includes: The audio saliency data is used to extract features through the recurrent convolutional layer to obtain saliency features; The saliency features are serialized through the sequence transformation layer to obtain a saliency sequence; The main melody of the human voice is extracted from the sample audio data using the fully connected layer, the saliency sequence, and the human voice discrimination information to obtain the main melody sequence of the human voice.

7. The method according to any one of claims 1 to 5, characterized in that, The step of performing loss calculations on the vocal melody sequence, the preset melody reference sequence, the vocal discrimination information, and the preset discrimination reference information to obtain target loss data specifically includes: Loss calculations are performed on the vocal melody sequence and the melody reference sequence to obtain melody loss data; Loss calculation is performed on the human voice discrimination information and the discrimination reference information to obtain discrimination loss data; The target loss data is obtained by weighting and summing the preset weight parameters, the melody loss data, and the discrimination loss data.

8. A device for extracting the main melody of human voices, characterized in that, The device includes: The data acquisition module is used to acquire sample audio data; The harmonic splicing module is used to perform harmonic splicing processing on the sample audio data to obtain harmonic splicing data; The data input module is used to input the harmonic splicing data into a preset original melody extraction model; wherein, the original melody extraction model includes: a convolutional neural network, a convolutional recurrent neural network, and a fully connected classifier. The data processing module is used to calculate the saliency of the harmonic splicing data through the convolutional neural network to obtain audio saliency data, and to perform human voice discrimination on the audio saliency data through the fully connected classifier to obtain human voice discrimination information. The human voice melody extraction module is used to extract the human voice melody from the sample audio data through the convolutional recurrent neural network, the audio saliency data and the human voice discrimination information to obtain a human voice melody sequence. The loss calculation module is used to perform loss calculation on the vocal melody sequence, the preset melody reference sequence, the vocal discrimination information, and the preset discrimination reference information to obtain the target loss data. The parameter adjustment module is used to adjust the parameters of the original melody extraction model according to the target loss data to obtain the target melody extraction model. The target melody extraction module is used to perform harmonic splicing processing on the acquired target audio data to obtain target spliced ​​data, and input the target spliced ​​data into the target melody extraction model to extract the main melody of human voice, thereby obtaining the target main melody sequence.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method for extracting the main melody of human voices as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for extracting the main melody of human voice as described in any one of claims 1 to 7.

Images