Methods for training vocal note recognition models and vocal note recognition methods.

Abstract

Invention details;

Description

Training method of human voice note recognition model, human voice note recognition method and device Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a training method for a human voice note recognition model, a human voice note recognition method, and a human voice note recognition device. Background Art

[0002] The vocal note recognition of a song refers to obtaining the vocal note sequence of the song based on the song with accompaniment.

[0003] In addition to vocals, songs often include accompaniment from various instruments. Some live performances also contain background noise or reverberation, which poses significant challenges for recognizing vocal notes. Related technologies use a vocal accompaniment separation algorithm to separate the vocal audio from the song. This is then processed using a vocal note recognition model to obtain the song's vocal note sequence.

[0004] However, the above method requires vocal note recognition based on the vocal accompaniment separation algorithm, which has high computational complexity.

[0005] Summary of the Invention

[0006] The present invention provides a method for training a human voice note recognition model, a method for human voice note recognition, and a device for human voice note recognition. The technical solution is as follows:

[0007] According to one aspect of an embodiment of the present application, a method for training a human voice note recognition model is provided, the method comprising:

[0008] Obtain at least one annotated vocal audio, vocal note annotation results corresponding to each of the annotated vocal audio, at least one pure vocal audio, and at least one accompaniment audio;

[0009] Training a first network based on the labeled vocal audio, the accompaniment audio, and the vocal note labeling results corresponding to the labeled vocal audio to obtain a trained first network; the first network is configured to output a vocal note recognition result corresponding to the labeled vocal audio based on a synthesized audio of the labeled vocal audio and the accompaniment audio;

[0010] Based on the trained first network, the pure human voice audio and the accompaniment audio, a second network is trained to obtain a human voice note recognition model; the second network is used to output a human voice note recognition result corresponding to the pure human voice audio based on the synthesized audio of the pure human voice audio and the accompaniment audio.

[0011] According to one aspect of an embodiment of the present application, a method for recognizing human voice notes is provided, the method comprising:

[0012] Acquire target audio with accompaniment, where the target audio includes human voice and accompaniment;

[0013] Acquire audio features of the target audio, where the audio features include features related to the target audio in the time and frequency domains;

[0014] Processing the audio features using a vocal note recognition model to obtain note features of the target audio, wherein the note features include features related to the vocal notes of the target audio;

[0015] Processing the note features using the vocal note recognition model to obtain a vocal note sequence of the target audio;

[0016] The vocal note recognition model is obtained by training a second network based on the trained first network, pure vocal audio and accompaniment audio; the first network is used to output the vocal note recognition result corresponding to the labeled vocal audio based on the synthesized audio of the labeled vocal audio and the accompaniment audio; the second network is used to output the vocal note recognition result corresponding to the pure vocal audio based on the synthesized audio of the pure vocal audio and the accompaniment audio.

[0017] According to one aspect of an embodiment of the present application, a training device for a human voice note recognition model is provided, the device comprising:

[0018] a sample acquisition module configured to acquire a first training sample set, a second training sample set, and a third training sample set, wherein the first training sample set includes at least one labeled vocal audio and a vocal note labeling result corresponding to the labeled vocal audio, the second training sample set includes at least one pure vocal audio, and the third training sample set includes at least one accompaniment audio;

[0019] a first network training module, configured to train a first network based on the labeled vocal audio, the accompaniment audio, and the vocal note labeling results corresponding to the labeled vocal audio, thereby obtaining a trained first network; the first network being configured to output a vocal note recognition result corresponding to the labeled vocal audio based on a synthesized audio of the labeled vocal audio and the accompaniment audio;

[0020] The second network training module is configured to train the second network based on the trained first network, the pure vocal audio, and the accompaniment audio to obtain a vocal note recognition model; the second network is configured to output a vocal note recognition result corresponding to the pure vocal audio based on the synthesized audio of the pure vocal audio and the accompaniment audio.

[0021] According to one aspect of an embodiment of the present application, a device for recognizing a human voice symbol is provided, the device comprising:

[0022] An audio acquisition module is used to acquire target audio with accompaniment, wherein the target audio includes human voice and accompaniment;

[0023] A feature acquisition module, configured to acquire audio features of the target audio, wherein the audio features include features related to the target audio in the time and frequency domains;

[0024] a feature extraction module, configured to process the audio features using a vocal note recognition model to obtain note features of the target audio, wherein the note features include features related to the vocal notes of the target audio;

[0025] A result obtaining module is used to process the note features through the vocal note recognition model to obtain the vocal note sequence of the target audio;

[0026] The vocal note recognition model is obtained by training a second network based on the trained first network, pure vocal audio and accompaniment audio; the first network is used to output the vocal note recognition result corresponding to the labeled vocal audio based on the synthesized audio of the labeled vocal audio and the accompaniment audio; the second network is used to output the vocal note recognition result corresponding to the pure vocal audio based on the synthesized audio of the pure vocal audio and the accompaniment audio.

[0027] According to one aspect of an embodiment of the present application, a computer device is provided, comprising a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-mentioned method for training a human voice note recognition model, or to implement the above-mentioned method for human voice note recognition.

[0028] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to be executed by a processor to implement the above-mentioned training method of the human voice note recognition model, or to implement the above-mentioned human voice note recognition method.

[0029] According to one aspect of an embodiment of the present application, a computer program product is provided, comprising computer instructions stored in a computer-readable storage medium, wherein a processor reads and executes the computer instructions from the computer-readable storage medium to implement the above-mentioned method for training a human voice note recognition model or to implement the above-mentioned method for human voice note recognition.

[0030] The technical solutions provided by the embodiments of the present application may have the following beneficial effects:

[0031] The vocal note recognition model obtained through the above training method can directly identify the corresponding vocal note sequence from the target audio with accompaniment. Therefore, during the model usage phase, there is no need to call the vocal accompaniment separation algorithm to extract the vocal audio from the target audio, which reduces the computational complexity of vocal note recognition. In addition, this application adopts a semi-supervised training method, training the first network with a small number of labeled samples, and then training the second network with the first network and a large number of unlabeled samples. In this way, only a small number of labeled samples are required to train a model with strong generalization performance, reducing the cost of obtaining training samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] FIG1 is a schematic diagram of an implementation environment of a solution provided by an embodiment of the present application;

[0033] FIG2 is a flow chart of a method for training a human voice symbol recognition model according to an embodiment of the present application;

[0034] FIG3 is a flowchart of a method for training a vocal note recognition model according to another embodiment of the present application;

[0035] FIG4 is a flowchart of a method for training a human voice note recognition model according to another embodiment of the present application;

[0036] FIG5 is a schematic diagram of a method for training a human voice note recognition model according to an embodiment of the present application;

[0037] FIG6 is a flowchart of a method for recognizing human voice symbols according to an embodiment of the present application;

[0038] FIG7 is a schematic diagram of a human voice note recognition model provided by one embodiment of the present application;

[0039] FIG8 is a block diagram of a training device for a human voice note recognition model provided by one embodiment of the present application;

[0040] FIG9 is a block diagram of a training device for a vocal note recognition model provided by another embodiment of the present application;

[0041] FIG10 is a block diagram of a human voice symbol recognition device provided by one embodiment of the present application;

[0042] FIG11 is a schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0044] Please refer to FIG1 , which shows a schematic diagram of a solution implementation environment provided by an embodiment of the present application. The solution implementation environment may include: a model using device 10 and a model training device 20 .

[0045] The model using device 10 is used to execute the human voice symbol recognition method in the embodiment of the present application. The model using device 10 can be a terminal device 11 or a server 12. The terminal device 11 can be an electronic device such as a mobile phone, a tablet computer, a game console, an e-book reader, a multimedia playback device, a wearable device, a PC (Personal Computer), a vehicle-mounted terminal, etc. The terminal device 11 can run a target application or a client of the target application. In the embodiment of the present application, the above-mentioned target application refers to an application that provides a human voice symbol recognition function. Optionally, the target application can be a system-level application, such as an operating system or a native application provided by the operating system; it can also be a third-party application, such as a third-party application downloaded and installed by the user, and the embodiment of the present application does not limit this.

[0046] The server 12 may be the background server of the target application, configured to provide background services for the target application in the terminal device 11. The server 12 may be a single server, a server cluster consisting of multiple servers, or a cloud computing service center. Optionally, the server 12 may provide background services for the target applications in multiple terminal devices 11 simultaneously.

[0047] The terminal device 11 and the server 12 can communicate with each other via a network 13. The network 13 can be a wired network or a wireless network.

[0048] In the method for recognizing human voice notes provided in an embodiment of the present application, the execution subject of each step can be a computer device, and the computer device refers to an electronic device with data calculation, processing and storage capabilities. For example, the human voice note recognition method can be executed by the terminal device 11 (such as the client of the target application installed and running in the terminal device 11 executes the human voice note recognition method), or the human voice note recognition method can be executed by the server 12, or the terminal device 11 and the server 12 interact and cooperate to execute it, and this application does not limit this. For example, the terminal device 11 obtains the target audio and sends the target audio to the server 12, and the server 12 executes the human voice note recognition method to obtain a human voice note sequence.

[0049] The model training device 20 is used to execute the training method for the vocal note recognition model in the embodiments of the present application. The model training device 20 can be a server or a computer device, which refers to an electronic device with data computing, processing, and storage capabilities. The model training device 20 trains the vocal note recognition model, and the trained vocal note recognition model is deployed in the model utilization device 10.

[0050] Please refer to Figure 2, which shows a flow chart of a method for training a human voice note recognition model according to an embodiment of the present application. The method may include at least one of the following steps 210 to 230.

[0051] Step 210 , obtaining at least one annotated vocal audio, vocal note annotating results corresponding to each annotated vocal audio, at least one pure vocal audio, and at least one accompaniment audio.

[0052] In some embodiments, a first training sample set, a second training sample set, and a third training sample set can be obtained. The first training sample set includes at least one labeled vocal audio and the vocal note labeling results corresponding to the labeled vocal audio. The second training sample set includes at least one pure vocal audio. The third training sample set includes at least one accompaniment audio.

[0053] Vocals refer to the parts of a song that are sung by the human voice, such as lyrics and harmony. Non-vocals refer to the parts of a song other than the human voice, such as accompaniment, reverb, noise, etc.

[0054] The labeled vocal audio refers to a cappella singing audio, in which the vocal notes corresponding to each audio frame contained in the audio are labeled. The vocal note labeling result corresponding to the labeled vocal audio refers to a vocal note sequence consisting of the vocal notes corresponding to each audio frame contained in the labeled vocal audio.

[0055] Pure vocal audio refers to the audio containing only vocals separated from song audio with accompaniment.

[0056] Accompaniment audio refers to the audio containing only the accompaniment obtained by separating the song audio with accompaniment.

[0057] In some embodiments, a vocal accompaniment separation algorithm can be used to separate pure vocal audio and accompaniment audio from songs with accompaniment. By performing the above separation operation on multiple songs, multiple pure vocal audios can be obtained to construct the second training sample set, and multiple accompaniment audios can be obtained to construct the third training sample set.

[0058] In some embodiments, the number of annotated human voice audios in the first training sample set is much smaller than the number of pure human voice audios in the second training sample set. For example, the first training sample set contains 100 annotated human voice audios, and the second training sample set contains 10,000 pure human voice audios.

[0059] The present application does not limit the number of accompaniment audios in the third training sample set. For example, the number of accompaniment audios in the third training sample set may be the same as or different from the number of pure vocal audios in the second training sample set.

[0060] Step 220 : Based on the labeled vocal audio, the accompaniment audio, and the vocal note labeling results corresponding to the labeled vocal audio, a first network is trained to obtain a trained first network; the first network is configured to output a vocal note recognition result corresponding to the labeled vocal audio based on the synthesized audio of the labeled vocal audio and the accompaniment audio.

[0061] The first network refers to an initialized vocal note recognition model. In some embodiments, the first network may also be referred to as a teacher network, and the second network may also be referred to as a student network.

[0062] In some embodiments, the accompaniment audio and the annotated vocal audio are synthesized to obtain a synthesized audio corresponding to the annotated vocal audio; based on the synthesized audio corresponding to the annotated vocal audio and the vocal note annotating results corresponding to the annotated vocal audio, the first network is trained to obtain a trained first network.

[0063] In some embodiments, the synthesized audio corresponding to the annotated vocal audio includes accompaniment audio and the annotated vocal audio.

[0064] In some embodiments, the synthesized audio corresponding to the labeled human voice audio is processed through the first network to obtain a human voice note recognition result corresponding to the labeled human voice audio, which is used as a first human voice note recognition result; based on the first human voice note recognition result and the human voice note labeling result, the first network is trained to obtain a trained first network.

[0065] The first recognition result of the human voice note refers to the human voice note sequence of the pure human voice audio obtained by the first network. By inputting the synthetic audio corresponding to the labeled human voice audio into the first network, the first network processes the synthetic audio corresponding to the labeled human voice audio, and outputs the first recognition result of the human voice note corresponding to the labeled human voice audio. In some embodiments, the first network is trained according to the loss function to obtain the trained first network. This application does not limit the specific loss function. For example, a cross entropy loss function, an exponential loss function, a log loss function, an absolute value loss function, a Focal-Loss loss function, etc. can be used.

[0066] In some embodiments, the parameters of the first network are adjusted by calculating the loss function value between the first voice note recognition result and the voice note labeling result to obtain the trained first network.

[0067] In some embodiments, the first network is trained by adjusting parameters of the first network by calculating a loss function value between the first voice note recognition result and the voice note labeling result.

[0068] In some embodiments, the first network includes an input layer, an intermediate layer, and an output layer. The input layer is configured to input audio features of synthesized audio corresponding to the labeled vocal audio; the intermediate layer is configured to extract note features of the synthesized audio corresponding to the labeled vocal audio based on the audio features; and the output layer is configured to obtain a vocal note sequence of the synthesized audio corresponding to the labeled vocal audio based on the note features.

[0069] In some embodiments, the input layer obtains the audio features of the synthesized audio corresponding to the labeled human voice audio based on the synthesized audio corresponding to the labeled human voice audio, and transmits the audio features to the middle layer.

[0070] In some embodiments, the input layer directly obtains the audio features of the synthesized audio corresponding to the labeled human voice audio and transmits them to the middle layer.

[0071] In some embodiments, the output layer is also used to identify the vocal and non-vocal parts of the note features.

[0072] In some embodiments, the first network is trained based on the vocal part of the note feature, the first vocal note recognition result, and the vocal note labeling result to obtain a trained first network.

[0073] In some embodiments, the first network is a neural network, and this application does not limit the specific network structure.

[0074] Step 230 : Based on the trained first network, the pure vocal audio, and the accompaniment audio, a second network is trained to obtain a vocal note recognition model; the second network is configured to output a vocal note recognition result corresponding to the pure vocal audio based on the synthesized audio of the pure vocal audio and the accompaniment audio.

[0075] In some embodiments, the second network is trained based on the trained first network, pure human voice audio, and accompaniment audio.

[0076] The second network refers to an initialized human voice note recognition model. In some embodiments, the second network is a neural network, and this application does not limit the specific network structure.

[0077] In some embodiments, the second network and the first network are two networks with the same structure and the same initialization parameters.

[0078] In some embodiments, the pure human voice audio is processed by the trained first network to obtain a human voice note recognition result corresponding to the pure human voice audio as a second human voice note recognition result; the second human voice note recognition result is determined as pseudo label information corresponding to the pure human voice audio; and the second network is trained based on the pseudo label information corresponding to the pure human voice audio, the accompaniment audio, and the pure human voice audio.

[0079] In some embodiments, the second recognition result of the human voice note can be directly determined as pseudo-label information. The solution is simple and easy to implement, and has low computational cost.

[0080] In some embodiments, the second vocal note recognition result is corrected, and the corrected vocal note sequence is determined as pseudo-label information. Correcting the second vocal note recognition result improves the accuracy of the pseudo-label information and further improves the accuracy of the trained vocal note recognition model.

[0081] In some embodiments, the accompaniment audio is synthesized with the pure human voice audio to obtain synthesized audio corresponding to the pure human voice audio; and the second network is trained based on the synthesized audio corresponding to the pure human voice audio and the pseudo-label information.

[0082] In some embodiments, the synthesized audio corresponding to the pure vocal audio includes accompaniment audio and the pure vocal audio.

[0083] In some embodiments, the synthesized audio corresponding to the pure human voice audio is processed by a second network to obtain a human voice note recognition result corresponding to the pure human voice audio, which is used as a third human voice note recognition result; and the second network is trained based on the third human voice note recognition result and the pseudo-label information. The third human voice note recognition result refers to the human voice note sequence of the pure human voice audio obtained by the second network. The synthesized audio corresponding to the pure human voice audio is input to the second network, and the second network processes the synthesized audio corresponding to the pure human voice audio and outputs the third human voice note recognition result.

[0084] In some embodiments, the second network is trained according to a loss function. This application does not limit the specific loss function. For example, a cross entropy loss function, an exponential loss function, a logarithmic loss function, an absolute value loss function, a Focal-Loss loss function, etc. can be used.

[0085] In some embodiments, the parameters of the second network are adjusted by calculating the loss function value between the third recognition result of the human voice note and the pseudo label information to obtain the human voice note recognition model.

[0086] In some embodiments, the parameters of the second network are adjusted and the second network is trained by calculating the loss function value between the third recognition result of the human voice note and the pseudo-label information.

[0087] In some embodiments, the second network includes an input layer, an intermediate layer, and an output layer. The input layer is configured to input audio features of the synthesized audio corresponding to the pure vocal audio; the intermediate layer is configured to extract note features of the synthesized audio corresponding to the pure vocal audio based on the audio features; and the output layer is configured to obtain a vocal note sequence of the synthesized audio corresponding to the pure vocal audio based on the note features.

[0088] In some embodiments, the output layer is also used to identify the vocal and non-vocal parts of the note features.

[0089] In some embodiments, the input layer is used to obtain audio features of the synthesized audio corresponding to the pure human voice audio based on the synthesized audio corresponding to the pure human voice audio, and transmit them to the middle layer.

[0090] In some embodiments, the input layer is used to directly obtain audio features of the synthesized audio corresponding to the pure human voice audio and transmit them to the middle layer.

[0091] In some embodiments, the second network is trained based on the vocal part of the note feature, the second vocal note recognition result, and the pseudo label information.

[0092] In some embodiments, the loss function for training the first network and the loss function for training the second network can be the same or different, and this application is not limited to this. Exemplarily, the loss function for training the first network and the loss function for training the second network are both cross-entropy loss functions. Exemplarily, the loss function for training the first network is a cross-entropy loss function, and the loss function for training the second network is an absolute value loss function.

[0093] A vocal note sequence is a sequence of notes representing the pitch range of a human voice. It includes the starting points, offset points, and pitch values ​​of each pitch range. The offset point is the end point of the pitch range and can be represented by its offset from the starting point, hence the name offset point. Pitch refers to the varying pitches of various sounds, or the height of a sound, and is one of the fundamental characteristics of sound. A pitch range is a section of audio that has the same pitch.

[0094] In some embodiments, the vocal note sequence is a MIDI (Musical Instrument Digital Interface) sequence.

[0095] In some embodiments, the training stopping condition is that the second network converges, that is, the second recognition result of the human voice note corresponding to the pure human voice audio obtained by the second network is infinitely close to the pseudo label information corresponding to the pure human voice audio.

[0096] In some embodiments, whether the second network meets the stop training condition is determined based on the loss function. For example, the stop training condition of the second network is that the loss function value reaches a minimum value.

[0097] In some embodiments, the training stop condition can be set as the number of iterations, and the training stop condition is satisfied when the set number of iterations is reached. The number of iterations can be calculated based on the number of executions of step 230.

[0098] In some embodiments, as shown in FIG3 , the method further includes step 232 of determining whether the second network meets a training stop condition; if so, determining the trained second network as a vocal note recognition model; if not, determining the trained second network as the trained first network, and executing step 230 again. That is, if the second network does not meet the training stop condition, determining the trained second network as the trained first network, and re-starting from the step of training the second network based on the trained first network, the pure vocal audio, and the accompaniment audio (step 230).

[0099] Exemplarily, the second network meets the training stop condition after the nth training. For the i-th training among the n trainings, the second network after the i-1th training is determined as the first network for the i-th training, and the step of training the second network based on the trained first network, pure human voice audio and accompaniment audio (step 230) is started again, where n is an integer greater than 2 and i is an integer greater than 1.

[0100] The technical solution provided by the embodiments of the present application, using the vocal note recognition model obtained through the above-mentioned training method, can directly identify the corresponding vocal note sequence from the target audio with accompaniment. Therefore, during the model usage phase, there is no need to call the vocal accompaniment separation algorithm to extract the vocal audio from the target audio, thereby reducing the computational complexity of vocal note recognition. In addition, the present application adopts a semi-supervised training method, training the first network with a small number of labeled samples, and then training the second network with the first network and a large number of unlabeled samples. In this way, only a small number of labeled samples are required to train a model with strong generalization performance, reducing the cost of obtaining training samples.

[0101] Please refer to Figure 4, which shows a flow chart of a method for training a human voice note recognition model according to another embodiment of the present application. The method may include at least one of the following steps 410 to 440.

[0102] Step 410: Obtain at least one annotated vocal audio, vocal note annotation results corresponding to each of the annotated vocal audio, at least one pure vocal audio, and at least one accompaniment audio.

[0103] In some embodiments, a cappella data set and a song data set are obtained, wherein the cappella data set includes at least one a cappella audio and a vocal note labeling result corresponding to the a cappella audio, and the song data set includes at least one song audio with accompaniment.

[0104] A cappella audio refers to vocal audio performed without accompaniment. The vocal note annotation result corresponding to the a cappella audio is a vocal note sequence consisting of the vocal notes corresponding to each audio frame contained in the a cappella audio.

[0105] Song audio refers to the audio combined with lyrics and accompaniment, which includes accompaniment and vocals. In certain embodiments, song audio also includes noise and reverberation.

[0106] In some embodiments, based on the a cappella audio and the vocal note labeling results corresponding to the a cappella audio, labeled vocal audio and the vocal note labeling results corresponding to the labeled vocal audio are generated to construct a first training sample set.

[0107] In some embodiments, a cappella audio is detected to obtain silent parts and unvoiced parts in the a cappella audio; the a cappella audio is determined as labeled vocal audio; and from the vocal note labeling results corresponding to the a cappella audio, the vocal note labeling results corresponding to the silent parts and the vocal note labeling results corresponding to the unvoiced parts are deleted to generate vocal note labeling results corresponding to the labeled vocal audio, thereby constructing a first training sample set.

[0108] In some embodiments, the a cappella audio is detected using a human voice detection algorithm to obtain silent parts and unvoiced parts in the a cappella audio.

[0109] By adopting the above method, it is ensured that the vocal note annotation result corresponding to the a cappella audio only has pitch in the vocal part, and the silent part and the unvoiced part have no pitch, thereby ensuring the accuracy of the vocal note annotation result corresponding to the a cappella audio.

[0110] In some embodiments, a vocal separation operation is performed on the song audio to obtain vocal audio and accompaniment audio; based on the vocal audio, pure vocal audio is generated to construct a second training sample set; based on the accompaniment audio, a third training sample set is constructed.

[0111] This application does not limit the specific method of performing vocal separation on song audio. For example, a vocal separation operation is performed on the song audio using a vocal accompaniment separation algorithm to obtain vocal audio and accompaniment audio.

[0112] In some embodiments, human voice audio is detected to obtain the non-human voice part in the human voice audio; the non-human voice part in the human voice audio is deleted to generate pure human voice audio; and a second training sample set is constructed based on the pure human voice audio.

[0113] In some embodiments, a vocal audio is detected using a vocal detection algorithm to obtain non-vocal portions of the vocal audio, which are then deleted to generate pure vocal audio. For example, a vocal audio is detected using a vocal detection algorithm to obtain non-vocal portions of the vocal audio, which are then deleted for more than 3 seconds to generate pure vocal audio. Generally, vocals only occupy a portion of a song, and the number of training samples in the second training sample set required for training is large. Deleting the non-vocal portions of the vocal audio can improve training efficiency and save storage space required for the second training sample set.

[0114] In some embodiments, all the obtained pure human voice audio is used to construct a second training sample set.

[0115] Since the vocal and accompaniment separation algorithm cannot guarantee the perfect separation of the vocals and accompaniment of each song, it is necessary to clean the pure vocal audio and remove the pure vocal audio with residual accompaniment.

[0116] In some embodiments, for each audio frame in the pure human voice audio, it is detected whether the audio frame is a human voice audio frame, and the energy of the audio frame is calculated; if the audio frame is not a human voice audio frame and the energy of the audio frame is less than a second threshold, the audio frame is determined to be an invalid frame; if the number of invalid frames in the pure human voice audio accounts for a proportion of the total number of audio frames contained in the pure human voice audio that is greater than a third threshold, the pure human voice audio is determined to be invalid pure human voice audio; and pure human voice audio is generated based on the pure human voice audio other than the invalid pure human voice audio.

[0117] In some embodiments, the specific values ​​of the second threshold and the third threshold can be set according to actual needs and are not limited in this application. For example, the value of the second threshold can be different for songs of different styles, for example, the second threshold for rock songs is higher than the second threshold for ancient style songs.

[0118] Exemplarily, the value of the third threshold is set to 30%. If the number of invalid frames in the pure human voice audio accounts for more than 30% of the total number of audio frames contained in the pure human voice audio, the pure human voice audio is determined to be invalid pure human voice audio.

[0119] In some embodiments, all pure human voice audios except invalid pure human voice audios are obtained to generate pure human voice audios.

[0120] Step 420 : synthesize the accompaniment audio and the annotated vocal audio to obtain a synthesized audio corresponding to the annotated vocal audio.

[0121] In some embodiments, an accompaniment audio is randomly selected from at least one accompaniment audio as a target accompaniment audio; data enhancement processing is performed on the labeled vocal audio to obtain processed labeled vocal audio; wherein the data enhancement processing includes at least one of the following: adding reverberation, changing the fundamental frequency; the target accompaniment audio and the processed labeled vocal audio are synthesized to obtain a synthesized audio corresponding to the labeled vocal audio.

[0122] In some embodiments, the accompaniment audio is randomly selected from the third training sample set as the target accompaniment audio.

[0123] When sound waves encounter obstacles during propagation, they are reflected by them, and each reflection absorbs some of the sound. This phenomenon is called reverberation, as the sound waves continue to reflect and absorb multiple times after the sound source stops. This phenomenon is known as reverberation. Adding reverberation to audio annotated with human voices can improve the sound quality of the audio.

[0124] Changing the fundamental frequency means changing the fundamental frequency of the marked vocal audio and the vocal note annotation result corresponding to the marked vocal audio within a certain range. This application does not limit the range of changing the fundamental frequency. For example, the fundamental frequency of the marked vocal audio is changed within the range of -200 to +300 cents, and the vocal note annotation result corresponding to the marked vocal audio is adjusted to the corresponding pitch. For example, the fundamental frequency of the marked vocal audio is increased by 200 cents, and the pitch of the vocal note annotation result corresponding to the marked vocal audio is also increased by 200 cents.

[0125] In some embodiments, the fundamental frequency of any one or more audio frames included in the annotated vocal audio and the pitch of the vocal note annotated results corresponding to the one or more audio frames may be changed.

[0126] Step 430 : Training the first network based on the synthesized audio corresponding to the labeled human voice audio and the human voice note labeling result corresponding to the labeled human voice audio to obtain a trained first network.

[0127] In some embodiments, the synthesized audio corresponding to the labeled human voice audio is processed by the first network to obtain a human voice note recognition result corresponding to the labeled human voice audio as a first human voice note recognition result; based on the first human voice note recognition result and the human voice note labeling result, the loss function value of the first network is determined; based on the loss function value of the first network, the parameters of the first network are adjusted to obtain the trained first network.

[0128] In some embodiments, the first network is trained using a cross entropy loss function.

[0129] In some embodiments, based on the synthesized audio corresponding to the labeled human voice audio and the human voice note labeling results, the first network is trained until convergence to obtain a trained first network.

[0130] Step 440 : Based on the trained first network, the pure human voice audio, and the accompaniment audio, the second network is trained to obtain a human voice note recognition model.

[0131] In some embodiments, the pure human voice audio is processed by the trained first network to obtain a human voice note recognition result corresponding to the pure human voice audio as a second human voice note recognition result; the second human voice note recognition result is determined as pseudo label information corresponding to the pure human voice audio; and the second network is trained based on the pure human voice audio, the accompaniment audio and the pseudo label information.

[0132] In some embodiments, the fundamental frequency of the pure human voice audio is extracted; and the second recognition result of the human voice note is corrected according to the fundamental frequency of the pure human voice audio to obtain pseudo label information corresponding to the pure human voice audio.

[0133] In some embodiments, the fundamental frequency of pure human voice audio is extracted through a fundamental frequency extraction algorithm.

[0134] In some embodiments, for each note included in the second recognition result of the vocal note, the pitch difference between the note and the fundamental frequency of the pronunciation position corresponding to the note is calculated; if the pitch difference is greater than a first threshold, the pitch of the note is corrected to the pitch of the fundamental frequency of the pronunciation position corresponding to the note; if the pitch difference is less than or equal to the first threshold, the pitch of the note is kept unchanged.

[0135] In some embodiments, this application does not limit the value of the first threshold.

[0136] Exemplarily, the value of the first threshold is 3 MIDI values. If the pitch difference between a note and the fundamental frequency of the pronunciation position corresponding to the note is greater than 3 MIDI values, the pitch of the note is corrected to the pitch of the fundamental frequency of the pronunciation position corresponding to the note; if the pitch difference is less than or equal to 3 MIDI values, the pitch of the note remains unchanged.

[0137] For example, if the fundamental frequency of the note's corresponding pronunciation position is 5 MIDI values, if the note's pitch is less than 2 MIDI values, or greater than 8 MIDI values, the note's pitch is corrected to 5 MIDI values; if the note's pitch is between 2 and 8 MIDI values, the note's pitch remains unchanged.

[0138] By correcting the second recognition result of the human voice note in the above manner, the accuracy of the pseudo-label information corresponding to the pure human voice audio is ensured, making the semi-supervised training method more efficient and stable.

[0139] In some embodiments, the accompaniment audio is synthesized with the pure human voice audio to obtain a synthesized audio corresponding to the pure human voice audio; the synthesized audio corresponding to the pure human voice audio is processed by a second network to obtain a human voice note recognition result corresponding to the pure human voice audio as a third human voice note recognition result; and the second network is trained based on the third human voice note recognition result and pseudo-label information.

[0140] In some embodiments, a loss function value of the second network is determined based on the third recognition result of the human voice note and the pseudo-label information; and parameters of the second network are adjusted based on the loss function value of the second network to obtain a human voice note recognition model.

[0141] In some embodiments, the second network is trained using a cross entropy loss function.

[0142] In some embodiments, the second network can also perform human voice recognition on the synthesized audio corresponding to the pure human voice audio, obtain the human voice part of the synthesized audio corresponding to the pure human voice audio and the non-human voice part of the synthesized audio corresponding to the pure human voice audio, and then train the second network based on the human voice part of the synthesized audio corresponding to the pure human voice audio, the non-human voice part of the synthesized audio corresponding to the pure human voice audio, and the pure human voice audio.

[0143] In some embodiments, a fully connected layer can be used to perform human voice recognition on the synthesized audio corresponding to the pure human voice audio to obtain the human voice portion of the synthesized audio corresponding to the pure human voice audio and the non-human voice portion of the synthesized audio corresponding to the pure human voice audio. For example, a softmax classifier can be used to classify the human voice portion of the synthesized audio corresponding to the pure human voice audio and the non-human voice portion of the synthesized audio corresponding to the pure human voice audio.

[0144] In some embodiments, the method further includes step 442, determining whether the second network meets the training stop condition; if so, determining the trained second network as the vocal note recognition model; if not, determining the trained second network as the trained first network, and executing the above step 440 again.

[0145] For example, please refer to FIG5 , which shows a schematic diagram of a method for training a human voice note recognition model provided by an embodiment of the present application.

[0146] Step 1: Randomly select accompaniment audio from the third training sample set (also known as data set 3) 511 as the target accompaniment audio; perform data enhancement processing on the labeled human voice audio in the first training sample set (also known as data set 1) 512 to obtain processed labeled human voice audio; synthesize the target accompaniment audio with the processed labeled human voice audio to obtain a synthesized audio corresponding to the labeled human voice audio.

[0147] The synthesized audio corresponding to the labeled vocal audio is processed through the teacher network 513 to obtain a vocal note recognition result corresponding to the labeled vocal audio as a first vocal note recognition result; based on the first vocal note recognition result and the vocal note labeling result corresponding to the labeled vocal audio, the loss function value 514 (cross entropy loss function) of the teacher network is determined; based on the loss function value 514 (cross entropy loss function) of the teacher network, the teacher network 513 is trained to obtain a trained teacher network 521.

[0148] Step 2: The pure human voice audio in the second training sample set (also referred to as data set 2) 522 is processed through the trained teacher network 521 to obtain the human voice note recognition result corresponding to the pure human voice audio, which is used as the human voice note second recognition result (also referred to as the pseudo label corresponding to the pure human voice audio) 523; based on the human voice note second recognition result 523, the pseudo label information corresponding to the pure human voice audio is determined (also referred to as the pseudo label correction corresponding to the pure human voice audio) 524.

[0149] Step 3: Randomly select accompaniment audio from the third training sample set 511 as the target accompaniment audio; perform data enhancement processing on the pure human voice audio in at least one pure human voice audio 522 to obtain processed pure human voice audio; synthesize the target accompaniment audio with the processed pure human voice audio to obtain a synthesized audio corresponding to the pure human voice audio.

[0150] The synthesized audio corresponding to the pure human voice audio is processed through the student network 525 to obtain the human voice note student recognition result corresponding to the pure human voice audio as the human voice note third recognition result (also called the prediction corresponding to the pure human voice audio) 526.

[0151] Step 4: Determine the loss function value 527 (cross entropy loss function) of the student network based on the human voice symbol student recognition result 526 corresponding to the pure human voice audio and the pseudo label information 524 corresponding to the pure human voice audio; train the student network 525 based on the loss function value 527 (cross entropy loss function) of the student network to obtain a trained student network 531.

[0152] Reasoning: If the trained student network 531 does not meet the training stop condition, the trained student network 531 is determined as the trained teacher network, and the process starts again from step 2. That is, the trained teacher network 521 in step 2 is replaced with the trained student network 531, and the process starts again from step 2.

[0153] When the trained student network 531 meets the training stop condition, the trained student network 531 is determined as a vocal note recognition model. A song with accompaniment is input, and the vocal note recognition model processes the song with accompaniment to obtain a vocal note sequence 533 corresponding to the song with accompaniment.

[0154] The technical solution provided in the embodiment of the present application, through the strategy of random data amplification, further expands the number of training samples on the basis of existing training samples to train the human voice note recognition model, thereby further improving the robustness of the human voice note recognition model.

[0155] Please refer to Figure 6, which shows a flow chart of a method for human voice character recognition provided by an embodiment of the present application. The method may include at least one of the following steps 610 to 640.

[0156] Step 610: Acquire target audio with accompaniment, where the target audio includes human voice and accompaniment.

[0157] In some embodiments, the target audio also includes noise and reverberation.

[0158] In some embodiments, the present application does not limit the type of target audio with accompaniment. For example, the target audio can be a song with accompaniment or a live song recording.

[0159] Step 620: Acquire audio features of the target audio, where the audio features include features related to the target audio in the time and frequency domains.

[0160] In some embodiments, a time-frequency transformation is performed on the target audio to obtain the frequency domain features of the target audio; and a filtering process is performed on the frequency domain features to obtain the audio features of the target audio.

[0161] This application does not limit the specific method for performing time-frequency transformation on the target audio. For example, a CWT-ESS (Continuous Wavelet Transform) algorithm, an STFT-ESS (Short-Time Fourier Transform) algorithm, an OpenGAN algorithm, etc. may be used.

[0162] This application does not limit the method of filtering the frequency domain features. For example, low-pass filtering, high-pass filtering, band-pass filtering, band-stop filtering, etc. can be used.

[0163] Step 630 : Process the audio features using a vocal note recognition model to obtain note features of the target audio, where the note features include features related to the vocal notes of the target audio.

[0164] The vocal note recognition model is obtained by training a second network based on the trained first network, pure vocal audio and accompaniment audio; the first network is used to output the vocal note recognition result corresponding to the labeled vocal audio based on the synthesized audio of the labeled vocal audio and the accompaniment audio; the second network is used to output the vocal note recognition result corresponding to the pure vocal audio based on the synthesized audio of the pure vocal audio and the accompaniment audio.

[0165] In some embodiments, for each audio frame contained in the target audio, the audio features of the audio frame and the context information of the audio features of the audio frame are processed by a human voice note recognition model to obtain a first intermediate feature corresponding to the audio frame; based on the first intermediate feature corresponding to the audio frame, the second intermediate feature corresponding to the audio frame is extracted; based on the second intermediate feature corresponding to the audio frame and the context information of the second intermediate feature corresponding to the audio frame, the note feature corresponding to the audio frame is obtained; wherein, the note feature of the target audio includes the note features corresponding to each audio frame contained in the target audio.

[0166] The first intermediate feature corresponding to the audio frame includes the audio feature corresponding to the audio frame and context information of the audio feature corresponding to the audio frame.

[0167] The second intermediate feature corresponding to the audio frame is used to represent the pitch feature of the audio frame.

[0168] The note feature corresponding to the audio frame includes the second intermediate feature corresponding to the audio frame and context information of the second intermediate feature corresponding to the audio frame.

[0169] Context information refers to the association information between the target audio frame and adjacent audio frames. Adjacent audio frames refer to adjacent audio frames and / or adjacent audio frames of the target audio frame. Adjacent audio frames refer to audio frames that do not contain other audio frames between them and the target audio frame. Adjacent audio frames refer to audio frames within a certain range of the target audio frame. For example, the five audio frames before and after the target audio frame can be called adjacent audio frames. This application does not limit the range of determining adjacent audio frames.

[0170] This application does not limit the method for obtaining the first intermediate feature corresponding to the audio frame based on the audio features of the audio frame and the contextual information of the audio features of the audio frame. Exemplarily, it can be implemented using a recursive neural network. For example, it can be implemented using an LSTM (Long Short Term Memory Network) model or a GRU (Gate Recurrent Unit) model.

[0171] This application does not limit the method for extracting the second intermediate features corresponding to the audio frame based on the first intermediate features corresponding to the audio frame. Exemplarily, it can be implemented through a convolutional neural network. For example, it can be implemented through a CNN (Convolutional Neural Network) or a residual convolutional neural network (ResNet).

[0172] This application does not limit the method for obtaining the note feature corresponding to the audio frame based on the second intermediate feature corresponding to the audio frame and the context information of the second intermediate feature corresponding to the audio frame. Exemplarily, a recursive neural network can be used for implementation. For example, it can be implemented using an LSTM (Long Short Term Memory Network) model or a GRU (Gate Recurrent Unit) model.

[0173] Step 640: Process the note features using a vocal note recognition model to obtain a vocal note sequence of the target audio.

[0174] In some embodiments, the note features of the target audio are classified and processed by a vocal note recognition model to obtain a vocal note sequence of the target audio.

[0175] In some embodiments, the note features of the target audio are classified according to the pitch of the note features of the target note to obtain a vocal note sequence of the target audio.

[0176] Exemplarily, the vocal note sequence of the target audio is a MIDI sequence, and the note features of the target audio are classified into different MIDI values ​​according to the pitches of the note features of the target notes to obtain the MIDI sequence of the target audio.

[0177] In some embodiments, the human voice note recognition model includes: an input layer, an intermediate layer, and an output layer.

[0178] The input layer is used to input the audio features of the target audio.

[0179] The middle layer is used to extract the note features of the target audio based on the audio features.

[0180] The intermediate layers include a first intermediate feature extraction layer, a second intermediate feature extraction layer and a note feature extraction layer.

[0181] For each audio frame contained in the target audio, the first intermediate feature extraction layer is used to obtain the first intermediate features corresponding to the audio frame based on the audio features of the audio frame and the contextual information of the audio features of the audio frame. The second intermediate feature extraction layer is used to extract the second intermediate features corresponding to the audio frame based on the first intermediate features corresponding to the audio frame. The note feature extraction layer is used to obtain the note features corresponding to the audio frame based on the second intermediate features corresponding to the audio frame and the contextual information of the second intermediate features corresponding to the audio frame.

[0182] In some embodiments, the first feature extraction layer is a bidirectional LSTM model, the second feature extraction layer is a CNN model, and the note feature extraction layer is a bidirectional LSTM model. In some embodiments, the second feature extraction layer can be configured with one or more CNN networks to form a CNN model as needed, which is not limited in this application. For example, a CNN model can be configured with a 5-layer CNN network.

[0183] The output layer is used to obtain the vocal note sequence of the target audio based on the note features.

[0184] In some embodiments, the output layer is a fully connected layer. In some embodiments, the output layer uses Softmax as a classifier.

[0185] 7 , the vocal note recognition model 700 includes an input layer 710 , an intermediate layer 720 , and an output layer 730 . The intermediate layer 720 includes a first intermediate feature extraction layer 721 , a second intermediate feature extraction layer 722 , and a note feature extraction layer 730 .

[0186] It should be noted that the above-mentioned embodiment of the method for recognizing a human voice note and the above-mentioned embodiment of the method for training a human voice note recognition model have the same concept. Please refer to the above-mentioned embodiment of the method for training a human voice note recognition model, and no further details will be given here.

[0187] The technical solution provided in the embodiment of the present application can identify the vocal note sequence of the target note with accompaniment through the vocal note recognition model, without calling the vocal accompaniment separation algorithm, reducing the complexity of calculation and thus reducing production costs. At the same time, the accuracy is not affected by the vocal accompaniment separation algorithm, thereby ensuring the accuracy of the vocal note sequence.

[0188] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0189] Please refer to Figure 8, which shows a block diagram of a training device for a human voice symbol recognition model provided by one embodiment of the present application. The device has the function of implementing the above-mentioned method example, and the function can be implemented by hardware or by hardware executing corresponding software. The device can be the terminal device described above, or it can be set in a terminal device. As shown in Figure 8, the device 800 can include: a sample acquisition module 810, a first network training module 820, and a second network training module 830.

[0190] The sample acquisition module 810 is configured to acquire at least one annotated vocal audio, vocal note annotation results corresponding to each of the annotated vocal audios, at least one pure vocal audio, and at least one accompaniment audio.

[0191] The first network training module 820 is configured to train a first network based on the labeled vocal audio, the accompaniment audio, and the vocal note labeling results corresponding to the labeled vocal audio to obtain a trained first network; the first network is configured to output a vocal note recognition result corresponding to the labeled vocal audio based on the synthesized audio of the labeled vocal audio and the accompaniment audio.

[0192] The second network training module 830 is configured to train a second network based on the trained first network, the pure vocal audio, and the accompaniment audio to obtain a vocal note recognition model; the second network is configured to output a vocal note recognition result corresponding to the pure vocal audio based on the synthesized audio of the pure vocal audio and the accompaniment audio.

[0193] In some embodiments, as shown in FIG9 , the first network training module 820 includes a first synthesis unit 821 and a first training unit 822 .

[0194] A first synthesis unit 821 is configured to synthesize the accompaniment audio with the annotated vocal audio to obtain a synthesized audio corresponding to the annotated vocal audio;

[0195] The first training unit 822 is configured to train the first network based on the synthesized audio corresponding to the labeled human voice audio and the human voice note labeling results corresponding to the labeled human voice audio to obtain the trained first network.

[0196] In some embodiments, the first synthesis unit 821 is used to randomly select an accompaniment audio from the at least one accompaniment audio as the target accompaniment audio; perform data enhancement processing on the labeled vocal audio to obtain processed labeled vocal audio; wherein the data enhancement processing includes at least one of the following: adding reverberation, changing the fundamental frequency; synthesizing the target accompaniment audio with the processed labeled vocal audio to obtain synthesized audio corresponding to the labeled vocal audio.

[0197] In some embodiments, the first training unit 822 is configured to process the synthesized audio corresponding to the labeled human voice audio through the first network to obtain a human voice note recognition result corresponding to the labeled human voice audio as a first human voice note recognition result; determine a loss function value of the first network based on the first human voice note recognition result and the human voice note labeling result; and adjust parameters of the first network based on the loss function value of the first network to obtain the trained first network.

[0198] In some embodiments, as shown in FIG9 , the second network training module 830 includes a first processing unit 831 , a determination unit 832 , a second synthesis unit 833 , a second processing unit 834 and a second training unit 835 .

[0199] The first processing unit 831 is configured to process the pure human voice audio through the trained first network to obtain a human voice note recognition result corresponding to the pure human voice audio as a second human voice note recognition result.

[0200] The determining unit 832 is configured to determine the second recognition result of the human voice note as pseudo label information corresponding to the pure human voice audio.

[0201] The second synthesis unit 833 is used to synthesize the accompaniment audio and the pure human voice audio to obtain synthesized audio corresponding to the pure human voice audio.

[0202] The second processing unit 834 is configured to process the synthesized audio corresponding to the pure human voice audio through the second network to obtain a human voice note recognition result corresponding to the pure human voice audio as a third human voice note recognition result.

[0203] The second training unit 835 is configured to train the second network according to the third recognition result of the human voice note and the pseudo label information corresponding to the pure human voice audio to obtain a human voice note recognition model.

[0204] In some embodiments, the determining unit 832 is configured to extract the fundamental frequency of the pure human voice audio; and modify the second recognition result of the human voice note according to the fundamental frequency of the pure human voice audio to obtain pseudo label information corresponding to the pure human voice audio.

[0205] In some embodiments, the determination unit 832 is configured to calculate, for each note included in the second vocal note recognition result, the pitch difference between the note and the fundamental frequency of the pronunciation position corresponding to the note; if the pitch difference is greater than a first threshold, correct the pitch of the note to the pitch of the fundamental frequency of the pronunciation position corresponding to the note; if the pitch difference is less than or equal to the first threshold, keep the pitch of the note unchanged; and determine the second vocal note recognition result after pitch adjustment as the pseudo-label information corresponding to the pure vocal audio.

[0206] In some embodiments, the second training unit 835 is configured to determine a loss function value of the second network based on the third vocal note recognition result and the pseudo-label information; and adjust parameters of the second network based on the loss function value of the second network to obtain the vocal note recognition model.

[0207] In some embodiments, the second network training module 830 is further configured to, when the second network does not meet the training stopping condition, determine the trained second network as the trained first network, and again start executing the step of training the second network based on the trained first network, the pure human voice audio, and the accompaniment audio.

[0208] In some embodiments, the sample acquisition module 810 is used to obtain at least one a cappella audio, the vocal note labeling results corresponding to each of the a cappella audios, and at least one song audio with accompaniment; based on the a cappella audio and the vocal note labeling results corresponding to the a cappella audio, generate the labeled vocal audio and the vocal note labeling results corresponding to the labeled vocal audio; perform a vocal separation operation on the song audio to obtain vocal audio and accompaniment audio; and generate the pure vocal audio based on the vocal audio.

[0209] In some embodiments, the sample acquisition module 810 is used to detect the a cappella audio to obtain the silent part and the unvoiced part in the a cappella audio; determine the a cappella audio as the labeled vocal audio; delete the vocal note labeling results corresponding to the silent part and the vocal note labeling results corresponding to the unvoiced part from the vocal note labeling results corresponding to the a cappella audio, and generate the vocal note labeling results corresponding to the labeled vocal audio.

[0210] In some embodiments, the sample acquisition module 810 is used to detect the human voice audio to obtain the non-human voice part in the human voice audio; delete the non-human voice part in the human voice audio to generate pure human voice audio; for each audio frame in the pure human voice audio, detect whether the audio frame is a human voice audio frame, and calculate the energy of the audio frame; if the audio frame is not the human voice audio frame and the energy of the audio frame is less than a second threshold, determine the audio frame as an invalid frame; if the number of invalid frames in the pure human voice audio accounts for a proportion of the total number of audio frames contained in the pure human voice audio that is greater than a third threshold, determine the pure human voice audio as invalid pure human voice audio; and generate the pure human voice audio based on the pure human voice audio other than the invalid pure human voice audio.

[0211] The technical solution provided by the embodiments of the present application, using the vocal note recognition model obtained through the above-mentioned training method, can directly identify the corresponding vocal note sequence from the target audio with accompaniment. Therefore, during the model usage phase, there is no need to call the vocal accompaniment separation algorithm to extract the vocal audio from the target audio, thereby reducing the computational complexity of vocal note recognition. In addition, the present application adopts a semi-supervised training method, training the first network with a small number of labeled samples, and then training the second network with the first network and a large number of unlabeled samples. In this way, only a small number of labeled samples are required to train a model with strong generalization performance, reducing the cost of obtaining training samples.

[0212] Please refer to Figure 10, which shows a block diagram of a human voice symbol recognition device provided by one embodiment of the present application. This device has the functions of implementing the above-mentioned method examples. These functions can be implemented through hardware or by hardware executing corresponding software implementations. This device can be the terminal device described above, or it can be provided in a terminal device. As shown in Figure 10, the device 1000 may include: an audio acquisition module 1010, a feature acquisition module 1020, a feature extraction module 1030, and a result acquisition module 1040.

[0213] The audio acquisition module 1010 is used to acquire target audio with accompaniment, where the target audio includes human voice and accompaniment.

[0214] The feature acquisition module 1020 is configured to acquire audio features of the target audio, where the audio features include features related to the target audio in the time and frequency domains.

[0215] The feature extraction module 1030 is configured to process the audio features using a vocal note recognition model to obtain note features of the target audio, where the note features include features related to the vocal notes of the target audio.

[0216] The result obtaining module 1040 is used to process the note features through the vocal note recognition model to obtain the vocal note sequence of the target audio; wherein the vocal note recognition model is obtained by training the second network based on the trained first network, pure vocal audio and accompaniment audio; the first network is used to output the vocal note recognition result corresponding to the labeled vocal audio based on the synthesized audio of the labeled vocal audio and the accompaniment audio; the second network is used to output the vocal note recognition result corresponding to the pure vocal audio based on the synthesized audio of the pure vocal audio and the accompaniment audio.

[0217] In some embodiments, the feature extraction module 1030 is used to obtain, for each audio frame contained in the target audio, a first intermediate feature corresponding to the audio frame based on the audio features of the audio frame and the context information of the audio features of the audio frame through the human voice note recognition model; extract a second intermediate feature corresponding to the audio frame based on the first intermediate feature corresponding to the audio frame; and obtain a note feature corresponding to the audio frame based on the second intermediate feature corresponding to the audio frame and the context information of the second intermediate feature corresponding to the audio frame; wherein the note feature of the target audio includes the note features corresponding to each audio frame contained in the target audio.

[0218] In some embodiments, the feature acquisition module 1020 is configured to perform time-frequency transformation on the target audio to obtain frequency domain features of the target audio; and perform filtering processing on the frequency domain features to obtain audio features of the target audio.

[0219] In some embodiments, the result obtaining module 1040 is configured to classify the note features of the target audio using the vocal note recognition model to obtain a vocal note sequence of the target audio.

[0220] In some embodiments, the vocal note sequence is obtained by a vocal note recognition model, which includes: an input layer, an intermediate layer, and an output layer; the input layer is used to input audio features of the target audio; the intermediate layer is used to extract note features of the target audio based on the audio features; and the output layer is used to obtain the vocal note sequence of the target audio based on the note features.

[0221] The technical solution provided in the embodiment of the present application can identify the vocal note sequence of the target note with accompaniment through the vocal note recognition model, without calling the vocal accompaniment separation algorithm, thereby reducing the complexity of the calculation. At the same time, the accuracy is not affected by the vocal accompaniment separation algorithm, thereby ensuring the accuracy of the vocal note sequence.

[0222] It should be noted that, when the device provided in the above embodiment realizes its function, it only uses the division of the above-mentioned functional modules as an example. In actual application, the above-mentioned functions can be distributed and completed by different functional modules according to actual needs, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0223] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0224] Please refer to Figure 11, which shows a schematic diagram of the structure of a computer device provided in one embodiment of the present application. The computer device can be any electronic device with data calculation, processing and storage functions. The computer device can be used to implement the training method of the human voice note recognition model provided in the above embodiment, or to implement the human voice note recognition method provided in the above embodiment. Specifically:

[0225] The computer device 1100 includes a central processing unit (CPU, central processing unit), GPU (graphics processing unit), and FPGA (field programmable gate array) 1101, a system memory 1104 including RAM (random-access memory) 1102 and ROM (read-only memory) 1103, and a system bus 1105 connecting the system memory 1104 and the central processing unit 1101. The computer device 1100 also includes a basic input / output system (I / O system) 1106 for facilitating information transmission between various components within the server, and a mass storage device 1107 for storing an operating system 1113, application programs 1114, and other program modules 1111.

[0226] In some embodiments, the basic input / output system 1106 includes a display 1108 for displaying information and an input device 1109, such as a mouse or keyboard, for user input. Both the display 1108 and the input device 1109 are connected to the central processing unit 1101 via an input / output controller 1110 connected to the system bus 1105. The basic input / output system 1106 may also include an input / output controller 1110 for receiving and processing input from a variety of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1110 also provides output to a display screen, printer, or other types of output devices.

[0227] The mass storage device 1107 is connected to the central processing unit 1101 via a mass storage controller (not shown) connected to the system bus 1105. The mass storage device 1107 and its associated computer-readable media provide non-volatile storage for the computer device 1100. In other words, the mass storage device 1107 may include a computer-readable medium (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0228] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other solid-state storage technology, CD-ROM, DVD (Digital Video Disc) or other optical storage, tape cassettes, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage media is not limited to the above-mentioned ones. The above-mentioned system memory 1104 and mass storage device 1107 can be collectively referred to as memory.

[0229] According to an embodiment of the present application, the computer device 1100 can also be connected to a remote computer on a network such as the Internet for operation. That is, the computer device 1100 can be connected to the network 1112 via the network interface unit 1111 connected to the system bus 1105. Alternatively, the network interface unit 1111 can be used to connect to other types of networks or remote computer systems (not shown).

[0230] The memory stores a computer program, which is loaded and executed by the processor to implement the above-mentioned method for training the human voice note recognition model or to implement the above-mentioned method for human voice note recognition.

[0231] In an exemplary embodiment, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the above-mentioned method for training a vocal note recognition model or to implement the above-mentioned method for recognizing vocal notes.

[0232] Optionally, the computer-readable storage medium may include: ROM (Read-Only Memory), RAM (Random-Access Memory), SSD (Solid State Drives), or an optical disk, etc. Among them, the random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0233] In an exemplary embodiment, a computer program product is further provided, comprising a computer program stored in a computer-readable storage medium, wherein a processor reads and executes the computer program from the computer-readable storage medium to implement the above-mentioned method for training a vocal note recognition model or the above-mentioned method for recognizing vocal notes.

[0234] In the description of the embodiments of the present application, the term "corresponding" may indicate a direct or indirect correspondence between the two, or an association relationship between the two, or a relationship between indication and being indicated, configuration and being configured, etc.

[0235] In this document, "plurality" refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0236] In addition, the step numbers described in this document only illustrate a possible execution order between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order of the diagram. The embodiments of the present application are not limited to this.

[0237] In addition, the embodiments provided herein may be arbitrarily combined to form new embodiments, which are all within the scope of protection of this application.

[0238] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0239] The above description is merely an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Images