A music score generating method, device, electronic device and storage medium

Abstract

The present disclosure relates to the field of speech processing technology, and in particular to a method, device, electronic device and storage medium for generating a musical score, which performs text recognition on audio data, determines various text information contained in the audio data, and time information corresponding to the various text information, wherein each text information contains at least one text character, and each time information represents the time period information occupied by the audio interval containing the corresponding text information in the audio data; based on a trained note recognition model, the audio features corresponding to the audio data and various time information are used as input parameters to respectively determine the musical note data contained in the audio data within the various time information; based on the determined various note data and various text information, generates musical score data corresponding to the audio data. In this way, the richness and completeness of the musical score can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of speech processing technology, and in particular to a music score generation method, device, electronic device and storage medium. Background Art

[0002] Currently, the processing technology for converting singing into musical scores can be applied to a variety of different scenarios, for example, it can be applied to the scenario of training singing synthesis models.

[0003] In the related art, when converting singing into music scores, digital signal processing methods are usually used to calculate the pitch information of frame-level audio data, and then extract the melody sequence from the audio data through post-processing such as smoothing estimation and averaging.

[0004] However, in the related art, since the melody sequence extracted from the audio data only contains the pitch information of the notes, if the audio data is a human voice, only the melody sequence can be extracted, but the singing text / lyrics corresponding to the human voice cannot be obtained, which will reduce the richness and completeness of the generated music score. Summary of the invention

[0005] The embodiments of the present disclosure provide a music score generation method, device, electronic device and storage medium to improve the richness and completeness of the generated music score.

[0006] The specific technical solutions provided by the embodiments of the present disclosure are as follows:

[0007] A music score generating method, comprising:

[0008] Performing text recognition on the audio data to determine each text information contained in the audio data and time information corresponding to each text information, wherein each text information contains at least one text character, and each time information represents the time period information occupied by the audio interval containing the corresponding text information in the audio data;

[0009] Based on the trained note recognition model, the audio features corresponding to the audio data and each time information are used as input parameters to respectively determine the note data contained in the audio data in each time information;

[0010] Based on the determined note data and text information, music score data corresponding to the audio data is generated.

[0011] Optionally, performing text recognition on the audio data to determine each text information contained in the audio data and time information corresponding to each text information includes:

[0012] Based on the trained text recognition model, taking the audio data as an input parameter, respectively determining the text information corresponding to each audio frame in the audio data, and based on the text information corresponding to each audio frame, determining each text information contained in the audio data, and the time information corresponding to each text information;

[0013] The audio data at least includes audio frames, and the text recognition model is obtained through iterative training based on audio data samples and corresponding text information labels.

[0014] Optionally, the audio features include at least a fundamental frequency sequence and a singing voice feature, and the step of respectively determining the note data included in the audio data in each time information includes:

[0015] Based on the fundamental frequency sequence and the singing voice feature, respectively determine the note data corresponding to each audio frame included in the audio data;

[0016] Based on the note data corresponding to each audio frame, the note data included in the audio data in each time information is determined respectively.

[0017] Optionally, the method further includes:

[0018] Divide the audio data into frames to obtain audio frames contained in the audio data;

[0019] Determine the fundamental frequency features corresponding to each of the audio frames respectively;

[0020] Generate a fundamental frequency sequence containing the characteristics of each fundamental frequency.

[0021] Optionally, generating the music score data corresponding to the audio data based on the determined note data and text information includes:

[0022] Determine the pitch information corresponding to the note data contained in the audio data in each time information respectively;

[0023] Based on each pitch information and each text information, music score data corresponding to the audio data is generated.

[0024] Optionally, the respectively determining the pitch information corresponding to the note data contained in the audio data in each time information includes:

[0025] For each piece of time information, perform the following operations respectively:

[0026] If it is determined that the total number of notes in the note data included in the audio data is 1 within any time information, it is determined that the corresponding text information matches the note data, and the pitch information corresponding to the note data is used as the pitch information corresponding to the note data included in the audio data within the time information;

[0027] If it is determined that the total number is greater than 1, then each note contained in the note data, except for the note at the preset position, is used as a sustained note of the note at the preset position, and based on the pitch information corresponding to the note at the preset position and the pitch information corresponding to each sustained note, the pitch information corresponding to the note data contained in the audio data is obtained.

[0028] Optionally, the training method of the note recognition model is:

[0029] Obtaining an audio sample set, wherein the audio sample set includes audio data samples, each audio data sample is marked with a time information sample, and a note label corresponding to each time information sample;

[0030] Extract features from each audio data sample respectively to obtain audio sample features corresponding to each audio data sample;

[0031] The audio sample features and the corresponding time information samples are respectively input into the initial note recognition model to determine the initial notes contained in each audio data sample, and based on the initial notes and note labels, the model parameters of the initial note recognition model are adjusted until the loss function of the initial note recognition model converges to obtain a trained note recognition model.

[0032] A music score generating device, comprising:

[0033] A first recognition module is used to perform text recognition on the audio data, determine each text information contained in the audio data, and time information corresponding to each text information, wherein each text information contains at least one text character, and each time information represents the time period information occupied by the audio interval containing the corresponding text information in the audio data;

[0034] A second recognition module is used to determine the note data contained in the audio data in each time information based on the trained note recognition model and taking the audio features corresponding to the audio data and each time information as input parameters;

[0035] The generating module is used to generate the music score data corresponding to the audio data based on the determined note data and text information.

[0036] Optionally, the first identification module is further used for:

[0037] Based on the trained text recognition model, taking the audio data as an input parameter, respectively determining the text information corresponding to each audio frame in the audio data, and based on the text information corresponding to each audio frame, determining each text information contained in the audio data, and the time information corresponding to each text information;

[0038] The audio data at least includes audio frames, and the text recognition model is obtained through iterative training based on audio data samples and corresponding text information labels.

[0039] Optionally, the audio features include at least a fundamental frequency sequence and a singing voice feature, and when respectively determining the note data included in the audio data in each time information, the second recognition module is further used to:

[0040] Based on the fundamental frequency sequence and the singing voice feature, respectively determine the note data corresponding to each audio frame included in the audio data;

[0041] Based on the note data corresponding to each audio frame, the note data included in the audio data in each time information is determined respectively.

[0042] Optionally, the method further comprises a processing module, wherein the processing module is configured to:

[0043] Divide the audio data into frames to obtain audio frames contained in the audio data;

[0044] Determine the fundamental frequency features corresponding to each of the audio frames respectively;

[0045] Generate a fundamental frequency sequence containing the characteristics of each fundamental frequency.

[0046] Optionally, the generating module is further used for:

[0047] Determine the pitch information corresponding to the note data contained in the audio data in each time information respectively;

[0048] Based on each pitch information and each text information, music score data corresponding to the audio data is generated.

[0049] Optionally, when respectively determining the pitch information corresponding to the note data contained in the audio data in each time information, the generating module is further used to:

[0050] For each piece of time information, perform the following operations respectively:

[0051] If it is determined that the total number of notes in the note data included in the audio data is 1 within any time information, it is determined that the corresponding text information matches the note data, and the pitch information corresponding to the note data is used as the pitch information corresponding to the note data included in the audio data within the time information;

[0052] If it is determined that the total number is greater than 1, then each note contained in the note data, except for the note at the preset position, is used as a sustained note of the note at the preset position, and based on the pitch information corresponding to the note at the preset position and the pitch information corresponding to each sustained note, the pitch information corresponding to the note data contained in the audio data is obtained.

[0053] Optionally, the device further comprises a training module, wherein the training module is used to:

[0054] Obtaining an audio sample set, wherein the audio sample set includes audio data samples, each audio data sample is marked with a time information sample, and a note label corresponding to each time information sample;

[0055] Extract features from each audio data sample respectively to obtain audio sample features corresponding to each audio data sample;

[0056] The audio sample features and the corresponding time information samples are respectively input into the initial note recognition model to determine the initial notes contained in each audio data sample, and based on the initial notes and note labels, the model parameters of the initial note recognition model are adjusted until the loss function of the initial note recognition model converges to obtain a trained note recognition model.

[0057] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the above-mentioned music score generating method are implemented when the processor executes the program.

[0058] A computer-readable storage medium stores a computer program, which implements the steps of the above-mentioned music score generation method when executed by a processor.

[0059] In the disclosed embodiment, text recognition is performed on audio data to determine the text information contained in the audio data and the time information corresponding to each text information. Based on the trained note recognition model, the audio features corresponding to the audio data and each time information are used as input parameters to respectively determine the note data contained in the audio data in each time information. Based on the determined note data and each text information, the music score data corresponding to the audio data is generated. In this way, when the audio data is human voice, the time information corresponding to each text information contained in the audio data and the audio features of the audio data are input into the note recognition model, and the note data contained in the audio data in each time information can be determined based on the note recognition model, so that the music score data can be generated according to the note data and the text information. Since the note data is the note contained in the audio data in the time information, the text information corresponding to the human voice can be obtained, and each text information is matched with the corresponding note data, so that the richness and completeness of the generated music score can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A schematic diagram of a process for training a musical note recognition model in an embodiment of the present disclosure;

[0061] Figure 2 A schematic diagram of a process of generating a music score in an embodiment of the present disclosure;

[0062] Figure 3 This is a first example diagram of text recognition in an embodiment of the present disclosure;

[0063] Figure 4 A second example diagram of text recognition in an embodiment of the present disclosure;

[0064] Figure 5 An example diagram of determining note data in an embodiment of the present disclosure;

[0065] Figure 6 An example diagram of generating music score data in an embodiment of the present disclosure;

[0066] Figure 7 is a schematic diagram of the structure of a music score generating system in an embodiment of the present disclosure;

[0067] Figure 8 A schematic diagram of the structure of a music score generating device provided in an embodiment of the present disclosure;

[0068] Fig. 9 Schematic diagram of the structure of an electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0069] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0070] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0071] It should be understood herein that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction rather than having any limiting meaning.

[0072] For ease of understanding, the terms involved in the embodiments of the present disclosure are explained below:

[0073] Audio frame: The audio data is divided into several short-term audio signals, each of which is an audio frame.

[0074] Note: The smallest unit of music, a note lasts for a certain period of time and has a pitch value.

[0075] Fundamental frequency: When a sound-emitting body makes a sound due to vibration, the sound can be decomposed into only a number of sine waves. That is, all natural sounds are basically composed of many sine waves with different frequencies, among which the sine wave with the lowest frequency is the fundamental frequency, and its unit is Hertz.

[0076] Sustained note: refers specifically to the sustained note in singing, which continues the pronunciation of the ending sound of the previous note, but the pitch is different from the previous note and has a certain duration.

[0077] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure. SUMMARY OF THE INVENTION

[0079] In the related art, when converting singing voice into music score, the digital signal processing method is usually used to calculate the pitch mark of the frame-level audio data, and extract the melody sequence from the audio data through post-processing such as smoothing estimation and averaging. However, since the melody sequence extracted from the audio data only contains the time and pitch information of the notes, when the audio data is human voice singing, the singing text / lyrics content corresponding to the human voice cannot be obtained, resulting in the generated music score being not rich and complete.

[0080] In order to solve the above problems, the present disclosure provides a method for generating a musical score, which specifically includes the following steps: performing text recognition on audio data, determining various text information contained in the audio data, and time information corresponding to each text information, based on a trained note recognition model, using the audio features corresponding to the audio data and various time information as input parameters, respectively determining the note data contained in the audio data in each time information, and generating musical score data corresponding to the audio data based on the determined note data and text information. In this way, the note data and text information are determined from the audio data, so that the note data and text information are matched based on the time information of the text information. Therefore, when the audio data is a human voice, the extracted note data can be matched with the text information of the human voice, thereby improving the richness and completeness of the musical score.

[0081] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0082] Application Scenario Overview

[0083] The method in the embodiments of the present disclosure can be applied to application scenarios in which music scores that require matching text with musical notes are used as training data. For example, it can be applied to the scenario of training a singing synthesis model. During the training process, it is necessary to obtain information such as singing text, singing duration, singing notes, etc. arranged in time sequence. Therefore, through the method in the embodiments of the present disclosure, music score data that matches singing text with singing notes can be obtained, thereby obtaining music score samples for training a singing synthesis model.

[0084] Exemplary Methods

[0085] The following describes the score generation method according to the exemplary embodiment of the present disclosure in combination with the above application scenarios. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0086] Based on the above embodiments, the training process of the note recognition model in the embodiment of the present disclosure is first described. Figure 1 FIG. 1 is a flow chart of training a note recognition model in an embodiment of the present disclosure, which specifically includes:

[0087] Step 100: Obtain an audio sample set.

[0088] The audio sample set includes various audio data samples, each audio data sample is marked with various time information samples, and the note label corresponding to each time information sample.

[0089] In the disclosed embodiment, an audio sample set for training a note recognition model is obtained.

[0090] The audio sample set in the embodiment of the present disclosure includes a plurality of audio data samples, each audio data sample contains text information tags, each text information tag corresponds to a time information sample, and each time information sample corresponds to at least one note tag.

[0091] It should be noted that, since the audio data may correspond to only one note or multiple notes in a time period, when the audio data corresponds to multiple notes, the notes other than the first note are sustained notes of the first note.

[0092] Step 110: extract features from each audio data sample to obtain audio sample features corresponding to each audio data sample.

[0093] In the embodiment of the present disclosure, a preset feature extraction method is adopted to perform feature extraction on each audio data sample respectively, so as to obtain audio sample features corresponding to each audio data sample.

[0094] Among them, in the embodiment of the present disclosure, in the process of extracting features from audio data samples, the fundamental frequency sequence and singing features of the audio data samples can be obtained, and the fundamental frequency sequence and singing features of the audio data samples can be used as audio sample features corresponding to the audio data samples.

[0095] Step 120: Input the audio sample features and the corresponding time information samples into the initial note recognition model respectively, determine the initial notes contained in each audio data sample, and adjust the model parameters of the initial note recognition model based on the initial notes and note labels until the loss function of the initial note recognition model converges to obtain a trained note recognition model.

[0096] In the disclosed embodiments, each audio sample feature, each time information sample corresponding to each audio sample feature, and each corresponding note label are used to train an initial note recognition model. Specifically, each audio sample feature and each time information sample corresponding to each audio sample feature are input into the initial note recognition model, and based on the audio sample feature and each time information sample corresponding to the audio sample feature, the note data of the audio data sample in the time information sample is determined, and based on the note data and the note data label, various model parameters of the initial note recognition model are adjusted until the loss function of the initial note recognition model converges, thereby obtaining a trained note recognition model.

[0097] Based on the above embodiments, see Figure 2 FIG. 1 is a flow chart of a method for generating a music score in an embodiment of the present disclosure, which specifically includes:

[0098] Step 200: Perform text recognition on the audio data to determine the text information contained in the audio data and the time information corresponding to each text information.

[0099] Each text information includes at least one text character, and each time information represents the time period information occupied by the audio interval containing the corresponding text information in the audio data.

[0100] In the disclosed embodiment, text recognition is performed on audio data to convert the audio data into text data, thereby obtaining the text information contained in the audio data and the time period information occupied by the audio interval containing the corresponding text information in the audio data.

[0101] For example, see Figure 3As shown, it is a first example diagram of text recognition in an embodiment of the present disclosure. After text recognition is performed on the audio data, it is determined that the text information contained in the audio data is "today", "day", "of", "day", and "empty", among which, the time information occupied by the audio interval containing "today" in the audio data is from the end of the 0th second to the end of the 1st second of the audio data, the time information occupied by the audio interval containing "day" in the audio data is from the end of the 1st second to the end of the 3rd second of the audio data, the time information occupied by the audio interval containing "of" in the audio data is from the end of the 3rd second to the end of the 4th second of the audio data, the time information occupied by the audio interval containing "day" in the audio data is from the end of the 4th second to the end of the 7th second of the audio data, and the time information occupied by the audio interval containing "empty" in the audio data is from the end of the 7th second to the end of the 10th second of the audio data.

[0102] Optionally, in the embodiment of the present disclosure, the text information contained in the audio data may be determined by a text recognition model. Therefore, a possible implementation method for performing text recognition on the audio data is provided, which specifically includes:

[0103] Based on the trained text recognition model, the audio data is used as an input parameter to determine the text information corresponding to each audio frame in the audio data. Based on the text information corresponding to each audio frame, the text information contained in the audio data and the time information corresponding to each text information are determined.

[0104] The audio data at least includes audio frames, and the text recognition model is obtained through iterative training based on audio data samples and corresponding text information labels.

[0105] First, the training process of the text recognition model in the embodiment of the present disclosure is explained: a sample set for training the text recognition model is obtained, the sample set includes multiple audio data samples, each audio data sample corresponds to multiple text information labels, each audio data sample is input into the initial text recognition model, the text information contained in the audio data sample is determined, and according to the text information labels corresponding to the audio data sample, the model parameters of the initial text recognition model are adjusted until the loss function of the text recognition model converges, thereby obtaining a trained text recognition model.

[0106] Then, after obtaining the trained text recognition model, the audio data is input into the trained text recognition model, the audio data is divided into frames, and the audio frames corresponding to the audio data are obtained. Then, text recognition is performed on each audio frame to obtain the text information corresponding to each audio frame. Based on the text information corresponding to each audio frame, the same text information is merged to determine the text information contained in the audio data and the time information corresponding to each text information.

[0107] For example, see Figure 4 As shown, this is a second example diagram of text recognition in an embodiment of the present disclosure. Assume that the text information corresponding to each audio frame contained in the audio data is "day", "day", "day", "empty", "empty", "empty", and "empty", respectively. Therefore, the 1st audio frame, the 2nd audio frame, and the 3rd audio frame are merged, and the 4th audio frame, the 5th audio frame, the 6th audio frame, and the 7th audio frame are merged, so as to determine that the text information contained in the audio data is "day" and "empty", respectively, wherein the time information corresponding to "day" is the time period information occupied by the 1st to 3rd audio frames in the audio data, and the time information corresponding to "empty" is the time period information occupied by the 4th to 7th audio frames in the audio data.

[0108] Step 210: Based on the trained note recognition model, the note data included in the audio data in each time information is determined respectively using the audio features corresponding to the audio data and each time information as input parameters.

[0109] In the disclosed embodiment, the audio features and each time information corresponding to the audio data are input into a trained note recognition model, and the note data contained in the audio data in each time information is output through the note recognition model.

[0110] It should be noted that the audio features in the embodiments of the present disclosure include at least one of the following: fundamental frequency sequence and singing voice features.

[0111] The fundamental frequency sequence includes fundamental frequency features corresponding to each audio frame of the audio data, and each fundamental frequency feature is obtained by extracting the fundamental frequency of the corresponding audio frame. For example, the fundamental frequency feature can be extracted using a frequency domain method. The fundamental frequency feature is a sine wave with the lowest frequency of the audio frame, which is used to determine the pitch information of the audio frame.

[0112] The singing feature is obtained after singing feature extraction of audio data, and represents the features of the audio data in each singing dimension. For example, the singing feature is a 10-dimensional vector in 10 different singing dimensions, which is not limited in the embodiments of the present disclosure.

[0113] The following first describes the process of determining the fundamental frequency sequence of audio data in the embodiment of the present disclosure:

[0114] The first step is to divide the audio data into frames to obtain the audio frames contained in the audio data.

[0115] In the embodiment of the present disclosure, the audio data is framed according to a preset time interval, thereby dividing the audio data into a plurality of audio frames.

[0116] The second step is to determine the fundamental frequency features corresponding to each audio frame.

[0117] In the disclosed embodiment, after obtaining each audio frame included in the audio data, fundamental frequency extraction is performed on each audio frame to obtain fundamental frequency features corresponding to each audio frame.

[0118] Among them, each audio frame corresponds to a fundamental frequency feature.

[0119] The third step is to generate a fundamental frequency sequence containing the characteristics of each fundamental frequency.

[0120] In the disclosed embodiment, a fundamental frequency sequence is generated based on the fundamental frequency feature corresponding to each audio frame. Therefore, the generated fundamental frequency sequence includes the fundamental frequency feature corresponding to each audio frame.

[0121] Optionally, in the embodiment of the present disclosure, when the audio features include singing features and fundamental frequency sequences, the note data can be determined based on the singing features and the fundamental frequency sequences. The process of determining the note data in the embodiment of the present disclosure is described below, specifically including:

[0122] S1: Based on the fundamental frequency sequence and the singing voice feature, respectively determine the note data corresponding to each audio frame included in the audio data.

[0123] In the disclosed embodiment, since the fundamental frequency sequence includes the fundamental frequency features of each audio frame, and the singing features of the audio data include the singing features of each audio frame, the following operations are performed for each audio frame: the fundamental frequency features corresponding to any audio frame are determined from the fundamental frequency sequence, and the singing features corresponding to the audio frame are determined from the singing features; based on the fundamental frequency features and singing features of the audio frame, the note data included in the audio frame is determined. In this way, the note data corresponding to each audio frame contained in the audio data can be obtained.

[0124] It should be noted that, in the embodiment of the present disclosure, each audio frame corresponds to a note data.

[0125] Furthermore, in the disclosed embodiments, when determining the note data corresponding to an audio frame, it can be obtained based on the correspondence between the fundamental frequency features in the fundamental frequency sequence, the singing voice features and the note data. That is, there is a corresponding relationship between the fundamental frequency features and the singing voice features in the fundamental frequency sequence and the note data, and thus the corresponding note data can be obtained based on the correspondence between the fundamental frequency features and the singing voice features and the note data.

[0126] S2: Based on the note data corresponding to each audio frame, the note data included in the audio data in each time information is determined respectively.

[0127] In the embodiments of the present disclosure, the note data contained in the audio interval corresponding to each time information is determined respectively. Taking any one time information (hereinafter referred to as time information i) as an example, the process of determining the note data in the embodiments of the present disclosure is introduced as follows: first, each audio frame corresponding to the time information i is determined, and then, based on the note data corresponding to each audio frame, the note data contained in the audio interval corresponding to the time information i is determined.

[0128] For example, see Figure 5 As shown, it is an example diagram for determining note data in an embodiment of the present disclosure, the audio data includes 5 audio frames in the time information i, among which the note data corresponding to the 1st audio frame is "1", the note data corresponding to the 2nd audio frame is "1", the note data corresponding to the 3rd audio frame is "1", the note data corresponding to the 4th audio frame is "1", and the note data corresponding to the 5th audio frame is "1", therefore, it is determined that the note data contained in the audio interval corresponding to the time information i is "1".

[0129] Step 220: Generate music score data corresponding to the audio data based on the determined note data and text information.

[0130] In the disclosed embodiment, after the musical note data and text information corresponding to each time information are obtained, the musical score data corresponding to the audio data is generated based on the determined musical note data and the corresponding text information.

[0131] Optionally, in an embodiment of the present disclosure, a possible implementation method is provided for generating music score data corresponding to audio data. The process of generating music score data in an embodiment of the present disclosure is described below:

[0132] First, the pitch information corresponding to the note data contained in the audio data in each time information is determined respectively.

[0133] In the disclosed embodiment, the pitch information corresponding to each note data included in the audio interval of the audio data within each time information is determined respectively.

[0134] Optionally, in the embodiment of the present disclosure, the pitch information may be determined based on the number of note data in the audio interval, which may be specifically divided into the following two cases. Taking the time information i as an example, the process of determining the pitch information in the embodiment of the present disclosure is described as follows:

[0135] Case 1: The total number of notes is 1.

[0136] Specifically, if it is determined that the total number of notes in the note data contained in the audio data within any time information is 1, then the corresponding text information is determined to match the note data, and the pitch information corresponding to the note data is used as the pitch information corresponding to the note data contained in the audio data within the time information.

[0137] In the disclosed embodiment, the total number of notes in the note data contained in the audio data within the time information i is counted, and when the total number is determined to be 1, the text information and the note data are matched one by one, that is, the audio data contains one note in the audio interval within the time information i. Therefore, based on the association between the note data and the pitch information, the pitch information corresponding to the note data is determined, and the determined pitch information is used as the pitch information corresponding to the note data contained in the audio data within the time information i.

[0138] For example, assuming that the audio data in time information i contains the note "7" and only contains the note "7", therefore, the pitch information of "7" is determined, and the determined pitch information is used as the pitch information corresponding to the note data contained in the audio data in time information i.

[0139] The second case: the total number of notes is greater than 1.

[0140] Specifically, if it is determined that the total number is greater than 1, then each note contained in the note data, except for the note at the preset position, is treated as a sustained note of the note at the preset position, and based on the pitch information corresponding to the note at the preset position and the pitch information corresponding to each sustained note, the pitch information corresponding to the note data contained in the audio data is obtained.

[0141] In the disclosed embodiment, the total number of notes in the note data contained in the audio data within the time information i is counted. When it is determined that the total number is greater than 1, it is determined that the audio data within the time information i contains multiple notes. Therefore, among the notes contained in the note data, the note located at a preset position is taken as the main note, and each note except the note at the preset position is taken as a sustained note of the main note. Then, based on the pitch information corresponding to the main note and the pitch information corresponding to each sustained note, the pitch information corresponding to the note data contained in the audio data is determined.

[0142] The note at the preset position may be the first note among all the notes, which is not limited in the embodiment of the present disclosure.

[0143] For example, assuming that the notes contained in the audio data within the time information i are "7", "1", and "2", therefore, take the first note "7" as the main note, take "1" and "2" as the sustained notes of note "7", determine the pitch information of "7", the pitch information of "1", and the pitch information of "2", and based on the pitch information of "7", the pitch information of "1", and the pitch information of "2", obtain the pitch information corresponding to the note data contained in the audio data within the time information i.

[0144] Then, based on each pitch information and each text information, music score data corresponding to the audio data is generated.

[0145] In the disclosed embodiment, after obtaining the pitch information and the text information, based on each piece of time information, each piece of pitch information and the corresponding text information are respectively combined to obtain the music score data corresponding to the audio data in each piece of time information, and then the music score data corresponding to the audio data in each piece of time information are merged to generate the music score data corresponding to the audio data.

[0146] For example, see Figure 6 As shown in FIG. 1 , it is an example diagram of generating music score data in an embodiment of the present disclosure, wherein the text information is “bird”, “child”, “in”, “high”, “sound”, “song”, and “sing”. Based on each pitch information and each text information, the following is generated: Figure 6 The music score data shown.

[0147] In the disclosed embodiments, score data is generated by using the note data of the audio data in each time information, and score tags with text information and note data can be generated for the audio data, thereby improving the completeness and richness of the score data, and can be used as training data in the field of singing synthesis research.

[0148] Based on the above embodiments, see Figure 7 FIG. 1 is a schematic diagram of the structure of the music score generation system in the embodiment of the present disclosure, which specifically includes:

[0149] The music score generation system in the embodiment of the present disclosure includes a model training module and an application module. The module training module includes a text recognition module and a note recognition module. The application module includes a trained text recognition module, a fundamental frequency extraction module, and a trained note recognition module. The model training module and the application module in the embodiment of the present disclosure are described below.

[0150] Model training module:

[0151] 1. Text recognition module.

[0152] In the disclosed embodiment, first, the boundaries of the text time are marked on the audio data samples respectively to obtain the time information mark, and each audio data sample is divided into frames, so that each audio data sample after framing is input into the initial text recognition model, thereby establishing a text recognition model with the frame-level audio data samples as input and the text information as the target.

[0153] 2. Note recognition module.

[0154] In the disclosed embodiment, each audio data sample with each time information sample is obtained, and the note data contained in the audio data sample in each time information sample is marked to obtain the note label corresponding to each time information sample. At the same time, the fundamental frequency sequence in frames is extracted from each audio data sample, and the audio data samples are frame-processed to obtain matching feature pairs of singing features, note labels, and fundamental frequency features of each audio frame, thereby establishing a note recognition model with frame-level audio data samples, fundamental frequency features, and singing features as input and note data as output.

[0155] Application Modules:

[0156] The first step of the application module execution process: the signal input into the music score generation system is arbitrary audio data. Through the trained text recognition model, the text information contained in the audio data and the time information corresponding to each text information are obtained.

[0157] The second step of the application module execution process is to extract the frame-level baseband sequence of the input audio data.

[0158] The third step of the application module execution process: input the singing features corresponding to the frame-level audio data, the time information obtained through the first step of the online use process, and the fundamental frequency sequence obtained through the second step of the online use process into the trained note recognition model, so as to obtain the note data contained in the audio data in each time information.

[0159] The fourth step of the application module execution process: output the music score data with text information according to the number of notes and pitch information in each time information.

[0160] Exemplary Devices

[0161] After introducing the method according to the exemplary embodiment of the present disclosure, the music score generating device according to the exemplary embodiment of the present disclosure will be introduced next.

[0162] like Figure 8 FIG. 8 is a schematic diagram of the structure of the music score generating device provided by the embodiment of the present disclosure. In one embodiment, the music score generating device 80 includes: a first recognition module 801 , a second recognition module 802 , a generation module 803 , a processing module 804 and a training module 805 .

[0163] The first recognition module 801 is used to perform text recognition on the audio data, determine each text information contained in the audio data, and the time information corresponding to each text information, wherein each text information contains at least one text character, and each time information represents the time period information occupied by the audio interval containing the corresponding text information in the audio data;

[0164] The second recognition module 802 is used to determine the note data contained in the audio data in each time information based on the trained note recognition model and taking the audio features corresponding to the audio data and each time information as input parameters;

[0165] The generating module 803 is used to generate the music score data corresponding to the audio data based on the determined note data and text information.

[0166] Optionally, the first identification module 801 is further used for:

[0167] Based on the trained text recognition model, taking the audio data as an input parameter, respectively determining the text information corresponding to each audio frame in the audio data, and based on the text information corresponding to each audio frame, determining each text information contained in the audio data, and the time information corresponding to each text information;

[0168] The audio data at least includes audio frames, and the text recognition model is obtained through iterative training based on audio data samples and corresponding text information labels.

[0169] Optionally, the audio features include at least a fundamental frequency sequence and a singing voice feature, and when respectively determining the note data included in the audio data in each time information, the second recognition module 802 is further used to:

[0170] Based on the fundamental frequency sequence and the singing voice feature, respectively determine the note data corresponding to each audio frame included in the audio data;

[0171] Based on the note data corresponding to each audio frame, the note data included in the audio data in each time information is determined respectively.

[0172] Optionally, the method further includes a processing module 804, wherein the processing module 804 is configured to:

[0173] Divide the audio data into frames to obtain audio frames contained in the audio data;

[0174] Determine the fundamental frequency features corresponding to each of the audio frames respectively;

[0175] Generate a fundamental frequency sequence containing the characteristics of each fundamental frequency.

[0176] Optionally, the generating module 803 is further used for:

[0177] Determine the pitch information corresponding to the note data contained in the audio data in each time information respectively;

[0178] Based on each pitch information and each text information, music score data corresponding to the audio data is generated.

[0179] Optionally, when respectively determining the pitch information corresponding to the note data contained in the audio data in each time information, the generating module 803 is further used to:

[0180] For each piece of time information, perform the following operations respectively:

[0181] If it is determined that the total number of notes in the note data included in the audio data is 1 within any time information, it is determined that the corresponding text information matches the note data, and the pitch information corresponding to the note data is used as the pitch information corresponding to the note data included in the audio data within the time information;

[0182] If it is determined that the total number is greater than 1, then each note contained in the note data, except for the note at the preset position, is used as a sustained note of the note at the preset position, and based on the pitch information corresponding to the note at the preset position and the pitch information corresponding to each sustained note, the pitch information corresponding to the note data contained in the audio data is obtained.

[0183] Optionally, the device further includes a training module 805, wherein the training module 805 is used to:

[0184] Obtaining an audio sample set, wherein the audio sample set includes audio data samples, each audio data sample is marked with a time information sample, and a note label corresponding to each time information sample;

[0185] Extract features from each audio data sample respectively to obtain audio sample features corresponding to each audio data sample;

[0186] The audio sample features and the corresponding time information samples are respectively input into the initial note recognition model to determine the initial notes contained in each audio data sample, and based on the initial notes and note labels, the model parameters of the initial note recognition model are adjusted until the loss function of the initial note recognition model converges to obtain a trained note recognition model.

[0187] The music score generating device provided in the embodiment of the present disclosure adopts the same inventive concept as the above-mentioned music score generating method and can achieve the same beneficial effects, which will not be described in detail here.

[0188] Based on the same inventive concept as the above-mentioned music score generation method, the present disclosure also provides an electronic device. Fig. 9 As shown, the electronic device 90 may include a processor 901 and a memory 902 .

[0189] The processor 901 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and may implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present disclosure. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present disclosure may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0190] The memory 902 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 902 in the disclosed embodiment can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0191] Exemplary Program Products

[0192] An embodiment of the present disclosure provides a computer-readable storage medium for storing computer program instructions used by the above-mentioned electronic device, which includes a program for executing the music score generating method in any exemplary embodiment of the present disclosure.

[0193] The above-mentioned computer storage media can be any available media or data storage devices that can be accessed by a computer, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)), etc.

[0194] In some possible implementations, various aspects of the present disclosure may also be implemented as a computer program product, which includes program code. When the computer program product is run on a server device, the computer program product is used to enable the server device to execute the steps of the music score generation method according to various exemplary implementations of the present disclosure described in the above “Exemplary Method” section of this specification.

[0195] The computer program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0196] According to the computer program product for instant messaging application of the embodiment of the present disclosure, it can adopt portable compact disk read-only memory (CD-ROM) and include program code, and can be run on server device. However, the program product of the present disclosure is not limited to this. In this document, computer readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device.

[0197] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein readable program code is carried. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0198] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0199] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0200] It should be noted that although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0201] In addition, although the operations of the disclosed method are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0202] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. A method for generating a musical score, characterized in that: include: Performing text recognition on the audio data to determine each text information contained in the audio data and time information corresponding to each text information, wherein each text information contains at least one text character, and each time information represents the time period information occupied by the audio interval containing the corresponding text information in the audio data; the time information is determined based on the audio interval of each text character in the corresponding text information in the audio data; Based on the trained note recognition model, the audio features corresponding to the audio data and each time information are used as input parameters to respectively determine the note data contained in the audio data in each time information; Based on the determined note data and text information, music score data corresponding to the audio data is generated.

2. The method according to claim 1, characterized in that The performing text recognition on the audio data to determine each text information contained in the audio data and the time information corresponding to each text information includes: Based on the trained text recognition model, taking the audio data as an input parameter, determining the text information corresponding to each audio frame in the audio data, and based on the text information corresponding to each audio frame, determining each text information contained in the audio data and the time information corresponding to each text information; The audio data at least includes audio frames, and the text recognition model is obtained through iterative training based on audio data samples and corresponding text information labels.

3. The method according to claim 1, characterized in that The audio features at least include a fundamental frequency sequence and a singing voice feature, and the step of respectively determining the note data included in the audio data in each time information includes: Based on the fundamental frequency sequence and the singing voice feature, respectively determine the note data corresponding to each audio frame included in the audio data; Based on the note data corresponding to each audio frame, the note data included in the audio data in each time information is determined respectively.

4. The method according to claim 3, characterized in that The method further comprises: Divide the audio data into frames to obtain audio frames contained in the audio data; Determine the fundamental frequency features corresponding to each of the audio frames respectively; Generate a fundamental frequency sequence containing the characteristics of each fundamental frequency.

5. The method according to claim 1, characterized in that The step of generating the music score data corresponding to the audio data based on the determined note data and text information includes: Determine the pitch information corresponding to the note data contained in the audio data in each time information respectively; Based on each pitch information and each text information, music score data corresponding to the audio data is generated.

6. The method according to claim 5, characterized in that The step of respectively determining the pitch information corresponding to the note data contained in the audio data in each time information includes: For each piece of time information, perform the following operations respectively: If it is determined that the total number of notes in the note data included in the audio data is 1 within any time information, it is determined that the corresponding text information matches the note data, and the pitch information corresponding to the note data is used as the pitch information corresponding to the note data included in the audio data within the time information; If it is determined that the total number is greater than 1, then each note contained in the note data, except for the note at the preset position, is used as a sustained note of the note at the preset position, and based on the pitch information corresponding to the note at the preset position and the pitch information corresponding to each sustained note, the pitch information corresponding to the note data contained in the audio data is obtained.

7. The method according to any one of claims 1 to 6, characterized in that: The training method of the note recognition model is: Obtaining an audio sample set, wherein the audio sample set includes audio data samples, each audio data sample is marked with a time information sample, and a note label corresponding to each time information sample; Extract features from each audio data sample respectively to obtain audio sample features corresponding to each audio data sample; The audio sample features and the corresponding time information samples are respectively input into the initial note recognition model to determine the initial notes contained in each audio data sample, and based on the initial notes and note labels, the model parameters of the initial note recognition model are adjusted until the loss function of the initial note recognition model converges to obtain a trained note recognition model.

8. A music score generating device, characterized in that: include: A first recognition module is used to perform text recognition on the audio data, determine each text information contained in the audio data, and time information corresponding to each text information, wherein each text information contains at least one text character, and each time information represents the time period information occupied by the audio interval containing the corresponding text information in the audio data; the time information is determined according to the audio interval of each text character in the corresponding text information in the audio data; A second recognition module is used to determine the note data contained in the audio data in each time information based on the trained note recognition model and taking the audio features corresponding to the audio data and each time information as input parameters; The generating module is used to generate the music score data corresponding to the audio data based on the determined note data and text information.

9. The device according to claim 8, characterized in that The first identification module is also used for: Based on the trained text recognition model, taking the audio data as an input parameter, determining the text information corresponding to each audio frame in the audio data, and based on the text information corresponding to each audio frame, determining each text information contained in the audio data and the time information corresponding to each text information; The audio data at least includes audio frames, and the text recognition model is obtained through iterative training based on audio data samples and corresponding text information labels.

10. The device according to claim 8, characterized in that The audio features at least include a fundamental frequency sequence and a singing voice feature. When the note data contained in the audio data in each time information is determined respectively, the second recognition module is further used to: Based on the fundamental frequency sequence and the singing voice feature, respectively determine the note data corresponding to each audio frame included in the audio data; Based on the note data corresponding to each audio frame, the note data included in the audio data in each time information is determined respectively.

11. The device according to claim 10, characterized in that The device further comprises a processing module, wherein the processing module is configured to: Divide the audio data into frames to obtain audio frames contained in the audio data; Determine the fundamental frequency features corresponding to each of the audio frames respectively; Generate a fundamental frequency sequence containing the characteristics of each fundamental frequency.

12. The device according to claim 8, characterized in that The generation module is also used for: Determine the pitch information corresponding to the note data contained in the audio data in each time information respectively; Based on each pitch information and each text information, music score data corresponding to the audio data is generated.

13. The device according to claim 12, characterized in that When determining the pitch information corresponding to the note data contained in the audio data in each time information respectively, the generating module is further used to: For each piece of time information, perform the following operations respectively: If it is determined that the total number of notes in the note data included in the audio data is 1 within any time information, it is determined that the corresponding text information matches the note data, and the pitch information corresponding to the note data is used as the pitch information corresponding to the note data included in the audio data within the time information; If it is determined that the total number is greater than 1, then each note contained in the note data, except for the note at the preset position, is used as a sustained note of the note at the preset position, and based on the pitch information corresponding to the note at the preset position and the pitch information corresponding to each sustained note, the pitch information corresponding to the note data contained in the audio data is obtained.

14. The device according to any one of claims 8 to 13, characterized in that The device also includes a training module, which is used to: Obtaining an audio sample set, wherein the audio sample set includes audio data samples, each audio data sample is marked with a time information sample, and a note label corresponding to each time information sample; Extract features from each audio data sample respectively to obtain audio sample features corresponding to each audio data sample; The audio sample features and the corresponding time information samples are respectively input into the initial note recognition model to determine the initial notes contained in each audio data sample, and based on the initial notes and note labels, the model parameters of the initial note recognition model are adjusted until the loss function of the initial note recognition model converges to obtain a trained note recognition model.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images