Singing annotation method, computer equipment and readable storage medium

Abstract

The invention relates to a song annotation method, computer equipment and a computer readable storage medium. The method comprises the following steps: acquiring a to-be-labeled song audio signal of a target song; acquiring a corresponding boundary audio frame of each note of the target song in the singing audio signal and a corresponding boundary audio frame of each syllable of the target song in the singing audio signal; according to the boundary audio frame of each syllable and the boundary audio frame of each note, obtaining pitch information of the notes included in the two boundary audio frame ranges of each syllable; and taking the two boundary audio frames of each syllable, the note boundary audio frames contained in the range of the two boundary audio frames of each syllable and the pitch information of the notes contained in the range of the two boundary audio frames of each syllable as song annotation information of the song audio signal. By adopting the method, the pitch information of each note contained in the syllables can be obtained by combining the note boundaries, and the song annotation is carried out by utilizing the pitch information of the notes, so that the accuracy of the song annotation is improved.

Description

Technical Field

[0001] The present application relates to the technical field of audio processing, and in particular to a singing annotation method, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the development of audio processing technology, a technology for realizing singing synthesis has emerged. This technology can mark the singing, for example, it can mark the pitch of the syllables contained in the singing, and by identifying the syllables contained in the singing and the duration of each syllable, the pitch information of each syllable can be determined.

[0003] In the above song annotation method, the time range of each syllable is usually directly identified, and the fundamental frequency signal within the syllable time range is averaged as the pitch information of each syllable. However, in this annotation method, the accuracy of the calculated pitch information is low, so the current singing annotation method has low accuracy of singing annotation. Summary of the invention

[0004] Based on this, it is necessary to provide a singing annotation method, device, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of singing annotation in order to solve the above technical problems.

[0005] In a first aspect, the present application provides a singing annotation method, comprising:

[0006] Obtaining a target song audio signal to be labeled;

[0007] Obtain two boundary audio frames corresponding to each note of the target song in the singing audio signal, and obtain two boundary audio frames corresponding to each syllable of the target song in the singing audio signal;

[0008] Determine the boundary audio frames of the notes contained within the two boundary audio frames of each syllable according to the boundary audio frames of each syllable and the boundary audio frames of each note, and obtain the pitch information of the notes contained within the two boundary audio frames of each syllable based on the boundary audio frames of the notes contained within the two boundary audio frames of each syllable;

[0009] The two boundary audio frames of each syllable, the note boundary audio frames contained in the two boundary audio frames of each syllable, and the pitch information of the notes contained in the two boundary audio frames of each syllable are used as the singing annotation information of the singing audio signal.

[0010] In one of the embodiments, obtaining two boundary audio frames corresponding to each note of the target song in the singing audio signal includes: obtaining the spectrum features and the fundamental frequency features of the singing audio signal; inputting the spectrum features and the fundamental frequency features into a pre-trained note prediction model, and obtaining two boundary audio frames corresponding to each note of the target song in the singing audio signal through the note prediction model.

[0011] In one of the embodiments, before inputting the spectral features and the fundamental frequency features into the pre-trained note prediction model, it also includes: obtaining a sample singing audio signal of a sample song, and sample spectral features and sample fundamental frequency features corresponding to the sample singing audio signal; inputting the sample spectral features and the sample fundamental frequency features into the note prediction model to be trained to obtain predicted boundary audio frames corresponding to each note of the sample song in the sample singing audio signal; obtaining actual boundary audio frames corresponding to each note of the sample song in the sample singing audio signal; training the note prediction model according to the difference between the predicted boundary audio frame and the actual boundary audio frame to obtain the pre-trained note prediction model.

[0012] In one of the embodiments, the pitch information of the notes contained in the two boundary audio frames of each syllable is obtained based on the boundary audio frames of the notes contained in the two boundary audio frames of each syllable, including: determining the current syllable of the target song, the current syllable being any syllable of the syllables of the target song; when there is a boundary audio frame of the note between the two boundary audio frames corresponding to the current syllable, obtaining the note duration of the note contained in the current syllable according to the two boundary audio frames corresponding to the current syllable and the existing boundary audio frames of the note; obtaining the pitch information of the note contained in the current syllable according to the note duration of the note contained in the current syllable.

[0013] In one of the embodiments, the method of obtaining the pitch information of the note contained in the current syllable according to the note duration of the note contained in the current syllable includes: obtaining a first sub-fundamental frequency signal that matches the note duration of the note contained in the current syllable from the fundamental frequency signal corresponding to the singing audio signal according to the note duration of the note contained in the current syllable; and obtaining the pitch information of the note contained in the current syllable according to the signal value of each of the first sub-fundamental frequency signals.

[0014] In one of the embodiments, after determining the current syllable of the target song, it also includes: when there is no boundary audio frame of a note between two boundary audio frames corresponding to the current syllable, obtaining the syllable duration of the current syllable according to the two boundary audio frames corresponding to the current syllable; according to the syllable duration, obtaining a second sub-baseband signal matching the syllable duration from the baseband signal corresponding to the singing audio signal; according to the signal value of the second sub-baseband signal, obtaining the pitch information of the current syllable, and using the pitch information of the current syllable as the pitch information of each note contained in the current syllable.

[0015] In one of the embodiments, obtaining two boundary audio frames corresponding to each syllable of the target song in the singing audio signal includes: obtaining lyrics text information of the target song, the lyrics text information including each syllable of the target song; and performing forced alignment processing on the singing audio signal and the lyrics text information to obtain two boundary audio frames corresponding to each syllable of the target song in the singing audio signal.

[0016] In a second aspect, the present application also provides a singing annotation device, comprising:

[0017] An audio signal acquisition module, used to acquire the singing audio signal to be annotated of the target song;

[0018] A note boundary acquisition module, used to acquire two boundary audio frames corresponding to each note of the target song in the singing audio signal, and to acquire two boundary audio frames corresponding to each syllable of the target song in the singing audio signal;

[0019] A pitch information acquisition module, for determining the boundary audio frames of the notes contained within the range of two boundary audio frames of each syllable according to the boundary audio frames of each syllable and the boundary audio frames of each note, and obtaining the pitch information of the notes contained within the range of two boundary audio frames of each syllable based on the boundary audio frames of the notes contained within the range of two boundary audio frames of each syllable;

[0020] The singing annotation acquisition module is used to use the two boundary audio frames of each syllable, the note boundary audio frames contained in the two boundary audio frames of each syllable, and the pitch information of the notes contained in the two boundary audio frames of each syllable as the singing annotation information of the singing audio signal.

[0021] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in any one of the embodiments of the first aspect when executing the computer program.

[0022] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of the embodiments of the first aspect.

[0023] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method described in any one of the embodiments of the first aspect.

[0024] The above-mentioned singing annotation method, device, computer equipment, computer-readable storage medium and computer program product can first obtain the boundary audio frame contained in the singing audio signal when annotating the singing audio signal, and then combine the boundary audio frame of each syllable contained in the singing audio signal with the boundary audio frame of each note to obtain the pitch information of each note contained in each syllable, so as to use the above-mentioned pitch information, the boundary audio frame of each note contained in each syllable and the boundary audio frame of each syllable as the singing annotation information. In this way, the boundary audio frame of the note can be combined to obtain the pitch information of each note contained in the syllable, and the pitch information of the note can be used for singing annotation, thereby improving the accuracy of singing annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 A schematic diagram of a flow chart of a singing voice marking method in one embodiment;

[0027] Figure 2 A schematic diagram of a process for training a note prediction model in one embodiment;

[0028] Figure 3 A schematic diagram of a process for obtaining pitch information of each note contained in each syllable in one embodiment;

[0029] Figure 4 A schematic diagram of training data annotation in one embodiment;

[0030] Figure 5 A schematic diagram of a training process of a note prediction model in one embodiment;

[0031] Figure 6 is a schematic diagram of a model structure of a note prediction model in one embodiment;

[0032] Figure 7 A schematic diagram of the overall process of singing voice labeling in one embodiment;

[0033] Figure 8 is a structural block diagram of a singing voice marking device in one embodiment;

[0034] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0036] In one embodiment, Figure 1 As shown, a singing annotation method is provided. This embodiment takes the method applied to a server as an example for illustration. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0037] Step S101, obtaining a singing audio signal to be annotated of a target song;

[0038] Step S102, obtaining two boundary audio frames corresponding to each note of the target song in the singing audio signal, and obtaining two boundary audio frames corresponding to each syllable of the target song in the singing audio signal.

[0039] Among them, the singing audio signal to be labeled refers to the singing audio signal of the target song that needs to complete the singing labeling, and the boundary audio frame corresponding to the note in the singing audio signal can also be called the syllable boundary, which refers to the audio frame corresponding to the boundary of the note of the target song in the singing audio signal. For example, the singing audio signal of a 2s target song is composed of 2 notes, where the duration of note 1 is 0s-1s, and the duration of note 2 is 1s-2s. Then the two boundary audio frames of note 1 of the target song can be: the singing audio signal frame corresponding to 0s and the singing audio signal frame corresponding to 1s, and the two boundary audio frames of note 2 can be: the singing audio signal frame corresponding to 1s and the singing audio signal frame corresponding to 2s, and the other audio frames of the singing audio signal are not used as boundary audio frames of notes.

[0040] The boundary audio frame corresponding to the syllable in the singing audio signal can also be called the syllable boundary, which refers to the audio frame corresponding to the boundary of the syllable of the target song in the singing audio signal, and can be used to distinguish different syllables in the singing audio signal. The syllable boundary can be obtained through the singing audio signal and the lyrics text corresponding to the singing audio signal.

[0041] Specifically, after obtaining the singing audio signal of the target song, the server can further extract two boundary audio frames of each note contained in the singing audio signal, and two boundary audio frames of each syllable.

[0042] Step S103, according to the boundary audio frames of each syllable and the boundary audio frames of each note, determine the boundary audio frames of the notes contained in the two boundary audio frames of each syllable, and based on the boundary audio frames of the notes contained in the two boundary audio frames of each syllable, obtain the pitch information of the notes contained in the two boundary audio frames of each syllable.

[0043] Since the same syllable can contain multiple notes, for example, if there is a note boundary between the boundaries of two syllables, it means that the syllable can be composed of two notes. Therefore, after the server obtains the boundary audio frames of each syllable and the boundary audio frames of each note, it can determine the boundary audio frames of the notes contained in the two boundary audio frames of each syllable, thereby determining the notes contained in the two boundary audio frames of each syllable, and further obtaining the pitch information of the notes contained in the two boundary audio frames of each syllable.

[0044] Step S104, taking two boundary audio frames of each syllable, the note boundary audio frames contained within the two boundary audio frames of each syllable, and the pitch information of the notes contained within the two boundary audio frames of each syllable as singing annotation information of the singing audio signal.

[0045] Finally, the server can also use the pitch information of each note contained in each syllable, the boundary audio frame of each note contained in each syllable, and the boundary audio frame corresponding to each syllable as the singing annotation information of the singing audio signal. In this way, the pitch information annotation at the note granularity can be achieved, thereby improving the accuracy of singing annotation.

[0046] For example, if the boundary audio frames of a syllable are the 10th and 30th frames, and the boundary audio frame of a note appears in the 20th frame, it means that the syllable contains two notes, and the syllable corresponds to two pitch information, wherein pitch information 1 is the pitch information corresponding to note 1 from the 10th to the 20th frames, and pitch information 2 is the pitch information corresponding to note 2 from the 20th to the 30th frames. If the existing technology is used for singing annotation, since only the syllable is recognized, the syllable will only correspond to an overall pitch information, that is, the pitch information of the syllable from the 10th to the 30th frame is directly annotated. It can be seen that the singing annotation method provided by this embodiment has a higher accuracy than the singing annotation method provided by the existing technology.

[0047] In the above-mentioned singing annotation method, when annotating the singing audio signal, the boundary audio frame contained in the singing audio signal can be obtained first, and then the boundary audio frame of each syllable contained in the singing audio signal and the boundary audio frame of each note can be combined to obtain the pitch information of each note contained in each syllable, so that the above-mentioned pitch information, the boundary audio frame of each note contained in each syllable and the boundary audio frame of each syllable are used as singing annotation information. In this way, the boundary audio frame of the note can be combined to obtain the pitch information of each note contained in the syllable, and the pitch information of the note can be used for singing annotation, thereby improving the accuracy of singing annotation.

[0048] In one embodiment, step S102 may further include: obtaining spectral features and fundamental frequency features of the singing audio signal; inputting the spectral features and fundamental frequency features into a pre-trained note prediction model, and obtaining two boundary audio frames corresponding to each note of the target song in the singing audio signal through the note prediction model.

[0049] The spectrum feature refers to the feature information extracted from the spectrum signal corresponding to the singing audio signal. The spectrum signal may refer to the Mel-spectrum signal corresponding to the singing audio signal, and the fundamental frequency feature refers to the feature information extracted from the fundamental frequency signal corresponding to the singing audio signal.

[0050] Specifically, when the server obtains the singing audio signal that needs to be annotated, it can also perform mel spectrum extraction and fundamental frequency extraction on the singing audio signal, so as to obtain the spectrum signal and fundamental frequency signal corresponding to the singing audio signal. Afterwards, the server can perform feature extraction on the spectrum signal and fundamental frequency signal, so as to obtain the spectrum feature corresponding to the spectrum signal and the fundamental frequency feature corresponding to the fundamental frequency signal.

[0051] The pre-trained note prediction model refers to a trained note prediction model, which can be used to identify the frame boundaries corresponding to each note contained in the singing audio signal, that is, the boundary audio frames of the notes. After the server obtains the spectral features and fundamental frequency features of the singing audio signal, it can input the above spectral features and fundamental frequency features into the pre-trained note prediction model, and the note prediction model identifies the two audio signal frames corresponding to the note boundaries in the singing audio signal.

[0052] In this embodiment, the note boundaries in the singing audio signal can be obtained by inputting the spectrum features and fundamental frequency features of the singing audio signal into a pre-trained note prediction model and outputting the note prediction model. In this way, the accuracy of obtaining the note boundaries can be improved.

[0053] In addition, if Figure 2 As shown, before step S101, the following may also be included:

[0054] Step S201, obtaining a sample singing audio signal of a sample song, and a sample frequency spectrum feature and a sample fundamental frequency feature corresponding to the sample singing audio signal.

[0055] Among them, the sample singing audio signal refers to the singing audio signal of the sample song used for training the note prediction model, the sample spectrum feature refers to the feature information extracted from the spectrum signal corresponding to the sample singing audio signal, the spectrum signal may refer to the Mel-spectrogram signal corresponding to the sample singing audio signal, and the sample fundamental frequency feature refers to the feature information extracted from the fundamental frequency signal corresponding to the sample singing audio signal.

[0056] Specifically, when the server is training the note prediction model, it can first collect the sample singing audio signal, and perform mel spectrum extraction and fundamental frequency extraction on the sample singing audio signal, so as to obtain the spectrum signal and fundamental frequency signal corresponding to the sample singing audio signal. After that, the server can perform feature extraction on the spectrum signal and fundamental frequency signal, so as to obtain the sample spectrum feature corresponding to the spectrum signal, and the sample fundamental frequency feature corresponding to the fundamental frequency signal.

[0057] Step S202: input the sample spectrum features and the sample fundamental frequency features into the note prediction model to be trained to obtain the predicted boundary audio frames corresponding to each note of the sample song in the sample singing audio signal.

[0058] The note prediction model to be trained refers to a note prediction model that requires neural network model training, which can be a convolutional neural network model, and the predicted boundary audio frames of each note are the audio signal frames corresponding to the note boundaries in the sample singing audio signal output by the note prediction model. Specifically, after obtaining the sample spectrum features and the sample fundamental frequency features, the server can input the above-mentioned sample spectrum features and the sample fundamental frequency features into the note prediction model to be trained, and the note prediction model outputs the predicted note frame boundaries of each note contained in the sample singing audio signal.

[0059] Step S203, obtaining actual boundary audio frames corresponding to each note of the sample song in the sample singing audio signal.

[0060] The actual boundary audio frame of each note refers to the note boundary actually contained in the sample singing audio signal. The note boundary can be obtained by the user marking the sample singing audio signal. For example, it can be obtained by marking the singing audio signal frame corresponding to the note boundary in the sample singing audio signal as the actual boundary audio frame of each note contained in the sample singing audio signal.

[0061] Step S204: training a note prediction model according to the difference between the predicted boundary audio frame and the actual boundary audio frame to obtain a pre-trained note prediction model.

[0062] After obtaining the predicted boundary audio frame and the actual boundary audio frame, the terminal can use the difference between the predicted boundary audio frame and the actual boundary audio frame to train the note prediction model, thereby obtaining a pre-trained note prediction model.

[0063] In this embodiment, the server can also use the sample singing audio signal and the actual note boundaries of the sample singing audio signal to train a note prediction model, through which the note boundaries can be accurately predicted, thereby improving the accuracy of note boundary acquisition.

[0064] In one embodiment, Figure 3 As shown, step S103 may further include:

[0065] Step S301, determining the current syllable of the target song, where the current syllable is any syllable among the syllables of the target song.

[0066] The current syllable refers to any one of the multiple syllables contained in the singing audio signal, and the two boundary audio frames corresponding to the current syllable are respectively used to distinguish the current syllable from the previous syllable and to distinguish the current syllable from the next syllable.

[0067] In this embodiment, a syllable can be uniquely determined between two adjacent syllable boundaries. After obtaining the current syllable, the server can also obtain a boundary audio frame for distinguishing the current syllable from the previous syllable, and a boundary audio frame for distinguishing the current syllable from the next syllable, respectively as the two boundary audio frames corresponding to the current syllable.

[0068] Step S302, when there is a boundary audio frame of a note between two boundary audio frames corresponding to the current syllable, obtain the note duration of the note contained in the current syllable according to the two boundary audio frames corresponding to the current syllable and the boundary audio frame of the existing note.

[0069] If there is a boundary audio frame of a note between the two boundary audio frames corresponding to the current syllable, it means that the current syllable contains at least two notes. The server can then obtain the note duration of the note contained in the current syllable based on the two boundary audio frames and the boundary audio frames of the existing notes.

[0070] For example, the two boundary audio frames of the current syllable are the 10th frame of the singing audio signal and the 30th frame of the singing audio signal, and a note boundary appears in the 20th frame of the singing audio signal, which indicates that the current syllable contains two notes, of which the note duration of note 1 in the current syllable is from the 10th frame to the 20th frame, and the note duration of note 2 in the current syllable is from the 20th frame to the 30th frame. In this way, the note duration of each note contained in the current syllable in the current syllable can be obtained.

[0071] Step S303, obtaining the pitch information of the note contained in the current syllable according to the note duration of the note contained in the current syllable.

[0072] After determining the note duration of each note contained in the current syllable, the pitch information of each note contained in the current syllable can be obtained based on the above note duration. For example, the pitch information of each note in the current syllable can be obtained by using the part of the singing audio signal that matches the duration of each note in the current syllable, that is, the pitch information of note 1 can be obtained by using the part of the 10th to 20th frames in the singing audio signal, and the pitch information of note 2 can be obtained by using the part of the 20th to 30th frames in the singing audio signal.

[0073] In this embodiment, if there is a boundary audio frame of a note between two boundary audio frames corresponding to a syllable, it indicates that the syllable contains at least two notes. Therefore, the two boundary audio frames corresponding to the current syllable and the boundary audio frames of the existing notes can be used to determine the note duration of each note contained in the current syllable, and then determine the pitch information of the notes contained in the current syllable. In this way, the accuracy of note marking can be improved.

[0074] Furthermore, step S303 may further include: according to the note duration of the note contained in the current syllable, obtaining a first sub-fundamental frequency signal matching the note duration of the note contained in the current syllable from the fundamental frequency signal corresponding to the singing audio signal; and obtaining the pitch information of the note contained in the current syllable according to the signal value of each first sub-fundamental frequency signal.

[0075] In this embodiment, the pitch information can be calculated by the fundamental frequency signal, and the first sub-fundamental frequency signal refers to the part of the fundamental frequency signal corresponding to the singing audio signal that matches the note duration of each note contained in the current syllable in the current syllable. Specifically, the server can obtain the corresponding fundamental frequency signal by performing fundamental frequency extraction on the singing audio signal, and extract the first sub-fundamental frequency signal that matches the audio duration from the fundamental frequency signal according to the note duration of each note contained in the current syllable, and use the signal value of the first sub-fundamental frequency signal to calculate the pitch information of each note contained in the current syllable, which can be the average value or median of the signal value as the pitch information of the corresponding note.

[0076] For example, the server may use the portion from the 10th frame to the 20th frame in the fundamental frequency signal as the first sub-fundamental frequency signal corresponding to note 1, and use the average value of the signal value corresponding to the first sub-fundamental frequency signal as the pitch information of note 1.

[0077] In this embodiment, the server can obtain a first sub-fundamental frequency signal that matches the note duration of each note from the fundamental frequency signal based on the note duration of each note, and then use the signal value of the first sub-fundamental frequency signal to obtain the pitch information of each note. In this way, the accuracy of obtaining the pitch information of the notes can be improved.

[0078] In addition, after step S301, it may also include: in a case where there is no boundary audio frame of a note between the two boundary audio frames corresponding to the current syllable, obtaining the syllable duration of the current syllable according to the two boundary audio frames corresponding to the current syllable; according to the syllable duration, obtaining a second sub-baseband signal matching the syllable duration from the baseband signal corresponding to the singing audio signal; according to the signal value of the second sub-baseband signal, obtaining the pitch information of the current syllable, and using the pitch information of the current syllable as the pitch information of each note contained in the current syllable.

[0079] If there is no boundary audio frame of a note between the two boundary audio frames corresponding to the current syllable, it means that the current syllable contains only one note instead of multiple notes. In this case, you only need to get the pitch information of the current syllable, which can be used as the pitch information of each note contained in the current syllable.

[0080] Specifically, if the server determines that there is no boundary audio frame with notes between the starting boundary audio frame and the ending boundary audio frame corresponding to the current syllable, the server can directly calculate the syllable duration of the current syllable based on the singing audio signal frames corresponding to the starting boundary audio frame and the ending boundary audio frame.

[0081] The second sub-baseband signal refers to the part of the baseband signal corresponding to the singing audio signal that matches the syllable duration in the current syllable. Specifically, the server can obtain the corresponding baseband signal by performing baseband extraction on the singing audio signal, and extract the second sub-baseband signal that matches the syllable duration from the baseband signal according to the syllable duration of the current syllable, and use the signal value of the second sub-baseband signal to calculate the pitch information of the current syllable, which can be the average or median of the signal value as the pitch information of the current syllable, and use the pitch information as the pitch information of each note contained in the current syllable.

[0082] For example, the starting boundary audio frame of the current syllable is the 30th frame of the singing audio signal, the ending boundary audio frame is the 50th frame of the singing audio signal, and there is no boundary audio frame of the note between the 30th and 50th frames of the singing audio signal, then it indicates that the syllable is composed of the same note, and the syllable duration is the 30th to 50th frames. The server can use the part of the 30th to 50th frames in the fundamental frequency signal as the second sub-fundamental frequency signal corresponding to the current syllable, and use the average value of the signal value corresponding to the second sub-fundamental frequency signal as the pitch information of the current syllable.

[0083] In this embodiment, if there is no boundary audio frame of a note between the first syllable boundary and the second syllable boundary, the server can directly use the pitch information of the current syllable as the pitch information of each note contained in the current syllable. In this way, the efficiency of obtaining the pitch information of each note contained in the current syllable can be improved.

[0084] In one embodiment, step S102 may further include: obtaining lyrics text information of the target song, the lyrics text information including each syllable of the target song; forcibly aligning the singing audio signal with the lyrics text information to obtain two boundary audio frames corresponding to each syllable of the target song in the singing audio signal.

[0085] The lyrics text information refers to the lyrics text corresponding to the singing audio signal. In this embodiment, the syllable boundary can be obtained by forcibly aligning the singing audio signal and the corresponding lyrics text. Specifically, after obtaining the singing audio signal, the server can also obtain the lyrics text corresponding to the singing audio signal as the lyrics text information, and then the singing audio signal and the lyrics text information can be forcibly aligned, so that each syllable contained in the singing audio signal and the two syllable boundaries corresponding to each syllable can be identified in this way.

[0086] In this embodiment, the syllable boundary can be obtained by the server through forced alignment processing of the singing audio signal and the lyrics text information corresponding to the singing audio signal. In this way, the accuracy of the syllable boundary acquisition can be ensured.

[0087] In one embodiment, a method for automatic annotation of singing voice is also provided, which uses a regression model to predict the boundaries of notes, combined with the syllable boundaries calculated by forced alignment, to achieve automatic generation of high-accuracy singing voice annotations. Compared with the prior art that takes the average value of the effective fundamental frequency within the syllable time range as the pitch of the syllable, the singing voice annotation method provided in this embodiment is more accurate, and can be specifically implemented by the following steps:

[0088] 1. Training data preparation:

[0089] (1) Audio and note annotated data pairs: The smallest granularity is the note. For example, there may be multiple notes in a syllable, so all the notes will be annotated. Figure 4 The syllable shen shown corresponds to multiple pitches, namely 61 and 59, while the syllable yi only corresponds to one pitch, namely 61.

[0090] (2) Performing fundamental frequency extraction and mel-spectrogram extraction on the audio signal. This process can be performed using existing tools.

[0091] 2. Model training:

[0092] In the entire process, forced alignment can be achieved using existing tools, while note prediction requires training a note prediction model.

[0093] The training steps of the note prediction model can be as follows Figure 5 As shown, the structure of the note prediction model used can be as follows Figure 6As shown in the figure, the input of the model is Mel feature and fundamental frequency feature. The fundamental frequency will be normalized before being sent to the model. The model will predict the probability of whether it is a note boundary frame by frame. This model is a regression model. The annotation of the model is derived from the annotation of whether it is a note boundary. Among them, the boundary of the initial and final vowels will not be marked as a note boundary, and the intersection of the syllable boundary and the note boundary will be used as the annotation of the model.

[0094] 3. Process reasoning:

[0095] The overall process of singing annotation is as follows Figure 7 As shown in the figure, the syllable boundary is provided by the result of forced alignment; if there is a transition in the syllable, the boundary of the note is output by the note prediction model, and then the corresponding fundamental frequency sequence is obtained according to the syllable boundary and the note boundary, and the fundamental frequency sequence is averaged or median to obtain the note value. For example, the syllable hao lasts from the 10th frame to the 30th frame, and the note boundary appears in the 20th frame, so hao has two pitches, one is mean(pitch[10:20]) and the other is mean(pitch[20:30]). In the above way, singing annotation can be completed, including: syllables and syllable boundaries, notes and note boundaries.

[0096] In this embodiment, a regression model is used to predict the boundaries of notes, combined with the syllable boundaries calculated by forced alignment, so as to improve the accuracy of singing annotation.

[0097] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0098] Based on the same inventive concept, the embodiment of the present application also provides a singing voice tagging device for implementing the singing voice tagging method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more singing voice tagging device embodiments provided below can refer to the limitations of the singing voice tagging method above, and will not be repeated here.

[0099] In one embodiment, Figure 8As shown, a singing voice annotation device is provided, comprising: an audio signal acquisition module 801, a note boundary acquisition module 802, a pitch information acquisition module 803 and a singing voice annotation acquisition module 804, wherein:

[0100] The audio signal acquisition module 801 is used to acquire the singing audio signal to be annotated of the target song;

[0101] The note boundary acquisition module 802 is used to acquire two boundary audio frames corresponding to each note of the target song in the singing audio signal, and to acquire two boundary audio frames corresponding to each syllable of the target song in the singing audio signal;

[0102] The pitch information acquisition module 803 is used to determine the boundary audio frames of the notes contained within the range of two boundary audio frames of each syllable according to the boundary audio frames of each syllable and the boundary audio frames of each note, and obtain the pitch information of the notes contained within the range of two boundary audio frames of each syllable based on the boundary audio frames of the notes contained within the range of two boundary audio frames of each syllable;

[0103] The singing annotation acquisition module 804 is used to use the two boundary audio frames of each syllable, the note boundary audio frames contained in the two boundary audio frames of each syllable, and the pitch information of the notes contained in the two boundary audio frames of each syllable as the singing annotation information of the singing audio signal.

[0104] Each module in the above-mentioned singing voice annotation device can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0105] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig. 9As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store singing audio signal data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a singing annotation method is implemented.

[0106] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0107] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0108] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0109] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0110] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0111] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0112] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0113] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A singing annotation method, characterized in that: The method comprises: Obtaining a target song audio signal to be labeled; Obtain two boundary audio frames corresponding to each note of the target song in the singing audio signal, and obtain two boundary audio frames corresponding to each syllable of the target song in the singing audio signal; Determine the boundary audio frames of the notes contained within the two boundary audio frames of each syllable according to the boundary audio frames of each syllable and the boundary audio frames of each note, and obtain the pitch information of the notes contained within the two boundary audio frames of each syllable based on the boundary audio frames of the notes contained within the two boundary audio frames of each syllable; The two boundary audio frames of each syllable, the note boundary audio frames contained in the two boundary audio frames of each syllable, and the pitch information of the notes contained in the two boundary audio frames of each syllable are used as the singing annotation information of the singing audio signal.

2. The method according to claim 1, characterized in that The step of obtaining two boundary audio frames corresponding to each note of the target song in the singing audio signal includes: Acquire the frequency spectrum characteristics and fundamental frequency characteristics of the singing audio signal; The frequency spectrum feature and the fundamental frequency feature are input into a pre-trained note prediction model, and two boundary audio frames corresponding to each note of the target song in the singing audio signal are obtained through the note prediction model.

3. The method according to claim 2, characterized in that Before inputting the spectrum feature and the fundamental frequency feature into the pre-trained note prediction model, the method further includes: Acquire a sample singing audio signal of a sample song, and a sample frequency spectrum feature and a sample fundamental frequency feature corresponding to the sample singing audio signal; Inputting the sample spectrum feature and the sample fundamental frequency feature into the note prediction model to be trained, and obtaining the predicted boundary audio frames corresponding to each note of the sample song in the sample singing audio signal; Obtaining actual boundary audio frames corresponding to each note of the sample song in the sample singing audio signal; The note prediction model is trained according to the difference between the predicted boundary audio frame and the actual boundary audio frame to obtain the pre-trained note prediction model.

4. The method according to claim 1, characterized in that: The step of obtaining pitch information of the notes contained within the two boundary audio frames of each syllable based on the boundary audio frames of the notes contained within the two boundary audio frames of each syllable comprises: Determine a current syllable of the target song, where the current syllable is any syllable of the syllables of the target song; In the case where there is a boundary audio frame of a musical note between two boundary audio frames corresponding to the current syllable, obtaining the note duration of the musical note contained in the current syllable within the current syllable according to the two boundary audio frames corresponding to the current syllable and the existing boundary audio frame of the musical note; According to the note duration of the note contained in the current syllable, the pitch information of the note contained in the current syllable is obtained.

5. The method according to claim 4, characterized in that The step of obtaining the pitch information of the note contained in the current syllable according to the note duration of the note contained in the current syllable includes: According to the note duration of the note contained in the current syllable, obtaining a first sub-baseband signal matching the note duration of the note contained in the current syllable from the baseband signal corresponding to the singing audio signal; The pitch information of the musical note contained in the current syllable is obtained according to the signal value of each of the first sub-baseband signals.

6. The method according to claim 4, characterized in that After determining the current syllable of the target song, the method further includes: When there is no boundary audio frame of a musical note between two boundary audio frames corresponding to the current syllable, acquiring the syllable duration of the current syllable according to the two boundary audio frames corresponding to the current syllable; According to the syllable duration, obtaining a second sub-baseband signal matching the syllable duration from the baseband signal corresponding to the singing audio signal; The pitch information of the current syllable is obtained according to the signal value of the second sub-baseband signal, and the pitch information of the current syllable is used as the pitch information of each note contained in the current syllable.

7. The method according to any one of claims 1 to 6, characterized in that: The step of obtaining two boundary audio frames corresponding to each syllable of the target song in the singing audio signal includes: Acquire lyrics text information of the target song, wherein the lyrics text information includes each syllable of the target song; The singing audio signal and the lyrics text information are forced to align to obtain two boundary audio frames corresponding to each syllable of the target song in the singing audio signal.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images