Song singing synthesis data labeling method and device, computer device, and storage medium

Abstract

The present disclosure relates to the technical field of computer processing, and discloses a singing synthesis data labeling method and device, computer equipment and a storage medium. The method provided by the present disclosure comprises the following steps: obtaining sample audio and lyric text content of the sample audio, wherein the voice content is obtained by pre-labeling through a preset plug-in; based on time information of the sample audio, displaying the voice content and the sample audio on the same interface; based on the display information of the interface, obtaining a labeling file comprising voice content, voice boundary and music information of the sample audio, wherein the voice boundary is obtained by labeling through a preset plug-in, and the music information is obtained through a digital audio workstation. The sample audio is aligned with the voice content by using the time information of the sample audio, and the voice content and the sample audio are displayed on the same interface by combining the time information and the preset plug-in, thereby removing the step of repeatedly labeling lyrics and improving the labeling efficiency of singing synthesis data.

Description

Technical Field

[0001] This disclosure relates to the field of computer processing, specifically to methods, apparatus, computer equipment, and storage media for labeling vocal synthesis data. Background Technology

[0002] Singing voice synthesis (SVS) is the synthesis of singing based on lyrics and musical score information. In singing voice synthesis technology, deep learning-based methods are currently the mainstream approach. This method uses deep neural networks to replace the decision tree clustering context-dependent Markov model. Since the quality of the synthesized voice depends on the training data, the quality and efficiency of the annotation of the singing voice synthesis data have become one of the bottlenecks in voice production. Because the annotation of singing voice synthesis data includes both musical and phonetic information, it requires annotation by personnel with backgrounds in music and linguistics, respectively. This process is complex and difficult to proofread, resulting in low annotation efficiency for singing voice synthesis data. Summary of the Invention

[0003] In view of this, this disclosure provides a method, apparatus, computer equipment, and storage medium for annotating singing synthesis data to solve the problem of low annotation efficiency.

[0004] Firstly, this disclosure provides a method for annotating singing synthesis data, the method comprising:

[0005] Obtain sample audio and the lyrics text content of the sample audio, wherein the lyrics text content is obtained through annotation by a preset plugin;

[0006] Based on the time information of the sample audio, the lyrics text and the sample audio are displayed on the same interface;

[0007] Based on the information displayed on the interface, an annotation file is obtained that includes the lyrics text content, speech boundaries, and music information of the sample audio. The speech boundaries are obtained through the preset plugin, and the music information is obtained through a digital audio workstation.

[0008] Secondly, this disclosure provides a labeling device for singing synthesis data, the device comprising:

[0009] The first acquisition module is used to acquire sample audio and the lyrics text content of the sample audio, wherein the lyrics text content is obtained by annotation through a preset plugin;

[0010] The display module is used to display the lyrics text content and the sample audio on the same interface based on the time information of the sample audio.

[0011] The second acquisition module is used to acquire, based on the display information of the interface, a labeled file including the lyrics text content, speech boundaries and music information of the sample audio. The speech boundaries are obtained by the preset plugin and the music information is obtained by the digital audio workstation.

[0012] Thirdly, this disclosure provides a computer device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the annotation method for singing synthesis data of the first aspect or any corresponding embodiment described above.

[0013] Fourthly, this disclosure provides a computer-readable storage medium storing computer instructions for causing a computer to execute the annotation method for singing synthesis data according to the first aspect or any corresponding embodiment described above.

[0014] The annotation method for singing synthesis data provided in this embodiment uses the time information of the sample audio to align the sample audio with the speech content. At the same time, it combines the time information and preset plugins to display the lyrics text content and sample audio on the same interface, eliminating the step of repeatedly annotating lyrics, realizing the integration of music and speech annotation, and improving the annotation efficiency of singing synthesis data. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of this disclosure, the accompanying drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a diagram illustrating the annotation of singing synthesis data in related technologies;

[0017] Figure 2 This is a flowchart illustrating a method for annotating singing synthesis data according to an embodiment of the present disclosure;

[0018] Figure 3 This is a flowchart illustrating another method for annotating singing synthesis data according to an embodiment of the present disclosure;

[0019] Figure 4 This is a flowchart illustrating a method for annotating synthesized singing data according to an embodiment of the present disclosure;

[0020] Figure 5 This is a schematic diagram of the annotation of singing synthesis data according to an embodiment of the present disclosure;

[0021] Figure 6 This is a structural block diagram of a device for labeling singing synthesis data according to an embodiment of the present disclosure;

[0022] Figure 7 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present disclosure. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0024] The annotation information for singing synthesis data is divided into music information and speech information. Music information requires annotation of melody, rhythm, and lyrics to produce a score file, such as a MusicXml file. Speech information requires annotation of the singing content and speech boundary information, generally including four layers of information: sentence, word, syllable, and phoneme, to produce a language file, such as a TextGrid file.

[0025] In related technologies, such as Figure 1 As shown, the annotation of singing synthesis data includes four steps: speech information annotation, music information annotation, annotation alignment, and annotation data cleaning. Specifically, speech information annotation includes content annotation and boundary annotation. Content annotation involves three steps: text annotation, sentence segmentation, and precise phoneme labeling, accurately annotating the actual singing content. Boundary annotation is used to delineate speech boundaries, also known as phoneme segmentation, accurately annotating the duration information of the actual singing content. Music information annotation includes Musical Instrument Digital Interface (MIDI) annotation and lyrics annotation. MIDI annotation is used to annotate the pitch and rhythm information of the actual singing content, while lyrics annotation is used to annotate the actual singing content down to the syllable level. Annotation alignment involves algorithm engineers aligning the shared lyrics information between the MusicXml and TextGrid files to generate error logs, and music annotators manually judging and modifying the reported errors based on the error logs to ensure complete consistency of shared information between the two files. Annotation data cleaning refers to algorithm engineers extracting the music and speech annotation information from the MusicXml and TextGrid files respectively, and integrating them into an annotation file used for training singing synthesis data.

[0026] In the aforementioned technical solution, the lack of integration between music and audio information during the annotation phase leads to a lack of understanding between the two groups of annotators regarding each other's annotations, resulting in a high error rate. To align the music and audio annotations, both groups must annotate lyrics, increasing workload. During the annotation file alignment phase, annotators need to open two separate software programs to locate and correct erroneous musical phrases, a process that requires significant manpower and results in low annotation efficiency.

[0027] Based on this, the annotation method for synthesized singing data provided in this disclosure, through the cooperation of a preset plugin and a digital audio workstation (DAW), integrates music information and speech information during the annotation stage. Furthermore, the music information and speech information (including lyrics and speech boundaries) are aligned using absolute time, thus eliminating the need for double annotation of lyrics information and improving annotation efficiency and accuracy. The digital audio workstation is audio processing software used for recording and mixing. Based on the computer's data processing capabilities, it uses the computer's hard drive as the primary storage unit and employs an audio card or digital signal processor. With software support, it performs functions such as recording, editing, playback, mixing, compression, and audio signal visualization.

[0028] According to an embodiment of this disclosure, a method for annotating singing synthesis data is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0029] This embodiment provides a method for annotating synthesized singing data, which can be used in the aforementioned computer devices, such as computers and mobile terminals. Figure 2 This is a flowchart of a method for annotating singing synthesis data according to an embodiment of the present disclosure, such as... Figure 2 As shown, the process includes the following steps:

[0030] Step S201: Obtain the sample audio and the lyrics text of the sample audio.

[0031] The lyrics text content is obtained through pre-annotation using a preset plugin.

[0032] The sample audio is an original a cappella recording of a song, with the vocals on a separate track; only the dry vocals were collected to obtain the sample audio. During recording, the singer provided supplementary audio to ensure vocal stability and pitch accuracy. The songs for the vocal recording were selected according to the following rules: songs containing non-Chinese characters were excluded; the tempo per minute of the remaining songs was calculated; songs with relatively low tempos were prioritized to ensure coverage of phonemes and phoneme combinations in the selected songs. It should be noted that the above is merely one method of obtaining sample audio and does not limit the scope of protection of this disclosure. The specific method of acquisition is set according to actual needs, and no limitations are made here.

[0033] The lyrics text content of the sample audio is obtained through pre-annotation using a preset plugin. The annotation of the lyrics text content includes text annotation, sentence segmentation, and phoneme annotation.

[0034] Text annotation involves filling in and correcting the lyrics of the sample audio. In some optional implementations, the lyrics are filled into corresponding intervals, with each interval corresponding to a complete line of lyrics, and each line lasting no more than 15 seconds. If the singing content of the sample audio is inconsistent with the lyrics, the actual singing content is used for text correction, ensuring there are no misspellings, omissions, or extra words. Text regularization is used, and punctuation marks are added at grammatical breaks.

[0035] Sentence segmentation involves dividing the lyrics of sample audio into segments, including segmentation annotation and file splitting. During segmentation annotation, silent segments must be preserved before and after each sentence, for example, 150ms to 300ms. If the silent segments before and after a sentence are too long, empty intervals must be cut out. Valid phonemes must be segmented completely; interval boundaries cannot be placed in the middle of phonemes. Inspiratory sounds should be placed completely at the end of the preceding sentence, and there should be no inspiratory sounds at the beginning of a sentence. File splitting uses a script to segment the audio and the results of segmentation annotation based on the annotated intervals, breaking the complete song into several short sentences.

[0036] Phoneme annotation involves checking the text, pinyin, and rests / pauses in the phonetic annotation files of a phonetic library. Phoneme annotation includes phoneme transcription, phoneme checking, and adding rests / pauses. Phoneme transcription uses a front-end dictionary to transcribe the text into phonemes, generating a file containing three layers of information: sentence level, text level, and phoneme transcription level. Phoneme checking verifies whether the phonemes and text information correspond to the actual performance content and makes corrections accordingly. In phonetic annotation, rests / pauses are represented using preset characters. For example, "sil" represents a long pause, appearing only at the beginning and end of a sentence, and "rest" represents a short pause, corresponding to rests in musical annotation. If there is an inspiratory sound between sentences, "rest" must be marked. If there is a noticeable pause within a sentence, "rest" must also be marked.

[0037] The preset plugin is used to implement the annotation function. It can save the user's annotation content and time boundary values ​​at the plugin level and display them on the interface. It also supports interaction with peripheral devices. These peripheral device actions include, but are not limited to, mouse dragging, clicking, and keyboard shortcuts.

[0038] As an optional implementation of the preset plugin, the preset plugin uses the JUCE framework and the Label class component as the basic structure for storing each segment of annotation information, thereby realizing the saving and modification of content annotations. At the same time, it inherits the MouseLister class to listen to mouse actions, realizing the interaction between the preset plugin and frequent mouse actions.

[0039] Step S202: Based on the time information of the sample audio, display the lyrics text and the sample audio on the same interface.

[0040] The time information of a sample audio refers to the time corresponding to the playback of the sample audio. For example, if the duration of a sample audio is 2 minutes, the time corresponding to the first playback of the lyrics "rainy day" is 25 seconds. These times are the time information of the sample audio.

[0041] The sample audio is displayed as a waveform and spectrum. Based on this, the lyrics text obtained in step S201 are displayed together with the waveform and spectrum of the sample audio on the same interface. The lyrics text is aligned with the sample audio during display to ensure synchronization between them.

[0042] Step S203: Based on the information displayed on the interface, obtain the annotation file including the lyrics text content, speech boundaries and music information of the sample audio.

[0043] The speech boundary is obtained by marking it through the preset plugin, and the music information is obtained through a digital audio workstation.

[0044] The lyrics text and sample audio are displayed on the same interface, thus ensuring the integration of annotations for speech and music information, and the annotation is performed at the same absolute time. The lyrics text of the sample audio is obtained in step S201 above, and the speech boundaries and music information are obtained based on the annotations of the content displayed on the same interface in step S202.

[0045] By using preset plugins and digital audio workstations to annotate on the same interface, speech boundaries and music information are obtained. This is based on automatic pre-annotation and combined with user annotation interaction to determine speech boundaries and music information.

[0046] The annotation method for singing synthesis data provided in this embodiment aligns the sample audio with the speech content using the time information of the sample audio. Simultaneously, it displays the lyrics text and sample audio on the same interface, eliminating the need for repetitive lyric annotation and integrating music and speech annotation, thus improving the annotation efficiency of singing synthesis data.

[0047] This embodiment provides a method for annotating synthesized singing data, which can be used in the aforementioned computer devices, such as computers and mobile terminals. Figure 3 This is a flowchart of a method for annotating singing synthesis data according to an embodiment of the present disclosure, such as... Figure 3 As shown, the process includes the following steps:

[0048] Step S301: Obtain the sample audio and the lyrics text of the sample audio.

[0049] The lyrics text content is obtained through annotation by a preset plugin. For details, please refer to [link / reference]. Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0050] Step S302: Based on the time information of the sample audio, display the lyrics text and the sample audio on the same interface.

[0051] Specifically, step S302 includes:

[0052] Step S3021: Import sample audio at the beginning of the project on the digital audio workstation and load preset plugins on the sample audio track.

[0053] The digital audio workstation is selected based on actual needs, and no limitations are imposed here. Sample audio is imported into the digital audio workstation, and preset plugins are loaded onto the tracks to segment phonemes with reference to the workstation's spectrum. Phoneme segmentation can be automatic or interactively performed by the user.

[0054] Step S3022: Align the lyrics text with the sample audio for display.

[0055] Since the lyrics text is annotated by a preset plugin, while the sample audio is annotated with music information after being imported into a digital audio workstation, these two are obtained through different objects (i.e., the preset plugin and the digital audio workstation). Specifically, the audio is imported into the beginning of the project in the digital audio workstation, ensuring that the audio's start time in the workstation is 0:00, achieving aligned display of the sample audio and lyrics text, and realizing the annotation of speech and music information through annotation fusion.

[0056] Step S303: Based on the information displayed on the interface, obtain the annotation file including the lyrics text content, speech boundaries, and music information of the sample audio.

[0057] The speech boundary is obtained by marking it through the preset plugin, and the music information is obtained through a digital audio workstation.

[0058] In some optional implementations, step S303 above includes:

[0059] Step a1: The preset plugin reads the playback time from the digital audio workstation.

[0060] Step a2: The preset plugin aligns and displays the lyrics text, speech boundaries, and music information based on the playback time.

[0061] The preset plugin needs to achieve transport synchronization with the digital audio workstation. It not only needs to acquire information unidirectionally from the digital audio workstation, but also needs to actively transmit information back to the digital audio workstation. That is, playback time switching and time range selection on the preset plugin should be synchronously reflected on the interface of the digital audio workstation. This is the key to the preset plugin's ability to integrate music and voice information annotation processes and improve annotation efficiency.

[0062] In related technologies, the plugins used for annotation run on real-time semaphore blocks, which can be processed between each render call. This also means that the plugins cannot read audio samples from the digital audio workstation outside of normal real-time audio time constraints, and therefore cannot achieve the synchronization functionality required by the default plugins.

[0063] The preset plugin achieves the aforementioned synchronization functionality based on ARA (Audio Random Access) technology. ARA is an extension of the established plugin standard, allowing digital audio workstations and plugins to exchange information about audio files, tempo, pitch, rhythm, and many other details, thus enabling closer collaboration and significantly improving plugin integration on digital audio workstations. It allows plugins to read audio samples from the digital audio workstation, enabling them to implement more complex processing algorithms, which was previously difficult to achieve when bound to a separate real-time buffer. Using the JUCE framework and ARA, the preset plugin reads audio samples from the digital audio workstation beyond the normal real-time audio time limits, operating like a sample editor without being bound to a single real-time buffer, and synchronizing playback time points and selection intervals with the digital audio workstation in real time, aligning musical and speech information.

[0064] Specifically, the preset plugin reads the playback time of sample audio from the digital audio workstation and uses this playback time to control the display of voice and music information, thereby achieving the aligned display of voice and music information.

[0065] The preset plugin enables reading playback time from the digital audio workstation. This reading method is not limited by a single real-time buffer. By synchronizing the playback time with the digital audio workstation in real time, the voice information is aligned with the music information.

[0066] For further details, please see Figure 2 Step S203 of the illustrated embodiment will not be described again here.

[0067] The annotation method for singing synthesis data provided in this embodiment combines a preset plugin with a digital audio workstation to integrate language annotation and music annotation. Since the time of the sample audio is absolute time, the alignment of speech information and music information is achieved through absolute time, which can ensure the accuracy of the alignment.

[0068] This embodiment provides a method for annotating synthesized singing data, which can be used in the aforementioned computer devices, such as computers and mobile terminals. Figure 4 This is a flowchart of a method for annotating singing synthesis data according to an embodiment of the present disclosure, such as... Figure 4 As shown, the process includes the following steps:

[0069] Step S401: Obtain the sample audio and the lyrics text of the sample audio.

[0070] The lyrics text content is obtained through annotation by a preset plugin. For details, please refer to [link / reference]. Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0071] Step S402: Based on the time information of the sample audio, display the lyrics text and the sample audio on the same interface. For details, please refer to [link to relevant documentation]. Figure 3 Step S302 of the illustrated embodiment will not be described again here.

[0072] Step S403: Based on the information displayed on the interface, obtain the annotation file including the lyrics text content, speech boundaries, and music information of the sample audio.

[0073] Specifically, step S403 includes:

[0074] Step S4031: Based on the sample audio displayed on the interface, obtain the phoneme boundary division result of the lyrics text content to obtain the speech boundary.

[0075] The speech boundaries are obtained based on annotations from a preset plugin.

[0076] Speech boundaries are obtained by segmenting the lyrics text content into phonemes. For example, the interface displays the spectrum of sample audio and the lyrics text content simultaneously. The phoneme boundaries of the lyrics text content are divided according to preset segmentation rules. Alternatively, the interface displays the spectrum of sample audio simultaneously, and the user obtains the speech boundaries by interacting with the interface.

[0077] In some optional implementations, step S4031 above includes:

[0078] Step b1: Obtain pre-annotated phonemes of the lyrics text.

[0079] Step b2: Based on the spectrum of the sample audio displayed on the interface, obtain the adjustment results of the pre-annotation of phonemes by the preset plugin, and obtain the speech boundary.

[0080] In the field of vocal synthesis, pre-annotation techniques can automatically annotate prosodic boundaries, providing initial annotations for various speech features in singing. For example, a pre-annotation model can be trained using sample data to provide prosodic and boundary predictions for new data. After obtaining the lyrics text, the trained pre-annotation model is used to predict the phoneme boundaries of the lyrics text, resulting in pre-annotated phonemes.

[0081] After obtaining the pre-annotated phonemes, the annotators adjust the pre-annotation results by combining them with the spectrum of the sample audio displayed on the interface, thereby obtaining accurate speech boundaries.

[0082] The existence of pre-annotated phonemes provides initial annotations for various speech features in singing. After adjustments are made based on these annotations, speech boundaries can be obtained. This not only significantly reduces the cost of standardization but also improves the overall quality of data, thereby helping to generate higher-quality synthesized singing voices.

[0083] Step S4032: Display the speech boundary and obtain the annotation results of music information on the sample audio to obtain music information.

[0084] The music information is obtained through annotation on a digital audio workstation.

[0085] After obtaining the speech boundaries, they are also displayed on the interface to annotate the actual musical information being sung. Specifically, using the speech boundaries displayed on the interface, users can achieve accurate frame-level annotation using a digital audio workstation. Only vowels with pitch information are annotated; initials without pitch information, such as affricates, are not annotated. Musical information is obtained through this annotation.

[0086] In some alternative implementations, the music information can be pre-annotated using a trained annotation model, and the pre-annotated music information can be displayed on the interface. The user then uses a digital audio workstation to adjust the pre-annotated music information based on the speech boundaries displayed on the interface, thus obtaining accurate music information.

[0087] Step S4033: Import music information into the preset plugin.

[0088] Step S4034: Use a preset plugin to convert the notes of each marked unit in the music information to obtain the pitch information of the sample audio.

[0089] The annotation file also includes pitch information, which is obtained through a preset plugin. Specifically, the music information output from the digital audio workstation is imported into the preset plugin to obtain the pitch information.

[0090] The music information output by a digital audio workstation includes multiple annotation units, each containing a start time, end time, and the note value. Since the file format output by a digital audio workstation is fixed, the notes in the annotation units need to be converted to obtain the pitch information. A preset plugin is used to generate pitch information within the annotation file, further enriching the content of the resulting annotation file.

[0091] In some optional implementations, step S4034 above includes:

[0092] Step c1: Obtain the initial annotation file containing lyrics text content, speech boundaries, and music information, as well as the time period of each annotation unit.

[0093] Step c2: Add a blank layer to the initial annotation file and map the time period to the blank layer.

[0094] Step c3 converts the pitch of the notes in each labeled unit into the instrument digital interface code.

[0095] Step c4: Fill the corresponding time unit of the blank layer with the instrument digital interface code, wherein discontinuous time intervals are filled with preset symbols to obtain the pitch information of the sample audio.

[0096] The music information in the initial annotation file is output by the digital audio workstation. The music information includes multiple annotation units, each with a start and end time, and correspondingly, each annotation unit has a time period.

[0097] A blank layer is added to the initial annotation file. The pitch of the notes in the annotation unit is converted into Musical Instrument Digital Interface (MIDI) encoding and filled into the corresponding time unit. The "rest" symbol is used to fill the discontinuous time intervals, thereby generating pitch information in the Textgrid.

[0098] Since the MIDI file format output by the digital audio workstation is fixed, each MIDI annotation unit contains start time, end time and note information. By adding a blank layer to the initial blank file, the pitch of the note in the MIDI annotation unit is converted into MIDI code and filled into the time unit corresponding to the blank layer, thereby generating pitch information in the initial annotation file and producing an annotation file containing pitch information.

[0099] Step S4035: Output a labeled file including lyrics text content, speech boundaries, and music information.

[0100] After the above annotation process, the lyrics text content, speech boundaries and music information of the sample audio are obtained. This information is stored in the same annotation file, which can be used to train the singing synthesis model in the future by outputting the annotation file.

[0101] The annotation method for singing synthesis data provided in this embodiment annotates music information based on the obtained speech boundaries, thereby obtaining music that accurately represents the actual singing.

[0102] As a specific application example of the embodiments of this disclosure, such as Figure 5 As shown, the annotation of singing synthesis data is performed on a computer. First, sample audio is acquired and its pitch is corrected. Then, a preset plugin is used to annotate the lyrics, including sentences, words, syllables, and phonemes. Next, based on the time information of the sample audio, the lyrics and sample audio are aligned and displayed on the same interface. The user interacts with this interface to annotate speech boundaries using the preset plugin and music information using a digital audio workstation, building upon the annotation and fusion process. Further, the music information obtained from the digital audio workstation is imported into the preset plugin to output pitch information. Based on this, an annotation file including lyrics, speech boundaries, music information, and pitch information can be output. This annotation file can be used for subsequent training of the singing synthesis model.

[0103] This embodiment also provides a device for labeling singing synthesis data, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0104] This embodiment provides a device for labeling singing synthesis data, such as... Figure 6 As shown, it includes:

[0105] The first acquisition module 601 is used to acquire sample audio and the lyrics text content of the sample audio. The lyrics text content is obtained by annotation through a preset plugin.

[0106] Display module 602 is used to display lyrics text and sample audio on the same interface based on the time information of the sample audio.

[0107] The second acquisition module 603 is used to acquire, based on the information displayed on the interface, a labeled file including the lyrics text content, speech boundaries, and music information of the sample audio. The speech boundaries are obtained by labeling through a preset plugin, and the music information is obtained through a digital audio workstation.

[0108] In some alternative implementations, the display module 602 includes:

[0109] An audio import unit is used to import the sample audio at the beginning of the project on the digital audio workstation and load the preset plugin on the track of the sample audio.

[0110] An alignment display unit is used to align the lyrics text content with the sample audio based on the time information of the sample audio.

[0111] In some alternative implementations, the second acquisition module 603 includes:

[0112] A reading unit is used for the preset plugin to read the playback time from the digital audio workstation;

[0113] An alignment unit is used by the preset plugin to align the lyrics text content with the sample audio based on the playback time.

[0114] In some alternative implementations, the second acquisition module 603 includes:

[0115] The boundary acquisition unit is used to obtain the phoneme boundary division result of the preset plugin on the lyrics text content based on the sample audio displayed on the interface, and obtain the speech boundary.

[0116] A boundary display unit is used to display the speech boundary and obtain the annotation results of the digital audio workstation on the sample audio for music information, thereby obtaining the music information;

[0117] The output unit is used to output a labeled file including the lyrics text content, the speech boundaries, and the music information.

[0118] In some alternative implementations, the second acquisition module 603 includes:

[0119] A pre-annotation acquisition unit is used to acquire pre-annotations of the phonemes in the lyrics text content;

[0120] The adjustment unit is used to obtain the adjustment result of the pre-labeled phonemes by the preset plugin based on the spectrum of the sample audio displayed on the interface, and to obtain the speech boundary.

[0121] In some optional implementations, the second acquisition module 603 further includes:

[0122] The music information import unit is used to import the music information into the preset plugin;

[0123] The conversion unit is used to convert the pitch of the notes in each annotation unit of the music information using the preset plugin to obtain the pitch information of the sample audio, and the annotation file includes the pitch information.

[0124] In some alternative implementations, the conversion unit includes:

[0125] The sub-unit is used to acquire an initial annotation file containing the lyrics text content, the speech boundaries, and the music information, as well as the time period of each annotation unit;

[0126] A mapping subunit is used to add a blank layer to the initial annotation file and map the time period to the blank layer;

[0127] The conversion subunit is used to convert the pitch of the notes in each annotation unit into the instrument data interface code;

[0128] The filling subunit is used to fill the instrument data interface encoding into the corresponding time unit of the blank layer, wherein discontinuous time intervals are filled with preset symbols to obtain the pitch information of the sample audio.

[0129] In this embodiment, the annotation device for singing synthesis data is presented in the form of a functional unit. Here, a unit refers to an ASIC circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0130] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0131] This disclosure also provides a computer device having the above-described features. Figure 7 The annotation device shown is for the singing synthesis data.

[0132] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of this disclosure, such as... Figure 7 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 7 Take a processor 10 as an example.

[0133] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0134] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0135] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created during the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0136] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0137] The computer device also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.

[0138] Input device 30 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the computer device, such as a touchscreen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 40 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touchscreen.

[0139] This disclosure also provides a computer-readable storage medium in which the methods described in this disclosure can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium after being downloaded over a network. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium may be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium may also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0140] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the method embodiments are basically similar to the device and system embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the device and system embodiments.

[0141] It is understood that in the specific embodiments of this disclosure, data such as audio is involved. When the above embodiments of this disclosure are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0142] Although embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for labeling synthesized singing data, characterized in that, The method includes: Obtain sample audio and the lyrics text content of the sample audio, wherein the lyrics text content is obtained through pre-annotation by a preset plugin; Based on the time information of the sample audio, the lyrics text and the sample audio are displayed on the same interface; Based on the information displayed on the interface, an annotation file is obtained that includes the lyrics text content, speech boundaries, and music information of the sample audio. The speech boundaries are obtained through the preset plugin, and the music information is obtained through a digital audio workstation. The step of displaying the lyrics text content and the sample audio on the same interface based on the time information of the sample audio includes: The sample audio is imported at the beginning of the project on the digital audio workstation, and the preset plugin is loaded on the track of the sample audio. The preset plugin is used to implement the annotation function, as well as to save the content and time boundary values ​​of the user annotation and display them on the interface. The lyrics text content is aligned and displayed with the sample audio.

2. The method according to claim 1, characterized in that, Based on the display information of the interface, the method for obtaining a labeled file including the lyrics text content, speech boundaries, and music information of the sample audio includes: The preset plugin reads the playback time from the digital audio workstation; The preset plugin aligns and displays the lyrics text, speech boundaries, and music information based on the playback time.

3. The method according to claim 1, characterized in that, Based on the display information of the interface, the method for obtaining a labeled file including the lyrics text content, speech boundaries, and music information of the sample audio includes: Based on the sample audio displayed on the interface, the phoneme boundary division result of the preset plugin on the lyrics text content is obtained, and the speech boundary is obtained. Display the speech boundary and obtain the annotation results of the digital audio workstation on the sample audio for music information, thereby obtaining the music information; The output includes a labeled file containing the lyrics text, the speech boundaries, and the music information.

4. The method according to claim 3, characterized in that, Based on the sample audio displayed on the interface, the phoneme boundary division result of the preset plugin on the lyrics text content is obtained to obtain the speech boundary, including: Obtain the pre-annotated phonemes of the lyrics text content; Based on the spectrum of the sample audio displayed on the interface, the adjustment result of the pre-annotation of the phonemes by the preset plugin is obtained, and the speech boundary is obtained.

5. The method according to claim 1, characterized in that, The process of obtaining the annotation file, which includes the lyrics text content, speech boundaries, and music information of the sample audio, based on the display information of the interface, further includes: Import the music information into the preset plugin; The pitch of the notes in each labeled unit of the music information is converted using the preset plugin to obtain the pitch information of the sample audio, and the labeled file includes the pitch information.

6. The method according to claim 5, characterized in that, The step of using the preset plugin to convert the pitch of the notes in each marked unit of the music information to obtain the pitch information of the sample audio includes: Obtain an initial annotation file containing the lyrics text content, the speech boundaries, and the music information, as well as the time period of each annotation unit; Add a blank layer to the initial annotation file and map the time period to the blank layer; Convert the pitch of the notes in each labeled unit into the instrument digital interface code; The instrument digital interface code is filled into the corresponding time unit of the blank layer, wherein discontinuous time intervals are filled with preset symbols to obtain the pitch information of the sample audio.

7. A device for labeling singing synthesis data, characterized in that, The device includes: The first acquisition module is used to acquire sample audio and the lyrics text content of the sample audio, wherein the lyrics text content is obtained by pre-annotation through a preset plugin; The display module is used to display the lyrics text content and the sample audio on the same interface based on the time information of the sample audio. The second acquisition module is used to acquire, based on the display information of the interface, a labeled file including the lyrics text content, speech boundaries and music information of the sample audio, wherein the speech boundaries are obtained by the preset plugin and the music information is obtained by the digital audio workstation. The display module includes: The audio import unit is used to import the sample audio at the beginning of the project on the digital audio workstation and load the preset plugin on the track of the sample audio. The preset plugin is used to implement the annotation function, and to save the content and time boundary values ​​of the user annotation and display them on the interface. An alignment display unit is used to align the lyrics text content with the sample audio for display.

8. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the annotation method for singing synthesis data according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the annotation method for singing synthesis data according to any one of claims 1 to 6.

Images