Song audio annotation and alignment model training method, device and storage medium

By separating and dividing the singing audio and lyric text, and automatically labeling it with the alignment model, the problems of low efficiency and high cost of song labeling are solved, and efficient and low-cost song audio labeling are achieved.

CN114373478BActive Publication Date: 2024-11-26BEIJING XIAOMI MOBILE SOFTWARE CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111538027.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-15
Publication Date
2024-11-26
Estimated Expiration
2041-12-15

AI Technical Summary

Technical Problem

Song labeling requires a lot of manual labeling, which is inefficient and costly. Especially due to the unfixed length of the song phoneme and wide range of the basic frequency, it is difficult for the existing technology to effectively automate the labeling.

Method used

By obtaining singing audio and lyric text, separating the vocals and accompaniment sounds, segmenting them according to the start timestamp of the lyrics clip and the energy information of the human voice audio, using the alignment model to automatically label the phoneme and time information, adjusting the fundamental frequency threshold to determine the mute The authenticity of phonemes.

Benefits of technology

It improves the efficiency and reduces costs of song audio labeling, realizes automatic labeling of song audio, and improves the accuracy and efficiency of labeling.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114373478B_ABST
    Figure CN114373478B_ABST
Patent Text Reader

Abstract

The embodiments of the present application provide a song audio annotation and alignment model training method, device and storage medium. In some embodiments of the present application, a song to be annotated is obtained, wherein the song to be annotated includes singing audio and lyrics text, and the singing audio contains human voice and accompaniment sound overlapping with human voice; the human voice and accompaniment sound in the singing audio are separated to obtain human voice audio; according to the start timestamp of each lyrics segment and the energy information of the human voice audio, the human voice audio is segmented at the sentence level to obtain at least one human voice audio segment, thereby improving the rationality of the segmentation of the human voice audio segment; the human voice audio segment and the corresponding lyrics segment are input into a preset alignment model to obtain the annotation result corresponding to each human voice audio segment, and the alignment model is used to automatically annotate the human voice audio segment and the corresponding lyrics segment, thereby improving the annotation efficiency and reducing the cost.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to a song audio tagging and model training method, device and storage medium. Background Art

[0002] As a branch of speech synthesis, song synthesis has a broad development prospect. The algorithm model used in song synthesis requires a large amount of labeled data during training. For song annotation, it is necessary to annotate the lyrics information of the song and the phoneme duration information of each pronunciation in the song.

[0003] The duration of phonemes in songs is not fixed, the fundamental frequency range is wide, and the difference from spoken speech is a difficult problem for song annotation. At present, song annotation is generally done manually, which has low annotation efficiency and consumes a lot of costs. Summary of the invention

[0004] The present application provides a song audio annotation and alignment model training method, device and storage medium, which can automatically annotate song audio with high annotation efficiency and low cost. The technical solution of the present application is as follows:

[0005] The present application embodiment provides a song audio tagging method, comprising:

[0006] Acquire song data to be annotated, wherein the song data to be annotated includes singing audio and lyrics text, the singing audio includes human voice and accompaniment sound overlapping with the human voice, the lyrics text corresponds to the content sung by the human voice in the singing audio, the lyrics text includes a plurality of segmented lyrics segments, and each of the lyrics segments includes a start timestamp;

[0007] Separating the human voice and the accompaniment sound in the singing audio to obtain human voice audio;

[0008] Determine the segmentation point of the vocal audio according to the start timestamp of each of the lyrics segments and the energy information of the vocal audio;

[0009] Segmenting the vocal audio according to the segmentation points to obtain at least one vocal audio segment, wherein each vocal audio segment corresponds to one lyric segment;

[0010] Each of the vocal audio clips and the corresponding lyrics clips are input into a preset alignment model to obtain a labeling result corresponding to each of the vocal audio clips, wherein each of the labeling results includes each phoneme in the vocal audio clip and the time information corresponding to each phoneme.

[0011] Optionally, determining the segmentation point of the vocal audio according to the start timestamp of each of the lyrics segments and the energy information of the vocal audio includes:

[0012] For each of the lyrics segments, taking audio data of a predetermined time length before and / or after the time node corresponding to the start timestamp from the vocal audio as sampling data;

[0013] Calculating the audio energy at each time node in each of the sampled data;

[0014] For each of the sampled data, the time node with the minimum audio energy is used as the segmentation point of the current sampled data.

[0015] Optionally, each of the lyrics segments includes a plurality of syllables;

[0016] After the human voice audio is segmented according to the segmentation points to obtain at least one human voice audio segment, the method further includes: for each of the lyrics segments, a silent phoneme is added between every two adjacent syllables to obtain an updated lyrics segment.

[0017] Optionally, after obtaining the labeling result corresponding to each of the human voice audio segments, the method further includes:

[0018] Extracting the fundamental frequency of each of the human voice audio segments;

[0019] The labeling result is adjusted according to the extracted fundamental frequency and a preset fundamental frequency threshold.

[0020] Optionally, adjusting the labeling result according to the extracted fundamental frequency and a preset fundamental frequency threshold includes:

[0021] According to the extracted fundamental frequency and the preset fundamental frequency threshold, determining whether each silent phoneme in the updated audio lyrics segment is a true silent phoneme;

[0022] If it is a true silent phoneme, retaining the time information corresponding to the silent phoneme;

[0023] If it is not a true silent phoneme, the time information corresponding to the silent phoneme is modified.

[0024] Optionally, extracting the fundamental frequency of each of the human voice audio segments comprises:

[0025] Dividing each of the audio segments into a plurality of audio frames according to a preset time length;

[0026] For each of the audio segments, a fundamental frequency corresponding to each audio frame is extracted in units of frames.

[0027] Optionally, judging whether each silent phoneme in the updated audio lyrics segment is a true silent phoneme according to the extracted fundamental frequency and a preset fundamental frequency threshold comprises:

[0028] If N audio frames whose fundamental frequency is greater than or equal to a preset fundamental frequency threshold appear continuously in the human voice audio segment corresponding to the time information of a silent phoneme, and the overall time length of the N audio frames is greater than the preset time threshold, then the silent phoneme is determined to be a true silent phoneme, where the value range of N is 2 or 3.

[0029] Optionally, if a silent phoneme is not a true silent phoneme, the time information corresponding to the silent phoneme is modified according to the following steps:

[0030] If there are no N consecutive audio frames whose fundamental frequencies are greater than or equal to a preset fundamental frequency threshold in the human voice audio segment corresponding to the time information of the silent phoneme, the silent phoneme is deleted from the annotation result, and the duration of the silent phoneme is divided into the previous phoneme of the current silent phoneme;

[0031] If N audio frames whose fundamental frequencies are greater than or equal to a preset fundamental frequency threshold appear continuously in the human voice audio segment corresponding to the time information of the silent phoneme, and the overall time length of the N audio frames is not greater than the preset time threshold, the silent phoneme is deleted from the annotation result, and the duration of the silent phoneme is divided into the next phoneme of the current silent phoneme.

[0032] The present application also provides an alignment model training method, including:

[0033] Collecting audio sample data and text sample data corresponding to the audio sample data;

[0034] Segmenting the audio sample data and the text sample data to obtain a first audio segment sample contained in the audio sample data and a first text segment sample corresponding to the first audio segment sample;

[0035] Filtering the first audio segment sample and the first text segment sample to obtain a second audio segment sample and a second text segment sample for model training;

[0036] Annotating the second audio segment sample to obtain a phoneme sample contained in the second audio segment sample and time information corresponding to the phoneme sample;

[0037] The initial alignment model is trained using the second audio segment sample, the second text segment sample, the phoneme sample included in the second audio segment sample, and time information corresponding to the phoneme sample to obtain a trained alignment model.

[0038] Optionally, the first audio segment sample and the first text segment sample are filtered to obtain a second audio segment sample and a second text segment sample for model training, including:

[0039] removing the audio segment sample of a set language and the corresponding text segment sample from the first audio segment sample and the first text segment sample to obtain the second audio segment sample and the second text segment sample;

[0040] or,

[0041] According to the fundamental frequency information of the first audio segment sample, the audio segment samples whose fundamental frequency information does not meet the fundamental frequency information condition and the corresponding text segment samples are removed from the first audio segment samples and the first text segment samples to obtain the second audio segment samples and the second text segment samples.

[0042] The present application also provides a song audio tagging device, including:

[0043] An acquisition module is used to acquire song data to be annotated, wherein the song data to be annotated includes singing audio and lyrics text, the singing audio includes human voice and accompaniment sound overlapping with the human voice, the lyrics text corresponds to the content sung by the human voice in the singing audio, and the lyrics text includes a plurality of segmented lyrics fragments, each of which includes a start timestamp;

[0044] A separation module, used for separating the human voice and the accompaniment sound in the singing audio to obtain the human voice audio;

[0045] A determination module, which determines the segmentation point of the vocal audio according to the start timestamp of each of the lyrics segments and the energy information of the vocal audio;

[0046] A segmentation module, which segments the vocal audio according to the segmentation points to obtain at least one vocal audio segment, wherein each vocal audio segment corresponds to one lyric segment;

[0047] The alignment model module inputs each of the vocal audio segments and the corresponding lyrics segments into a preset alignment model to obtain a labeling result corresponding to each of the vocal audio segments, wherein each of the labeling results includes each phoneme in the vocal audio segment and the time information corresponding to each phoneme.

[0048] An embodiment of the present application also provides an electronic device, including: a memory and a processor;

[0049] The memory is used to store computer programs;

[0050] The processor is used to execute the computer program to perform the steps in the above method.

[0051] The present application also provides an alignment model training device, including:

[0052] A collection module, used to collect audio sample data and text sample data corresponding to the audio sample data;

[0053] A segmentation module, configured to segment the audio sample data and the text sample data to obtain a first audio segment sample contained in the audio sample data and a first text segment sample corresponding to the first audio segment sample;

[0054] A filtering module, used for filtering the first audio segment sample and the first text segment sample to obtain a second audio segment sample and a second text segment sample for model training;

[0055] a labeling module, configured to label the second audio segment sample to obtain a phoneme sample contained in the second audio segment sample and time information corresponding to the phoneme sample;

[0056] A training module is used to train an initial alignment model using the second audio segment sample, the second text segment sample, the phoneme sample contained in the second audio segment sample, and the time information corresponding to the phoneme sample to obtain a trained alignment model.

[0057] An embodiment of the present application also provides an electronic device, including: a memory and a processor;

[0058] The memory is used to store computer programs;

[0059] The processor is used to execute the computer program to implement each step in the above method.

[0060] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements each step of the above method when executed by a processor.

[0061] An embodiment of the present application also provides a computer program product, including a computer program / instruction, which implements the steps in the above method when executed by a processor.

[0062] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects:

[0063] In some embodiments of the present application, a song to be annotated is obtained, wherein the song to be annotated includes singing audio and lyrics text, and the singing audio includes human voice and accompaniment sound overlapping with the human voice; the human voice and the accompaniment sound in the singing audio are separated to obtain human voice audio; according to the start timestamp of each lyrics segment and the energy information of the human voice audio, the human voice audio is segmented at the sentence level to obtain at least one human voice audio segment, thereby improving the rationality of the segmentation of the human voice audio segment; the human voice audio segment and the corresponding lyrics segment are input into a preset alignment model to obtain the annotation result corresponding to each vocal audio segment, and the alignment model is used to automatically annotate the human voice audio segment and the corresponding lyrics segment, thereby improving the annotation efficiency and reducing the cost.

[0064] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application, and do not constitute an improper limitation on the present application.

[0066] Figure 1 A flowchart of a song audio tagging method provided for an exemplary embodiment of the present application;

[0067] Figure 2 A flowchart of an alignment model training method provided for an exemplary embodiment of the present application;

[0068] Figure 3 A flowchart of another song audio tagging method provided for an exemplary embodiment of the present application;

[0069] Figure 4 A flowchart of another song audio tagging method provided for an exemplary embodiment of the present application;

[0070] Figure 5 is a structural block diagram of a song audio tagging device provided by an exemplary embodiment of the present application;

[0071] Figure 6 is a structural block diagram of an alignment model training device provided by an exemplary embodiment of the present application;

[0072] Figure 7 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of the present application;

[0073] Figure 8 A schematic structural diagram of an electronic device provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0074] In order to enable ordinary persons in the art to better understand the technical solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0075] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0076] The acquisition and automatic labeling of songs has always been a bottleneck in the research and development of singing synthesis and related fields. There is very little singing data available for training, and it is usually recorded by a dedicated person, which is very costly. In addition, the labor and time costs of labeling singing data are very high, and the labeling efficiency is low.

[0077] There are many methods for acquiring and automatically annotating spoken speech, but they are not suitable for singing data. The phonemes in songs are not fixed in duration, have a wide range of fundamental frequencies, and are different from spoken speech, which is a problem for song annotation. Currently, song annotation is generally done manually, which is inefficient and costs a lot of money.

[0078] In response to the above-mentioned technical problems, in some embodiments of the present application, a song to be annotated is obtained, wherein the song to be annotated includes a singing audio and a lyric text, and the singing audio contains a human voice and an accompaniment sound that overlaps with the human voice; the human voice and the accompaniment sound in the singing audio are separated to obtain the human voice audio; according to the start timestamp of each lyric segment and the energy information of the human voice audio, the human voice audio is segmented at the sentence level to obtain at least one human voice audio segment, thereby improving the rationality of the segmentation of the human voice audio segment; the human voice audio segment and the corresponding lyric segment are input into a preset alignment model to obtain the annotation result corresponding to each human voice audio segment, and the alignment model is used to automatically annotate the human voice audio segment and the corresponding lyric segment, thereby improving the annotation efficiency and reducing the cost.

[0079] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0080] Figure 1 The following is a flowchart of a song audio tagging method provided by an exemplary embodiment of the present application. Figure 1 As shown, the method includes:

[0081] S101: Acquire song data to be annotated, wherein the song data to be annotated includes singing audio and lyrics text, the singing audio includes human voice and accompaniment sound overlapping with the human voice, the lyrics text corresponds to the content sung by the human voice in the singing audio, and the lyrics text includes a plurality of segmented lyrics segments, each of which includes a start timestamp;

[0082] S102: separating the human voice and the accompaniment sound in the singing audio to obtain the human voice audio;

[0083] S103: determining a segmentation point of the vocal audio according to the start timestamp of each lyric segment and the energy information of the vocal audio;

[0084] S104: Segment the vocal audio according to the segmentation points to obtain at least one vocal audio segment, wherein each vocal audio segment corresponds to a lyrics segment;

[0085] S105: Input each vocal audio segment and the corresponding lyrics segment into a preset alignment model to obtain a labeling result corresponding to each vocal audio segment, wherein each labeling result includes each phoneme in the vocal audio segment and time information corresponding to each phoneme.

[0086] In this embodiment, the execution subject of the above method includes but is not limited to any one of the following: a personal computer, a tablet computer, a laptop computer, a mobile phone, a smart bracelet, a smart TV, a smart watch and a smart speaker.

[0087] In this embodiment, a song to be labeled is obtained, wherein the song to be labeled includes singing audio and lyrics text, and the singing audio includes human voice and accompaniment sound overlapping with the human voice; the human voice and the accompaniment sound in the singing audio are separated to obtain human voice audio; according to the start timestamp of each lyrics segment and the energy information of the human voice audio, the human voice audio is segmented at the sentence level to obtain at least one human voice audio segment, thereby improving the rationality of the segmentation of the human voice audio segment; the human voice audio segment and the corresponding lyrics segment are input into a preset alignment model to obtain the labeling result corresponding to each human voice audio segment, and the alignment model is used to automatically label the human voice audio segment and the corresponding lyrics segment, thereby improving the labeling efficiency and reducing the cost.

[0088] It should be noted that the lyrics text of the song to be annotated includes a plurality of lyrics segments that are pre-segmented in units of sentences, and each lyrics segment includes a start timestamp.

[0089] The energy information of human voice audio is the energy generated by the vibration of human voice audio in the propagation medium.

[0090] The phonemes corresponding to the human voice audio clip include but are not limited to the following: the phoneme texts corresponding to the human voice audio clip, the initial consonant text and the final consonant text in the pinyin text corresponding to the human voice audio clip, and the phoneme audios corresponding to the human voice audio clip.

[0091] In this embodiment, the songs to be marked are obtained, including but not limited to the following ways:

[0092] Acquisition method 1: crawl multiple songs from the Internet, and select songs to be labeled from the multiple songs.

[0093] Acquisition method 2: Acquire the locally stored songs to be annotated.

[0094] Acquisition method three: respond to the song recording operation and obtain the recorded song to be marked.

[0095] In the above-mentioned acquisition method 1, an optional embodiment is to use a crawler tool to crawl multiple songs from the Internet, and randomly select a song from the multiple songs as the song to be marked; in another embodiment, a crawler tool is used to crawl multiple songs from the Internet, and a song that meets the set conditions is selected from the multiple songs as the song to be marked. In the embodiment of the present application, a crawler tool is used to crawl multiple songs from the Internet, and the acquisition of song data is fast and low in cost. It should be noted that the embodiment of the present application does not limit the setting conditions, and the setting conditions can be adjusted according to actual conditions. For example, the setting conditions include but are not limited to the following: setting a sequence number, the longest duration, the shortest duration, the highest song energy value and the highest song score.

[0096] In the above-mentioned second acquisition method, any song can be obtained from the locally stored song database as the song to be marked.

[0097] In the above-mentioned acquisition method three, a song recording application is used to record a song, and a song recording operation is responded to to obtain the recorded song to be marked.

[0098] In this embodiment, the human voice and the accompaniment sound in the singing audio are separated to obtain the human voice audio. One achievable way is to input the singing audio into an existing human voice separation model, separate the human voice audio and the accompaniment audio, and obtain the human voice audio. Another achievable way is to use a separation tool to separate the human voice audio and the accompaniment audio, and obtain the human voice audio.

[0099] In this embodiment, the vocal audio and lyrics are re-segmented according to the start timestamp of the lyrics segment and the energy information of the vocal audio in the lyrics to obtain at least one vocal audio segment and at least one vocal audio segment. One achievable manner is to determine the segmentation point of the vocal audio according to the start timestamp of each lyrics segment and the energy information of the vocal audio; and segment the vocal audio according to the segmentation point to obtain at least one vocal audio segment.

[0100] In the above embodiment, the start timestamp of each lyric segment and the energy information of the vocal audio are used to determine the segmentation point of the vocal audio; the vocal audio is segmented according to the segmentation point to obtain at least one vocal audio segment. One achievable method is that, for each lyric segment, audio data of a predetermined time length before and / or after the time node corresponding to the start timestamp is taken from the vocal audio as sampling data; the audio energy of each time node in each sampling data is calculated; for each sampling data, the time node with the smallest audio energy is used as the segmentation point of the current sampling data; the vocal audio is segmented using the segmentation point to obtain at least one vocal audio segment.

[0101] For example, according to the start timestamp of the lyrics segment in the lyrics, the human voice audio is segmented to obtain 10 audio segments; the audio data corresponding to the start timestamp is used as the sampling data according to the step length of 500ms; the energy value of each time node in each sampling data is calculated respectively; for each sampling data, the time node with the smallest audio energy is used as the segmentation point of the current sampling data, and the human voice audio is segmented to obtain 10 audio segments. It should be noted that the embodiment of the present application does not limit the setting of the audio step length, and the setting of the audio step length can be limited according to actual conditions.

[0102] After obtaining at least one vocal audio segment, the vocal audio segment and the corresponding lyrics segment are input into the preset alignment model to obtain the annotation results corresponding to each vocal audio segment, wherein each annotation result includes each phoneme in the vocal audio segment and the time information corresponding to each phoneme. Among them, the time information corresponding to each phoneme includes the start and end time corresponding to each phoneme. The alignment model outputs two categories of data: one part is the high-quality target vocal audio segment that can be aligned, and the other part is the low-quality vocal audio segment that cannot be aligned. The reasons for the failure to align may include: lyrics text errors, high noise, unclean accompaniment separation, chorus and duet, poor sound quality, etc.

[0103] In the above and following embodiments, the target vocal audio segment and the lyrics segment corresponding to the target vocal audio segment are used to train the existing alignment model to obtain an updated alignment model. Among them, the alignment model is iteratively trained using high-quality target vocal audio segments, making the alignment model more robust and more tolerant to noisy data and data with poor sound quality. At the same time, the data diversity of the entire alignment model is expanded, so that the alignment effect is improved.

[0104] In the above and following embodiments, after the vocal audio is segmented according to the segmentation points to obtain at least one vocal audio segment, for each lyric segment, a silent phoneme is added between every two adjacent syllables to obtain an updated lyric segment.

[0105] In the above and following embodiments, after obtaining the annotation results corresponding to the human voice audio segment, the fundamental frequency is extracted for each human voice audio segment; the annotation results are adjusted according to the extracted fundamental frequency and the preset fundamental frequency threshold. One achievable method is to divide each audio segment into multiple audio frames according to the preset time length; for each audio segment, the fundamental frequency corresponding to each audio frame is extracted in units of frames; according to the extracted fundamental frequency and the preset fundamental frequency threshold, it is determined whether each silent phoneme in the updated audio lyrics segment is a true silent phoneme; if it is a true silent phoneme, the time information corresponding to the silent phoneme is retained; if it is not a true silent phoneme, the time information corresponding to the silent phoneme is modified. It should be noted that the present application does not limit the preset time, and the preset time can be 1 millisecond, 2 milliseconds, 5 milliseconds, etc., and the preset time can be adjusted according to actual conditions. The present application does not limit the preset fundamental frequency threshold, and the preset fundamental frequency threshold can be 20 Hz, 50 Hz, 60 Hz, etc., and the preset fundamental frequency threshold can be adjusted according to actual conditions.

[0106] In the above embodiment, whether each silent phoneme in the updated audio lyrics segment is a true silent phoneme is determined based on the extracted fundamental frequency and the preset fundamental frequency threshold. One achievable method is that if N audio frames with fundamental frequencies greater than or equal to the preset fundamental frequency threshold appear continuously in the human voice audio segment corresponding to the time information of a silent phoneme, and the overall time length of the N audio frames is greater than the preset time threshold, then the silent phoneme is determined to be a true silent phoneme, where the value range of N is 2 or 3; otherwise, the silent phoneme is not a true silent phoneme. It should be noted that the present application does not limit the preset time threshold, and the preset time threshold can be 50 milliseconds, 100 milliseconds, 200 milliseconds, etc. The preset time threshold can be adjusted according to actual conditions.

[0107] In the above embodiment, if a silent phoneme is not a true silent phoneme, the time information corresponding to the silent phoneme is modified according to the following steps. One achievable method is that if there are no N audio frames with a fundamental frequency greater than or equal to a preset fundamental frequency threshold value appearing continuously in the human voice audio segment corresponding to the time information of the silent phoneme, the silent phoneme is deleted in the annotation result, and the duration of the silent phoneme is divided into the previous phoneme of the current silent phoneme; if there are N audio frames with a fundamental frequency greater than or equal to the preset fundamental frequency threshold value appearing continuously in the human voice audio segment corresponding to the time information of the silent phoneme, and the overall time length of the N audio frames is not greater than the preset time threshold value, the silent phoneme is deleted in the annotation result, and the duration of the silent phoneme is divided into the next phoneme of the current silent phoneme.

[0108] For example, for the first audio segment, multiple audio frames in the first audio segment are obtained, the fundamental frequency corresponding to each audio frame is extracted in units of frames, and the number of continuous audio frames whose fundamental frequency corresponding to the audio frame is less than 60Hz is counted. If the number of continuous frames is less than three frames, it is not true silence, and the silent phoneme is deleted in the annotation result, and the duration of the silent phoneme is divided into the previous phoneme of the current silent phoneme. If the number of continuous audio frames is greater than or equal to three frames, the overall time length of multiple audio frames is continued to be determined. If the overall time length of multiple audio frames is less than 100ms, it is not true silence, and the silent phoneme is deleted in the annotation result, and the duration of the silent phoneme is divided into the next phoneme of the current silent phoneme; if the duration of the first silent phoneme is greater than 100ms, it is considered that the segment is a true silent phoneme, and the first silent phoneme is retained.

[0109] In the above embodiment, the corrected annotation results corresponding to each vocal audio segment can be used for model training to obtain a model with better quality. For example, the corrected annotation results corresponding to each vocal audio segment can be used for model training to obtain a speech synthesis model.

[0110] In the above and the following embodiments, before using the alignment model, it is necessary to train the alignment model using training sample data. Figure 2 A flowchart of an alignment model training method provided by an exemplary embodiment of the present application. Figure 2 As shown, the alignment model training method includes:

[0111] S201: Collecting audio sample data and text sample data corresponding to the audio sample data;

[0112] S202: segmenting the audio sample data and the text sample data to obtain a first audio segment sample contained in the audio sample data and a first text segment sample corresponding to the first audio segment sample;

[0113] S203: Filter the first audio segment sample and the first text segment sample to obtain a second audio segment sample and a second text segment sample for model training;

[0114] S204: marking the second audio segment sample to obtain a phoneme sample contained in the second audio segment sample and time information corresponding to the phoneme sample;

[0115] S205: Train the initial alignment model using the second audio segment sample, the second text segment sample, the phoneme sample included in the second audio segment sample, and the time information corresponding to the phoneme sample to obtain a trained alignment model.

[0116] In this embodiment, a large amount of audio sample data is first collected to cover different types of audio that may be encountered in various audio processing scenarios as much as possible to improve sample coverage. For example, the audio sample data can be audio of normal speaking intonation or audio of singing intonation.

[0117] In this embodiment, the audio sample data and the text sample data are segmented to obtain a first audio segment sample contained in the audio sample data and a first text segment sample corresponding to the first audio segment sample.

[0118] In this embodiment, the first audio segment sample and the first text segment sample are filtered to obtain the second audio segment sample and the second text segment sample for model training. The filtering methods include but are not limited to the following:

[0119] Filtering method 1, removing the audio segment sample of a set language and the corresponding text segment sample from the first audio segment sample and the first text segment sample to obtain a second audio segment sample and a second text segment sample;

[0120] Filtering method 2: According to the fundamental frequency information of the first audio segment sample, the audio segment samples and the corresponding text segment samples whose fundamental frequency information does not meet the fundamental frequency information condition are removed from the first audio segment sample and the first text segment sample to obtain the second audio segment sample and the second text segment sample.

[0121] In the above-mentioned filtering method one, for example, if only Chinese human voice audio clips are needed in actual situations, the English audio clips in the first audio clip sample are filtered, and the English audio clip samples and the corresponding text clip samples are removed to obtain the second audio clip sample and the second text clip sample.

[0122] In the above filtering method 2, the fundamental frequency information includes but is not limited to the following: the proportion of low frequencies in the fundamental frequency of the non-silent segment, the proportion of high frequencies in the fundamental frequency of the non-silent segment, and the proportion of frames with zero fundamental frequency to the total number of frames, the fundamental frequency mean, fundamental frequency variance, and fundamental frequency standard deviation. For example, the proportion of low frequencies in the fundamental frequency of the non-silent segment is less than or equal to 0.04%, the proportion of high frequencies in the fundamental frequency of the non-silent segment is less than or equal to 0.04%, the proportion of frames with zero fundamental frequency to the total number of frames is less than or equal to 0.3%, the fundamental frequency mean is greater than or equal to 150, the fundamental frequency variance is greater than or equal to 10 and less than 100, and the fundamental frequency standard deviation is less than or equal to 500. According to actual conditions, the audio segment samples in the first audio segment sample that do not meet at least one of the above fundamental frequency information are filtered to obtain a second audio segment sample and a second text segment sample. It should be noted that the embodiment of the present application does not limit the fundamental frequency information conditions, and can be adjusted according to actual conditions.

[0123] In this embodiment, the second audio segment sample is labeled to obtain the phoneme sample contained in the second audio segment sample and the time information corresponding to the phoneme sample; the second audio segment sample, the second text segment sample, the phoneme sample contained in the second audio segment sample and the time information corresponding to the phoneme sample are input into the initial alignment model. Inside the initial alignment model, the training sample is calculated according to the model parameters, and the labeling result of the audio segment sample is output by the output layer of the initial alignment model. Then, the loss function layer of the initial alignment model can calculate the loss function according to the difference between the labeling result of the audio segment sample of the output layer and the actual labeling result on the training sample. If the loss function does not meet the set requirements, the model parameters can be adjusted to continue iterative training. When the loss function of the initial alignment model meets the set requirements, a trained alignment model can be obtained.

[0124] Figure 3 The following is a flow chart of another song audio tagging method provided by an exemplary embodiment of the present application. Figure 3 As shown, the method includes:

[0125] S301: crawling songs to be annotated from the Internet, wherein the song data to be annotated includes singing audio and lyrics text, the singing audio includes human voice and accompaniment sound overlapping with the human voice, the lyrics text corresponds to the content sung by the human voice in the singing audio, and the lyrics text includes multiple segmented lyrics fragments, each of which includes a start timestamp;

[0126] S302: separating the human voice and the accompaniment sound in the singing audio of the song to be marked to obtain the human voice audio;

[0127] S303: Determine a segmentation point of the vocal audio according to the start timestamp of each lyric segment and the energy information of the vocal audio;

[0128] S304: Segment the vocal audio according to the segmentation points to obtain at least one vocal audio segment, wherein each vocal audio segment corresponds to a lyrics segment;

[0129] S305: Input each vocal audio segment and the corresponding lyrics segment into a preset alignment model to obtain a labeling result corresponding to each vocal audio segment, wherein each labeling result includes each phoneme in the vocal audio segment and time information corresponding to each phoneme.

[0130] In this embodiment, the implementation of each execution step of the above method can refer to the description of the corresponding parts of the above embodiments, which will not be repeated here.

[0131] Figure 4 The following is a flow chart of another song audio tagging method provided by an exemplary embodiment of the present application. Figure 4 As shown, the method includes:

[0132] S401: crawling songs to be annotated from the Internet, wherein the song data to be annotated includes singing audio and lyrics text, the singing audio includes human voice and accompaniment overlapping with the human voice, the lyrics text corresponds to the content sung by the human voice in the singing audio, and the lyrics text includes a plurality of segmented lyrics fragments, each of which includes a start timestamp;

[0133] S402: separating the human voice and the accompaniment sound in the singing audio of the song to be marked to obtain the human voice audio;

[0134] S403: segmenting the vocal audio according to the start timestamp of each lyric segment and the energy information of the vocal audio to obtain at least one vocal audio segment, wherein each vocal audio segment corresponds to a lyric segment;

[0135] S404: Input each vocal audio segment and the corresponding lyrics segment into a preset alignment model to obtain a labeling result corresponding to each vocal audio segment; wherein each labeling result includes each phoneme in the vocal audio segment and time information corresponding to each phoneme.

[0136] In the above method embodiment of the present application, a song to be annotated is obtained, wherein the song to be annotated includes singing audio and lyrics text, and the singing audio includes human voice and accompaniment sound overlapping with the human voice; the human voice and the accompaniment sound in the singing audio are separated to obtain human voice audio; according to the start timestamp of each lyrics segment and the energy information of the human voice audio, the human voice audio is segmented at the sentence level to obtain at least one human voice audio segment, thereby improving the rationality of the segmentation of the human voice audio segment; the human voice audio segment and the corresponding lyrics segment are input into a preset alignment model to obtain the annotation result corresponding to each human voice audio segment, and the alignment model is used to automatically annotate the human voice audio segment and the corresponding lyrics segment, thereby improving the annotation efficiency and reducing the cost.

[0137] Figure 5 is a structural block diagram of a song audio tagging device 50 provided by an exemplary embodiment of the present application. Figure 5 The device includes an acquisition module 51, a separation module 52, a determination module 53, a segmentation module 54 and an alignment model module 55.

[0138] The acquisition module 51 is used to acquire the song data to be annotated, wherein the song data to be annotated includes singing audio and lyrics text, the singing audio includes human voice and accompaniment sound overlapping with the human voice, the lyrics text corresponds to the content sung by the human voice in the singing audio, and the lyrics text includes a plurality of segmented lyrics segments, each of which includes a start timestamp;

[0139] A separation module 52 is used to separate the human voice and the accompaniment sound in the singing audio to obtain the human voice audio;

[0140] A determination module 53 determines a segmentation point of the vocal audio according to a start timestamp of each lyric segment and energy information of the vocal audio;

[0141] A segmentation module 54 segments the vocal audio according to the segmentation points to obtain at least one vocal audio segment, wherein each vocal audio segment corresponds to a lyrics segment;

[0142] The alignment model module 55 inputs each vocal audio segment and the corresponding lyrics segment into a preset alignment model to obtain a labeling result corresponding to each vocal audio segment, wherein each labeling result includes each phoneme in the vocal audio segment and the time information corresponding to each phoneme.

[0143] Optionally, when determining the segmentation point of the vocal audio based on the start timestamp of each lyric segment and the energy information of the vocal audio, the determination module 53 is used to: for each lyric segment, take audio data of a predetermined time length before and / or after the time node corresponding to the start timestamp in the vocal audio as sampling data; calculate the audio energy of each time node in each sampling data; for each sampling data, take the time node with the smallest audio energy as the segmentation point of the current sampling data.

[0144] Optionally, each lyric segment includes multiple syllables; after the segmentation module 54 segments the vocal audio according to the segmentation points to obtain at least one vocal audio segment, it is also used to: for each lyric segment, add a silent phoneme between every two adjacent syllables to obtain an updated lyric segment.

[0145] Optionally, after obtaining the annotation results corresponding to each vocal audio segment, the alignment model module 55 is further used to: extract the fundamental frequency of each vocal audio segment; and adjust the annotation results according to the extracted fundamental frequency and a preset fundamental frequency threshold.

[0146] Optionally, when adjusting the annotation result according to the extracted fundamental frequency and the preset fundamental frequency threshold, the alignment model module 55 is used to: determine whether each silent phoneme in the updated audio lyrics segment is a true silent phoneme according to the extracted fundamental frequency and the preset fundamental frequency threshold; if it is a true silent phoneme, retain the time information corresponding to the silent phoneme; if it is not a true silent phoneme, modify the time information corresponding to the silent phoneme.

[0147] Optionally, the alignment model module 55 extracts the fundamental frequency for each vocal audio segment for: dividing each audio segment into a plurality of audio frames according to a preset time length; and for each audio segment, extracting the fundamental frequency corresponding to each audio frame in units of frames.

[0148] Optionally, when the alignment model module 55 determines whether each silent phoneme in the updated audio lyrics segment is a true silent phoneme based on the extracted fundamental frequency and the preset fundamental frequency threshold, it is used to: if N audio frames with fundamental frequencies greater than or equal to the preset fundamental frequency threshold appear continuously in the human voice audio segment corresponding to the time information of a silent phoneme, and the overall time length of the N audio frames is greater than the preset time threshold, then the silent phoneme is determined to be a true silent phoneme, where the value range of N is 2 or 3.

[0149] Optionally, if a silent phoneme is not a true silent phoneme, the alignment model module 55 modifies the time information corresponding to the silent phoneme according to the following steps: if there are no consecutive N audio frames with a fundamental frequency greater than or equal to a preset fundamental frequency threshold in the vocal audio segment corresponding to the time information of the silent phoneme, the silent phoneme is deleted in the annotation result, and the duration of the silent phoneme is divided into the previous phoneme of the current silent phoneme; if N audio frames with a fundamental frequency greater than or equal to the preset fundamental frequency threshold appear consecutively in the vocal audio segment corresponding to the time information of the silent phoneme, and the overall time length of the N audio frames is not greater than the preset time threshold, the silent phoneme is deleted in the annotation result, and the duration of the silent phoneme is divided into the next phoneme of the current silent phoneme.

[0150] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0151] Figure 6 60 is a structural block diagram of an alignment model training device 60 provided by an exemplary embodiment of the present application. Figure 6 The alignment model training device includes a collection module 61, a segmentation module 62, a filtering module 63, a labeling module 64 and a training module 65.

[0152] Wherein, the collecting module 61 is used to collect audio sample data and text sample data corresponding to the audio sample data;

[0153] A segmentation module 62 is used to segment the audio sample data and the text sample data to obtain a first audio segment sample contained in the audio sample data and a first text segment sample corresponding to the first audio segment sample;

[0154] A filtering module 63, used for filtering the first audio segment sample and the first text segment sample to obtain a second audio segment sample and a second text segment sample for model training;

[0155] a labeling module 64, configured to label the second audio segment sample to obtain a phoneme sample contained in the second audio segment sample and time information corresponding to the phoneme sample;

[0156] The training module 65 is used to train the initial alignment model using the second audio segment sample, the second text segment sample, the phoneme sample contained in the second audio segment sample and the time information corresponding to the phoneme sample to obtain a trained alignment model.

[0157] Optionally, when filtering the first audio segment sample and the first text segment sample to obtain the second audio segment sample and the second text segment sample for model training, the filtering module 63 is used to:

[0158] removing the audio segment sample of a set language and the corresponding text segment sample from the first audio segment sample and the first text segment sample to obtain a second audio segment sample and a second text segment sample;

[0159] or,

[0160] According to the fundamental frequency information of the first audio segment sample, the audio segment samples whose fundamental frequency information does not meet the fundamental frequency information condition and the corresponding text segment samples are removed from the first audio segment sample and the first text segment sample to obtain the second audio segment sample and the second text segment sample.

[0161] Figure 7 The following is a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application. Figure 7 As shown, the electronic device includes: a memory 701 and a processor 702. In addition, the electronic device also includes necessary components such as a power supply component 703, a communication component 704 and a display screen 705.

[0162] The memory 701 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application program or method used to operate on the electronic device.

[0163] Memory 701 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0164] The communication component 704 is used for data transmission with other devices.

[0165] The processor 702 can execute computer instructions stored in the memory 701 to: obtain song data to be annotated, wherein the song data to be annotated includes singing audio and lyrics text, the singing audio includes human voice and accompaniment sounds overlapping with the human voice, the lyrics text corresponds to the content sung by the human voice in the singing audio, and the lyrics text includes multiple segmented lyrics fragments, each of which includes a start timestamp; separate the human voice and accompaniment sounds in the singing audio to obtain human voice audio; determine the segmentation point of the human voice audio according to the start timestamp of each lyrics fragment and the energy information of the human voice audio; segment the human voice audio according to the segmentation point to obtain at least one human voice audio fragment, wherein each human voice audio fragment corresponds to a lyrics fragment; input each human voice audio fragment and the corresponding lyrics fragment into a preset alignment model to obtain a labeling result corresponding to each human voice audio fragment, wherein each labeling result includes each phoneme in the human voice audio fragment and the time information corresponding to each phoneme.

[0166] Optionally, when determining the segmentation point of the vocal audio according to the start timestamp of each lyric segment and the energy information of the vocal audio, the processor 702 is configured to:

[0167] For each lyric segment, audio data of a predetermined time length before and / or after a time node corresponding to the start timestamp is taken from the vocal audio as sampling data;

[0168] Calculate the audio energy at each time node in each sampled data;

[0169] For each sampled data, the time node with the minimum audio energy is used as the segmentation point of the current sampled data.

[0170] Optionally, each lyric segment includes a plurality of syllables;

[0171] After the processor 702 divides the vocal audio according to the division point to obtain at least one vocal audio segment, it can also be used to: for each lyric segment, add a silent phoneme between every two adjacent syllables to obtain an updated lyric segment.

[0172] Optionally, after obtaining the labeling result corresponding to each vocal audio segment, the processor 702 is further configured to:

[0173] Extract the fundamental frequency for each vocal audio clip;

[0174] The annotation results are adjusted according to the extracted fundamental frequency and the preset fundamental frequency threshold.

[0175] Optionally, when adjusting the labeling result according to the extracted fundamental frequency and the preset fundamental frequency threshold, the processor 702 is configured to:

[0176] According to the extracted fundamental frequency and the preset fundamental frequency threshold, determining whether each silent phoneme in the updated audio lyrics segment is a true silent phoneme;

[0177] If it is a true silent phoneme, the time information corresponding to the silent phoneme is retained;

[0178] If it is not a true silent phoneme, the time information corresponding to the silent phoneme is modified.

[0179] Optionally, the processor 702 extracts the base frequency of each vocal audio segment for:

[0180] Divide each audio clip into multiple audio frames according to a preset time length;

[0181] For each audio segment, the fundamental frequency corresponding to each audio frame is extracted in units of frames.

[0182] Optionally, when the processor 702 determines whether each silent phoneme in the updated audio lyrics segment is a true silent phoneme according to the extracted fundamental frequency and the preset fundamental frequency threshold, it is configured to:

[0183] If N audio frames with fundamental frequencies greater than or equal to a preset fundamental frequency threshold appear continuously in the human voice audio segment corresponding to the time information of a silent phoneme, and the overall time length of the N audio frames is greater than the preset time threshold, the silent phoneme is determined to be a true silent phoneme, where the value range of N is 2 or 3.

[0184] Optionally, if a silent phoneme is not a true silent phoneme, the processor 702 modifies the time information corresponding to the silent phoneme according to the following steps:

[0185] If there are no N consecutive audio frames whose fundamental frequency is greater than or equal to the preset fundamental frequency threshold in the human voice audio segment corresponding to the time information of the silent phoneme, the silent phoneme is deleted from the annotation result, and the duration of the silent phoneme is divided into the previous phoneme of the current silent phoneme;

[0186] If N audio frames whose fundamental frequencies are greater than or equal to a preset fundamental frequency threshold appear continuously in the human voice audio segment corresponding to the time information of the silent phoneme, and the overall time length of the N audio frames is not greater than the preset time threshold, the silent phoneme is deleted from the annotation result, and the duration of the silent phoneme is divided into the next phoneme of the current silent phoneme.

[0187] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program. When the computer-readable storage medium stores the computer program and the computer program is executed by one or more processors, the one or more processors execute Figure 1 Each step in the method embodiment.

[0188] Accordingly, the embodiment of the present application also provides a computer program product, the computer program product includes a computer program / instruction, and the computer program / instruction is executed by a processor. Figure 1 Each step in the method embodiment.

[0189] Figure 8 The following is a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application. Figure 8 As shown, the electronic device includes: a memory 801 and a processor 802. In addition, the electronic device also includes necessary components such as a power supply component 803 and a communication component 804.

[0190] The memory 801 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application program or method used to operate on the electronic device.

[0191] Memory 801 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0192] The communication component 804 is used for data transmission with other devices.

[0193] The processor 802 can execute computer instructions stored in the memory 801 to: collect audio sample data and text sample data corresponding to the audio sample data; segment the audio sample data and the text sample data to obtain a first audio segment sample contained in the audio sample data and a first text segment sample corresponding to the first audio segment sample; filter the first audio segment sample and the first text segment sample to obtain a second audio segment sample and a second text segment sample for model training; annotate the second audio segment sample to obtain a phoneme sample contained in the second audio segment sample and time information corresponding to the phoneme sample; train the initial alignment model using the second audio segment sample, the second text segment sample, the phoneme sample contained in the second audio segment sample and the time information corresponding to the phoneme sample to obtain a trained alignment model.

[0194] Optionally, when the processor 802 filters the first audio segment sample and the first text segment sample to obtain the second audio segment sample and the second text segment sample for model training, it is used to: remove the audio segment sample of a set language and the corresponding text segment sample from the first audio segment sample and the first text segment sample to obtain the second audio segment sample and the second text segment sample; or, based on the fundamental frequency information of the first audio segment sample, remove the audio segment sample and the corresponding text segment sample whose fundamental frequency information does not meet the fundamental frequency information condition from the first audio segment sample and the first text segment sample to obtain the second audio segment sample and the second text segment sample.

[0195] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program. When the computer-readable storage medium stores the computer program and the computer program is executed by one or more processors, the one or more processors execute Figure 2 Each step in the method embodiment.

[0196] Accordingly, the embodiment of the present application also provides a computer program product, the computer program product includes a computer program / instruction, and the computer program / instruction is executed by a processor. Figure 2 Each step in the method embodiment.

[0197] Above Figure 7 and Figure 8 The communication component in is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0198] Above Figure 7 and Figure 8 The power supply component in the device provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0199] Above Figure 7The display screen in the embodiment includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0200] The electronic device mentioned above also includes an audio component.

[0201] The audio component may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal may be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0202] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0203] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0204] In the above-mentioned apparatus, device, storage medium and program product embodiments of the present application, a song to be labeled is obtained, wherein the song to be labeled includes singing audio and lyrics text, and the singing audio contains human voice and accompaniment sound overlapping with the human voice; the human voice and the accompaniment sound in the singing audio are separated to obtain human voice audio; according to the start timestamp of each lyrics segment and the energy information of the human voice audio, the human voice audio is segmented at the sentence level to obtain at least one human voice audio segment, thereby improving the rationality of the segmentation of the human voice audio segment; the human voice audio segment and the corresponding lyrics segment are input into a preset alignment model to obtain the labeling result corresponding to each human voice audio segment, and the alignment model is used to automatically label the human voice audio segment and the corresponding lyrics segment, thereby improving the labeling efficiency and reducing the cost.

[0205] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0206] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0207] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0208] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0209] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0210] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0211] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A song audio tagging method, characterized in that: include: Acquire song data to be annotated, wherein the song data to be annotated includes singing audio and lyrics text, the singing audio includes human voice and accompaniment sound overlapping with the human voice, the lyrics text corresponds to the content sung by the human voice in the singing audio, the lyrics text includes a plurality of segmented lyrics segments, and each of the lyrics segments includes a start timestamp; Separating the human voice and the accompaniment sound in the singing audio to obtain human voice audio; Determine the segmentation point of the vocal audio according to the start timestamp of each of the lyrics segments and the energy information of the vocal audio; Segmenting the vocal audio according to the segmentation points to obtain at least one vocal audio segment, wherein each vocal audio segment corresponds to one lyric segment; Each of the vocal audio clips and the corresponding lyrics clips are input into a preset alignment model to obtain a labeling result corresponding to each of the vocal audio clips, wherein each of the labeling results includes each phoneme in the vocal audio clip and the time information corresponding to each phoneme.

2. The method according to claim 1, characterized in that The step of determining the segmentation point of the vocal audio according to the start timestamp of each of the lyrics segments and the energy information of the vocal audio comprises: For each of the lyrics segments, taking audio data of a predetermined time length before and / or after the time node corresponding to the start timestamp from the vocal audio as sampling data; Calculating the audio energy at each time node in each of the sampled data; For each of the sampled data, the time node with the minimum audio energy is used as the segmentation point of the current sampled data.

3. The method according to claim 1, characterized in that Each of the lyrics segments includes a plurality of syllables; After the human voice audio is segmented according to the segmentation points to obtain at least one human voice audio segment, the method further includes: for each of the lyrics segments, a silent phoneme is added between every two adjacent syllables to obtain an updated lyrics segment.

4. The method according to claim 3, characterized in that After obtaining the labeling result corresponding to each of the human voice audio segments, the method further includes: Extracting the fundamental frequency of each of the human voice audio segments; The labeling result is adjusted according to the extracted fundamental frequency and a preset fundamental frequency threshold.

5. The method according to claim 4, characterized in that The step of adjusting the labeling result according to the extracted fundamental frequency and the preset fundamental frequency threshold comprises: According to the extracted fundamental frequency and the preset fundamental frequency threshold, determining whether each silent phoneme in the updated audio lyrics segment is a true silent phoneme; If it is a true silent phoneme, retaining the time information corresponding to the silent phoneme; If it is not a true silent phoneme, the time information corresponding to the silent phoneme is modified.

6. The method according to claim 5, characterized in that Extracting the fundamental frequency of each of the human voice audio segments comprises: Dividing each of the audio segments into a plurality of audio frames according to a preset time length; For each of the audio segments, a fundamental frequency corresponding to each audio frame is extracted in units of frames.

7. The method according to claim 6, characterized in that The step of judging whether each silent phoneme in the updated audio lyrics segment is a true silent phoneme according to the extracted fundamental frequency and the preset fundamental frequency threshold comprises: If N audio frames whose fundamental frequency is greater than or equal to a preset fundamental frequency threshold appear continuously in the human voice audio segment corresponding to the time information of a silent phoneme, and the overall time length of the N audio frames is greater than the preset time threshold, then the silent phoneme is determined to be a true silent phoneme, where the value range of N is 2 or 3.

8. The method according to claim 7, characterized in that If a silent phoneme is not a true silent phoneme, the time information corresponding to the silent phoneme is modified according to the following steps: If there are no N consecutive audio frames whose fundamental frequencies are greater than or equal to a preset fundamental frequency threshold in the human voice audio segment corresponding to the time information of the silent phoneme, the silent phoneme is deleted from the annotation result, and the duration of the silent phoneme is divided into the previous phoneme of the current silent phoneme; If N audio frames whose fundamental frequencies are greater than or equal to a preset fundamental frequency threshold appear continuously in the human voice audio segment corresponding to the time information of the silent phoneme, and the overall time length of the N audio frames is not greater than the preset time threshold, the silent phoneme is deleted from the annotation result, and the duration of the silent phoneme is divided into the next phoneme of the current silent phoneme.

9. The method according to claim 1, characterized in that: The training method of the alignment model includes: Collecting audio sample data and text sample data corresponding to the audio sample data; Segmenting the audio sample data and the text sample data to obtain a first audio segment sample contained in the audio sample data and a first text segment sample corresponding to the first audio segment sample; Filtering the first audio segment sample and the first text segment sample to obtain a second audio segment sample and a second text segment sample for model training; Annotating the second audio segment sample to obtain a phoneme sample contained in the second audio segment sample and time information corresponding to the phoneme sample; The initial alignment model is trained using the second audio segment sample, the second text segment sample, the phoneme sample included in the second audio segment sample, and time information corresponding to the phoneme sample to obtain a trained alignment model.

10. The method according to claim 9, characterized in that Filtering the first audio segment sample and the first text segment sample to obtain a second audio segment sample and a second text segment sample for model training includes: removing the audio segment sample of a set language and the corresponding text segment sample from the first audio segment sample and the first text segment sample to obtain the second audio segment sample and the second text segment sample; or, According to the fundamental frequency information of the first audio segment sample, the audio segment samples whose fundamental frequency information does not meet the fundamental frequency information condition and the corresponding text segment samples are removed from the first audio segment samples and the first text segment samples to obtain the second audio segment samples and the second text segment samples.

11. A song audio tagging device, characterized in that: include: An acquisition module is used to acquire song data to be annotated, wherein the song data to be annotated includes singing audio and lyrics text, the singing audio includes human voice and accompaniment sound overlapping with the human voice, the lyrics text corresponds to the content sung by the human voice in the singing audio, and the lyrics text includes a plurality of segmented lyrics fragments, each of which includes a start timestamp; A separation module, used for separating the human voice and the accompaniment sound in the singing audio to obtain the human voice audio; A determination module, which determines the segmentation point of the vocal audio according to the start timestamp of each of the lyrics segments and the energy information of the vocal audio; A segmentation module, which segments the vocal audio according to the segmentation points to obtain at least one vocal audio segment, wherein each vocal audio segment corresponds to one lyric segment; The alignment model module inputs each of the vocal audio segments and the corresponding lyrics segments into a preset alignment model to obtain a labeling result corresponding to each of the vocal audio segments, wherein each of the labeling results includes each phoneme in the vocal audio segment and the time information corresponding to each phoneme.

12. An electronic device, characterized in that: include: Memory and processor; The memory is used to store computer programs; The processor is used to execute the computer program to implement each step in the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps in the method according to any one of claims 1 to 10 are implemented.