Method and device for generating rhythmic annotation data

By extracting audio and text from videos, and using audio text alignment model and pronunciation recognition rules to generate pronunciation labeling data, the problem of high construction cost of pronunciation labeling data sets in the existing technology is solved, and low-cost and efficient pronunciation is achieved Label dataset generation.

CN117012178BActive Publication Date: 2024-11-01ALIPAY (HANGZHOU) INFORMATION TECH CO LTD
View PDF 15 Cites 0 Cited by

Patent Information

Application Number
CN202310954803.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-07-31
Publication Date
2024-11-01
Estimated Expiration
2043-07-31

AI Technical Summary

Technical Problem

In the prior art, the construction of prosthetic annotation data set is expensive and relies on manual annotation, and there is a lack of low-cost solutions to generate high-quality prosthetic annotation data.

Method used

By extracting audio and text from the target video, the pre-trained audio text alignment model is used to perform word-level alignment processing, audio features are extracted, and each word is labeled according to the preset rhythm recognition rules to generate rhythm label data.

Benefits of technology

It realizes the construction of massive rhythmic annotation data sets at low cost, reduces the dependence on manual annotation, improves the labeling efficiency and accuracy, and is suitable for processing Internet-shanghai video data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117012178B_ABST
    Figure CN117012178B_ABST
Patent Text Reader

Abstract

The embodiments of the present specification provide a method and device for generating prosody annotation data, the method comprising: extracting a target audio from a target video, and extracting a target text corresponding to the target audio; using a pre-trained audio-text alignment model to perform word-level alignment processing on the target audio and the target text to obtain an alignment result; based on the alignment result, performing feature extraction processing on the target audio to obtain audio features corresponding to each word in the target text; using a preset prosody recognition rule associated with multiple prosody tags to determine the prosody tags corresponding to each word based on the audio features; marking each word in the target text with the recognized prosody tag, and using the marked target text as prosody annotation data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of computer technology, and in particular, to a method and device for generating prosody annotation data. Background Art

[0002] Real people's pronunciation has rhythmic characteristics, that is, real people's speech has pause changes and rhythm. At present, in order to simulate the rhythm of real people's speech and make the synthesized speech sound more realistic, rhythm prediction technology is applied in speech synthesis. Prosody prediction technology can improve the quality of speech synthesis and make the synthesized speech more fluent and natural. In the existing prosody prediction technology, a prosody prediction model is generally used for prosody prediction. Among them, the prosody prediction model usually needs to be trained based on a large amount of prosody annotated data corpus.

[0003] There is an urgent need for a reasonable and reliable solution to construct a prosody annotation dataset at low cost. Summary of the invention

[0004] The embodiments of this specification provide a prosody annotation data generation solution, which can construct a prosody annotation dataset at a low cost.

[0005] In a first aspect, an embodiment of the present specification provides a method for generating prosody annotation data, comprising: extracting a target audio from a target video, and extracting a target text corresponding to the target audio; using a pre-trained audio-text alignment model to perform word-level alignment processing on the target audio and the target text to obtain an alignment result; based on the alignment result, performing feature extraction processing on the target audio to obtain audio features corresponding to each word in the target text; using a preset prosody recognition rule associated with multiple prosody tags to determine the prosody tags corresponding to each word based on the audio features; marking the identified prosody tags for each word in the target text, and using the marked target text as prosody annotation data.

[0006] In some embodiments, extracting the target text corresponding to the target audio includes: separating the target video into an audio file and a video file; extracting subtitles from the video file; and generating the target text based on the subtitles.

[0007] In some embodiments, extracting subtitles from the video file includes: extracting subtitles from the video file using an optical character recognition algorithm.

[0008] In some embodiments, the feature extraction processing of the target audio includes: extracting the pronunciation duration features of each of the characters.

[0009] In some embodiments, the alignment result includes the start timestamp and the end timestamp of the audio corresponding to each of the characters; and the extraction of the pronunciation duration features of each of the characters includes: determining the pronunciation duration features of each of the characters based on the start timestamp and the end timestamp.

[0010] In some embodiments, the feature extraction processing of the target audio further includes: extracting the pitch features and energy features of the audio corresponding to each of the characters.

[0011] In some embodiments, extracting the pitch characteristics and energy characteristics of the audio corresponding to each of the characters includes: using a wavelet prosody analysis toolkit to extract the pitch characteristics and energy characteristics of the audio corresponding to each of the characters.

[0012] In some embodiments, the feature extraction processing of the target audio further includes: extracting significant features of the audio corresponding to each of the characters.

[0013] In some embodiments, extracting the significant features of the audio corresponding to each of the characters includes: interpolating and normalizing blank segments between adjacent characters in the target audio; combining the normalized blank segments by addition or multiplication; performing continuous wavelet transform on the processed target audio, and determining the significant features of the audio corresponding to each of the characters based on the transform results.

[0014] In some embodiments, the prosody recognition rule includes multiple conditions corresponding to the multiple prosody tags; and the use of preset prosody recognition rules associated with the multiple prosody tags to determine the prosody tags corresponding to the respective characters based on the audio features includes: for each of the respective characters, determining a target condition satisfied by the audio features corresponding to the character among the multiple conditions, and determining the prosody tag corresponding to the target condition as the prosody tag corresponding to the character.

[0015] In some embodiments, a single prosodic tag is used to represent any of: a non-prosodic boundary, a prosodic word boundary, a prosodic phrase boundary, an intonation phrase boundary.

[0016] In some embodiments, the target video comprises a video downloaded from a public channel.

[0017] In some embodiments, the target video includes any one of the following: a short video, a long video, or a live video.

[0018] In some embodiments, the target video belongs to a target business field, and the rhythm recognition rule is configured for the target business field.

[0019] In a second aspect, an embodiment of the present specification provides a prosody annotation data generation device, comprising: an audio-text extraction unit, configured to extract a target audio from a target video, and to extract a target text corresponding to the target audio; an audio-text alignment unit, configured to use a pre-trained audio-text alignment model to perform word-level alignment processing on the target audio and the target text to obtain an alignment result; a feature extraction unit, configured to perform feature extraction processing on the target audio based on the alignment result, thereby obtaining audio features corresponding to each word in the target text; a prosody recognition unit, configured to use a preset prosody recognition rule associated with a plurality of prosody tags to determine the prosody tags corresponding to each word based on the audio features; and a prosody annotation unit, configured to annotate the recognized prosody tags for each word in the target text, and use the annotated target text as prosody annotation data.

[0020] In a third aspect, an embodiment of the present specification provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, the computer is caused to execute the method described in any implementation manner in the first aspect.

[0021] In a fourth aspect, an embodiment of the present specification provides a computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in any implementation manner in the first aspect is implemented.

[0022] In a fifth aspect, an embodiment of the present specification provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the method described in any implementation manner in the first aspect.

[0023] The solution provided by the above-mentioned embodiments of this specification extracts the target audio from the target video, and extracts the target text corresponding to the target audio, and then uses the pre-trained audio-text alignment model to perform word-level alignment processing on the target audio and the target text to obtain an alignment result, and performs feature extraction processing on the target audio based on the alignment result, thereby obtaining the audio features corresponding to each word in the target text, so as to use the preset prosody recognition rules associated with multiple prosody tags, determine the prosody tags corresponding to each word based on the audio features, and mark the recognized prosody tags for each word in the target text, and use the marked target text as prosody annotation data. This solution can be used to process a large amount of video data on the Internet, thereby generating a large amount of prosody annotation data, thereby realizing the low-cost construction of prosody annotation datasets without relying on manual annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the multiple embodiments disclosed in this specification, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only the multiple embodiments disclosed in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 It is a schematic diagram of an application scenario in which the embodiments of this specification can be applied;

[0026] Figure 2 is a flow chart of a method for generating prosody annotation data in an embodiment of this specification;

[0027] Figure 3 It is a flow chart of the feature extraction process;

[0028] Figure 4 is a flow chart of a method for generating prosody annotation data in an embodiment of this specification;

[0029] Figure 5 It is a structural diagram of a prosody annotation data generating device in an embodiment of this specification. DETAILED DESCRIPTION

[0030] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. The described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of this application.

[0031] It should be noted that, for the convenience of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments in this specification can be combined with each other.

[0032] As mentioned above, in the existing prosody prediction technology, a prosody prediction model is generally used for prosody prediction. Among them, the prosody prediction model usually needs to be trained based on a large amount of prosody annotated data corpus. At present, institutions (such as enterprises, etc.) that have the need to build a prosody prediction model generally need to purchase data from data companies through legal channels to obtain prosody annotated data. Taking the Chinese speech synthesis scenario as an example, there is currently no open source Chinese prosody annotated data set. Chinese prosody annotated data can only be obtained by purchasing data from data companies. The cost of obtaining prosody annotated data through procurement is relatively high.

[0033] Based on this, the embodiments of this specification provide a prosody annotation data generation solution, which can construct a prosody annotation dataset at a low cost.

[0034] Figure 1 Schematic diagram of an application scenario in which the embodiments of this specification can be applied. Figure 1 In the application scenario shown, the target video (eg Figure 1 A video downloaded from a public channel as shown in Figure 1), a pre-trained audio-text alignment model (e.g. Figure 1 The MFA (Montreal Forced Aligner) model shown in FIG. 1 ) and preset prosody recognition rules associated with multiple prosody tags.

[0035] In terms of data composition structure, video A may include audio files and video files. In terms of video type, video A may include any of short videos, long videos, and live videos. In terms of language, video A may include but is not limited to Chinese videos.

[0036] The MFA model is an audio-text forced alignment model that can be used for word-level forced alignment processing. In one example, the MFA model can be trained based on an open source speech synthesis dataset (such as aishell).

[0037] The prosody recognition rules can be used to identify the prosody tag corresponding to the word in the text based on the audio feature corresponding to the word. The audio feature may include, for example, the pronunciation duration feature of the word. Further, the audio feature may also include the pitch feature and energy feature of the audio corresponding to the word; and / or include the saliency feature of the audio corresponding to the word. In one example, the saliency feature may be represented by the frequency domain intensity of the audio. A single prosody tag may, for example, represent any one of the following: a non-prosody boundary, a prosody word boundary, a prosody phrase boundary, and an intonation phrase boundary. As an example, the tag name of the prosody tag for representing a non-prosody boundary may be, for example, "#0", the tag name of the prosody tag for representing a prosody word boundary may be, for example, "#1", the tag name of the prosody tag for representing a prosody phrase boundary may be, for example, "#2", and the tag name of the prosody tag for representing an intonation phrase boundary may be, for example, "#3".

[0038] The execution subject of the method for generating prosody annotation data can be any device, platform or device cluster with data storage, computing and processing capabilities. The execution subject can include the video A, MFA model and prosody recognition rules as described above. In the process of constructing prosody annotation data based on video A, the execution subject can be as follows: Figure 1As shown in , audio A1 is extracted from video A, and text A2 corresponding to audio A1 is extracted. Afterwards, the pre-trained MFA model can be used to perform word-level alignment processing on audio A1 and text A2 to obtain an alignment result. Then, based on the alignment result, feature extraction processing can be performed on audio A1 to obtain audio features corresponding to each word in text A2. Then, the prosody recognition rules can be used to determine the prosody labels corresponding to each word based on the audio features. Then, the recognized prosody labels can be marked for each word in text A2, and the marked text A2 can be used as prosody marking data.

[0039] By adopting the above-described prosody annotation data generation process, a large amount of video data on the Internet can be processed to generate a large amount of prosody annotation data, and a prosody annotation dataset can be constructed at a low cost without relying on manual annotation.

[0040] The specific implementation steps of the above method are described below in conjunction with specific embodiments.

[0041] See also Figure 2 , which is a flow chart of the method for generating prosody annotation data in the embodiment of this specification. The execution subject of the method can be any device, platform or device cluster with data storage, computing and processing capabilities. The method includes the following steps: step S201, extracting the target audio from the target video, and extracting the target text corresponding to the target audio; step S203, using the pre-trained audio-text alignment model, performing word-level alignment processing on the target audio and the target text to obtain the alignment result; step S205, based on the alignment result, performing feature extraction processing on the target audio, so as to obtain the audio features corresponding to each word in the target text; step S207, using the preset prosody recognition rules associated with multiple prosody tags, determining the prosody tags corresponding to each word based on the audio features; step S209, marking the recognized prosody tags for each word in the target text, and using the marked target text as prosody annotation data.

[0042] It should be noted that, in terms of data composition structure, the target video may include audio files and video files. In terms of video source, the target video may include videos downloaded from public channels. In terms of video type, the target video may include any of short videos, long videos and live videos. In terms of language, the target video may include but is not limited to Chinese videos.

[0043] In one example, the target video is included in a video set. The video set may include a large number of videos, such as a large number of videos downloaded from public channels. Each video in the video set may be used as a target video, and steps S201-S209 may be performed for the target video.

[0044] Next, steps S201-S209 are further described.

[0045] In step S201, target audio may be extracted from a target video, and target text corresponding to the target audio may be extracted.

[0046] Specifically, the target video can be separated into an audio file and a video file, for example, the target video can be separated into an audio file and a video file using the ffmpeg tool, and the audio file can be used as the target audio. Among them, the ffmpeg tool is a set of open source computer programs that can be used to record and convert digital audio and video, and can convert them into streams. In addition, when the video file includes subtitles, the subtitles can be extracted from the video file, for example, the subtitles can be extracted from the video file using an OCR (Optical Character Recognition) algorithm, and the target text is generated based on the subtitles. When the video file does not include subtitles, the target audio can be converted into the target text using, for example, a speech recognition algorithm.

[0047] Next, in step S203, the target audio and target text may be input into a pre-trained audio-text alignment model, so that the audio-text alignment model may be used to perform word-level alignment processing on the target audio and target text to obtain an alignment result.

[0048] The audio-text alignment model may include an MFA model. The alignment result may include time-related information of the audio corresponding to each word in the target text. The time-related information may include, for example, a start timestamp and an end timestamp of the audio. It should be understood that the alignment result may be used to determine the audio (also referred to as an audio segment) corresponding to each word in the target audio.

[0049] Next, in step S205, feature extraction processing may be performed on the target audio based on the alignment result, so as to obtain audio features corresponding to each word in the target text.

[0050] Since the alignment result includes the time-related information of the audio corresponding to each word in the target text, the pronunciation duration of each word can be easily determined based on the time-related information, and the pronunciation duration can be used for prosody recognition. Therefore, in step S205, the following can be performed: Figure 3 In the sub-step S2051 shown in FIG, the pronunciation duration feature of each word is extracted. Specifically, in the sub-step S2051, the pronunciation duration feature of each word can be determined based on the start timestamp and the end timestamp of the audio corresponding to each word in the target text included in the alignment result. Figure 3 It is a flow chart of the feature extraction process.

[0051] Furthermore, in step S205, the following steps may be performed: Figure 3 Sub-steps S2053 and / or S2055 shown in.

[0052] In sub-step S2053, the tone features and energy features of the audio corresponding to each word in the target text can be extracted. Specifically, the tone features and energy features of the audio corresponding to each word can be extracted using a wavelet prosody analysis toolkit (such as Wavelet Prosody Analyzer toolkit 2).

[0053] In sub-step S2055, the significant features of the audio corresponding to each word in the target text can be extracted. The significant features can be represented by the frequency domain intensity of the audio. Specifically, when there are blank segments between adjacent words in the target audio, in order to ensure the accuracy of the extracted significant features, in sub-step S2055, the blank segments between adjacent words in the target audio can be interpolated and normalized, and then the normalized blank segments can be combined by addition or multiplication, and then the processed target audio can be subjected to continuous wavelet transform (Continues Wavelet Transform, CWT), and the significant features of the audio corresponding to each word can be determined based on the transformation results.

[0054] Next, in step S207, the preset prosodic recognition rules associated with the plurality of prosodic labels can be used to determine the prosodic labels corresponding to the respective characters in the target text based on the audio features corresponding to the respective characters. It should be noted that when a single prosodic label is used to represent any of the following: a non-prosodic boundary, a prosodic word boundary, a prosodic phrase boundary, or an intonation phrase boundary, the prosodic label corresponding to the character can be used to indicate whether the character is a non-prosodic boundary, or a boundary of a prosodic word, a prosodic phrase, or an intonation phrase.

[0055] In one example, the prosody recognition rule may include multiple conditions corresponding to the multiple prosody tags. For each of the individual characters, a target condition satisfied by the audio feature corresponding to the character may be determined in the multiple conditions, and the prosody tag corresponding to the target condition may be determined as the prosody tag corresponding to the character. Further, each of the multiple conditions may include a feature value interval set for at least one feature of the audio features. For each of the individual characters, when the at least one feature of the audio features corresponding to the character is within the feature value interval corresponding to the condition, it may be determined that the condition is the target condition satisfied by the audio feature.

[0056] It should be noted that the multiple prosodic labels can be divided into levels. For example, the four prosodic labels #0, #1, #2, and #3 described in the previous text have their respective levels increasing in sequence. In practice, the higher the pitch, the greater the energy, and the longer the pronunciation time of the word, the greater the probability of being marked with a higher level of prosodic label; the more significant the word is compared with the surrounding information after the continuous wavelet change, that is, the higher the significance of the word, the greater the probability of being marked with a higher level of prosodic label. It should be understood that the content of the prosodic recognition rules can be designed according to actual needs, and no specific limitation is made here.

[0057] Next, in step S209, the recognized prosodic label can be marked for each character in the target text, and the marked target text is used as prosodic labeling data. Specifically, the recognized prosodic label can be marked for each character according to the actual marking requirements. For example, if the marking requirement is to mark the prosodic label of the character after the character, the prosodic label corresponding to the character can be inserted after each character in the target text; if the marking requirement is to mark the prosodic label of the character before the character, the prosodic label corresponding to the character can be inserted before each character in the target text.

[0058] Figure 2 The solution provided by the corresponding embodiment extracts the target audio from the target video, and extracts the target text corresponding to the target audio, and then uses the pre-trained audio-text alignment model to perform word-level alignment processing on the target audio and the target text to obtain an alignment result, and performs feature extraction processing on the target audio based on the alignment result, thereby obtaining the audio features corresponding to each word in the target text, so as to use the preset rhythm recognition rules associated with multiple rhythm tags, determine the rhythm tags corresponding to each word based on the audio features, and mark the recognized rhythm tags for each word in the target text, and use the marked target text as rhythm annotation data. This solution can be used to process a large amount of video data on the Internet, such as a large amount of expressive short videos, long videos, and live videos, so as to generate a large amount of rhythm annotation data, thereby realizing the low-cost construction of rhythm annotation data sets without relying on manual annotation.

[0059] In addition, when the audio features corresponding to a character include pronunciation duration features, pitch features, energy features, and significance features, the solution can integrate the features of these four dimensions to automatically label each character with a rhythmic tag, which can effectively improve the tagging efficiency and ensure the accuracy of the tagging results.

[0060] In practice, different business fields may involve different entity nouns, long sentences, etc. The prosody annotation data obtained by purchasing data from data companies described in the previous article is generally general corpus, and the coverage of corpus related to certain business fields (such as the financial field, etc.) is relatively low. Therefore, the prosody prediction model trained based on general corpus is not effective on the scripts in this business field. Taking the digital human broadcasting scenario in the financial field as an example, in this scenario, using the prosody prediction model trained based on general corpus will encounter prosody prediction failures for entity nouns (such as fund names, etc.) and long sentences in the digital human broadcast.

[0061] In order to help improve the prediction effect of the rhythm prediction model in a certain business field on the script in the business field, and then improve the naturalness and anthropomorphism of speech synthesis in the business field, and improve the overall effect of digital human broadcasting, the inventors thought of collecting videos in the business field, and configuring rhythm recognition rules applicable to the business field, and then adopting Figure 2 The video is processed in a similar manner to the method described in the corresponding embodiment to generate prosody annotation data suitable for the business field.

[0062] Specifically, see Figure 4 , which is a flow chart of the method for generating prosody annotation data in an embodiment of this specification. The execution subject of the method can be any device, platform or device cluster with data storage, computing and processing capabilities. The method includes the following steps: Step S401, extracting the target audio from the target video belonging to the target business field, and extracting the target text corresponding to the target audio; Step S403, using the pre-trained audio-text alignment model, performing word-level alignment processing on the target audio and the target text to obtain the alignment result; Step S405, based on the alignment result, performing feature extraction processing on the target audio, so as to obtain the audio features corresponding to each word in the target text; Step S407, using the prosody recognition rules associated with multiple prosody tags preset for the target business field, determining the prosody tags corresponding to each word based on the audio features; Step S409, marking the recognized prosody tags for each word in the target text, and using the marked target text as prosody annotation data.

[0063] For explanation of steps S401-S409, please refer to Figure 2 The relevant descriptions of steps S201-S209 in the corresponding embodiment are not repeated here.

[0064] Further references Figure 5 , which is a schematic diagram of a structure of a prosody annotation data generation device in an embodiment of the present specification. The device can be applied to any device, platform or device cluster with data storage, computing and processing capabilities.

[0065] like Figure 5 As shown, the prosody annotation data generating device 500 of this embodiment includes: an audio-text extraction unit 501, an audio-text alignment unit 502, a feature extraction unit 503, a prosody recognition unit 504 and a prosody annotation unit 505. Among them, the audio-text extraction unit 501 is configured to extract the target audio from the target video, and extract the target text corresponding to the target audio; the audio-text alignment unit 502 is configured to use the pre-trained audio-text alignment model to perform word-level alignment processing on the target audio and the target text to obtain an alignment result; the feature extraction unit 503 is configured to perform feature extraction processing on the target audio based on the alignment result, so as to obtain the audio features corresponding to each word in the target text; the prosody recognition unit 504 is configured to use a preset prosody recognition rule associated with multiple prosody tags to determine the prosody tags corresponding to each word based on the audio features; the prosody annotation unit 505 is configured to mark the recognized prosody tags for each word in the target text, and use the marked target text as prosody annotation data.

[0066] In one implementation, the audio-text extraction unit 501 may be further configured to: separate the target video into an audio file and a video file; extract subtitles from the video file; and generate the target text based on the subtitles.

[0067] In one implementation, the audio text extraction unit 501 may be further configured to extract subtitles from the video file using an optical character recognition algorithm.

[0068] In one implementation, the feature extraction unit 503 may be further configured to extract the pronunciation duration features of each of the characters.

[0069] In one embodiment, the alignment result may include the start timestamp and end timestamp of the audio corresponding to each of the above-mentioned characters; and the feature extraction unit 503 may be further configured to: determine the pronunciation duration features of each of the above-mentioned characters based on the start timestamp and end timestamp.

[0070] In one implementation, the feature extraction unit 503 may be further configured to extract the tone features and energy features of the audio corresponding to each of the above characters.

[0071] In one implementation, the feature extraction unit 503 may be further configured to: utilize a wavelet prosody analysis toolkit to extract the tone features and energy features of the audio corresponding to each of the above-mentioned characters.

[0072] In one implementation, the feature extraction unit 503 may be further configured to extract significant features of the audio corresponding to each of the above characters.

[0073] In one embodiment, the feature extraction unit 503 can be further configured to: interpolate and normalize the blank segments between adjacent words in the target audio; combine the normalized blank segments by addition or multiplication; perform continuous wavelet transform on the processed target audio, and determine the significant features of the audio corresponding to each of the above words based on the transform results.

[0074] In one embodiment, the prosody recognition rule includes multiple conditions corresponding to the above-mentioned multiple prosody tags; and the prosody recognition unit 504 can be further configured to: for each of the above-mentioned characters, determine the target condition satisfied by the audio feature corresponding to the character in the multiple conditions, and determine the prosody tag corresponding to the target condition as the prosody tag corresponding to the character.

[0075] In one embodiment, a single prosodic tag is used to represent any of the following: a non-prosodic boundary, a prosodic word boundary, a prosodic phrase boundary, an intonation phrase boundary.

[0076] In one implementation, the target video includes a video downloaded from a public channel.

[0077] In one implementation, the target video includes any one of the following: a short video, a long video, or a live video.

[0078] In one implementation, the target video belongs to a target business field, and the above-mentioned rhythm recognition rule is configured for the target business field.

[0079] exist Figure 5 In the corresponding device embodiment, the specific processing of each unit and the technical effects it brings can refer to the relevant description in the relevant method embodiment in the previous text, and will not be repeated here.

[0080] The embodiments of the present specification also provide a computer-readable storage medium on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method for generating prosody annotation data described in the method embodiment above.

[0081] The embodiment of the present specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method for generating prosody annotation data described in the method embodiment above is implemented.

[0082] The embodiments of this specification also provide a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the method for generating prosody annotation data described in the method embodiment above.

[0083] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the multiple embodiments disclosed in this specification can be implemented by hardware, software, firmware, or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0084] In some cases, the actions or steps described in the claims may be performed in a different order than in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0085] The specific implementation methods described above further illustrate in detail the purposes, technical solutions and beneficial effects of the multiple embodiments disclosed in this specification. It should be understood that the above description is only the specific implementation methods of the multiple embodiments disclosed in this specification, and is not used to limit the protection scope of the multiple embodiments disclosed in this specification. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the multiple embodiments disclosed in this specification should be included in the protection scope of the multiple embodiments disclosed in this specification.

Claims

1. A method for generating prosody annotation data, comprising: Extracting target audio from a target video belonging to a target business field, and extracting target text corresponding to the target audio; wherein the target video includes a video downloaded from a public channel; Using a pre-trained audio-text alignment model, the target audio and the target text are aligned at the word level to obtain an alignment result; Based on the alignment result, feature extraction processing is performed on the target audio, so as to obtain audio features corresponding to each word in the target text; Using a prosody recognition rule associated with a plurality of prosody tags configured for the target business field, the prosody tags corresponding to the respective characters are determined based on the audio features; wherein the audio features include pronunciation duration features, pitch features, and energy features, the plurality of prosody tags are divided into levels, and the prosody recognition rule reflects that the higher the pitch, the greater the energy, and the longer the pronunciation duration of the character, the greater the probability of being marked with a prosody tag of a higher level; The recognized prosody tags are respectively marked for the respective characters in the target text, and the marked target text is used as prosody marking data for training the prosody prediction model in the target business field.

2. The method according to claim 1, wherein: The extracting the target text corresponding to the target audio includes: Separating the target video into an audio file and a video file; Extract subtitles from the video file; The target text is generated based on the subtitles.

3. The method according to claim 2, wherein: The step of extracting subtitles from the video file comprises: Subtitles are extracted from the video file using an optical character recognition algorithm.

4. The method according to claim 1, wherein: The performing feature extraction processing on the target audio includes: The pronunciation duration features of the respective characters are extracted.

5. The method according to claim 4, wherein: The alignment result includes the start timestamp and the end timestamp of the audio corresponding to each word; as well as The step of extracting the pronunciation duration features of each of the characters comprises: Based on the start timestamp and the end timestamp, the pronunciation duration characteristics of each of the characters are determined.

6. The method according to claim 4, wherein: The feature extraction process of the target audio further includes: Extract the tone features and energy features of the audio corresponding to each of the characters.

7. The method according to claim 6, wherein: The step of extracting the tone features and energy features of the audio corresponding to each of the characters comprises: The wavelet prosody analysis toolkit is used to extract the pitch characteristics and energy characteristics of the audio corresponding to each of the characters.

8. The method according to any one of claims 4 to 7, wherein: The feature extraction process of the target audio further includes: The significant features of the audio corresponding to each of the characters are extracted, and the significant features are represented by the frequency domain intensity of the audio.

9. The method according to claim 8, wherein: The extracting the significant features of the audio corresponding to each of the characters includes: Performing interpolation and normalization processing on blank segments between adjacent words in the target audio; Combining the normalized blank segments by addition or multiplication; The processed target audio is subjected to continuous wavelet transform, and the significant features of the audio corresponding to each of the characters are determined based on the transform result.

10. The method according to claim 1, wherein: The prosody identification rule includes a plurality of conditions corresponding to the plurality of prosody tags; and The method of using the prosody recognition rules associated with the plurality of prosody tags configured for the target business field to determine the prosody tags corresponding to the respective characters based on the audio features includes: For each of the characters, a target condition satisfied by the audio feature corresponding to the character is determined among the multiple conditions, and a prosodic label corresponding to the target condition is determined as the prosodic label corresponding to the character.

11. The method according to claim 1, wherein: A single prosodic tag is used to represent any of the following: non-prosodic boundaries, prosodic word boundaries, prosodic phrase boundaries, intonation phrase boundaries.

12. The method according to claim 1, wherein: The target video includes any one of the following: a short video, a long video, and a live video.

13. A device for generating prosody annotation data, comprising: An audio-text extraction unit is configured to extract a target audio from a target video belonging to a target business field, and to extract a target text corresponding to the target audio; wherein the target video includes a video downloaded from a public channel; An audio-text alignment unit is configured to use a pre-trained audio-text alignment model to perform word-level alignment processing on the target audio and the target text to obtain an alignment result; A feature extraction unit is configured to perform feature extraction processing on the target audio based on the alignment result, so as to obtain audio features corresponding to each word in the target text; The prosody recognition unit is configured to determine the prosody labels corresponding to the respective characters based on the audio features using the prosody recognition rules associated with the plurality of prosody labels configured for the target business field; wherein the audio features include pronunciation duration features, pitch features, and energy features, the plurality of prosody labels are divided into levels, and the prosody recognition rules reflect that the higher the pitch, the greater the energy, and the longer the pronunciation duration of the character, the greater the probability of being marked with a prosody label of a higher level; The prosody tagging unit is configured to tag the recognized prosody tags for the respective characters in the target text, and use the annotated target text as prosody tagging data for training the prosody prediction model in the target business field.

14. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 12.

15. A computing device comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 12 is implemented.

Citation Information

Patent Citations

  • Spoken Chinese stress automatic detection method

    CN101751919A

  • Broadcast receiving apparatus and method of determining presence of noise performed by broadcast receiving apparatus

    CN102163978A

  • Sample marking method and device based on artificial intelligence prosody prediction

    CN106601228A

  • Automatic rhythm extracting method, system and application thereof in natural language processing

    CN106683667A

  • Voice labeling method and system thereof

    CN110556093A