Text alignment information acquisition method and device, and computer device

Abstract

The application provides a text alignment information acquisition method and device and computer equipment. The method comprises the following steps: obtaining phoneme sequence information of a target text and audio information associated with the target text; based on the phoneme sequence information and the audio information, determining phoneme boundary information of each phoneme in the phoneme sequence information and silence segment information in the audio information; wherein the silence segment information comprises first silence boundary information of known punctuation and second silence boundary information of unknown punctuation; analyzing the silence segment information to correct the phoneme boundary information and the silence segment information, thereby obtaining first pronunciation duration of each phoneme and second pronunciation duration of each known punctuation; based on the first pronunciation duration and the second pronunciation duration, adding preset prosodic labels to the first silence boundary information and the second silence boundary information, thereby obtaining alignment information of the target text. By using the method, the reliability of the alignment effect can be effectively improved, and text alignment information with high accuracy can be obtained.

Description

Technical Field

[0001] This application relates to the field of intelligent voice technology, specifically to a method, apparatus, and computer device for obtaining text alignment information. Background Technology

[0002] Intelligent voice technology enables human-computer language communication, including Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) technologies. It is now widely used in daily life, such as home voice assistants and intelligent voice customer service. It not only improves the convenience of people's lives but also helps enterprises save costs and improve work efficiency, and has a wide range of application scenarios.

[0003] However, the aforementioned application scenarios rely heavily on the training of speech models. During model training, data is a crucial factor influencing model performance. Typically, the required data consists of one audio clip corresponding to one text clip. However, for some models, in addition to the audio and its corresponding text, finer-grained alignment information is also needed. Generally, the average person speaks 160-260 Chinese characters per minute. If alignment information is required, then an average of 5-9 phoneme boundaries need to be labeled per second of audio. For datasets ranging from tens to hundreds of hours, relying entirely on manual alignment labeling is extremely time-consuming and resource-intensive, making it impractical.

[0004] Therefore, there are two common methods for obtaining alignment information: one is to directly obtain alignment information using machine learning alignment tools; the other is to use machine learning tools for pre-labeling, followed by manual checking and modification. However, the alignment results of both methods are relatively unreliable, either due to the influence of the alignment tool or human experience. Therefore, how to obtain highly reliable alignment information has become an urgent problem to be solved. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, apparatus, and computer device for obtaining text alignment information to address the aforementioned technical problems. This would improve the reliability of alignment results and obtain more accurate text alignment information by refining the alignment information acquisition process and improving alignment pre-annotation tools.

[0006] Firstly, this application provides a method for obtaining text alignment information, including:

[0007] Obtain the phoneme sequence information of the target text, as well as the audio information associated with the target text;

[0008] Based on phoneme sequence information and audio information, the phoneme boundary information of each phoneme in the phoneme sequence information and the silence segment information in the audio information are determined; wherein, the silence segment information includes the first silence boundary information of known punctuation and the second silence boundary information of unknown punctuation.

[0009] Analyze the silence segment information to correct the boundaries of the phoneme boundary information and the silence segment information, and obtain the first pronunciation duration of each phoneme and the second pronunciation duration of each known punctuation mark;

[0010] Based on the first and second pronunciation durations, preset prosodic tags are added to the first and second silence boundary information to obtain the alignment information of the target text.

[0011] Secondly, this application provides a text alignment information acquisition device, comprising:

[0012] The information acquisition module is used to acquire the phoneme sequence information of the target text, as well as the audio information associated with the target text;

[0013] The boundary determination module is used to determine the phoneme boundary information of each phoneme in the phoneme sequence information and the silence segment information in the audio information based on the phoneme sequence information and the audio information; wherein, the silence segment information includes the first silence boundary information of known punctuation and the second silence boundary information of unknown punctuation.

[0014] The boundary correction module is used to analyze the silence segment information to correct the boundary information of phonemes and silence segments, so as to obtain the first pronunciation duration of each phoneme and the second pronunciation duration of each known punctuation mark.

[0015] The alignment acquisition module is used to add preset prosodic tags to the first silence boundary information and the second silence boundary information based on the first pronunciation duration and the second pronunciation duration, so as to obtain the alignment information of the target text.

[0016] Thirdly, this application also provides a computer device, comprising:

[0017] One or more processors;

[0018] The memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the text alignment information acquisition method described above.

[0019] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to execute the above-described text alignment information acquisition method.

[0020] Fifthly, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the text alignment information acquisition method provided in the first aspect.

[0021] The aforementioned text alignment information acquisition method, apparatus, and computer equipment acquire the phoneme sequence information of the target text and the audio information associated with the target text. Based on the phoneme sequence information and audio information, they determine the phoneme boundary information of each phoneme in the phoneme sequence information and the silence segment information in the audio information. They then analyze the silence segment information, correct the boundary information of the phonemes and the silence segment information, obtain the first pronunciation duration of each phoneme and the second pronunciation duration of each known punctuation mark. Based on the first and second pronunciation durations, they add preset prosodic labels to the silence segment information, including the first silence boundary information of known punctuation marks and the second silence boundary information of unknown punctuation marks, ultimately obtaining higher-precision alignment information. By changing the analysis from text and audio to phoneme sequence and audio, errors caused by the text-to-phoneme conversion process are avoided, as are subsequent correlation errors. Furthermore, silence detection solves the problem of errors easily occurring at silence boundaries, increasing the reliability of the alignment results and improving the accuracy of alignment annotation. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is an application scenario diagram of the text alignment information acquisition method provided in the embodiments of this application;

[0024] Figure 2 This is a flowchart illustrating the text alignment information acquisition method provided in the embodiments of this application;

[0025] Figure 3 This is a schematic diagram of the interface of the text alignment information acquisition method provided in the embodiments of this application;

[0026] Figure 4 This is a schematic diagram illustrating the specific process of the text alignment information acquisition method provided in the embodiments of this application;

[0027] Figure 5 This is a schematic diagram of the structure of the text alignment information acquisition device provided in the embodiments of this application;

[0028] Figure 6 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] In the description of this application, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0031] It should be noted that the text alignment information acquisition method provided in this application embodiment can be applied to, for example, Figure 1The text alignment information acquisition system shown includes a terminal 102 and a server 104. The terminal 102 can be a device that includes both receiving and transmitting hardware, i.e., a device with receiving and transmitting hardware capable of performing bidirectional communication over a bidirectional communication link. Such a device can include cellular or other communication devices with single-line displays, multi-line displays, or no multi-line displays. Specifically, the terminal 102 can be a desktop terminal or a mobile terminal; it can also be a mobile phone, tablet computer, or laptop computer. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Furthermore, the terminal 102 and the server 104 establish a communication connection through a network, which can be any of a wide area network (WAN), local area network (LAN), or metropolitan area network (MAN).

[0032] Those skilled in the art will understand that Figure 1 The application environment shown is merely one applicable scenario for the solution in this application and does not constitute a limitation on the application scenario of the solution in this application. Other application environments may include more than one. Figure 1 The number of devices shown may be more or less. For example, Figure 1 Only one server is shown. It is understood that this text alignment information acquisition system may also include one or more other devices, which are not specifically limited here. Additionally, the text alignment information acquisition system may also include a memory for storing data, such as alignment information for various types of text.

[0033] It should also be noted that, Figure 1 The schematic diagram of the text alignment information acquisition system shown is merely an example. The text alignment information acquisition system and scenario described in this embodiment are for the purpose of more clearly illustrating the technical solutions of this embodiment and do not constitute a limitation on the technical solutions provided by this embodiment. As those skilled in the art will know, with the evolution of text alignment information acquisition systems and the emergence of new business scenarios, the technical solutions provided by this embodiment are also applicable to similar technical problems.

[0034] Furthermore, as mentioned above, alignment information ("alignment information" refers to the temporal boundaries of each phoneme in the audio, i.e., the start and end times) can be directly obtained through machine learning alignment tools. A common technique is to use the GMM-HMM framework for modeling. On a certain amount of speech and text data, a pre-trained phoneme model is obtained through maximum likelihood estimation iteration. Then, this model is applied to calculate the phonemes corresponding to each frame in the audio (e.g., ...). Figure 3 As shown, the phoneme pronounced from second "2.277207" to second "2.420640" in the audio is "h", thus obtaining the time boundary of the phoneme. Currently, some open-source alignment tools are based on the above algorithm and provide some usable pre-trained models, generating alignment information simply and efficiently, but the alignment results are not very reliable. Therefore, when model training requires alignment information with higher accuracy, a method of tool pre-annotation + manual inspection and modification is adopted, that is, the above tool annotation results are distributed to annotators for inspection, and problems are found and corrected in combination with the audio. Thus, whether the tool annotation results are used directly or tool pre-annotation + manual inspection and modification are used, alignment tools are indispensable. The quality of the alignment tool output results is a key factor affecting the final result and work efficiency. Even if manual inspection is added later, if the pre-annotation errors are greatly reduced, the efficiency of manual annotation will be greatly improved. Therefore, this application proposes to improve the reliability of alignment results by starting from the alignment annotation process and alignment tools, which will be described in detail below.

[0035] See Figure 2 This application provides a method for obtaining text alignment information. This embodiment mainly applies this method to the above-mentioned... Figure 1 Taking server 104 as an example, the method includes steps S201 to S204, as follows:

[0036] S201, Obtain the phoneme sequence information of the target text, as well as the audio information associated with the target text.

[0037] The target text can be any selected Chinese text for which alignment information is to be obtained, and the text type is not specifically limited in this embodiment. For example, the target text can be a randomly selected excerpt from a news article, or it can be specified content, such as "Today, I also have a book to read."

[0038] Wherein, the phoneme sequence information may be an ordered array composed of at least two phonemes, and a phone, in the technical practice of speech synthesis, can be understood as the smallest unit for speech modeling. In practical applications, technicians can determine the phoneme set (all speech units that need to be modeled) based on data, resources, project requirements, etc. Its theoretical basis is the pinyin initial-final scheme or IPA (International Phonetic Alphabet), but may be modified. The phoneme set used by the current text alignment information acquisition system splits a syllable into the form of initial, medial + final + tone. For example, "啊" (ā) has only one phoneme "a1", and "代" (dài) has two phonemes "d" and "ai4".

[0039] Wherein, the audio information may be an information carrier with frequency and amplitude changes of human speech acoustic waves, and this embodiment names the file thereof as "example_1.wav".

[0040] In specific implementation, the server 104 may acquire the target text and the audio information associated with the target text from the terminal 102, and may also acquire the target text and the audio information thereof through the terminal 102, that is, regard the terminal 102 as an intermediate device for information transmission. In practical application scenarios, the acquisition methods of the target text and the audio information include but are not limited to the following methods: 1. In a common network structure, the server 104 acquires the target text and the audio information thereof from the terminal 102 or other cloud devices with established network connection; 2. In a preset blockchain network, the server 104 may synchronously acquire the target text and the audio information thereof from a terminal node, and the blockchain network may be a public chain, a private chain, etc.; 3. In a preset tree structure, the server 104 may request the target text and the audio information thereof from an upper-level server, or poll the target text and the audio information thereof from a lower-level server, wherein said upper-level server or lower-level server are both local servers associated with the terminal 102, so said server 104 may be an edge server. Of course, those skilled in the art can also acquire the target text and the audio information thereof through other methods, which can be specifically determined according to the actual application scenario or requirements.

[0041] Further, when the server 104 acquires the phoneme sequence information of the target text, it needs to acquire the target text first, and the acquisition method of the target text has been described in detail above. As for the acquisition method of the phoneme sequence information, the embodiment of the present application proposes that phoneme conversion processing needs to be performed on the target text, and the specific steps will be described in detail below.

[0042] In one embodiment, this step includes: acquiring interrelated text to be processed and audio information; performing speech recognition on the audio information to obtain speech-recognized text; analyzing the speech-recognized text and the text to be processed based on a preset edit distance algorithm, so as to correct the text to be processed when it is determined that there is an error in the text to be processed, and obtaining the target text; and performing phoneme conversion on the target text to obtain the phoneme sequence information of the target text.

[0043] In specific implementation, the association between the associated text and audio mentioned in the embodiments of this application is reflected in the fact that the audio is a multimedia voice file recorded for the text content. The text to be processed refers to the original data without correction processing. That is, in order to obtain the phoneme sequence information of the target text and the audio information associated with the target text, the server 104 can first obtain the text to be processed and its corresponding audio information, and then use speech recognition technology to identify the speech content contained in the audio information and output the speech recognition text. Finally, the edit distance algorithm is used to calculate the word error rate (CER) of the speech recognition text and the text to be processed. The judgment criteria are: if the two are completely the same, the CER should be 0; if the two are completely different, the CER should be 1. The larger the CER, the more different parts the two have. The system determines whether the text to be processed differs from the speech recognition text in terms of text content. When it is determined that the text to be processed has an error, the text to be processed is corrected to obtain the target text required at the moment. The specific steps will be described in detail below.

[0044] Furthermore, server 104 can use the G2P tool to convert the target text into a phoneme sequence. The conversion process considers handling polyphonic characters and tone sandhi to obtain the required phoneme sequence information. The G2P tool is a component of the speech synthesis system's front-end, namely the character-to-phonetic conversion system. It is primarily responsible for converting Chinese character sequences into reasonable pinyin (or phoneme) sequences while preserving the original text's punctuation. This process considers polyphonic character conversion, tone sandhi (tone sandhi for third-tone characters, no tone for first-tone characters, and neutral tone), and combines techniques such as a phonetic dictionary, deep learning models, and decision tree rules.

[0045] In one embodiment, based on a preset edit distance algorithm, the speech recognition text and the text to be processed are analyzed to correct the text to be processed when an error is determined to exist, thereby obtaining the target text. This includes: removing punctuation from both the speech recognition text and the text to be processed to obtain a first text corresponding to the speech recognition text and a second text corresponding to the text to be processed; converting the first text and the second text into pinyin to obtain a first pinyin sequence corresponding to the first text and a second pinyin sequence corresponding to the second text; analyzing the first pinyin sequence and the second pinyin sequence based on the preset edit distance algorithm to obtain a character error rate; and determining whether the text to be processed contains text errors based on the character error rate, so that the text to be processed is corrected when an error is determined to exist, thereby obtaining the target text.

[0046] In the specific implementation, before using the ASR tool developed based on the edit distance algorithm, server 104 can first remove the punctuation from the speech recognition text and the text to be processed, and then convert them into pinyin, obtaining two pinyin sequences. Then, the edit distance algorithm is used to calculate the word error rate (CER) of both. Finally, by combining the length of the pinyin sequence to analyze the magnitude of the word error rate, the text to be processed can be corrected when errors are determined to exist, thus obtaining the target text. The specific steps will be explained in detail below.

[0047] In one embodiment, determining whether there are text errors in the text to be processed based on the word error rate, and correcting the text to be processed to obtain the target text when an error is determined to exist, includes: obtaining the maximum sequence length of the first pinyin sequence and the second pinyin sequence; determining a target word error rate threshold from a plurality of pre-stored word error rate thresholds based on the maximum sequence length; if the word error rate is greater than or equal to the target word error rate threshold, determining that there are errors in the text to be processed, and correcting the text to be processed based on the speech recognition text to obtain the target text.

[0048] In the specific implementation, the numerical value of the character error rate is analyzed based on the length of the pinyin sequence. This is achieved by obtaining the maximum sequence length of the first and second pinyin sequences. For example, if the length of the first pinyin sequence is "13" and the length of the second pinyin sequence is "15", then the maximum sequence length is "15". At this point, the server 104 can use this maximum sequence length to determine the target character error rate threshold from a set of pre-stored character error rate thresholds. For example, for text with a pinyin sequence length less than "15", the CER threshold is set to "0.1"; for text with a pinyin sequence length between "15-30", the CER threshold is set to "0.06"; and for text with a pinyin sequence length greater than "30", the CER threshold is set to "0.04". Therefore, the target character error rate threshold should be "0.06". Finally, by comparing the character error rate with the target character error rate threshold, if the character error rate is greater than or equal to the target character error rate threshold, it is determined that the text to be processed contains errors. The text to be processed is then corrected based on the speech recognition data to obtain the target text. For example, if the current character error rate is "0.2" and the target character error rate threshold should be "0.06", then it can be determined that there is an error in the text to be processed. At this time, the server 104 can generate a text correction prompt and send the prompt to the terminal 102 to obtain the target text submitted by the staff through the terminal 102.

[0049] For example, see Figure 4 After receiving the text to be processed, “Today, I also have a book to read.” and its audio information “example_1.wav”, server 104 can use the ASR tool to determine whether there are text errors in the text to be processed. After determining that there are text errors in the text to be processed based on word error rate analysis, it prompts the staff to modify the text through terminal 102 to obtain the accurate and valid target text, “Today, I also have a book to read.”.

[0050] It is understandable that if it is determined that there are no text errors in the text to be processed, then the text to be processed can be identified as the target text and used as the basis for subsequent analysis, such as converting the target text into a phoneme sequence. The conversion strategy has been described in detail above and will not be repeated in this embodiment.

[0051] However, it should be noted that after obtaining the phoneme sequence information of the target text, server 104 can also generate a phoneme check prompt and send this prompt back to terminal 102 to obtain the target phoneme sequence submitted by the staff through terminal 102. Therefore, this target phoneme sequence may be the original phoneme sequence information, meaning that the phoneme sequence information submitted to terminal 102 is correct; at the same time, this target phoneme sequence may not be the original phoneme sequence information, meaning that the phoneme sequence information submitted to terminal 102 contains some errors, but has been corrected by the staff.

[0052] For example, refer to Figure 4 , in the embodiment of the present application, the phoneme set adopted adopts a form where a syllable is split into an initial, a medial + final + tone, Figure 4 as shown, the phoneme of the Chinese character "看" is divided into "@k" and "@an1", but its actual tone should be the fourth tone, so "@an1" is corrected to "@an4" in the target phoneme sequence.

[0053] S202, based on the phoneme sequence information and the audio information, determining phoneme boundary information of each phoneme in the phoneme sequence information and mute segment information in the audio information; wherein the mute segment information comprises first mute boundary information corresponding to known punctuation marks and second mute boundary information corresponding to unknown punctuation marks.

[0054] Wherein, the phoneme boundary information can be represented by the boundary start time and boundary end time of the phoneme, and the mute segment information can also be represented by the mute start time and mute end time.

[0055] In specific implementation, analysis and practice find that when an alignment tool takes text and audio as inputs, due to data collection problems, it is difficult to avoid errors of extra characters or missing characters in the text. Moreover, conversion from text to phonemes generally adopts a fixed dictionary or an open-source G2P tool, which is prone to conversion errors for polyphonic characters and tone sandhi phenomena common in Chinese, thereby outputting an incorrect phoneme sequence. Errors in the phoneme sequence will affect the accuracy of phoneme boundary prediction for the whole sentence. Therefore, the embodiment of the present application proposes that phoneme sequence information and audio information can be taken as inputs for model analysis, that is, changing input from text and audio to inputting phoneme sequence and audio, so as to avoid errors caused by the text-to-phoneme conversion process. This strategy is regarded as an improvement to the alignment pre-annotation tool, and relatively accurate phoneme boundary information can be obtained.

[0056] Further, since the alignment tool is relatively sensitive to mute segments, some mute segments are very short and are actually a part of adjacent phonemes, but they will be segmented separately, leading to errors in phoneme boundary division. In addition, for alignment required for speech synthesis, prosody labels (phrase pause labels, clause pause labels and full sentence pause labels) need to be given according to mute duration and speaker's rhythm, but current alignment tools cannot provide this function. Therefore, the embodiment of the present application proposes that after obtaining the mute segment information in the audio information, in-depth analysis is performed on the mute segment information to obtain accurate alignment information, which will be described in detail below.

[0057] In one embodiment, this step comprises: inputting the phoneme sequence information and audio information into a trained alignment model, and outputting phoneme boundary information of each phoneme; determining, based on the phoneme boundary information, speech segments in the audio information that do not correspond to any phoneme; determining punctuation marks included in target text as known punctuation marks, determining first silence boundary information in the speech segments according to the known punctuation marks, and performing screening to obtain second silence boundary information; and taking the first silence boundary information and the second silence boundary information as silence segment information.

[0058] In specific implementation, embodiments of the present application propose that the GMM-HMM alignment model can be updated through maximum likelihood estimation to improve the boundary prediction effect of the alignment tool on current data. Then the alignment tool reads the phoneme sequence information, audio information and the updated alignment model to predict the phoneme boundary information of each phoneme and fine-grained silence segment information. Herein, updating the GMM-HMM alignment model through maximum likelihood estimation actually means using an existing model to calculate the maximum likelihood of the alignment result of new data, then updating model parameters based on this value and recalculating the alignment result of the new data, and the updated alignment model can be obtained after multiple iterations.

[0059] Further, the logic for the alignment tool to output silence segment information is based on the phoneme boundary information of phonemes, first determining speech segments in the audio information that do not correspond to any phoneme, such as Figure 4 shown, the phoneme boundary information of the phoneme "@j" is "0.19-0.27s", and the speech segment "0.0-0.19s" before this segment, as well as "0.69-1.24s" after this segment and "2.35-2.59s" are speech segments that do not correspond to any phoneme. At this time, based on the known punctuation marks "," and "." in the target text "Today, I also have a book to read.", it can be determined that the first silence boundary information should be "0.69-1.24s" corresponding to ",", and "2.35-2.59s" corresponding to ".", and the second silence boundary information includes "0.0-0.19s".

[0060] S203, analyzing the silence segment information to correct boundaries of the phoneme boundary information and the silence segment information, so as to obtain a first pronunciation duration of each phoneme and a second pronunciation duration of each known punctuation mark.

[0061] Wherein, both the first pronunciation duration and the second pronunciation duration can refer to an average pronunciation duration, that is, when there are multiple identical phonemes, the average value of the pronunciation durations of the identical phonemes can be counted as the first pronunciation duration. Similarly, when there are multiple identical known punctuation marks, the average value of the pronunciation durations of each identical punctuation mark can be counted as the second pronunciation duration.

[0062] In the specific implementation, after analyzing and obtaining the phoneme boundary information and silence segment information, server 104 can perform in-depth analysis of the silence segment information to obtain more accurate alignment information, thereby overcoming the problem of incorrect division of silence boundaries by the alignment tool. Thus, after accurately determining the time boundaries of the phoneme boundary information and silence segment information, the first pronunciation duration of each phoneme and the second pronunciation duration of each known punctuation mark can be calculated. This allows for the subsequent addition of prosodic tags, resulting in alignment information containing prosodic tags.

[0063] In one embodiment, this step includes: analyzing audio information based on a preset speech activity detection algorithm to obtain silent segments in the audio information as silent segment information to be identified; if the time boundaries of the silent segment information to be identified and the silent segment information are inconsistent, then the phoneme boundary information and the silent segment information are corrected to obtain the first pronunciation duration and the second pronunciation duration.

[0064] Among them, Voice Activity Detection (VAD), also known as voice endpoint detection or voice boundary detection, aims to identify and eliminate long periods of silence in the audio signal stream. Silence suppression can save valuable bandwidth resources and help reduce the end-to-end latency perceived by the user.

[0065] In practice, the phoneme boundary information and the time boundary of the silence segment information are accurately determined by analyzing the silence segment information. This mainly involves merging excessively short silence segments into the phoneme boundary and correcting the time boundary of longer silence segments. The algorithm used in this process is the VDA algorithm.

[0066] Specifically, the audio framing proposed in this application is based on "10ms" units. The alignment tool predicts the corresponding phoneme for each frame, that is, predicts a phoneme once every "10ms". For each frame, it may be a certain phoneme in the defined phoneme set, or it may be a silent segment, while the smallest unit of VAD detection is currently "30ms". Based on this, the VDA algorithm predicts a phoneme once every "30ms", compared to the alignment model predicting a phoneme once every "10ms". For a certain audio segment, if VAD detects a silent segment from "900-960ms", while the alignment model detects a silent segment from "910-960ms", it means that the time boundary of the silent segment information to be identified is inconsistent with the silent segment information. At this time, the server 104 needs to analyze the audio amplitude of "900-910ms" to correct the boundary between the phoneme boundary information and the silent segment information, thereby obtaining the first pronunciation duration and the second pronunciation duration, which will be explained in detail below.

[0067] In one embodiment, if the time boundaries of the silence segment information to be identified are inconsistent with those of the silence segment information, boundary correction is performed on the phoneme boundary information and the silence segment information to obtain a first pronunciation duration and a second pronunciation duration. This includes: if the time boundaries of the silence segment information to be identified are inconsistent with those of the silence segment information, and there is an overlapping part of the speech segments, then the non-overlapping part of the speech segments, excluding the overlapping part, is analyzed within the maximum range of the speech segments. Based on the non-overlapping part of the speech segments, boundary correction is performed on the phoneme boundary information and the silence segment information to obtain a first pronunciation duration and a second pronunciation duration; if the time boundaries of the silence segment information to be identified are inconsistent with those of the silence segment information, and there is no overlapping part of the speech segments, then the boundary correction is performed on the phoneme boundary information and the silence segment information based on the phoneme boundary information of the preceding adjacent phoneme of the silence segment information to obtain a first pronunciation duration and a second pronunciation duration.

[0068] In a specific implementation, taking the values ​​shown in the previous embodiment as an example, VAD detects that the "900-960ms" segment is a silence segment, and the alignment model detects that the "910-960ms" segment is a silence segment. At this time, it can be determined that there is an overlapping part of the speech segments "910-960ms". Then, the non-overlapping part of the speech segment within the maximum range of the speech segment "900-960ms" excluding the overlapping part "910-960ms" is "900-910ms". By analyzing the audio amplitude of this part, it can be determined whether it meets the silence condition. Then, if it is determined that it meets the silence condition, the alignment result "900-960ms" is corrected to be a silence segment.

[0069] For example, see Figure 4 The silence segment detected by the alignment model includes "0.0-0.19s". If the VDA detects that the silence segment to be identified is "0.0-0.22s", then the non-overlapping part of the speech segment, excluding the overlapping part "0.0-0.19s", within the maximum range of "0.0-0.22s" is "0.19-0.22s". If the analysis of the audio amplitude of this part can determine that it meets the silence condition, then the silence segment information needs to be corrected to "0.0-0.22s", and the phoneme boundary information of the phoneme "@j" needs to be corrected to "0.22-0.27s".

[0070] Furthermore, for cases where the time boundaries of the silence segment information to be identified and the silence segment information are inconsistent, and there is no overlapping part of the speech segment (generally less than 30ms), that is, when the alignment model detects "910-920ms" as a silence segment, while VAD detects "910-920ms" as a non-silent segment, "910-920ms" can be regarded as a too short silence segment and needs to be merged into the preceding phoneme boundary information. That is, the phoneme boundary information of the adjacent preceding phoneme corresponding to "910-920ms" is expanded to "920ms".

[0071] In one embodiment, boundary correction is performed on phoneme boundary information and silence segment information to obtain a first pronunciation duration and a second pronunciation duration, including: determining the corrected phoneme boundary information and the corrected silence segment information; wherein the corrected phoneme boundary information includes a first boundary start time and a first boundary stop time, and the corrected silence segment information includes a second boundary start time and a second boundary stop time; calculating the difference between the first boundary stop time and the first boundary start time, and averaging the differences for the same phoneme to obtain the first pronunciation duration; calculating the difference between the second boundary stop time and the second boundary start time, and averaging the differences for the same known punctuation mark to obtain the second pronunciation duration.

[0072] In practice, for each phoneme and each known punctuation mark, the difference between the boundary stop time and the boundary start time gives the pronunciation duration of that phoneme or punctuation mark. This calculation is performed on all data, and the average pronunciation duration of the same phoneme or punctuation mark is taken to obtain the first and second pronunciation durations.

[0073] For example, in the same target text, the phoneme boundary information of a certain phoneme "@j" is "0.22-0.27s" and "0.55-0.58s". The pronunciation duration of the former is "0.05s" and the pronunciation duration of the latter is "0.03s". After averaging, the pronunciation duration of the first sound is "0.04s".

[0074] S204, based on the first pronunciation duration and the second pronunciation duration, add preset prosodic tags to the first silence boundary information and the second silence boundary information to obtain the alignment information of the target text.

[0075] The prosodic tags can include: the first prosodic tag (phrase pause tag "#2"), the second prosodic tag (clause pause tag "#3"), and the third prosodic tag (entire sentence pause tag "#4").

[0076] In the specific implementation, server 104 executes the steps described in the above embodiment to obtain accurate phoneme boundary information and silence segment information, and then needs to add prosodic tags. The second prosodic tag "#3" and the third prosodic tag "#4" are determined based on the corresponding positions of known punctuation marks in the text. Only the first prosodic tag "#2" (phrase-level pause without reference punctuation) requires threshold calculation for determination. Therefore, a threshold needs to be determined based on the first and second pronunciation durations. Then, based on the threshold, the first prosodic tag is added to the second silence boundary information, and the second or third prosodic tag is added to the first silence boundary information based on the position of the known punctuation marks, thereby obtaining alignment information containing prosodic tags.

[0077] In one embodiment, this step includes: determining a pause threshold based on a first pronunciation duration and a second pronunciation duration; if the pronunciation duration corresponding to the second silence boundary information is greater than or equal to the pause threshold, then adding a preset first prosodic tag at the second silence boundary information; and determining and adding a preset second prosodic tag or a third prosodic tag to the first silence boundary information according to the known punctuation type to obtain the alignment information of the target text; wherein, the first prosodic tag is a phrase pause tag, the second prosodic tag is a clause pause tag, and the third prosodic tag is a whole sentence pause tag.

[0078] In this embodiment, the first pronunciation duration applies to all phonemes, but phonemes include initials, medials, and finals. Therefore, the first pronunciation duration corresponding to an initial is defined as "L1", and the first pronunciation duration corresponding to a medial + final is defined as "L2". The second pronunciation duration applies to all known punctuation marks. This application defines commas and semicolons as first punctuation marks, and question marks, exclamation marks, and periods as second punctuation marks. Therefore, the second pronunciation duration corresponding to the first punctuation mark can be defined as "L3", and the second pronunciation duration corresponding to the second punctuation mark can be defined as "L4". The pause threshold is also defined as "thresold".

[0079] In specific implementation, this application embodiment limits "thresold = min(a1*L1 + a2*L2, a3*L3 + a4*L4)", where "a1 / a2 / a3 / a4" are empirical values ​​obtained from statistical analysis of past data, and this application embodiment does not impose any specific limitations. Therefore, if the pronunciation duration corresponding to a certain segment of second silence boundary information is greater than or equal to the pause threshold, a first prosodic label "#2" can be added at the second silence boundary information. Finally, according to the known punctuation type, a second prosodic label "#3" or a third prosodic label "#4" is added at each first silence boundary information.

[0080] In one embodiment, determining and adding a preset second or third prosodic tag to the first silence boundary information based on the known punctuation type to obtain the alignment information of the target text includes: if the known punctuation type is a first punctuation, then adding a second prosodic tag at the first silence boundary information; if the known punctuation type is a second punctuation, then adding a third prosodic tag at the first silence boundary information to obtain the alignment information of the target text; wherein the first punctuation includes at least one of the following: comma, semicolon; the second punctuation includes at least one of the following: question mark, exclamation mark, period.

[0081] For specific implementation details, please refer to [link / reference]. Figure 4Server 104 can add a second prosodic tag "#3" at the first silence boundary corresponding to a comma or semicolon, such as adding "#3" before "0.69-1.24s" corresponding to a comma in the text. A third prosodic tag "#4" can be added at the first silence boundary corresponding to a question mark, exclamation mark, or period, such as adding "#4" before "2.35-2.59s" corresponding to a period at the end of the text. This will output the alignment information for the file named "example_1.TextGrid".

[0082] The text alignment information acquisition method in the above embodiments obtains the phoneme sequence information of the target text and the audio information associated with the target text. Based on the phoneme sequence information and the audio information, it determines the phoneme boundary information of each phoneme in the phoneme sequence information and the silence segment information in the audio information. It then analyzes the silence segment information, corrects the boundary information of the phonemes and the silence segment information, obtains the first pronunciation duration of each phoneme and the second pronunciation duration of each known punctuation mark, and adds preset prosodic tags to the silence segment information, including the first silence boundary information of known punctuation marks and the second silence boundary information of unknown punctuation marks, ultimately obtaining higher-precision alignment information. Changing the analysis from text and audio to phoneme sequence and audio not only avoids errors in the text-to-phoneme conversion process but also avoids subsequent association errors. Furthermore, silence detection solves the problem of errors easily occurring at silence boundaries, increasing the reliability of the alignment results and improving the accuracy of alignment annotation.

[0083] It should be understood that, although Figure 2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0084] To better implement the text alignment information acquisition method provided in the embodiments of this application, based on the text alignment information acquisition method proposed in the embodiments of this application, this application also provides a text alignment information acquisition device, such as... Figure 5 As shown, the text alignment information acquisition device 500 includes:

[0085] The information acquisition module 510 is used to acquire the phoneme sequence information of the target text, as well as the audio information associated with the target text;

[0086] The boundary determination module 520 is used to determine the phoneme boundary information of each phoneme in the phoneme sequence information and the silence segment information in the audio information based on the phoneme sequence information and the audio information; wherein, the silence segment information includes the first silence boundary information of known punctuation and the second silence boundary information of unknown punctuation.

[0087] The boundary correction module 530 is used to analyze the silence segment information to correct the boundary information of phonemes and silence segment information, so as to obtain the first pronunciation duration of each phoneme and the second pronunciation duration of each known punctuation mark.

[0088] The alignment acquisition module 540 is used to add preset prosodic tags to the first silence boundary information and the second silence boundary information based on the first pronunciation duration and the second pronunciation duration, so as to obtain the alignment information of the target text.

[0089] In one embodiment, the information acquisition module 510 is further configured to acquire interrelated text to be processed and audio information; perform speech recognition on the audio information to obtain speech-recognized text; analyze the speech-recognized text and the text to be processed based on a preset edit distance algorithm, so as to correct the text to be processed when it is determined that there is an error in the text to be processed, and obtain the target text; and perform phoneme conversion on the target text to obtain the phoneme sequence information of the target text.

[0090] In one embodiment, the information acquisition module 510 is further configured to remove punctuation from the speech recognition text and the text to be processed, respectively, to obtain a first text corresponding to the speech recognition text and a second text corresponding to the text to be processed; to convert the first text and the second text into pinyin, to obtain a first pinyin sequence corresponding to the first text and a second pinyin sequence corresponding to the second text; to analyze the first pinyin sequence and the second pinyin sequence based on a preset edit distance algorithm to obtain a character error rate; and to determine whether there are text errors in the text to be processed based on the character error rate, so as to correct the text to be processed when it is determined that there are errors, and obtain the target text.

[0091] In one embodiment, the information acquisition module 510 is further configured to acquire the maximum sequence length of the first pinyin sequence and the second pinyin sequence; based on the maximum sequence length, determine the target character error rate threshold from a plurality of pre-stored character error rate thresholds; if the character error rate is greater than or equal to the target character error rate threshold, determine that the text to be processed has an error, and correct the text to be processed based on the speech recognition text to obtain the target text.

[0092] In one embodiment, the boundary determination module 520 is further configured to input phoneme sequence information and audio information into a trained alignment model, output phoneme boundary information of each phoneme; determine speech segments in the audio information that do not correspond to phonemes based on the phoneme boundary information; determine punctuation marks contained in the target text as known punctuation marks, and determine the first silence boundary information in the speech segment based on the known punctuation marks, and filter to obtain the second silence boundary information; and use the first silence boundary information and the second silence boundary information as silence segment information.

[0093] In one embodiment, the boundary correction module 530 is further configured to analyze audio information based on a preset speech activity detection algorithm to obtain silent segments in the audio information as silent segment information to be identified; if the time boundaries of the silent segment information to be identified and the silent segment information are inconsistent, the phoneme boundary information and the silent segment information are corrected to obtain the first pronunciation duration and the second pronunciation duration.

[0094] In one embodiment, the boundary correction module 530 is further configured to, if the time boundaries of the silence segment information to be identified are inconsistent with those of the silence segment information and there is an overlapping part of the speech segments, analyze the non-overlapping part of the speech segments within the maximum range of the speech segments, excluding the overlapping part of the speech segments, and perform boundary correction on the phoneme boundary information and the silence segment information based on the non-overlapping part of the speech segments to obtain the first pronunciation duration and the second pronunciation duration; if the time boundaries of the silence segment information to be identified are inconsistent with those of the silence segment information and there is no overlapping part of the speech segments, perform boundary correction on the phoneme boundary information and the silence segment information based on the phoneme boundary information of the preceding adjacent phonemes of the silence segment information to obtain the first pronunciation duration and the second pronunciation duration.

[0095] In one embodiment, the boundary correction module 530 is further configured to determine the corrected phoneme boundary information and the corrected silence segment information; wherein the corrected phoneme boundary information includes a first boundary start time and a first boundary stop time, and the corrected silence segment information includes a second boundary start time and a second boundary stop time; calculate the difference between the first boundary stop time and the first boundary start time, and calculate the average of the differences for the same phoneme to obtain a first pronunciation duration; calculate the difference between the second boundary stop time and the second boundary start time, and calculate the average of the differences for the same known punctuation mark to obtain a second pronunciation duration.

[0096] In one embodiment, the alignment acquisition module 540 is further configured to determine a pause threshold based on a first pronunciation duration and a second pronunciation duration; if the pronunciation duration corresponding to the second silence boundary information is greater than or equal to the pause threshold, then a preset first prosodic label is added at the second silence boundary information; and determine and add a preset second prosodic label or a preset third prosodic label to the first silence boundary information according to the punctuation type of a known punctuation mark, thereby obtaining the alignment information of the target text; wherein the first prosodic label is a phrase pause label, the second prosodic label is a clause pause label, and the third prosodic label is a whole sentence pause label.

[0097] In one embodiment, the alignment acquisition module 540 is further configured to add a second prosody tag at the first silence boundary information if the known punctuation type is a first punctuation; and add a third prosody tag at the first silence boundary information if the known punctuation type is a second punctuation, thereby obtaining the alignment information of the target text; wherein the first punctuation includes at least one of the following: comma, semicolon; and the second punctuation includes at least one of the following: question mark, exclamation mark, period.

[0098] In the above embodiments, by acquiring the phoneme sequence information of the target text and the audio information associated with the target text, and based on the phoneme sequence information and audio information, determining the phoneme boundary information of each phoneme in the phoneme sequence information and the silence segment information in the audio information, the silence segment information can be analyzed. Boundary correction can be performed on the phoneme boundary information and the silence segment information to obtain the first pronunciation duration of each phoneme and the second pronunciation duration of each known punctuation mark. Then, based on the first and second pronunciation durations, preset prosodic tags are added to the silence segment information, including the first silence boundary information of known punctuation marks and the second silence boundary information of unknown punctuation marks, ultimately obtaining higher-precision alignment information. Changing the analysis from text and audio to phoneme sequence and audio not only avoids errors in the text-to-phoneme conversion process but also avoids subsequent association errors. Furthermore, silence detection solves the problem of errors easily occurring at silence boundaries, increasing the reliability of the alignment results and improving the accuracy of alignment annotation.

[0099] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a method for acquiring text-aligned information. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0100] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0101] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0102] Obtain the phoneme sequence information of the target text, as well as the audio information associated with the target text;

[0103] Based on phoneme sequence information and audio information, the phoneme boundary information of each phoneme in the phoneme sequence information and the silence segment information in the audio information are determined; wherein, the silence segment information includes the first silence boundary information of known punctuation and the second silence boundary information of unknown punctuation.

[0104] Analyze the silence segment information to correct the boundaries of the phoneme boundary information and the silence segment information, and obtain the first pronunciation duration of each phoneme and the second pronunciation duration of each known punctuation mark;

[0105] Based on the first and second pronunciation durations, preset prosodic tags are added to the first and second silence boundary information to obtain the alignment information of the target text.

[0106] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0107] The system acquires the interrelated text and audio information to be processed; performs speech recognition on the audio information to obtain the speech-recognized text; analyzes the speech-recognized text and the text to be processed based on a preset edit distance algorithm, and corrects the text to be processed when it is determined that there is an error in the text to be processed to obtain the target text; performs phoneme conversion on the target text to obtain the phoneme sequence information of the target text.

[0108] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0109] Punctuation is removed from both the speech recognition text and the text to be processed, resulting in a first text corresponding to the speech recognition text and a second text corresponding to the text to be processed. The first and second texts are then converted into pinyin to obtain a first pinyin sequence corresponding to the first text and a second pinyin sequence corresponding to the second text. Based on a preset edit distance algorithm, the first and second pinyin sequences are analyzed to obtain the character error rate. Based on the character error rate, it is determined whether there are text errors in the text to be processed. If errors are found in the text to be processed, the text to be processed is corrected to obtain the target text.

[0110] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0111] Obtain the maximum sequence length of the first and second pinyin sequences; based on the maximum sequence length, determine the target character error rate threshold from multiple pre-stored character error rate thresholds; if the character error rate is greater than or equal to the target character error rate threshold, it is determined that the text to be processed has errors, and the text to be processed is corrected based on the speech recognition text to obtain the target text.

[0112] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0113] Phoneme sequence information and audio information are input into a trained alignment model, and the phoneme boundary information of each phoneme is output. Based on the phoneme boundary information, speech segments in the audio information that do not correspond to phonemes are identified. Punctuation marks contained in the target text are identified as known punctuation marks, and based on the known punctuation marks, the first silence boundary information in the speech segment is determined, and the second silence boundary information is obtained by filtering. The first silence boundary information and the second silence boundary information are used as silence segment information.

[0114] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0115] Based on a preset speech activity detection algorithm, the audio information is analyzed to obtain the silence segments in the audio information as the silence segment information to be identified; if the time boundary of the silence segment information to be identified is inconsistent with the silence segment information, the phoneme boundary information and the silence segment information are corrected to obtain the first pronunciation duration and the second pronunciation duration.

[0116] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0117] If the time boundaries of the silence segment information to be identified are inconsistent with the silence segment information, and there is an overlapping part of the speech segments, then the non-overlapping part of the speech segments, excluding the overlapping part, is analyzed within the maximum range of the speech segments. Based on the non-overlapping part of the speech segments, the phoneme boundary information and the silence segment information are corrected to obtain the first pronunciation duration and the second pronunciation duration. If the time boundaries of the silence segment information to be identified are inconsistent with the silence segment information, and there is no overlapping part of the speech segments, then the phoneme boundary information and the silence segment information are corrected based on the phoneme boundary information of the preceding adjacent phonemes of the silence segment information to obtain the first pronunciation duration and the second pronunciation duration.

[0118] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0119] Determine the corrected phoneme boundary information and the corrected silence segment information; wherein, the corrected phoneme boundary information includes the first boundary start time and the first boundary stop time, and the corrected silence segment information includes the second boundary start time and the second boundary stop time; calculate the difference between the first boundary stop time and the first boundary start time, and calculate the average of the differences for the same phoneme to obtain the first pronunciation duration; calculate the difference between the second boundary stop time and the second boundary start time, and calculate the average of the differences for the same known punctuation mark to obtain the second pronunciation duration.

[0120] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0121] Based on the first and second pronunciation durations, a pause threshold is determined; if the pronunciation duration corresponding to the second silence boundary information is greater than or equal to the pause threshold, a preset first prosodic label is added at the second silence boundary information; and a preset second or third prosodic label is added to the first silence boundary information according to the known punctuation type to obtain the alignment information of the target text; wherein, the first prosodic label is a phrase pause label, the second prosodic label is a clause pause label, and the third prosodic label is a whole sentence pause label.

[0122] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0123] If the punctuation type is known to be the first punctuation mark, then add a second prosody tag at the first silence boundary information; if the punctuation type is known to be the second punctuation mark, then add a third prosody tag at the first silence boundary information to obtain the alignment information of the target text; wherein, the first punctuation mark includes at least one of the following: comma, semicolon; the second punctuation mark includes at least one of the following: question mark, exclamation mark, period.

[0124] In the above embodiments, the analysis of phoneme sequences and audio is changed from analyzing text and audio. This not only avoids errors caused by the text-to-phoneme process, but also avoids subsequent correlation errors. In addition, silence detection solves the problem of errors easily occurring at silence boundaries, increases the reliability of alignment results, and improves the accuracy of alignment annotation.

[0125] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0126] Obtain the phoneme sequence information of the target text, as well as the audio information associated with the target text;

[0127] Based on phoneme sequence information and audio information, the phoneme boundary information of each phoneme in the phoneme sequence information and the silence segment information in the audio information are determined; wherein, the silence segment information includes the first silence boundary information of known punctuation and the second silence boundary information of unknown punctuation.

[0128] Analyze the silence segment information to correct the boundaries of the phoneme boundary information and the silence segment information, and obtain the first pronunciation duration of each phoneme and the second pronunciation duration of each known punctuation mark;

[0129] Based on the first and second pronunciation durations, preset prosodic tags are added to the first and second silence boundary information to obtain the alignment information of the target text.

[0130] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0131] The system acquires the interrelated text and audio information to be processed; performs speech recognition on the audio information to obtain the speech-recognized text; analyzes the speech-recognized text and the text to be processed based on a preset edit distance algorithm, and corrects the text to be processed when it is determined that there is an error in the text to be processed to obtain the target text; performs phoneme conversion on the target text to obtain the phoneme sequence information of the target text.

[0132] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0133] Punctuation is removed from both the speech recognition text and the text to be processed, resulting in a first text corresponding to the speech recognition text and a second text corresponding to the text to be processed. The first and second texts are then converted into pinyin to obtain a first pinyin sequence corresponding to the first text and a second pinyin sequence corresponding to the second text. Based on a preset edit distance algorithm, the first and second pinyin sequences are analyzed to obtain the character error rate. Based on the character error rate, it is determined whether there are text errors in the text to be processed. If errors are found in the text to be processed, the text to be processed is corrected to obtain the target text.

[0134] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0135] Obtain the maximum sequence length of the first and second pinyin sequences; based on the maximum sequence length, determine the target character error rate threshold from multiple pre-stored character error rate thresholds; if the character error rate is greater than or equal to the target character error rate threshold, it is determined that the text to be processed has errors, and the text to be processed is corrected based on the speech recognition text to obtain the target text.

[0136] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0137] Phoneme sequence information and audio information are input into a trained alignment model, and the phoneme boundary information of each phoneme is output. Based on the phoneme boundary information, speech segments in the audio information that do not correspond to phonemes are identified. Punctuation marks contained in the target text are identified as known punctuation marks, and based on the known punctuation marks, the first silence boundary information in the speech segment is determined, and the second silence boundary information is obtained by filtering. The first silence boundary information and the second silence boundary information are used as silence segment information.

[0138] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0139] Based on a preset speech activity detection algorithm, the audio information is analyzed to obtain the silence segments in the audio information as the silence segment information to be identified; if the time boundary of the silence segment information to be identified is inconsistent with the silence segment information, the phoneme boundary information and the silence segment information are corrected to obtain the first pronunciation duration and the second pronunciation duration.

[0140] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0141] If the time boundaries of the silence segment information to be identified are inconsistent with the silence segment information, and there is an overlapping part of the speech segments, then the non-overlapping part of the speech segments, excluding the overlapping part, is analyzed within the maximum range of the speech segments. Based on the non-overlapping part of the speech segments, the phoneme boundary information and the silence segment information are corrected to obtain the first pronunciation duration and the second pronunciation duration. If the time boundaries of the silence segment information to be identified are inconsistent with the silence segment information, and there is no overlapping part of the speech segments, then the phoneme boundary information and the silence segment information are corrected based on the phoneme boundary information of the preceding adjacent phonemes of the silence segment information to obtain the first pronunciation duration and the second pronunciation duration.

[0142] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0143] Determine the corrected phoneme boundary information and the corrected silence segment information; wherein, the corrected phoneme boundary information includes the first boundary start time and the first boundary stop time, and the corrected silence segment information includes the second boundary start time and the second boundary stop time; calculate the difference between the first boundary stop time and the first boundary start time, and calculate the average of the differences for the same phoneme to obtain the first pronunciation duration; calculate the difference between the second boundary stop time and the second boundary start time, and calculate the average of the differences for the same known punctuation mark to obtain the second pronunciation duration.

[0144] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0145] Based on the first and second pronunciation durations, a pause threshold is determined; if the pronunciation duration corresponding to the second silence boundary information is greater than or equal to the pause threshold, a preset first prosodic label is added at the second silence boundary information; and a preset second or third prosodic label is added to the first silence boundary information according to the known punctuation type to obtain the alignment information of the target text; wherein, the first prosodic label is a phrase pause label, the second prosodic label is a clause pause label, and the third prosodic label is a whole sentence pause label.

[0146] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0147] If the punctuation type is known to be the first punctuation mark, then add a second prosody tag at the first silence boundary information; if the punctuation type is known to be the second punctuation mark, then add a third prosody tag at the first silence boundary information to obtain the alignment information of the target text; wherein, the first punctuation mark includes at least one of the following: comma, semicolon; the second punctuation mark includes at least one of the following: question mark, exclamation mark, period.

[0148] In the above embodiments, the analysis of phoneme sequences and audio is changed from analyzing text and audio. This not only avoids errors caused by the text-to-phoneme process, but also avoids subsequent correlation errors. In addition, silence detection solves the problem of errors easily occurring at silence boundaries, increases the reliability of alignment results, and improves the accuracy of alignment annotation.

[0149] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0150] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0151] The foregoing has provided a detailed description of a text alignment information acquisition method, apparatus, and computer device provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for obtaining text alignment information, characterized in that, include: The method involves obtaining phoneme sequence information of a target text and audio information associated with the target text. The step of obtaining the phoneme sequence information of the target text includes: obtaining interrelated text to be processed and audio information, wherein the text to be processed refers to the original data without correction processing; performing speech recognition on the audio information to obtain speech-recognized text; analyzing the speech-recognized text and the text to be processed based on a preset edit distance algorithm, and correcting the text to be processed when an error is determined to exist, to obtain the target text; and performing phoneme conversion on the target text to obtain the phoneme sequence information of the target text. Based on the phoneme sequence information and the audio information, determine the phoneme boundary information of each phoneme in the phoneme sequence information, and the silence segment information in the audio information; wherein, the silence segment information includes the first silence boundary information of known punctuation and the second silence boundary information of unknown punctuation. The silence segment information is analyzed to perform boundary correction on the phoneme boundary information and the silence segment information, so as to obtain the first pronunciation duration of each phoneme and the second pronunciation duration of each known punctuation mark; Based on the first pronunciation duration and the second pronunciation duration, a pause threshold is determined; Based on the pause threshold, preset prosodic tags are added to the first silence boundary information and the second silence boundary information to obtain the alignment information of the target text.

2. The method as described in claim 1, characterized in that, The algorithm based on a preset edit distance analyzes the speech recognition text and the text to be processed, and corrects the text to be processed when an error is determined to exist, thereby obtaining the target text, including: Punctuation is removed from the speech-recognized text and the text to be processed, respectively, to obtain a first text corresponding to the speech-recognized text and a second text corresponding to the text to be processed; The first text and the second text are converted into pinyin to obtain a first pinyin sequence corresponding to the first text and a second pinyin sequence corresponding to the second text; Based on a preset edit distance algorithm, the first pinyin sequence and the second pinyin sequence are analyzed to obtain the character error rate; Based on the word error rate, it is determined whether there are text errors in the text to be processed. If it is determined that there are errors in the text to be processed, the text to be processed is corrected to obtain the target text.

3. The method as described in claim 2, characterized in that, The step of determining whether the text to be processed contains text errors based on the word error rate, and correcting the text to be processed when errors are determined to exist, to obtain the target text, includes: Obtain the maximum sequence length of the first pinyin sequence and the second pinyin sequence; Based on the maximum sequence length, the target word error rate threshold is determined from a plurality of pre-stored word error rate thresholds; If the word error rate is greater than or equal to the target word error rate threshold, it is determined that the text to be processed has an error, and the text to be processed is corrected based on the speech recognition text to obtain the target text.

4. The method as described in claim 1, characterized in that, The step of determining the phoneme boundary information of each phoneme in the phoneme sequence information and the silence segment information in the audio information based on the phoneme sequence information includes: The phoneme sequence information and the audio information are input into the trained alignment model, and the phoneme boundary information of each phoneme is output. Based on the phoneme boundary information, determine the speech segments in the audio information that do not correspond to the phoneme; The punctuation marks contained in the target text are identified as the known punctuation marks, and based on the known punctuation marks, the first silence boundary information in the speech segment is determined, and the second silence boundary information is obtained by filtering. The first silence boundary information and the second silence boundary information are used as the silence segment information.

5. The method as described in claim 1 or 4, characterized in that, The analysis of the silence segment information, to perform boundary correction on the phoneme boundary information and the silence segment information, to obtain the first pronunciation duration of each phoneme and the second pronunciation duration of each known punctuation mark, includes: Based on a preset voice activity detection algorithm, the audio information is analyzed to obtain silent segments in the audio information as silent segment information to be identified; If the time boundary of the silence segment information to be identified is inconsistent with that of the silence segment information, then the boundary correction is performed on the phoneme boundary information and the silence segment information to obtain the first pronunciation duration and the second pronunciation duration.

6. The method as described in claim 5, characterized in that, If the time boundary of the silence segment information to be identified is inconsistent with that of the silence segment information, then boundary correction is performed on the phoneme boundary information and the silence segment information to obtain the first pronunciation duration and the second pronunciation duration, including: If the time boundaries of the silence segment information to be identified are inconsistent with those of the silence segment information, and there is an overlapping part of the speech segments, then the non-overlapping part of the speech segments, excluding the overlapping part of the speech segments, is analyzed within the maximum range of the speech segments. Based on the non-overlapping part of the speech segments, the phoneme boundary information and the silence segment information are corrected to obtain the first pronunciation duration and the second pronunciation duration. If the time boundary of the silence segment information to be identified is inconsistent with that of the silence segment information, and there is no overlapping part of the speech segment, then based on the phoneme boundary information of the preceding adjacent phonemes of the silence segment information, the phoneme boundary information and the silence segment information are corrected to obtain the first pronunciation duration and the second pronunciation duration.

7. The method as described in claim 6, characterized in that, The step of performing boundary correction on the phoneme boundary information and the silence segment information to obtain the first pronunciation duration and the second pronunciation duration includes: Determine the corrected phoneme boundary information and the corrected silence segment information; wherein, the corrected phoneme boundary information includes a first boundary start time and a first boundary stop time, and the corrected silence segment information includes a second boundary start time and a second boundary stop time. Calculate the difference between the first boundary stop time and the first boundary start time, and calculate the average of the differences for the same phoneme to obtain the first pronunciation duration; Calculate the difference between the second boundary stop time and the second boundary start time, and calculate the average of the differences for the same known punctuation marks to obtain the second pronunciation duration.

8. The method as described in claim 1, characterized in that, The step of adding preset prosodic tags to the first silence boundary information and the second silence boundary information based on the pause threshold to obtain the alignment information of the target text includes: If the pronunciation duration corresponding to the second silence boundary information is greater than or equal to the pause threshold, then a preset first prosodic label is added at the location of the second silence boundary information; and Based on the known punctuation type, a preset second or third prosodic tag is added to the first silence boundary information to obtain the alignment information of the target text; The first prosodic tag is a phrase pause tag, the second prosodic tag is a clause pause tag, and the third prosodic tag is a whole sentence pause tag.

9. The method as described in claim 8, characterized in that, The step of determining and adding a preset second or third prosodic tag to the first silence boundary information based on the known punctuation type to obtain the alignment information of the target text includes: If the known punctuation mark is of the first punctuation mark type, then the second prosody tag is added at the location of the first silence boundary information; If the known punctuation mark is of the second punctuation mark type, then the third prosody tag is added at the first silence boundary information to obtain the alignment information of the target text; The first punctuation mark includes at least one of the following: comma, semicolon; the second punctuation mark includes at least one of the following: question mark, exclamation mark, period.

10. A text alignment information acquisition device, characterized in that, include: An information acquisition module is used to acquire phoneme sequence information of a target text and audio information associated with the target text. The step of acquiring the phoneme sequence information of the target text includes: acquiring interrelated text to be processed and audio information, wherein the text to be processed refers to the original data without correction processing; performing speech recognition on the audio information to obtain speech-recognized text; analyzing the speech-recognized text and the text to be processed based on a preset edit distance algorithm, so as to correct the text to be processed when an error is determined to exist, thereby obtaining the target text; and performing phoneme conversion on the target text to obtain the phoneme sequence information of the target text. A boundary determination module is used to determine the phoneme boundary information of each phoneme in the phoneme sequence information and the silence segment information in the audio information based on the phoneme sequence information and the audio information; wherein, the silence segment information includes first silence boundary information of known punctuation and second silence boundary information of unknown punctuation. A boundary correction module is used to analyze the silence segment information to correct the boundary information of the phonemes and the silence segment information, so as to obtain the first pronunciation duration of each phoneme and the second pronunciation duration of each known punctuation mark. The alignment acquisition module is used to determine a pause threshold based on the first pronunciation duration and the second pronunciation duration; and to add preset prosodic tags to the first silence boundary information and the second silence boundary information based on the pause threshold to obtain the alignment information of the target text.

11. A computer device, characterized in that, include: One or more processors; Memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the text alignment information acquisition method of any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, It stores a computer program, which is loaded by a processor to perform the steps in the text alignment information acquisition method according to any one of claims 1 to 9.

Images