A punctuation prediction method, device, apparatus and storage medium

Abstract

The application provides a punctuation prediction method, device and equipment and a storage medium, wherein the method comprises: obtaining a recognition result as a target text every time a speech recognition system recognizes a speech segment in target speech and outputs a recognition result; obtaining local context information of each word in the target text, and predicting punctuation in the target text based on the local context information of each word in the target text to obtain a prediction result as a preliminary punctuation prediction result of the target text; and determining a final punctuation prediction result of the target text based on a recognition result attribute of the target text and the preliminary punctuation prediction result of the target text. The punctuation prediction method provided by the application can be used to predict punctuation of a target text, and can obtain a relatively accurate and stable punctuation prediction result. The punctuation prediction method provided by the application has high prediction efficiency and good user experience.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a punctuation prediction method, apparatus, device, and storage medium. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence technology, speech recognition technology has made great breakthroughs and has been widely applied in all aspects of society, bringing great convenience to people's production and life. Simultaneous interpretation, live-streaming e-commerce, and video subtitle generation all have a demand for long-form speech-to-text conversion, which demonstrates the broad market potential of speech recognition technology.

[0003] Standard speech recognition systems typically generate text without any punctuation marks from the speech, resulting in poor readability and negatively impacting the performance of downstream tasks (such as translation, entity recognition, and other natural language processing tasks). Predicting punctuation marks in the recognized text can solve this problem, and how to predict punctuation marks in the recognition result is a pressing issue that needs to be addressed. Summary of the Invention

[0004] In view of this, the present invention provides a punctuation prediction method, apparatus, device, and storage medium for predicting punctuation in speech recognition results, the technical solution of which is as follows:

[0005] A punctuation prediction method, comprising:

[0006] Whenever the speech recognition system recognizes a speech segment in the target speech and outputs a recognition result, the recognition result is obtained as the target text;

[0007] The local context information of each word in the target text is obtained, and based on the local context information of each word in the target text, the punctuation in the target text is predicted. The prediction result is used as the preliminary punctuation prediction result of the target text.

[0008] Based on the recognition result attributes of the target text and the preliminary punctuation prediction result of the target text, the final punctuation prediction result of the target text is determined.

[0009] Optionally, obtaining the local context information of each word in the target text includes:

[0010] Obtain the representation vector of each word in the target text, wherein the representation vector of a word contains the semantic information of the word itself and the position information of the word in the target text;

[0011] For each word in the target text, a context vector corresponding to the word is determined based on the word's representation vector, or based on the representation vectors of the first M words and / or the representation vectors of the last N words and the word's representation vector. The context vector corresponding to the word can represent the local context information of the word, and M and N are both preset integers.

[0012] Optionally, determining the final punctuation prediction result of the target text based on the recognition result attributes of the target text and the preliminary punctuation prediction result of the target text includes:

[0013] If the target text is an intermediate recognition result of the speech segment, then the punctuation after the last word in the preliminary punctuation prediction result of the target text is removed, and the prediction result after removing the punctuation is determined as the final punctuation prediction result of the target text.

[0014] If the target text is the final recognition result of the speech segment, then the preliminary punctuation prediction result of the target text is determined as the final punctuation prediction result of the target text.

[0015] Optionally, determining the final punctuation prediction result of the target text based on the recognition result attributes of the target text and the preliminary punctuation prediction result of the target text includes:

[0016] If the target text is the final recognition result of the speech segment, and the speech segment is the last speech segment of the target speech, then the preliminary punctuation prediction result of the target text is determined as the final punctuation prediction result of the target text.

[0017] If the target text is not the final recognition result of the last speech segment of the target speech, then the punctuation after the last word in the preliminary punctuation prediction result of the target text is removed, and the prediction result after removing the punctuation is determined as the final punctuation prediction result of the target text.

[0018] If the target text is the first intermediate recognition result of the speech segment, and the speech segment has forward-adjacent speech segments, the method further includes:

[0019] The first N words of the target text are concatenated with the last recognition result of the forward adjacent speech segment to obtain the concatenated text;

[0020] Based on the concatenated text, punctuation prediction is performed again on the last recognition result of the forward adjacent speech segment.

[0021] Optionally, predicting punctuation in the target text based on the local context information of each word in the target text includes:

[0022] For each word in the target text:

[0023] Based on the local context information of the word, predict the probability that the word is followed by punctuation.

[0024] If the probability of a word being followed by punctuation is greater than a preset probability threshold, then based on the local context information of the word, the probability of the punctuation following the word being one of the preset punctuation marks is further predicted.

[0025] Based on the probability that the punctuation mark following the word is one of the set punctuation marks, the punctuation mark following the word is determined from each set punctuation mark.

[0026] Optionally, obtaining the local context information of each word in the target text and predicting punctuation in the target text based on the local context information of each word includes:

[0027] Based on the pre-trained punctuation prediction model, the context vectors corresponding to each word in the target text are determined, wherein the context vector corresponding to a word can represent the local context information of the word;

[0028] Based on the context vectors corresponding to each word in the target text, the punctuation in the target text is predicted using the punctuation prediction model.

[0029] The punctuation prediction model is trained using multiple punctuated training texts from the training dataset.

[0030] Optionally, the training dataset is obtained by mixing first training text annotated with punctuation from a first training data subset with second training text annotated with punctuation from a second training data subset;

[0031] The second training text in the second training data subset is obtained by processing the first training text in the first training data subset. The first training text in the first training data subset is a semantically complete and noise-free training text, while the second training text in the second training data subset is a semantically incomplete and / or noise-containing training text.

[0032] Optionally, the process of processing the first training text in the first training data subset to obtain the second training text includes:

[0033] For each first training text in all or part of the first training text in the first training data subset:

[0034] Randomly select segmentation points from the first training text;

[0035] Based on the selected segmentation points, a portion of the content in the first training text is deleted, and the first training text after deleting the portion of the content is used as the second training text.

[0036] Optionally, the process of processing the first training text in the first training data subset to obtain the second training text includes:

[0037] Extract a portion of the first training text from the first training data subset;

[0038] Replace some words in the first extracted training text with random words;

[0039] And / or, replace some words in the extracted first training text with words from the corresponding acoustic nearest word set;

[0040] And / or, insert random words after a subset of words in the extracted first training text;

[0041] And / or, delete some words from the extracted first training text.

[0042] A punctuation prediction device includes: a text acquisition module, a preliminary punctuation prediction result determination module, and a final punctuation prediction result determination module;

[0043] The text acquisition module is used to acquire the recognition result as the target text whenever the speech recognition system recognizes a speech segment in the target speech and outputs a recognition result.

[0044] The preliminary punctuation prediction result determination module is used to obtain the local context information of each word in the target text, and predict the punctuation in the target text based on the local context information of each word in the target text. The prediction result is used as the preliminary punctuation prediction result of the target text.

[0045] The final punctuation prediction result determination module is used to determine the final punctuation prediction result of the target text based on the recognition result attributes of the target text and the preliminary punctuation prediction result of the target text.

[0046] Optionally, the preliminary punctuation prediction result determination module is specifically used for:

[0047] Based on the pre-trained punctuation prediction model, the context vectors corresponding to each word in the target text are determined, and based on the context vectors corresponding to each word in the target text, the punctuation in the target text is predicted using the punctuation prediction model.

[0048] Among them, the context vector corresponding to a word can represent the local context information of that word;

[0049] The punctuation prediction model is trained using multiple training texts annotated with punctuation from the training dataset. The training dataset is obtained by mixing the first training texts annotated with punctuation from the first training data subset with the second training texts annotated with punctuation from the second training data subset. The second training texts in the second training data subset are obtained by processing the first training texts in the first training data subset. The first training texts in the first training data subset are semantically complete and noise-free training texts, while the second training texts in the second training data subset are semantically incomplete and / or contain noise.

[0050] A punctuation prediction device includes: a memory and a processor;

[0051] The memory is used to store programs;

[0052] The processor is configured to execute the program to implement each step of the punctuation prediction method described above.

[0053] A readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the punctuation prediction method described in any of the preceding claims.

[0054] As can be seen from the above scheme, the punctuation prediction method, apparatus, device, and storage medium provided by the present invention, whenever the speech recognition system recognizes a speech segment in the target speech and outputs a recognition result, acquires the recognition result as the target text, then acquires the local context information of each word in the target text, and predicts the punctuation in the target text based on the local context information of each word in the target text. The obtained prediction result is used as the preliminary punctuation prediction result of the target text. Finally, based on the recognition result attributes of the target text and the preliminary punctuation prediction result of the target text, the final punctuation prediction result of the target text is determined. The punctuation prediction method provided by the present invention determines the preliminary punctuation prediction result of the target text based on the local context information of each word in the target text. On this basis, it combines the recognition result attributes of the target text to determine the final prediction result of the target text. This prediction method can provide relatively accurate and stable punctuation prediction results, and has high prediction efficiency and a good user experience. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0056] Figure 1This is a schematic diagram of the hardware architecture involved in the present invention;

[0057] Figure 2 This is a flowchart illustrating the punctuation prediction method provided in an embodiment of the present invention.

[0058] Figure 3 This is a schematic diagram illustrating the process of predicting punctuation marks in target text based on a punctuation prediction model, provided in an embodiment of the present invention.

[0059] Figure 4 This is a schematic diagram illustrating an example of the punctuation prediction model provided in an embodiment of the present invention.

[0060] Figure 5 This is a schematic diagram of the punctuation prediction device provided in an embodiment of the present invention;

[0061] Figure 6 This is a schematic diagram of the punctuation prediction device provided in an embodiment of the present invention. Detailed Implementation

[0062] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] To achieve punctuation prediction, the inventors conducted research. The initial approach was to pre-train a punctuation prediction model using semantically complete, noise-free training text with punctuation annotations at the sentence level. After obtaining the recognition result of a speech segment from a speech recognition system, the punctuation prediction model was used to obtain the complete contextual information of each word in the recognition result, and then the punctuation in the recognition result was predicted based on this contextual information. It should be noted that when recognizing a speech segment, a speech endpoint detection system is typically used first to detect speech endpoints, breaking long speech segments into shorter segments, which are then input into the speech recognition system for recognition.

[0064] In scenarios with high real-time requirements, speech recognition systems typically employ a word-by-word recognition method to identify speech segments. This method results in multiple intermediate recognition results and a final recognition result during the recognition process. Below is an example of multiple recognition results generated by a speech recognition system when recognizing a speech segment:

[0065] Yes (1st intermediate recognition result) → Very (2nd intermediate recognition result) → Very large from → Very large from 5 kilometers to 5 → Very large from 5 kilometers to 500 meters → Very large from 5 kilometers to 500 meters (6th intermediate recognition result) → Very large from 5 kilometers to 500 meters coverage → Very large from 5 kilometers to 500 meters coverage → Very large from 5 kilometers to 500 meters coverage requirement (9th intermediate recognition result) → Very large from 5 kilometers to 500 meters coverage requirement this → Very large from 5 kilometers to 500 meters coverage requirement this is (11th intermediate recognition result) → Very large from 5 kilometers to 500 meters coverage requirement this is the problem (final recognition result)

[0066] When performing punctuation prediction, each time the speech recognition system outputs a recognition result for a speech segment in a speech, punctuation prediction is performed based on the punctuation prediction model.

[0067] The inventors of this case, through research on the aforementioned punctuation prediction scheme, discovered that as recognition progresses, the number of characters in the recognition result increases, and the following text becomes longer. Often, textual changes lead to semantic shifts. Predicting punctuation based on the complete contextual information of each word in the recognition result is highly likely to result in repeated changes to punctuation at the same position. Furthermore, the aforementioned punctuation prediction scheme is limited by the influence of speech endpoint detection. Each text used for punctuation prediction is a short, independent sentence. This means that when the speaker's pace is slow or there are long pauses, the speech endpoint detection system is highly likely to break the sentence in the middle. Because the punctuation prediction model is trained using semantically complete training text, it is prone to predicting a period after the last word of each sentence, even if the sentence is not finished. In scenarios with high real-time requirements, the word-by-word incremental recognition characteristic results in most recognition results being short, semantically incomplete sentences. These short sentences are often identified as having a period at the end, but when new words are added to the end of the sentence, these periods are often erased.

[0068] Taking the following semantically incomplete text as an example, the prediction result of the above punctuation prediction scheme is as follows:

[0069] Sometimes → Sometimes I don't know. → Sometimes I don't say, he says → Sometimes I don't say, he says, "Xiao Ming." → Sometimes I don't say, he says, "Xiao Ming, I → Sometimes I don't say, he says, "Xiao Ming, I hope → Sometimes I don't say, he says, "Xiao Ming, I hope you → Sometimes I don't say, he says, "Xiao Ming, I hope you rest." → Sometimes I don't say, he says, "Xiao Ming, I hope you rest but I don't want you to." → Sometimes I don't say, he says, "Xiao Ming, I hope you rest, but I don't want you to stop streaming." → Sometimes I don't say, he says, "Xiao Ming, I hope you rest, but I don't want you to stop streaming, so..."

[0070] In view of the many shortcomings of the above-mentioned punctuation prediction schemes, the inventors of this case conducted further research and, through continuous research, finally proposed a punctuation prediction method with better performance. This punctuation prediction method can give relatively accurate and stable punctuation prediction results for the text to be predicted.

[0071] Before introducing the punctuation prediction method provided by this invention, the hardware architecture involved in this invention will be described first.

[0072] In one possible implementation, such as Figure 1 As shown, the hardware architecture involved in this invention may include: electronic device 101 and server 102.

[0073] For example, electronic device 101 can be any electronic product that can interact with a user, such as PC, laptop, tablet, PDA, mobile phone, wearable device, etc.

[0074] It should be noted that, Figure 1 This is just one example; there can be many types of electronic devices, not limited to... Figure 1 The laptop in the middle.

[0075] For example, server 102 can be a single server, a server cluster consisting of multiple servers, or a cloud computing server center. Server 102 may include processors, memory, and network interfaces, etc.

[0076] For example, electronic device 101 can establish a connection and communicate with server 102 through a wireless communication network; for example, electronic device 101 can establish a connection and communicate with server 102 through a wired communication network.

[0077] For example, electronic device 101 acquires voice and sends the acquired voice to server 102. Server 102 first performs voice endpoint detection on the received voice to segment the voice into voice segments, then identifies the voice segments, and then predicts the punctuation on the recognition results according to the punctuation prediction method provided by the present invention.

[0078] In another possible implementation, the hardware architecture involved in this invention may include an electronic device. The electronic device is one with strong data processing capabilities.

[0079] For example, an electronic device can be any electronic product that can interact with a user through one or more means such as a keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device, such as a PC, mobile phone, laptop, tablet computer, PDA, personal computer, etc.

[0080] The electronic device acquires speech, then performs speech endpoint detection on the acquired speech to segment the speech into speech segments, then recognizes the speech segments to obtain the recognition results, and finally predicts the punctuation on the recognition results according to the punctuation prediction method provided by the present invention.

[0081] Those skilled in the art should understand that the above-described electronic devices and servers are merely examples, and other existing or future electronic devices or servers that are applicable to this invention should also be included within the scope of protection of this invention, and are hereby incorporated by reference.

[0082] The punctuation prediction method provided by the present invention will be described in the following embodiments.

[0083] First Embodiment

[0084] Please see Figure 2 The diagram illustrates a flowchart of a punctuation prediction method provided in an embodiment of the present invention. This method may include:

[0085] Step S201: Whenever the speech recognition system recognizes a speech segment in the target speech and outputs a recognition result, the recognition result is obtained as the target text.

[0086] The target text may be an intermediate recognition result of a speech segment in the target speech, or it may be the final recognition result.

[0087] It should be noted that the speech recognition system uses word-by-word recognition to identify speech segments in the target speech. This recognition method results in multiple recognition results (multiple intermediate recognition results and one final recognition result) during the recognition of a speech segment. The multiple recognition results are output one by one, and each output recognition result has new content compared to the previous recognition result. Each time the speech recognition system outputs a recognition result, it can obtain that recognition result and use it as the target text for punctuation prediction.

[0088] Step S202: Obtain the local context information of each word in the target text, and based on the local context information of each word in the target text, predict the punctuation in the target text. The prediction result is used as the preliminary punctuation prediction result of the target text.

[0089] The process of obtaining local context information for each word in the target text may include:

[0090] Step a1: Obtain the representation vector of each word in the target text.

[0091] The representation vector of a word contains the semantic information of the word itself and the position information of the word in the target text.

[0092] Specifically, the target text is segmented to obtain each word in the target text. For each word in the target text, a vector representing the semantic information of the word itself is obtained, and a vector representing the position information of the word in the target text is also obtained. Then, these two vectors are fused (for example, by adding the two vectors together), and the fused vector is used as the representation vector of the word.

[0093] Step a2: For each word in the target text, determine the context vector corresponding to the word based on the word's representation vector, or based on the representation vectors of the first M words and / or the representation vectors of the last N words and the word's representation vector.

[0094] Among them, the context vector corresponding to a word can represent the local context information of that word.

[0095] The context vector corresponding to a word can be determined based on the word's representation vector. Alternatively, the context vector corresponding to a word can be determined based on the representation vectors of the first M words and the word's representation vector. Or, the context vector corresponding to a word can be determined based on the representation vectors of the last N words and the word's representation vector. Furthermore, the context vector corresponding to a word can be determined based on the representation vectors of the first M words, the last N words, and the word's representation vector.

[0096] In order to achieve better prediction results, this invention preferably determines the context vector corresponding to the word based on the representation vectors of the first M words, the representation vectors of the last N words, and the representation vector of the word itself. The context vector determined in this way contains the contextual relationship information between the word and the first M words and the last N words.

[0097] In this embodiment, when determining the context vector corresponding to a word, it does not focus on the entire context, but rather on a local context. For example, it only focuses on the M words preceding the word and the N words following the word. It should be noted that if there are no words after a word (for example, the last word of the target text), then only the M words preceding the word are considered; if there are no words before a word (for example, the first word of the target text), then only the N words following the word are considered; if there are fewer than M words before a word, then all words preceding the word are considered; if there are fewer than N words following a word, then all words following the word are considered; if there are no words before or after a word, then only the word itself is considered.

[0098] Both M and N mentioned above are preset integers. For example, M can be 42 and N can be 2. It should be noted that this embodiment does not limit the specific values ​​of M and N, and the specific values ​​of M and N can be set according to the specific application scenario.

[0099] When predicting punctuation in a target text based on the local context information of each word, for each word in the target text, the local context information of the word is used to predict whether there is punctuation after the word, and if so, what kind of punctuation it is, so as to obtain the punctuation prediction result of the word. After obtaining the punctuation prediction results of each word in the target text, the punctuation in the target text can be determined based on the punctuation prediction results of each word in the target text.

[0100] In the process of realizing this case, the inventors discovered that for punctuation prediction tasks, the more distant the context relationship, the smaller the impact on punctuation prediction, and the closer the context relationship, the greater the impact on punctuation prediction. When a word is far enough away from the target word (the word whose punctuation is to be predicted), it can be basically determined that the word has zero impact on the target word. In addition, since the speech recognition system uses word-by-word recognition to recognize speech segments, for the first recognized word, as the recognition progresses, its context becomes longer and longer. If the entire context is considered during prediction, it is very likely that the punctuation at the same position will change repeatedly.

[0101] To address the above findings, to ensure that punctuation doesn't change with increasing context, we can predict punctuation based solely on the preceding text. To ensure accuracy, we can predict punctuation based solely on the following text. However, ignoring the following text significantly reduces the accuracy of punctuation prediction, while ignoring the preceding text significantly reduces its stability. To simultaneously ensure both accuracy and stability, and while maintaining prediction efficiency, for each word in the target text, when predicting the punctuation after that word, we preferably consider both the preceding and following text. Given that distant context has a relatively small impact on punctuation, we limit the scope of the preceding and following text (only considering the M words preceding the word and the N words following it). To quickly provide accurate and stable punctuation prediction results, the preceding text can be longer, and the following text shorter. This strategy allows for a balance between punctuation accuracy and stability, while also maintaining prediction efficiency.

[0102] Step S203: Based on the recognition result attributes of the target text and the preliminary punctuation prediction result of the target text, determine the final punctuation prediction result of the target text.

[0103] The recognition result attribute of the target text is either "intermediate recognition result" or "final recognition result".

[0104] To avoid the punctuation after the last word of the target text changing repeatedly as subsequent recognition content increases, this embodiment does not directly use the punctuation prediction result as the final punctuation prediction result of the target text after obtaining the punctuation prediction result through step S202. Instead, based on the punctuation prediction result obtained through step S202, it further combines the recognition result attributes of the target text to determine the final punctuation prediction result of the target text.

[0105] The punctuation prediction method provided in this invention involves acquiring the recognition result as the target text whenever the speech recognition system recognizes a speech segment in the target speech and outputs a recognition result. Then, it acquires the local context information of each word in the target text and predicts the punctuation in the target text based on this local context information. The obtained prediction result serves as the preliminary punctuation prediction result for the target text. Finally, based on the recognition result attributes of the target text and the preliminary punctuation prediction result, the final punctuation prediction result of the target text is determined. This punctuation prediction method, based on the local context information of each word in the target text to determine the preliminary punctuation prediction result, and then combining this with the recognition result attributes of the target text to determine the final prediction result, provides relatively accurate and stable punctuation prediction results with high prediction efficiency and a good user experience.

[0106] Second Embodiment

[0107] This embodiment describes the implementation process of "step S202: obtaining local context information of each word in the target text, and predicting punctuation in the target text based on the local context information of each word in the target text" in the above embodiment.

[0108] In one possible implementation, step S202 can be implemented based on a model. However, this embodiment is not limited to this, meaning that this embodiment does not limit the specific implementation form of step S202. The following section will describe the specific implementation process of step S202 based on a model, using the example of implementing step S202 based on a model.

[0109] Please see Figure 3 This diagram illustrates a process for predicting punctuation marks in target text based on a punctuation prediction model, which may include:

[0110] Step S301: Based on the pre-trained punctuation prediction model, determine the context vector corresponding to each word in the target text.

[0111] For example, such as Figure 4As shown, the punctuation prediction model in this embodiment may include a representation vector acquisition module 401 and an attention module 402. The representation vector acquisition module 401 can acquire the representation vector of each word in the target text. After obtaining the representation vectors of each word in the target text, the representation vectors of each word in the target text can be input into the attention module 402. For each word in the target text, the attention module 402 determines the context vector corresponding to the word based on the word's representation vector, or based on the representation vectors of the preceding M words and / or the representation vectors of the following N words and the word's representation vector. The representation vector of a word contains the semantic information of the word itself and the positional information of the word in the target text, while the context vector corresponding to a word can represent the local context information of the word.

[0112] When the representation vector acquisition module 401 based on the punctuation prediction model acquires the representation vector of each word in the target text, for each word in the target text, it can acquire the word embedding vector (a vector representing the semantic information of the word itself) and the word position embedding vector (a vector representing the position information of the word in the target text) based on the representation vector acquisition module 401. The acquired word embedding vector and position embedding vector are then fused, and the fused vector is used as the representation vector of the word. When acquiring the word embedding vector of a word, the word ID of the word can be searched in a pre-prepared dictionary, and then the word embedding vector of the word can be obtained based on the word ID. It should be noted that if the word ID of the word is not found, the word embedding vector can be obtained by searching for the word ID in a pre-prepared dictionary. <unk>The word ID is replaced by the word ID. When obtaining the position embedding vector of a word, the position embedding vector of the word can be obtained through sine and cosine position encoding.

[0113] Preferably, for each word in the target text, when the attention module 402 based on the punctuation prediction model determines the context vector corresponding to the word, the attention module 402 only focuses on a portion of the words in the target text, such as the first M words and the last N words of the word. For example, M is 42, N is 2, and the attention module 402 can include 6 attention layers (such as self-attention layers). The left field of view of the first attention layer is 7 and the right field of view is 1, that is, the first attention layer only focuses on the 7 words above and the 1 word below. The left field of view of the second attention layer is 7 and the right field of view is 1, that is, the second attention layer also only focuses on the 7 words above and the 1 word below. The left field of view of the third to sixth attention layers is 7 and the right field of view is 0, that is, the third to sixth attention layers only focus on the 7 words above. By superimposing the left field of view and the right field of view of the 6 attention layers, for each word in the target text, the 6 attention layers can focus on 42 (6*7) words above and 2 words below. For each word in the target text, in order to make the attention module 402 of the punctuation prediction model only focus on the first M words and the last N words of the word, words that do not need to be focused on can be masked out based on the masking mechanism.

[0114] Step S302: Based on the context vectors corresponding to each word in the target text, predict the punctuation in the target text using the punctuation prediction model.

[0115] Based on the context vectors corresponding to each word in the target text, and using a punctuation prediction model, there are multiple ways to predict punctuation in the target text. This embodiment provides the following two optional implementation methods:

[0116] The first implementation method: For each word in the target text, firstly, based on the context vector corresponding to the word, predict the probability P that there will be no punctuation after the word using a punctuation prediction model. 10 And the probability P that the punctuation following the word is one of the specified punctuation marks. 11 (The probability that the punctuation mark following the word is the first specified punctuation mark) ~ P 1n (The probability that the punctuation mark following the word is the nth specified punctuation mark, where n is the number of specified punctuation marks), P 10 +P 11 +…+P 1n =1, the probability P that there is no punctuation after obtaining the word. 10 And the probability P that the punctuation following the word is one of the specified punctuation marks. 11 ~P 1n Then, based on the probability P... 10 P 11 ~P 1n Determine if there is punctuation after the word, and if so, which of the various punctuation marks to use. For example, starting from P... 10 P 11 ~P 1n If the maximum probability is P, then... 10 If the probability is P, then it is determined that there is no punctuation after the word. 1i If i is an integer in [1, n], then the punctuation mark after the word is determined to be the i-th set punctuation mark.

[0117] The second implementation method: For each word in the target text, firstly, based on the context vector corresponding to the word, predict the probability P that there will be punctuation after the word using a punctuation prediction model. 20 Then determine the probability P that the word is followed by punctuation. 20 Is it greater than the preset probability threshold P? th If not, then it is determined that there is no punctuation after the word; if so, then it is determined that there is punctuation after the word. Further, based on the context vector corresponding to the word, the probability P of the punctuation following the word being one of the specified punctuation marks is predicted. 21 (The probability that the punctuation mark following the word is the first specified punctuation mark) ~ P 2n (The probability that the punctuation mark following the word is the nth predetermined punctuation mark), and finally, based on the probability P that the punctuation mark following the word is any of the predetermined punctuation marks. 21 ~P 2n Determine the punctuation mark following the word from among the various set punctuation marks, for example, from P 21 ~P 2n The maximum probability is determined, and the punctuation mark corresponding to the maximum probability is set as the punctuation mark after the word.

[0118] like Figure 4 As shown, in addition to the representation vector acquisition module 401 and attention module 402 mentioned above, the punctuation prediction model also includes a probability prediction module 403, which may include a linear classification layer and a softmax layer.

[0119] In the first implementation described above, the output of the attention module 402, which is the context vector corresponding to each word in the target text, is input to the linear classification layer of the probability prediction module 403. All outputs of the linear classification layer are input to the softmax layer. The softmax layer outputs the predicted probability for each word in the target text. The predicted probability for a word is the probability that there is no punctuation after that word and the probability that the punctuation after that word is one of the specified punctuation marks. Assuming there are n specified punctuation marks, the linear classification layer mentioned above includes n+1 nodes. For each word in the target text, the context vector corresponding to that word is input to the linear classification layer, which will produce the outputs of n+1 nodes of the linear classification layer. The outputs of n+1 nodes of the linear classification layer are input to the softmax layer, which outputs n+1 probabilities, namely the probability that there is no punctuation after that word and the probability that the punctuation after that word is one of the specified punctuation marks.

[0120] For the second implementation described above, the output of the attention module 402 is the context vector corresponding to each word in the target text, which is then input into the linear classification layer of the probability prediction module 403. Assuming there are n punctuation marks, the linear classification layer includes n+1 nodes. For each word in the target text, the context vector corresponding to that word is input into the linear classification layer, resulting in the output of the n+1 nodes of the linear classification layer. The output of the first node of the linear classification layer is used as the probability that there is punctuation after the word. If the probability that there is punctuation after the word is greater than a preset probability threshold, the outputs of the other n nodes of the linear classification layer are input into the softmax layer. The softmax layer outputs n probabilities, which are the probabilities of the punctuation marks after the word.

[0121] The second implementation method described above can flexibly control the number of punctuation marks by adjusting the probability threshold, thereby adapting to different usage scenarios and user habits.

[0122] The punctuation prediction model in this embodiment is trained using multiple annotated training texts from the training dataset. The training process of the punctuation prediction model will be described below.

[0123] Before introducing the training process of the punctuation prediction model, we will first introduce the training dataset of the punctuation prediction model.

[0124] In one possible implementation, the training texts in the training dataset of the punctuation prediction model are all semantically complete and noise-free texts.

[0125] Considering that the recognition results output by speech recognition systems are mostly semantically incomplete texts that may contain recognition errors, in order to train a punctuation prediction model with better performance, in another possible implementation, the training dataset can include semantically complete and noise-free training texts annotated with punctuation, as well as semantically incomplete and / or noisy training texts annotated with punctuation.

[0126] Optionally, the training dataset can be obtained by mixing first training text annotated with punctuation from a first training data subset with second training text annotated with punctuation from a second training data subset. The first training text in the first training data subset is semantically complete and noise-free, while the second training text in the second training data subset is semantically incomplete and / or noisy. The second training text in the second training data subset is obtained by processing the first training text in the first training data subset.

[0127] Optionally, several texts with punctuation can be crawled directly from the Internet. A first training data subset is constructed based on the crawled texts. After obtaining the first training data subset, the second training text is obtained by processing the first training text in the first training data subset.

[0128] Optionally, the first training text in the first training data subset can be randomly segmented and / or noise-added to obtain the second training text. It should be noted that the purpose of randomly segmenting the first training text is to process the semantically complete first training text into semantically incomplete text, and the purpose of adding noise to the first training text is to introduce noise into the clean first training text to simulate speech recognition errors.

[0129] The process of randomly segmenting the first training text in the first training data subset to obtain the second training text may include: for each first training text in all or part of the first training text in the first training data subset, randomly selecting a segmentation point from the first training text, deleting part of the content in the first training text based on the selected segmentation point, and the first training text after deleting part of the content is semantically incomplete text, which is used as the second training text.

[0130] For example, the first training text is "A batch of public service positions will be developed in the fields of grassroots social management and public services to provide transitional placement for college graduates facing difficulties." For a complete sentence, a segmentation point can be randomly selected. Assuming the segmentation point is after "difficult," the part after the segmentation point can be deleted, resulting in "A batch of public service positions will be developed in the fields of grassroots social management and public services to provide transitional placement for college graduates facing difficulties." Alternatively, the part before the segmentation point can be deleted, resulting in "Transitional placement for college graduates." For a complete sentence, multiple segmentation points can also be randomly selected. For example, if two segmentation points are selected, in the first training text above, assuming the part after "grassroots social management" is the first segmentation point and the part after "graduates" is the second segmentation point, the part before the first segmentation point and the part after the second segmentation point can be deleted, resulting in "A batch of public service positions will be developed in the fields of public services to provide transitional placement for college graduates facing difficulties." Optionally, after segmenting a complete sentence, the minimum number of whole words is 5, and the maximum is 100.

[0131] Optionally, for a first training text, multiple different second training texts can be obtained by deleting different parts based on the selected segmentation points. For a first training text, multiple different second training texts can also be obtained by selecting different numbers and / or different positions of segmentation points.

[0132] The process of adding noise to the first training text in the first training data subset to obtain the second training text may include: extracting a portion of the first training text from the first training data subset; replacing some words in the extracted first training text with random words; and / or replacing some words in the extracted first training text with words from the corresponding acoustic nearest word set; and / or inserting random words after some words in the extracted first training text; and / or deleting some words in the extracted first training text.

[0133] For example, 30% of the first training text can be extracted from the first training data subset. For 10% of the words in the extracted first training text, 10% of the words are replaced with random words, 40% of the words are replaced with words from the corresponding acoustic nearest word set, 25% of the words are deleted, and 25% of the words are replaced with random words. For instance, if the first training data subset includes 100 first training texts, then 30 first training texts are extracted from the first training data subset. Assuming that the 30 first training texts include 200 words, for 20 of the 200 words, 2 words are replaced with random words, 8 words are replaced with words from the corresponding acoustic nearest word set, 5 words are deleted, and 5 words are replaced with random words.

[0134] It should be noted that the above implementation method adds noise to all the first training texts extracted from the first training data subset as a whole. In addition to this method, other methods can also be used. For example, noise can be added to each first training text extracted from the first training data subset. That is, for each extracted first training text, some words in the first training text can be replaced with random words, and / or, some words in the first training text can be replaced with words from the corresponding acoustic nearest word set, and / or, random words can be inserted after some words in the first training text, and / or, some words in the first training text can be deleted.

[0135] It should be noted that when replacing a word in the first training text, if there is punctuation after the word, the punctuation is retained after replacement. When deleting a word in the first training text, if there is punctuation after the word, the punctuation is moved to the word preceding the word. When inserting a random word after a word in the first training text, if there is punctuation after the word, the punctuation after the word is moved to the word following the inserted word.

[0136] When adding noise to the first practice text, an acoustic nearest word set is used. The process of obtaining the acoustic nearest word set includes: obtaining a pronunciation dictionary, which includes multiple words and their corresponding phoneme sequences; for each word, determining the first acoustic similarity value and the second acoustic similarity value between the word and each of the other words based on the phoneme sequence corresponding to the word and the phoneme sequences corresponding to each of the other words; and forming the acoustic nearest word set corresponding to the word by selecting the X (e.g., 10) words with the highest first acoustic similarity value and the X words with the highest second acoustic similarity value.

[0137] For a word A, its first acoustic similarity value with another word B is Corr = S / G, where S is the number of identical phonemes in the phoneme sequence corresponding to word A and the phoneme sequence corresponding to word B, and G is the total number of phonemes in the phoneme sequence corresponding to word B; for a word A, its second acoustic similarity value with another word B is Acc = (SI) / G, where I is the number of redundant phonemes in the phoneme sequence corresponding to word A compared to the phoneme sequence corresponding to word B.

[0138] The training process of training a punctuation prediction model using the training dataset will be described next.

[0139] The process of training a punctuation prediction model using a training dataset can include: obtaining training text from the training dataset; determining the context vectors corresponding to each word in the training text based on the punctuation prediction model; predicting punctuation in the training text based on the context vectors corresponding to each word in the training text; determining the prediction loss (e.g., cross-entropy loss) of the punctuation prediction model based on the punctuation predicted by the punctuation prediction model for the training text and the punctuation annotated in the training text; and updating the parameters of the punctuation prediction model based on the prediction loss of the punctuation prediction model. The punctuation prediction model is trained iteratively multiple times in the above manner until the training termination condition is met (e.g., model convergence, or reaching a preset number of training iterations).

[0140] The specific process of determining the context vector corresponding to each word in the training text based on the punctuation prediction model, and predicting the punctuation in the training text based on the context vector corresponding to each word in the training text, is similar to the specific implementation process of steps S301 to S302 above, and will not be described in detail here.

[0141] After training, the performance of the punctuation prediction model can be evaluated using the evaluation dataset. The evaluation dataset includes multiple evaluation data, each of which includes a first text and a second text annotated with punctuation. The first text is the speech recognition result, which may contain recognition errors, and the second text is the standard text corresponding to the first text, which is the speech-annotated text.

[0142] During evaluation, for each piece of evaluation data in the evaluation dataset, the first text is input into the punctuation prediction model for punctuation prediction to obtain the punctuation prediction result of the first text. If the first text is inconsistent with the second text, the first text and the second text are aligned by edit distance. Specifically, for insertion errors, the extra words in the first text are deleted; for deletion errors, the extra words in the second text are deleted; for substitution errors, the incorrect words in the first text are retained. For example, if the first text is "天气真粗去玩你说好吗” and the second text is "天气真好出去玩好吗”, they are aligned by edit distance. After alignment, the first text becomes "天气真粗去玩好吗” and the second text becomes "天气真出去玩好吗”. Based on the alignment situation of the first text and the second text, the punctuation prediction result of the first text and the punctuation annotation information of the second text are adjusted. Suppose the prediction result of the first text in the above example is "o o!o o o o o o?” and the annotation information of the second text in the above example is "o o o!o o o o?”. After adjustment based on the alignment situation, the prediction result of the first text is adjusted to "o o!o o o o?” and the annotation information of the second text is adjusted to "o o!o o o o?”. Based on the adjusted punctuation prediction result and the adjusted punctuation annotation information, the prediction effect characterization value of the punctuation prediction model on this piece of evaluation data is determined (which reflects the consistency between the annotation prediction result and the annotation). If the first text is consistent with the second text, the prediction accuracy characterization value of the punctuation prediction model on this piece of evaluation data is directly determined according to the punctuation prediction result of the first text and the punctuation annotation information of the second text. After obtaining the prediction effect characterization values of the punctuation prediction model on each piece of evaluation data in the evaluation dataset, the performance of the punctuation prediction model is determined according to the prediction effect characterization values of the punctuation prediction model on each piece of evaluation data in the evaluation dataset.

[0143] Third Embodiment

[0144] This embodiment introduces the specific implementation process of "Step S103: Based on the recognition result attributes of the target text and the preliminary punctuation prediction result of the target text, determine the final punctuation prediction result" in the above embodiment.

[0145] There are multiple ways to determine the final punctuation prediction result of the target text based on the recognition result attributes of the target text and the preliminary punctuation prediction result of the target text. This embodiment provides the following two optional implementation methods:

[0146] The first implementation method: If the target text is the intermediate recognition result of a speech segment in the target speech, then remove the punctuation after the last word in the preliminary punctuation prediction result of the target text, and determine the prediction result after removing the punctuation as the final punctuation prediction result of the target text; if the target text is the final recognition result of the current speech segment, then determine the preliminary punctuation prediction result of the target text as the final punctuation prediction result of the target text.

[0147] In the case where the target text is the intermediate recognition result of a speech segment in the target speech, since there are no other words after the last word of the target text, therefore, when predicting the punctuation after the last word, only the context information of this word is used, and the punctuation predicted only using the context information of this word may be inaccurate. To avoid the situation where the punctuation after this word jumps repeatedly, the punctuation after this word can be removed.

[0148] Exemplarily, the target text is "is very large, from five kilometers to five hundred meters", which is the first recognition result of a speech segment in the target speech. The word segmentation result of the target text is "is very large, from five kilometers to five hundred meters", and the preliminary punctuation prediction result of the target text is "is very large, from five kilometers to five hundred meters,". Since the punctuation after the last word "meters" is predicted only based on the context information of "meters", therefore, the punctuation after "meters" may be inaccurate. Therefore, remove the punctuation after "meters", and use "is very large, from five kilometers to five hundred meters" as the final punctuation prediction result of the first recognition result "is very large, from five kilometers to five hundred meters" of the speech segment.

[0149] Exemplarily, the target text is "is a very large demand for coverage from five kilometers to five hundred meters", which is the final recognition result of a speech segment in the target speech. The preliminary punctuation prediction result of the target text is "is very large, from five kilometers to five hundred meters, such a demand for coverage.", and since the target text is the final recognition result of the speech segment, therefore, directly determine "is very large, from five kilometers to five hundred meters, such a demand for coverage." as the final punctuation prediction result of the target text.

[0150] For the above implementation method, since the punctuation of the last word in the final recognition result of the speech segment is predicted only using the context information, therefore, the prediction result of the last word in the final recognition result of the speech segment may be inaccurate. For the last word in the final recognition result of the speech segment, in order to obtain an accurate punctuation prediction result, this embodiment provides a second implementation method for determining the final punctuation prediction result of the target text based on the recognition result attribute of the target text and the preliminary punctuation prediction result of the target text:

[0151] If the target text is the final recognition result of a speech segment in the target speech, and that speech segment is the last speech segment of the target speech, then the preliminary punctuation prediction result of the target text is determined as the final punctuation prediction result of the target text; if the target text is not the final recognition result of the last speech segment of the target speech, then the punctuation after the last word in the preliminary punctuation prediction result of the target text is removed, and the prediction result after removing the punctuation is determined as the final punctuation prediction result of the target text.

[0152] If the target text is the first intermediate recognition result of a speech segment in the target speech, and the speech segment has a forward adjacent speech segment, the first N words of the target text are concatenated with the last recognition result of the forward adjacent speech segment of the speech segment to obtain the concatenated text; based on the concatenated text, the punctuation prediction of the forward adjacent speech segment of the speech segment is performed again.

[0153] For example, the target text is "Needs to solve this problem that has been bothering us for a long time", and its word segmentation result is "Needs to solve this problem that has been bothering us for a long time". The target text is the first recognition result of a speech segment in the target speech. The final recognition result of the forward adjacent speech segment of this speech segment is "A very large coverage requirement from five kilometers to five hundred meters". Then, the first N words of the target text are concatenated with "A very large coverage requirement from five kilometers to five hundred meters". Assuming N is 2, the first two words "Needs to solve" in the target text are concatenated after "A very large coverage requirement from five kilometers to five hundred meters". Based on the concatenated text, punctuation prediction is performed again on the phrase "a very large coverage requirement from five kilometers to five hundred meters." Thus, the accurate punctuation can be predicted for the last word in this phrase. Simultaneously, punctuation prediction is performed on the target text. Assuming the initial prediction result for the target text is "It needs to be solved; this has been troubling us for a long time.", since the target text is the first recognition result of the speech segment, the punctuation after the last word in the initial prediction result is removed, and "It needs to be solved; this has been troubling us for a long time" is taken as the final punctuation prediction result for the target text.

[0154] Fourth embodiment

[0155] Based on the above embodiments, this embodiment further illustrates the process of predicting punctuation for each recognition result of each speech segment of the target speech.

[0156] The process of predicting punctuation for each recognition result of each segment of the target speech includes:

[0157] (1) Obtain the first recognition result result11 of the first speech segment VAD1 of the target speech.

[0158] (2) Obtain the local context information of each word contained in result11, predict the punctuation in result11 based on the local context information of each word contained in result11, so as to obtain the preliminary punctuation prediction result of result11, remove the punctuation after the last word in the preliminary punctuation prediction result of result11, and obtain the final punctuation prediction result of result1.

[0159] Optionally, preliminary punctuation prediction results for result11 can be obtained based on the punctuation prediction model described above.

[0160] (3) Obtain the second recognition result of the speech segment VAD1, result12.

[0161] (4) Obtain the local context information of each word contained in result12, predict the punctuation in result12 based on the local context information of each word contained in result12, so as to obtain the preliminary punctuation prediction result of result12. Assuming that result12 is the final recognition result of speech segment VAD1, remove the punctuation after the last word in the preliminary punctuation prediction result of result2.

[0162] (5) Obtain the first recognition result result21 of the second speech segment VAD2 of the target speech.

[0163] (6) Concatenate the first N words (e.g. the first two words) of result21 with the last recognition result of VAD1, result12. Based on the concatenated text, re-predict the punctuation of result12. Also, obtain the local context information of each word contained in result21. Based on the local context information of each word contained in result21, predict the punctuation in result21 to obtain the preliminary punctuation prediction result of result21. Remove the punctuation after the last word in the preliminary punctuation prediction result of result21 to obtain the final punctuation prediction result of result21.

[0164] (7) Obtain the second recognition result of the speech segment VAD2, result22.

[0165] (8) Obtain the local context information of each word contained in result22, and predict the punctuation in result22 based on the local context information of each word contained in result22 to obtain the preliminary punctuation prediction result of result22. Remove the punctuation after the last word in the preliminary punctuation prediction result of result22 to obtain the final punctuation prediction result of result22.

[0166] (9) Obtain the third recognition result of speech segment VAD2, result23.

[0167] (10) Obtain the local context information of each word contained in result23, and predict the punctuation in result23 based on the local context information of each word contained in result23 to obtain the preliminary punctuation prediction result of result23. Assuming that result23 is the final recognition result of VAD2, remove the punctuation after the last word in the preliminary punctuation prediction result of result23 to obtain the final punctuation prediction result of result23.

[0168] (11) Obtain the first recognition result of the speech segment VAD3, result31.

[0169] (12) Concatenate the first N words (e.g. the first two words) of result31 with the last recognition result of VAD2, result23. Based on the concatenated text, re-predict the punctuation of result23. Also, obtain the local context information of each word contained in result31. Based on the local context information of each word contained in result31, predict the punctuation in result31 to obtain the preliminary punctuation prediction result of result31. Remove the punctuation after the last word in the preliminary punctuation prediction result of result31 to obtain the final punctuation prediction result of result31.

[0170] ...

[0171] Obtain the last recognition result of the last speech segment of the target speech, obtain the local context information of each word contained in the recognition result, and predict the punctuation in the recognition result based on the local context information of each word contained in the recognition result. The obtained prediction result is used as the final punctuation prediction result of the recognition result.

[0172] As can be seen from the above embodiments, the punctuation prediction method provided by the present invention has the following advantages: First, when predicting the punctuation information of each word in the target text, it does not utilize the global context information of each word, but rather utilizes local context information, such as the contextual relationship between each word and the preceding M words (e.g., the preceding 42 words) and the following N words (e.g., the following 2 words). Utilizing the local context information of each word for punctuation prediction can balance the accuracy and stability of punctuation prediction. Furthermore, compared to using global context information for prediction, it can improve the efficiency of punctuation prediction. Second, for the last word of the last recognition result of a speech segment, to avoid repeated changes in punctuation due to the addition of subsequent text, after obtaining the preliminary punctuation prediction result, the punctuation after the last word in the preliminary punctuation prediction result is removed. After obtaining the first recognition result of the next speech segment, punctuation prediction is performed again for the last recognition result of that speech segment. Third, based on each word in the target text... When predicting probabilities using local contextual information, an adjustable probability threshold is introduced. For each word in the target text, the probability of punctuation following the word is first obtained. After determining that the probability of punctuation following the word is greater than the probability threshold, the probability of the punctuation following the word being any of the set punctuation marks is further obtained, thereby determining the punctuation marks following the word. The introduction of the probability threshold allows the punctuation prediction method provided by this invention to adapt to different usage scenarios and user habits (adjusting the probability threshold can change the number of punctuation marks). Fourth, considering that the speech recognition results output by the speech recognition system are mostly semantically incomplete and may contain text with recognition errors, in order to enable the punctuation prediction model to have better prediction performance, during training, the training data of the punctuation prediction model includes not only semantically complete and noise-free training text, but also semantically incomplete and / or noisy training text. Training with semantically incomplete and / or noisy training text makes the punctuation prediction model more adaptable to the actual situation and enhances the robustness of the punctuation prediction model.

[0173] Fifth embodiment

[0174] This invention also provides a punctuation prediction device. The punctuation prediction device provided in this invention will be described below. The punctuation prediction device described below can be referred to in correspondence with the punctuation prediction method described above.

[0175] Please see Figure 5 The diagram shows a schematic of the structure of the punctuation prediction device provided in an embodiment of the present invention, which may include: a text acquisition module 501, a preliminary punctuation prediction result determination module 502, and a final punctuation prediction result determination module 503.

[0176] The text acquisition module 501 is used to acquire the recognition result as the target text whenever the speech recognition system recognizes a speech segment in the target speech and outputs a recognition result.

[0177] The preliminary punctuation prediction result determination module 502 is used to obtain the local context information of each word in the target text, and predict the punctuation in the target text based on the local context information of each word in the target text, and the obtained prediction result is used as the preliminary punctuation prediction result of the target text.

[0178] The final punctuation prediction result determination module 503 is used to determine the final punctuation prediction result of the target text based on the recognition result attributes of the target text and the preliminary punctuation prediction result of the target text.

[0179] Optionally, the preliminary punctuation prediction result determination module 502, when acquiring the local context information of each word in the target text, is specifically used for:

[0180] Obtain the representation vector of each word in the target text, wherein the representation vector of a word contains the semantic information of the word itself and the position information of the word in the target text;

[0181] For each word in the target text, a context vector corresponding to the word is determined based on the word's representation vector, or based on the representation vectors of the first M words and / or the representation vectors of the last N words and the word's representation vector. The context vector corresponding to the word can represent the local context information of the word, and M and N are both preset integers.

[0182] Optionally, when determining the final punctuation prediction result based on the recognition result attributes of the target text and the preliminary punctuation prediction result of the target text, the final punctuation prediction result determination module 503 is specifically used for:

[0183] If the target text is an intermediate recognition result of the speech segment, then the punctuation after the last word in the preliminary punctuation prediction result of the target text is removed, and the prediction result after removing the punctuation is determined as the final punctuation prediction result of the target text.

[0184] If the target text is the final recognition result of the speech segment, then the preliminary punctuation prediction result of the target text is determined as the final punctuation prediction result of the target text.

[0185] Optionally, when determining the final punctuation prediction result based on the recognition result attributes of the target text and the preliminary punctuation prediction result of the target text, the final punctuation prediction result determination module 503 is specifically used for:

[0186] If the target text is the final recognition result of the speech segment, and the speech segment is the last speech segment of the target speech, then the preliminary punctuation prediction result of the target text is determined as the final punctuation prediction result of the target text.

[0187] If the target text is not the final recognition result of the last speech segment of the target speech, then the punctuation after the last word in the preliminary punctuation prediction result of the target text is removed, and the prediction result after removing the punctuation is determined as the final punctuation prediction result of the target text.

[0188] Optionally, the punctuation prediction device provided in this embodiment of the invention may further include a text splicing module and a punctuation prediction module.

[0189] The text concatenation module is used to concatenate the first N words of the target text with the last recognition result of the preceding adjacent speech segment when the target text is the first intermediate recognition result of the speech segment and the speech segment has a preceding adjacent speech segment, to obtain the concatenated text.

[0190] The preliminary punctuation prediction result determination module 502 is also used to re-predict punctuation based on the spliced ​​text and the last recognition result of the forward adjacent speech segment.

[0191] Optionally, when the preliminary punctuation prediction result determination module 502 predicts punctuation in the target text based on the local context information of each word in the target text, it is specifically used for...

[0192] For each word in the target text:

[0193] Based on the local context information of the word, predict the probability that the word is followed by punctuation.

[0194] If the probability of a word being followed by punctuation is greater than a preset probability threshold, then based on the local context information of the word, the probability of the punctuation following the word being one of the preset punctuation marks is further predicted.

[0195] Based on the probability that the punctuation mark following the word is one of the set punctuation marks, the punctuation mark following the word is determined from each set punctuation mark.

[0196] Optionally, when the preliminary punctuation prediction result determination module 502 acquires the local context information of each word in the target text and predicts the punctuation in the target text based on the local context information of each word in the target text, it is specifically used for:

[0197] Based on the pre-trained punctuation prediction model, the context vectors corresponding to each word in the target text are determined, wherein the context vector corresponding to a word can represent the local context information of the word;

[0198] Based on the context vectors corresponding to each word in the target text, the punctuation in the target text is predicted using the punctuation prediction model.

[0199] The punctuation prediction model is trained using multiple punctuated training texts from the training dataset.

[0200] Optionally, the training dataset is obtained by mixing first training text annotated with punctuation from a first training data subset with second training text annotated with punctuation from a second training data subset;

[0201] The second training text in the second training data subset is obtained by processing the first training text in the first training data subset. The first training text in the first training data subset is a semantically complete and noise-free training text, while the second training text in the second training data subset is a semantically incomplete and / or noise-containing training text.

[0202] Optionally, the punctuation prediction method provided in this embodiment of the invention may further include: a first text processing module. The first text processing module is used to process a first training text in the first training data subset to obtain a second training text. Specifically, when processing the first training text in the first training data subset, the first text processing module is used to:

[0203] For each first training text in all or part of the first training text in the first training data subset:

[0204] Randomly select segmentation points from the first training text;

[0205] Based on the selected segmentation points, a portion of the content in the first training text is deleted, and the first training text after deleting the portion of the content is used as the second training text.

[0206] Optionally, the punctuation prediction method provided in this embodiment of the invention may further include: a second text processing module. When processing the first training text in the first training data subset, the second text processing module is specifically used for:

[0207] Extract a portion of the first training text from the first training data subset;

[0208] Replace some words in the first extracted training text with random words;

[0209] And / or, replace some words in the extracted first training text with words from the corresponding acoustic nearest word set;

[0210] And / or, insert random words after a subset of words in the extracted first training text;

[0211] And / or, delete some words from the extracted first training text.

[0212] The punctuation prediction device provided in this embodiment of the invention can quickly predict relatively accurate and stable punctuation marks whenever the speech recognition system recognizes a speech segment in the target speech and outputs a recognition result, resulting in a better user experience.

[0213] Sixth Embodiment

[0214] This invention also provides a punctuation prediction device; please refer to [link / reference]. Figure 6 The diagram shows the structure of the punctuation prediction device, which may include: at least one processor 601, at least one communication interface 602, at least one memory 603 and at least one communication bus 604.

[0215] In this embodiment of the invention, the number of processor 601, communication interface 602, memory 603, and communication bus 604 is at least one, and processor 601, communication interface 602, and memory 603 communicate with each other through communication bus 604.

[0216] The processor 601 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0217] The memory 603 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0218] The memory stores a program, which the processor can call. The program is used for:

[0219] Whenever the speech recognition system recognizes a speech segment in the target speech and outputs a recognition result, the recognition result is obtained as the target text;

[0220] The local context information of each word in the target text is obtained, and based on the local context information of each word in the target text, the punctuation in the target text is predicted. The prediction result is used as the preliminary punctuation prediction result of the target text.

[0221] Based on the recognition result attributes of the target text and the preliminary punctuation prediction result of the target text, the final punctuation prediction result of the target text is determined.

[0222] Optionally, the refined and extended functions of the program can be found in the description above.

[0223] Seventh Embodiment

[0224] This invention also provides a readable storage medium that stores a program suitable for execution by a processor, the program being used for:

[0225] Whenever the speech recognition system recognizes a speech segment in the target speech and outputs a recognition result, the recognition result is obtained as the target text;

[0226] The local context information of each word in the target text is obtained, and based on the local context information of each word in the target text, the punctuation in the target text is predicted. The prediction result is used as the preliminary punctuation prediction result of the target text.

[0227] Based on the recognition result attributes of the target text and the preliminary punctuation prediction result of the target text, the final punctuation prediction result of the target text is determined.

[0228] Optionally, the refined and extended functions of the program can be found in the description above.

[0229] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0230] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0231] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.< / unk>

Claims

1. A punctuation prediction method, characterized in that, include: Whenever the speech recognition system recognizes a speech segment in the target speech and outputs a recognition result, the recognition result is obtained as the target text; The local context information of each word in the target text is obtained, and based on the local context information of each word in the target text, the punctuation in the target text is predicted. The prediction result is used as the preliminary punctuation prediction result of the target text. Based on the recognition result attributes of the target text and the preliminary punctuation prediction results of the target text, the final punctuation prediction result of the target text is determined; the recognition result attribute is one of the intermediate recognition result and the final recognition result; The step of obtaining the local context information of each word in the target text includes: Obtain the representation vector of each word in the target text, wherein the representation vector of a word contains the semantic information of the word itself and the position information of the word in the target text; For each word in the target text, a context vector corresponding to the word is determined based on the word's representation vector, or based on the representation vectors of the first M words and / or the representation vectors of the last N words and the word's representation vector. The context vector corresponding to the word can represent the local context information of the word, and M and N are both preset integers.

2. The punctuation prediction method according to claim 1, characterized in that, The step of determining the final punctuation prediction result of the target text based on the recognition result attributes of the target text and the preliminary punctuation prediction result of the target text includes: If the target text is an intermediate recognition result of the speech segment, then the punctuation after the last word in the preliminary punctuation prediction result of the target text is removed, and the prediction result after removing the punctuation is determined as the final punctuation prediction result of the target text. If the target text is the final recognition result of the speech segment, then the preliminary punctuation prediction result of the target text is determined as the final punctuation prediction result of the target text.

3. The punctuation prediction method according to claim 1, characterized in that, The step of determining the final punctuation prediction result of the target text based on the recognition result attributes of the target text and the preliminary punctuation prediction result of the target text includes: If the target text is the final recognition result of the speech segment, and the speech segment is the last speech segment of the target speech, then the preliminary punctuation prediction result of the target text is determined as the final punctuation prediction result of the target text. If the target text is not the final recognition result of the last speech segment of the target speech, then the punctuation after the last word in the preliminary punctuation prediction result of the target text is removed, and the prediction result after removing the punctuation is determined as the final punctuation prediction result of the target text. If the target text is the first intermediate recognition result of the speech segment, and the speech segment has forward-adjacent speech segments, the method further includes: The first N words of the target text are concatenated with the last recognition result of the forward adjacent speech segment to obtain the concatenated text; Based on the concatenated text, punctuation prediction is performed again on the last recognition result of the forward adjacent speech segment.

4. The punctuation prediction method according to claim 1, characterized in that, The step of predicting punctuation in the target text based on the local context information of each word in the target text includes: For each word in the target text: Based on the local context information of the word, predict the probability that the word is followed by punctuation. If the probability of a word being followed by punctuation is greater than a preset probability threshold, then based on the local context information of the word, the probability of the punctuation following the word being one of the preset punctuation marks is further predicted. Based on the probability that the punctuation mark following the word is one of the set punctuation marks, the punctuation mark following the word is determined from each set punctuation mark.

5. The punctuation prediction method according to any one of claims 1 to 4, characterized in that, The step of obtaining local context information of each word in the target text and predicting punctuation in the target text based on the local context information of each word includes: Based on the pre-trained punctuation prediction model, the context vectors corresponding to each word in the target text are determined. Based on the context vectors corresponding to each word in the target text, the punctuation in the target text is predicted using the punctuation prediction model. The punctuation prediction model is trained using multiple punctuated training texts from the training dataset.

6. The punctuation prediction method according to claim 5, characterized in that, The training dataset is obtained by mixing the first training text labeled with punctuation from the first training data subset with the second training text labeled with punctuation from the second training data subset; The second training text in the second training data subset is obtained by processing the first training text in the first training data subset. The first training text in the first training data subset is a semantically complete and noise-free training text, while the second training text in the second training data subset is a semantically incomplete and / or noise-containing training text.

7. The punctuation prediction method according to claim 6, characterized in that, The process of processing the first training text in the first training data subset to obtain the second training text includes: For each first training text in all or part of the first training text in the first training data subset: Randomly select segmentation points from the first training text; Based on the selected segmentation points, a portion of the content in the first training text is deleted, and the first training text after deleting the portion of the content is used as the second training text.

8. The punctuation prediction method according to claim 6, characterized in that, The process of processing the first training text in the first training data subset to obtain the second training text includes: Extract a portion of the first training text from the first training data subset; Replace some words in the first extracted training text with random words; And / or, replace some words in the extracted first training text with words from the corresponding acoustic nearest word set; And / or, insert random words after a subset of words in the extracted first training text; And / or, delete some words from the extracted first training text.

9. A punctuation prediction device, characterized in that, include: The module includes a text acquisition module, a preliminary punctuation prediction result determination module, and a final punctuation prediction result determination module. The text acquisition module is used to acquire the recognition result as the target text whenever the speech recognition system recognizes a speech segment in the target speech and outputs a recognition result. The preliminary punctuation prediction result determination module is used to obtain the local context information of each word in the target text, and predict the punctuation in the target text based on the local context information of each word in the target text. The prediction result is used as the preliminary punctuation prediction result of the target text. The final punctuation prediction result determination module is used to determine the final punctuation prediction result of the target text based on the recognition result attributes of the target text and the preliminary punctuation prediction result of the target text; the recognition result attribute is one of the intermediate recognition result and the final recognition result; The preliminary punctuation prediction result determination module, when acquiring the local context information of each word in the target text, is specifically used for: Obtain the representation vector of each word in the target text, wherein the representation vector of a word contains the semantic information of the word itself and the position information of the word in the target text; For each word in the target text, a context vector corresponding to the word is determined based on the word's representation vector, or based on the representation vectors of the first M words and / or the representation vectors of the last N words and the word's representation vector. The context vector corresponding to the word can represent the local context information of the word, and M and N are both preset integers.

10. The punctuation prediction device according to claim 9, characterized in that, The preliminary punctuation prediction result determination module is specifically used for: Based on the pre-trained punctuation prediction model, the context vectors corresponding to each word in the target text are determined, and based on the context vectors corresponding to each word in the target text, the punctuation in the target text is predicted using the punctuation prediction model. The punctuation prediction model is trained using multiple training texts annotated with punctuation from the training dataset. The training dataset is obtained by mixing the first training texts annotated with punctuation from the first training data subset with the second training texts annotated with punctuation from the second training data subset. The second training texts in the second training data subset are obtained by processing the first training texts in the first training data subset. The first training texts in the first training data subset are semantically complete and noise-free training texts, while the second training texts in the second training data subset are semantically incomplete and / or contain noise.

11. A punctuation prediction device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement the various steps of the punctuation prediction method as described in any one of claims 1 to 8.

12. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements each step of the punctuation prediction method as described in any one of claims 1 to 8.

Images