Punctuation prediction method, device, computer equipment, readable storage medium and program product for hybrid speech recognition text

Abstract

The present application relates to a punctuation prediction method, device, computer equipment, readable storage medium and program product for mixed speech recognition text. The text segments corresponding to the languages ​​in the speech are identified by a language text recognition model, and multiple punctuation prediction models are trained by text segment samples of multiple languages. The punctuation prediction models of the corresponding languages ​​output corresponding punctuation prediction results according to the input text segments, and the corresponding punctuation information is added to the corresponding text segments according to the punctuation prediction results. Compared with the traditional method of adding punctuation according to a fixed language, this scheme trains multiple punctuation prediction models by using text segment samples of multiple languages, recognizes text segments of the corresponding languages ​​for speech mixed with multiple languages, and predicts punctuation for the text segments according to the languages ​​of the text segments by using the corresponding punctuation prediction models, thereby improving the accuracy of punctuation prediction for recognition text mixed with multiple speech.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a method, apparatus, computer device, computer-readable storage medium and computer program product for punctuation prediction of mixed speech recognition text. Background Art

[0002] When processing electricity business involving users who speak Cantonese, it is necessary to perform text recognition and punctuation on the user's voice. However, there is a mixture of Mandarin and Cantonese in the user's voice, and it is necessary to punctuate the recognized text of Cantonese and Mandarin in the voice to distinguish Cantonese and Mandarin speech. Currently, the recognized text of Cantonese and Mandarin mixed speech is usually punctuated by the annotation method of Mandarin text. However, the grammatical rules and phrase usage of Cantonese are quite different from those of Mandarin. Punctuation by the annotation rules of Mandarin text will reduce the accuracy of punctuation.

[0003] Therefore, the current punctuation method for mixed speech recognition text has the defect of low accuracy. Summary of the invention

[0004] Based on this, it is necessary to provide a punctuation prediction method, device, computer equipment, computer-readable storage medium and computer program product for hybrid speech recognition text that can improve accuracy in order to solve the above technical problems.

[0005] In a first aspect, the present application provides a punctuation prediction method for a hybrid speech recognition text, the method comprising:

[0006] Acquire speech to be predicted, input the speech into a trained language text recognition model, and use the language text recognition model to recognize each text segment corresponding to each language in the speech;

[0007] Input each of the text segments into each trained punctuation prediction model of the corresponding language, and each of the punctuation prediction models outputs corresponding punctuation prediction results according to the input text segment; wherein each of the punctuation prediction models is trained based on text segment samples of each language;

[0008] According to each of the punctuation prediction results, corresponding punctuation information is added to each of the corresponding text segments.

[0009] In one embodiment, the step of inputting the speech into a trained language text recognition model, and allowing the language text recognition model to recognize each text segment corresponding to each language in the speech, includes:

[0010] The speech is input into a trained language-text recognition model, and the language-text recognition model recognizes each first text segment corresponding to the speech segment of the first language in the speech and each second text segment corresponding to the speech segment of the second language in the speech through a speech activity detection system.

[0011] In one embodiment, the identifying, by the language text recognition model through a voice activity detection system, each first text segment corresponding to a speech segment of the first language in the speech, and each second text segment corresponding to a speech segment of the second language, comprises:

[0012] The language text recognition model recognizes each voice character in the voice through a voice activity detection system, and obtains a voice character sequence based on a plurality of the voice characters;

[0013] Taking the first phonetic character in the phonetic character sequence as the current phonetic character, and obtaining the probability that the current phonetic character belongs to the first language according to the comparison result between the current phonetic character and the text dictionary corresponding to the first language;

[0014] If the probability is greater than or equal to a preset threshold, it is determined that the current voice character belongs to the first language; otherwise, it is determined that the current voice character belongs to the second language;

[0015] The step of taking the next voice character corresponding to the current voice character in the voice character sequence as a new current voice character, and returning the comparison result between the current voice character and the text dictionary corresponding to the first language to obtain the probability that the current voice character belongs to the first language, until each voice character in the voice character sequence is recognized;

[0016] According to each continuous voice character in the voice character sequence belonging to the first language, each first text segment corresponding to the first language is obtained; according to each continuous voice character in the voice character sequence belonging to the second language, each second text segment corresponding to the second language is obtained.

[0017] In one embodiment, the step of inputting each of the text segments into each of the trained punctuation prediction models of the corresponding language, and having each of the punctuation prediction models output corresponding punctuation prediction results according to the input text segments, comprises:

[0018] For each text segment, determine a corresponding trained target punctuation prediction model according to the language corresponding to the text segment, and input the text segment into the target punctuation prediction model;

[0019] The target punctuation prediction model extracts a text feature sequence corresponding to the text segment, and matches the text feature sequence with a preset punctuation set through a conditional random field to obtain each target character in the text segment that needs to be marked with punctuation and a target punctuation corresponding to each target character;

[0020] According to each of the target characters and each of the target punctuation marks corresponding to each of the target characters, a punctuation prediction result corresponding to the text segment is output.

[0021] In one embodiment, the method further comprises:

[0022] Obtaining each text segment sample corresponding to each language and a punctuation prediction model to be trained corresponding to each of the languages;

[0023] For the punctuation prediction model corresponding to each language, a text segment sample corresponding to the language is input into the punctuation prediction model, and the punctuation prediction model extracts a text feature sample sequence corresponding to the text segment sample through a conditional random field;

[0024] Matching the text feature sample sequence with a preset punctuation set, and determining a target preset punctuation corresponding to each text feature in the text feature sample sequence from each preset punctuation in the preset punctuation set;

[0025] Determine a corresponding probability distribution function according to each of the text features and each of the target preset punctuation marks corresponding to each of the text features;

[0026] The model parameters of the punctuation prediction model are adjusted according to the probability distribution function until a preset training end condition is met, thereby obtaining a trained punctuation prediction model corresponding to the language.

[0027] In one embodiment, adjusting the model parameters of the punctuation prediction model according to the probability distribution function until a preset training end condition is met to obtain a trained punctuation prediction model corresponding to the language includes:

[0028] Determining a corresponding log-likelihood function according to the inverse of the logarithm of the probability distribution function;

[0029] The model parameters of the punctuation prediction model are adjusted according to the output value of the log-likelihood function until the output value of the log-likelihood function is minimized, thereby obtaining a trained punctuation prediction model corresponding to the language.

[0030] In a second aspect, the present application provides a punctuation prediction device for a hybrid speech recognition text, the device comprising:

[0031] An acquisition module, used for acquiring speech to be predicted, inputting the speech into a trained language text recognition model, and allowing the language text recognition model to recognize each text segment corresponding to each language in the speech;

[0032] A prediction module, used for inputting each of the text segments into each of the trained punctuation prediction models of the corresponding language, and each of the punctuation prediction models outputs corresponding punctuation prediction results according to the input text segments; wherein each of the punctuation prediction models is trained based on text segment samples of each language;

[0033] The adding module is used to add corresponding punctuation information in corresponding text segments according to the punctuation prediction results.

[0034] In a third aspect, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0035] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0036] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0037] The punctuation prediction method, device, computer equipment, computer-readable storage medium and computer program product for mixed speech recognition text are used to identify the text segments corresponding to the languages ​​in the speech to be predicted through the language text recognition model, train multiple punctuation prediction models through the text segment samples of multiple languages, input each text segment into each trained punctuation prediction model of the corresponding language, and each punctuation prediction model outputs the corresponding punctuation prediction results according to the input text segment, and add the corresponding punctuation information to the corresponding text segment according to the punctuation prediction results. Compared with the traditional method of adding punctuation according to a fixed language, this scheme trains multiple punctuation prediction models through the text segment samples of multiple languages, recognizes the text segment of the corresponding language for the speech mixed with multiple languages, and predicts the punctuation of the text segment according to the language of the text segment through the corresponding punctuation prediction model, thereby improving the accuracy of punctuation prediction for the recognition text mixed with multiple speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A schematic diagram of a flow chart of a punctuation prediction method for a hybrid speech recognition text in one embodiment;

[0040] Figure 2 A schematic diagram of the structure of a conditional random field model in an embodiment;

[0041] Figure 3 A schematic flow chart of a punctuation prediction method for a hybrid speech recognition text in another embodiment;

[0042] Figure 4 It is a structural block diagram of a punctuation prediction device for hybrid speech recognition text in one embodiment;

[0043] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] In one embodiment, Figure 1 As shown, a punctuation prediction method for hybrid speech recognition text is provided. This embodiment takes the method applied to a server as an example for illustration. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server, including the following steps S202 to S206. Among them:

[0046] Step S202, obtaining speech to be predicted, inputting the speech into a trained language text recognition model, and allowing the language text recognition model to recognize each text segment corresponding to each language in the speech.

[0047] Among them, the speech to be predicted may be a speech that needs to be subjected to text recognition and punctuation prediction. The above-mentioned speech to be predicted may include speech parts of multiple languages, that is, a speech of a mixed language. Since the recognition strategies of different languages ​​are different, different strategies need to be adopted to perform text recognition and punctuation prediction on the above-mentioned mixed language speech to be predicted. Among them, the terminal can use speech samples of multiple languages ​​and corresponding text samples to train the language text recognition model to be trained, and adjust the model parameters of the language text recognition model according to the comparison result of the text segment prediction result output by the language text recognition model based on the speech sample and the above-mentioned text sample, and then obtain the trained language text recognition model when the preset training end condition is met. Among them, the preset training end condition may include within the preset training times, the similarity of the above-mentioned comparison result is greater than or equal to the preset similarity threshold, or the training times reach the above-mentioned preset training times.

[0048] After the terminal obtains the speech to be predicted, the speech can be input into the trained language text recognition model, and the language text recognition model can recognize the text segments corresponding to the languages ​​in the speech. For example, the language text recognition model performs text recognition on the speech segments of each language based on the dictionary corresponding to each language in the speech to obtain the text segments corresponding to each language.

[0049] Step S204, input each text segment into each trained punctuation prediction model of the corresponding language, and each punctuation prediction model outputs corresponding punctuation prediction results according to the input text segment; wherein each punctuation prediction model is trained based on text segment samples of each language.

[0050] Among them, the terminal can pre-train the punctuation prediction model to be trained based on the text segment samples of each language, so as to obtain the punctuation prediction model of the corresponding language. Among them, there can be multiple punctuation prediction models, and each punctuation prediction model can be trained based on the text segment samples of the corresponding language. The punctuation prediction model is used to predict punctuation for the text segment of the corresponding language, determine which characters in the text segment containing continuous characters should be followed by punctuation, and the type of punctuation. The terminal can determine the languages ​​corresponding to the above-mentioned text segments, and then the terminal can input the above-mentioned text segments into the punctuation prediction models of the corresponding languages, and the punctuation prediction models output the corresponding punctuation prediction results according to the input text segments. Among them, the punctuation prediction results can be a punctuation set, which includes the character information of each character in the corresponding text segment to which punctuation needs to be added, and the type of punctuation that needs to be added after the character information.

[0051] For example, the above-mentioned speech may be a mixture of Cantonese and Mandarin, and the terminal may respectively identify the text segments corresponding to Cantonese and the text segments corresponding to Mandarin through the language text recognition model, wherein each text segment represents a continuous segment of characters in the same language. The terminal may respectively input the text segments corresponding to Cantonese into the punctuation prediction model trained with Cantonese text segment samples, and input the text segments corresponding to Mandarin into the punctuation prediction model trained with Mandarin text segment samples. Thus, the terminal may obtain the punctuation prediction results output by the punctuation prediction models of the various languages. Among them, the text segment samples of the various languages ​​may include multiple characters with pre-marked punctuation.

[0052] Step S206: adding corresponding punctuation information to corresponding text segments according to the punctuation prediction results.

[0053] Among them, the above-mentioned punctuation prediction result can be a punctuation set. The punctuation set includes character information that needs to be connected to punctuation marks and the types of punctuation connected to the character information. Different text segments correspond to different punctuation prediction results. Therefore, for each punctuation prediction result, the terminal can add corresponding punctuation information to the text segment corresponding to the punctuation prediction result. For example, the terminal determines the character position corresponding to the above-mentioned character information in the corresponding text segment based on the character information in the punctuation prediction result, and adds the punctuation corresponding to the character information after the character position. After adding the corresponding punctuation information to each text segment based on each punctuation prediction result, the terminal combines each text segment according to the output order of the language text recognition model. For the recognition text of two languages, the terminal can connect them with a period, so that the terminal adds punctuation to the text segment of the same language based on the punctuation prediction result, and after combining the above-mentioned text segments with punctuation, the recognition text with punctuation added corresponding to the above-mentioned speech to be predicted can be obtained.

[0054] In the punctuation prediction method for mixed speech recognition text, the language text recognition model is used to identify the text segments corresponding to the languages ​​in the speech to be predicted, multiple punctuation prediction models are trained through text segment samples of multiple languages, each text segment is input into each trained punctuation prediction model of the corresponding language, each punctuation prediction model outputs corresponding punctuation prediction results according to the input text segment, and corresponding punctuation information is added to each corresponding text segment according to each punctuation prediction result. Compared with the traditional method of adding punctuation according to a fixed language, this scheme trains multiple punctuation prediction models through text segment samples of multiple languages, recognizes text segments of the corresponding languages ​​for speech mixed with multiple languages, and predicts punctuation for the text segments according to the languages ​​of the text segments through corresponding punctuation prediction models, thereby improving the accuracy of punctuation prediction for recognition text mixed with multiple speech.

[0055] In one embodiment, speech is input into a trained language-text recognition model, and the language-text recognition model recognizes each text segment corresponding to each language in the speech, including: speech is input into a trained language-text recognition model, and the language-text recognition model recognizes each first text segment corresponding to a speech segment of a first language in the speech, and each second text segment corresponding to a speech segment of a second language in the speech through a speech activity detection system.

[0056] In this embodiment, the terminal can perform text recognition of speech through a trained language text recognition model. The above-mentioned language text recognition model can be used to recognize each language in a speech containing mixed languages, divide the speech segments corresponding to each language, and recognize each text segment corresponding to each speech segment. The text segment recognized by the language text recognition model can be a text segment that does not contain punctuation marks, and each text segment represents a text segment of speech in the same language. The language text recognition model can use different dictionaries to recognize speech in different languages.

[0057] For example, the terminal may input the above-mentioned speech to be predicted into a trained language text recognition model, and the language text recognition model may recognize the speech segments corresponding to each language in the speech through the VAD (Voice Activity Detection) system. The VAD system is used to recognize when there is speech activity in the audio signal and when it is silent or background noise. The speech to be predicted may include two different languages. The terminal recognizes the speech segments of the first language and the speech segments of the second language in the above-mentioned speech through the VAD system. The terminal may use VAD to recognize the first text segments corresponding to the speech segments of the first language, and recognize the second text segments corresponding to the speech segments of the second language. The above-mentioned speech to be predicted may be a mixed speech of Cantonese and Mandarin. The terminal may respectively recognize the speech segments and the text segments corresponding to Cantonese, and recognize the speech segments and the text segments corresponding to Mandarin through the VAD system.

[0058] Through this embodiment, the terminal can use the voice activity detection system in the language text recognition model to identify voice fragments of each language and the corresponding text segments of the speech containing mixed languages, so that the terminal can predict punctuation for each text segment corresponding to each recognized language, thereby improving the accuracy of punctuation prediction for the recognized text of mixed languages.

[0059] In one embodiment, a language text recognition model recognizes each first text segment corresponding to a speech segment of a first language in a speech and each second text segment corresponding to a speech segment of a second language through a speech activity detection system, including: the language text recognition model recognizes each voice character in the speech through the speech activity detection system, and obtains a voice character sequence according to multiple voice characters; takes the first voice character in the voice character sequence as the current voice character, and obtains the probability that the current voice character belongs to the first language according to the comparison result of the current voice character with the text dictionary corresponding to the first language; if the probability is greater than or equal to a preset threshold, determines that the current voice character belongs to the first language, otherwise it is determined that the current voice character belongs to the second language; taking the next voice character corresponding to the current voice character in the voice character sequence as the new current voice character, returning the comparison result between the current voice character and the text dictionary corresponding to the first language, and obtaining the probability that the current voice character belongs to the first language, until each voice character in the voice character sequence is recognized; obtaining each first text segment corresponding to the first language according to each consecutive voice character in the voice character sequence and belonging to the first language, and obtaining each second text segment corresponding to the second language according to each consecutive voice character in the voice character sequence and belonging to the second language.

[0060] In this embodiment, the terminal can recognize the speech text by probability when recognizing the speech text. For example, after the terminal inputs the above-mentioned speech to be predicted into the trained language text recognition model, the language text recognition model can recognize each speech character in the speech through the above-mentioned VAD system. Among them, the speech character refers to the character obtained after recognizing the speech segment corresponding to a single character. The terminal can obtain a speech character sequence based on multiple speech characters in the language text recognition model.

[0061] Taking the above speech including speech in two languages ​​as an example, the terminal can identify the language of each character in the speech character sequence. For example, the terminal uses the language text recognition model to take the first speech character in the speech character sequence as the current speech character, obtains the comparison result of the current speech character with the text dictionary corresponding to the first language, and the terminal can obtain the probability that the current speech character belongs to the first language based on the comparison result of the current speech character with the text dictionary corresponding to the first language.

[0062] Among them, if the probability is greater than or equal to the preset threshold, the terminal determines that the current voice character belongs to the first language through the language text recognition model, otherwise it determines that the current voice character belongs to the second language. If the language text recognition model detects that the current voice character is not the voice of the last character in the above-mentioned voice to be predicted, the language text recognition model can repeat the above-mentioned recognition process. For example, the language text recognition model uses the next voice character corresponding to the current voice character in the voice character sequence as the new current voice character, and returns the step of obtaining the probability that the current voice character belongs to the first language based on the comparison result of the current voice character and the text dictionary corresponding to the first language, until each voice character in the voice character sequence is recognized. Thus, the terminal can obtain each first text segment corresponding to the first language according to each voice character that is continuous and belongs to the first language in the voice character sequence, and obtain each second text segment corresponding to the second language according to each voice character that is continuous and belongs to the second language in the voice character sequence. For example, the terminal uses the language text recognition model and the corresponding terminal text dictionary to perform text recognition on each voice character segment.

[0063] Specifically, the terminal performs speech recognition and text recognition in various languages ​​through the VAD system. After the terminal inputs the speech to be predicted into the language text recognition model, the VAD system can divide the speech, and a speech segment divided by the VAD system can be considered to be of the same language. Taking the example that the speech to be predicted includes Cantonese and Mandarin, the above probability can be a CTC (Connectionist Temporal Classification) probability. While the Mandarin-Cantonese mixed speech recognition system outputs speech characters, it can obtain the CTC output probability. The speech character sequence recognized by the terminal through the VAD system can be expressed as (x1, x2,…, x n ). Among them, x n represents the voice character recognized by the nth character voice recognition. The voice character sequence can be a character sequence corresponding to the mixed voice of Cantonese and Mandarin. Taking Cantonese as the first language as an example, the terminal determines x through the language text recognition model i Whether it belongs to the Cantonese process can be specifically expressed as: f(x i )={0,x i does not belong to W; θ(x i )}. Where W is the text dictionary corresponding to the first language, such as the Cantonese text dictionary; θ(x i ) is x i The probability of the CTC output is P(X)=(∑ (i=1) n f(x i)) / n. The above preset threshold can be expressed as threshold. When the terminal detects that P(X)≥threshhold, it determines that the speech is Cantonese, otherwise it is Mandarin.

[0064] Through this embodiment, the terminal can use VAD and CTC probability to perform character recognition on speech in mixed languages, thereby obtaining text segments corresponding to each language, and performing punctuation prediction based on text segments in the same language, thereby improving the accuracy of punctuation prediction.

[0065] In one embodiment, each text segment is input into each trained punctuation prediction model of the corresponding language, and each punctuation prediction model outputs corresponding punctuation prediction results according to the input text segment, including: for each text segment, according to the language corresponding to the text segment, determining the corresponding trained target punctuation prediction model, and inputting the text segment into the target punctuation prediction model; extracting the text feature sequence corresponding to the text segment by the target punctuation prediction model, matching the text feature sequence with a preset punctuation set through a conditional random field, and obtaining each target character that needs to be marked with punctuation in the text segment and the target punctuation corresponding to each target character; and outputting the punctuation prediction result corresponding to the text segment according to each target character and each target punctuation corresponding to each target character.

[0066] In this embodiment, the terminal can perform punctuation prediction on the text segment of the corresponding language based on the punctuation prediction models corresponding to different languages. For example, there can be multiple text segments, wherein, since speech segments of different languages ​​can appear alternately in the speech to be predicted, there can be multiple text segments of the same language and text segments of different languages. For each text segment, the terminal can determine the corresponding trained target punctuation prediction model according to the language corresponding to the text segment, and input the text segment into the target punctuation prediction model.

[0067] In the target punctuation prediction model, the target punctuation prediction model extracts the text feature sequence corresponding to the text segment, and matches the text feature sequence with the preset punctuation set through CRF (Conditional Random Field) to obtain the target characters that need to be marked with punctuation in the text segment and the target punctuation corresponding to each target character. Among them, the preset punctuation set includes multiple types of punctuation, such as question marks, commas, periods, and non-punctuation marks. The target punctuation prediction model can output the punctuation prediction results corresponding to the text segment based on each target character and each target punctuation corresponding to each target character. That is, the character information in the text segment that needs to be connected with punctuation marks and the type of punctuation connected to the character information.

[0068] Specifically, taking the above speech including Cantonese and Mandarin as an example, the terminal can pre-construct a preset punctuation set, specifically represented by Y = {3, 2, 1, 0}, which respectively represent question mark (?), comma (,), period (.) and no punctuation. The terminal extracts the text feature sequence of the text segment through the target punctuation prediction model of the language corresponding to the text segment, where the text feature sequence includes multiple vectors, specifically represented by x=(x1,x2,…,x n ). The punctuation can be represented by the function f i (x,y) to represent, if the sample (x n ,y n ) needs punctuation, then f i (x,y)=1,2,3, otherwise f i (x,y)=0. Consider a random process y i , it outputs a certain y with a certain probability based on the observable vector x, and y belongs to the above finite set Y.

[0069] In the target punctuation prediction model, the terminal predicts punctuation based on the above x and y, where x represents all the features corresponding to the current text segment, and y indicates whether the character information to be predicted should be followed by punctuation, and what kind of punctuation should be followed. The terminal can use CRF for predictive annotation. CRF is also called Markov random field. CRF can be regarded as an undirected graph model that can examine the conditional probability of the annotation sequence given the input text feature sequence. Let x be a random variable on the sequence that needs punctuation prediction, and y be a random variable in the corresponding punctuation sequence. Assume that all components of y are y n Included in a fixed symbol set Y. The CRF model structure is as follows Figure 2 As shown, Figure 2 Schematic diagram of the structure of a conditional random field model in an embodiment, wherein x is the observation sequence and y is the label sequence.

[0070] Then the terminal can obtain the corresponding CRF probability distribution function: p(Y|X,λ)=1 / (z(X))exp(∑ i n ∑ j λ j Λ j (x i ,y i ,y i-1 )); Z(X)=exp(∑ y∈Y ∑ i n ∑ j λ j Λ j (x i ,y i ,y i-1))). Where Z(X) is the normalization term, j is the number of feature functions Λ, n is the length of the input sequence, λ j is the weight of the jth feature function.

[0071] Thus, the terminal uses the target punctuation prediction model to determine the target characters that need to be punctuated and the target punctuation that needs to be added after each target character for the input text X based on the CRF probability distribution. For example, through the target punctuation prediction model, according to the result of the hybrid speech recognition text language information detection, the model of Cantonese or Mandarin is selected for prediction to obtain the prediction result Y n ∈{0,1,2,3,4}, n is the length of the input text, if Y n ≠0, at X n Add corresponding punctuation afterward.

[0072] Through this embodiment, the terminal can use CRF probability and the target punctuation prediction model corresponding to each language to determine the target characters that need to be punctuated and the type of punctuation that needs to be added after the target characters, thereby improving the accuracy of adding punctuation to the mixed speech recognition text.

[0073] In one embodiment, it also includes: obtaining each text segment sample corresponding to each language and the punctuation prediction model to be trained corresponding to each language; for the punctuation prediction model corresponding to each language, inputting the text segment sample corresponding to the language into the punctuation prediction model, and extracting the text feature sample sequence corresponding to the text segment sample through the conditional random field by the punctuation prediction model; matching the text feature sample sequence with the preset punctuation set, and determining the target preset punctuation corresponding to each text feature in the text feature sample sequence from each preset punctuation in the preset punctuation set; determining the corresponding probability distribution function according to each text feature and each target preset punctuation corresponding to each text feature; adjusting the model parameters of the punctuation prediction model according to the probability distribution function until the preset training end condition is met, and the trained punctuation prediction model corresponding to the language is obtained.

[0074] In this embodiment, the terminal may use text segment samples of various languages ​​in advance to train the punctuation prediction model. For example, the terminal may obtain various text segment samples corresponding to various languages ​​and punctuation prediction models to be trained corresponding to various languages. For the punctuation prediction model corresponding to each language, the terminal may input the text segment samples corresponding to the language into the punctuation prediction model, and the punctuation prediction model may extract the text feature sample sequence corresponding to the text segment sample through the conditional random field.

[0075] Among them, the terminal can match the text feature sample sequence with the preset punctuation set, determine the target preset punctuation corresponding to each text feature in the text feature sample sequence from each preset punctuation in the preset punctuation set, and determine the corresponding probability distribution function according to each text feature and each target preset punctuation corresponding to each text feature.

[0076] Thus, the terminal can adjust the model parameters of the punctuation prediction model according to the probability distribution function until the preset training end condition is met, thereby obtaining a trained punctuation prediction model corresponding to the language. The preset training end condition can be determined based on the output value of the log-likelihood function.

[0077] In one embodiment, the model parameters of the punctuation prediction model are adjusted according to the probability distribution function until a preset training end condition is met, thereby obtaining a trained punctuation prediction model corresponding to the language, including: determining the corresponding log-likelihood function according to the inverse of the logarithm of the probability distribution function; adjusting the model parameters of the punctuation prediction model according to the output value of the log-likelihood function until the output value of the log-likelihood function is minimized, thereby obtaining a trained punctuation prediction model corresponding to the language.

[0078] In this embodiment, the terminal can construct a log-likelihood function based on the probability distribution function in the above training process. For example, the terminal determines the corresponding log-likelihood function according to the inverse of the logarithm of the probability distribution function. The terminal can adjust the model parameters of the punctuation prediction model according to the output value of the log-likelihood function until the output value of the log-likelihood function is minimized, thereby obtaining a trained punctuation prediction model corresponding to the language.

[0079] Specifically, the above probability distribution function can be the probability distribution function corresponding to CRF, which can be specifically expressed as p(Y|X,λ)=1 / (z(X))exp(∑ i n ∑ j λ j Λ j (x i ,y i ,y i-1 )); Z(X)=exp(∑ y∈Y ∑ i n ∑ j λ j Λ j (x i ,y i ,y i-1 ))). Where Z(X) is the normalization term, j is the number of feature functions Λ, n is the length of the input sequence, λ j is the weight of the jth feature function.

[0080] The terminal can construct a log-likelihood function as a loss function through each text feature in the text segment sample and the target preset punctuation corresponding to each text feature, and adjust the model parameter λ. The terminal estimates the parameter λ through the maximum likelihood estimation algorithm, assuming that the training set composed of text segment samples is D=[(x 1 ,y 1 ),(x 2 ,y 2 ),⋯,(x m ,y m )], m is the number of training set sequences, x m Represents text features, y m represents the target preset punctuation, and the log-likelihood function can be specifically expressed as:

[0081] .

[0082] When the output value of the log-likelihood function is the minimum, the trained punctuation prediction model is determined based on the model parameters at this time. Thus, the terminal can use the text segment samples corresponding to Cantonese and Mandarin to train the punctuation prediction model to be trained and obtain the corresponding punctuation prediction model.

[0083] Through the above embodiments, the terminal can use conditional random fields to train the punctuation prediction model of the corresponding language, and then obtain a punctuation prediction and annotation model that can automatically realize the text of the corresponding language, thereby improving the accuracy of punctuation prediction for speech recognition text of mixed languages.

[0084] In an exemplary embodiment, Figure 3 As shown, Figure 3 The flowchart of the punctuation prediction method of mixed speech recognition text in another embodiment is shown in FIG. In this embodiment, taking the mixed language of Cantonese and Mandarin as an example, the terminal can perform text recognition on the speech of the mixed language, recognize the text corresponding to each language, and obtain each text segment, including each text segment corresponding to Cantonese and each text segment corresponding to Mandarin.

[0085] The terminal detects the language of each text segment. For a Cantonese text segment, the terminal uses the punctuation prediction model corresponding to Cantonese and combines it with the Cantonese text segment to output the punctuation prediction result; for a Mandarin text segment, the terminal uses the punctuation prediction model corresponding to Mandarin and combines it with the Mandarin text segment to output the punctuation prediction result. This enables punctuation prediction for speech recognition text in mixed languages.

[0086] Through the above embodiments, the terminal can train multiple punctuation prediction models through text segment samples in multiple languages, recognize text segments in corresponding languages ​​for speech mixed with multiple languages, and predict punctuation for the text segments through corresponding punctuation prediction models according to the languages ​​of the text segments, thereby improving the accuracy of punctuation prediction for recognized text mixed with multiple languages.

[0087] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0088] Based on the same inventive concept, the embodiment of the present application also provides a punctuation prediction device for a hybrid speech recognition text for implementing the punctuation prediction method for a hybrid speech recognition text involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more punctuation prediction devices for hybrid speech recognition text provided below can refer to the limitations of the punctuation prediction method for hybrid speech recognition text above, and will not be repeated here.

[0089] In an exemplary embodiment, Figure 4 As shown, a punctuation prediction device for a hybrid speech recognition text is provided, comprising: an acquisition module 500, a prediction module 502 and an adding module 504, wherein:

[0090] The acquisition module 500 is used to acquire the speech to be predicted, input the speech into the trained language text recognition model, and use the language text recognition model to recognize each text segment corresponding to each language in the speech.

[0091] The prediction module 502 is used to input each text segment into each trained punctuation prediction model of the corresponding language, and each punctuation prediction model outputs corresponding punctuation prediction results according to the input text segment; wherein each punctuation prediction model is trained based on text segment samples of each language.

[0092] The adding module 504 is used to add corresponding punctuation information in corresponding text segments according to the punctuation prediction results.

[0093] In one embodiment, the acquisition module 500 is used to input speech into a trained language-text recognition model, and the language-text recognition model recognizes each first text segment corresponding to the speech segment of the first language in the speech, and each second text segment corresponding to the speech segment of the second language in the speech through a speech activity detection system.

[0094] In one embodiment, the acquisition module 500 is used to recognize each voice character in the speech through a voice activity detection system by a language text recognition model, and obtain a voice character sequence according to multiple voice characters; take the first voice character in the voice character sequence as the current voice character, and obtain the probability that the current voice character belongs to the first language according to the comparison result of the current voice character with the text dictionary corresponding to the first language; if the probability is greater than or equal to a preset threshold, determine that the current voice character belongs to the first language, otherwise determine that the current voice character belongs to the second language; take the next voice character corresponding to the current voice character in the voice character sequence as the new current voice character, return to the step of obtaining the probability that the current voice character belongs to the first language according to the comparison result of the current voice character with the text dictionary corresponding to the first language, until each voice character in the voice character sequence is recognized; obtain each first text segment corresponding to the first language according to each consecutive voice character in the voice character sequence and belonging to the first language, and obtain each second text segment corresponding to the second language according to each consecutive voice character in the voice character sequence and belonging to the second language.

[0095] In one embodiment, the above-mentioned prediction module 502 is used to determine the corresponding trained target punctuation prediction model for each text segment according to the language corresponding to the text segment, and input the text segment into the target punctuation prediction model; the target punctuation prediction model extracts the text feature sequence corresponding to the text segment, and matches the text feature sequence with a preset punctuation set through a conditional random field to obtain each target character that needs to be marked with punctuation in the text segment and the target punctuation corresponding to each target character; according to each target character and each target punctuation corresponding to each target character, the punctuation prediction result corresponding to the text segment is output.

[0096] In one embodiment, the above-mentioned device also includes: a training module, which is used to obtain each text segment sample corresponding to each language and the punctuation prediction model to be trained corresponding to each language; for the punctuation prediction model corresponding to each language, the text segment sample corresponding to the language is input into the punctuation prediction model, and the punctuation prediction model extracts the text feature sample sequence corresponding to the text segment sample through the conditional random field; the text feature sample sequence is matched with the preset punctuation set, and the target preset punctuation corresponding to each text feature in the text feature sample sequence is determined from each preset punctuation in the preset punctuation set; according to each text feature and each target preset punctuation corresponding to each text feature, the corresponding probability distribution function is determined; according to the probability distribution function, the model parameters of the punctuation prediction model are adjusted until the preset training end condition is met, and the trained punctuation prediction model corresponding to the language is obtained.

[0097] In one embodiment, the above-mentioned training module is used to determine the corresponding log-likelihood function based on the inverse of the logarithm of the probability distribution function; adjust the model parameters of the punctuation prediction model according to the output value of the log-likelihood function until the output value of the log-likelihood function is minimized, thereby obtaining a trained punctuation prediction model corresponding to the language.

[0098] Each module in the punctuation prediction device for hybrid speech recognition text can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0099] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 5As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a punctuation prediction method for a hybrid speech recognition text is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0100] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0101] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above-mentioned punctuation prediction method for hybrid speech recognition text when executing the computer program.

[0102] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the punctuation prediction method for the hybrid speech recognition text is implemented.

[0103] In one embodiment, a computer program product is provided, including a computer program, which implements the above-mentioned punctuation prediction method for mixed speech recognition text when executed by a processor.

[0104] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0105] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0106] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0107] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A punctuation prediction method for hybrid speech recognition text, characterized in that: The method comprises: Acquire speech to be predicted, input the speech into a trained language text recognition model, and use the language text recognition model to recognize each text segment corresponding to each language in the speech; Input each of the text segments into each trained punctuation prediction model of the corresponding language, and each of the punctuation prediction models outputs corresponding punctuation prediction results according to the input text segments, including: for each text segment, determine the corresponding trained target punctuation prediction model according to the language corresponding to the text segment, and input the text segment into the target punctuation prediction model; extract the text feature sequence corresponding to the text segment by the target punctuation prediction model, match the text feature sequence with a preset punctuation set through a conditional random field, and obtain each target character that needs to be marked with punctuation in the text segment and the target punctuation corresponding to each target character; output the punctuation prediction result corresponding to the text segment according to each of the target characters and each of the target punctuation corresponding to each of the target characters; wherein the training process of the punctuation prediction model includes: obtaining each text segment sample corresponding to each language and each of the A punctuation prediction model to be trained corresponding to a language; for the punctuation prediction model corresponding to each language, a text segment sample corresponding to the language is input into the punctuation prediction model, and the punctuation prediction model extracts a text feature sample sequence corresponding to the text segment sample through a conditional random field; the text feature sample sequence is matched with a preset punctuation set, and the target preset punctuation corresponding to each text feature in the text feature sample sequence is determined from each preset punctuation in the preset punctuation set; a corresponding probability distribution function is determined according to each text feature and each target preset punctuation corresponding to each text feature; a corresponding log-likelihood function is determined according to the inverse of the logarithm of the probability distribution function; a model parameter of the punctuation prediction model is adjusted according to the output value of the log-likelihood function until the output value of the log-likelihood function is minimized, thereby obtaining a trained punctuation prediction model corresponding to the language; According to each of the punctuation prediction results, corresponding punctuation information is added to each of the corresponding text segments.

2. The method according to claim 1, characterized in that The step of inputting the speech into a trained language text recognition model, and allowing the language text recognition model to recognize each text segment corresponding to each language in the speech, comprises: The speech is input into a trained language-text recognition model, and the language-text recognition model recognizes each first text segment corresponding to the speech segment of the first language in the speech and each second text segment corresponding to the speech segment of the second language in the speech through a speech activity detection system.

3. The method according to claim 2, characterized in that The method of recognizing each first text segment corresponding to the speech segment of the first language in the speech and each second text segment corresponding to the speech segment of the second language by the language text recognition model through a voice activity detection system includes: The language text recognition model recognizes each voice character in the voice through a voice activity detection system, and obtains a voice character sequence based on a plurality of the voice characters; Taking the first phonetic character in the phonetic character sequence as the current phonetic character, and obtaining the probability that the current phonetic character belongs to the first language according to the comparison result between the current phonetic character and the text dictionary corresponding to the first language; If the probability is greater than or equal to a preset threshold, it is determined that the current voice character belongs to the first language; otherwise, it is determined that the current voice character belongs to the second language; The step of taking the next voice character corresponding to the current voice character in the voice character sequence as a new current voice character, and returning the comparison result between the current voice character and the text dictionary corresponding to the first language to obtain the probability that the current voice character belongs to the first language, until each voice character in the voice character sequence is recognized; According to each continuous voice character in the voice character sequence belonging to the first language, each first text segment corresponding to the first language is obtained; according to each continuous voice character in the voice character sequence belonging to the second language, each second text segment corresponding to the second language is obtained.

4. A punctuation prediction device for hybrid speech recognition text, characterized in that: The device comprises: An acquisition module, used for acquiring speech to be predicted, inputting the speech into a trained language text recognition model, and allowing the language text recognition model to recognize each text segment corresponding to each language in the speech; The prediction module is used to input each of the text segments into each of the trained punctuation prediction models of the corresponding language, and each of the punctuation prediction models outputs corresponding punctuation prediction results according to the input text segments. Specifically, for each text segment, according to the language corresponding to the text segment, a corresponding trained target punctuation prediction model is determined, and the text segment is input into the target punctuation prediction model; the target punctuation prediction model extracts the text feature sequence corresponding to the text segment, and matches the text feature sequence with a preset punctuation set through a conditional random field to obtain each target character that needs to be marked with punctuation in the text segment and the target punctuation corresponding to each target character; according to each of the target characters and each of the target punctuation corresponding to each of the target characters, the punctuation prediction result corresponding to the text segment is output; wherein, it also includes a training module for obtaining each of the text segment samples corresponding to each language and each of the target punctuation corresponding to the target characters. a punctuation prediction model to be trained corresponding to the language; for the punctuation prediction model corresponding to each language, input the text segment sample corresponding to the language into the punctuation prediction model, and extract the text feature sample sequence corresponding to the text segment sample by the punctuation prediction model through a conditional random field; match the text feature sample sequence with a preset punctuation set, and determine the target preset punctuation corresponding to each text feature in the text feature sample sequence from each preset punctuation in the preset punctuation set; determine the corresponding probability distribution function according to each of the text features and each target preset punctuation corresponding to each of the text features; determine the corresponding log-likelihood function according to the inverse of the logarithm of the probability distribution function; adjust the model parameters of the punctuation prediction model according to the output value of the log-likelihood function until the output value of the log-likelihood function is minimized, thereby obtaining the trained punctuation prediction model corresponding to the language; The adding module is used to add corresponding punctuation information in corresponding text segments according to the punctuation prediction results.

5. The device according to claim 4, characterized in that The acquisition module is used to: The speech is input into a trained language-text recognition model, and the language-text recognition model recognizes each first text segment corresponding to the speech segment of the first language in the speech and each second text segment corresponding to the speech segment of the second language in the speech through a speech activity detection system.

6. The device according to claim 5, characterized in that The acquisition module is used to: The language text recognition model recognizes each voice character in the voice through a voice activity detection system, and obtains a voice character sequence based on a plurality of the voice characters; Taking the first phonetic character in the phonetic character sequence as the current phonetic character, and obtaining the probability that the current phonetic character belongs to the first language according to the comparison result between the current phonetic character and the text dictionary corresponding to the first language; If the probability is greater than or equal to a preset threshold, it is determined that the current voice character belongs to the first language; otherwise, it is determined that the current voice character belongs to the second language; The step of taking the next voice character corresponding to the current voice character in the voice character sequence as a new current voice character, and returning the comparison result between the current voice character and the text dictionary corresponding to the first language to obtain the probability that the current voice character belongs to the first language, until each voice character in the voice character sequence is recognized; According to each continuous voice character in the voice character sequence belonging to the first language, each first text segment corresponding to the first language is obtained; according to each continuous voice character in the voice character sequence belonging to the second language, each second text segment corresponding to the second language is obtained.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 3 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images