A text-smooth speech recognition method, system and storage medium

Abstract

The present invention relates to a text-smooth speech recognition method, system and storage medium, belonging to the field of speech recognition technology; it solves the problem that the oral transcription text output after speech recognition is not easy to read and has low fidelity; the text-smooth speech recognition method of the present invention comprises: obtaining speech to be recognized; converting the speech to be recognized into text to be smoothed; using a text smoothness model to obtain a smooth text after recognition from the converted text to be smoothed; wherein the text smoothness model is used to output an unsmooth ordered label of the current unsmooth text and a smooth text corresponding to the current unsmooth text; the unsmooth ordered label is predicted by a CRF method through an unsmooth label prediction task; the smooth text is searched by a beam search method through a smooth text generation task; the speech recognition method of the present invention solves the problem that the recognition results of existing speech recognition technologies are not accurate enough, the readability is not strong, and it is difficult to meet the needs of complex scenarios.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a text-smooth speech recognition method, system, and storage medium. Background Art

[0002] As an important entry point for human-computer interaction, speech recognition has been widely used in various scenarios, such as voice input, voice search, voice translation, smart home, etc. Some of these scenarios have high requirements for real-time online recognition, such as voice control and meeting minutes.

[0003] The text generated by automatic speech recognition often contains numerous artifacts, which can severely interfere with subsequent natural language understanding systems. The text smoothing task, a task between the speech recognition and natural language understanding systems, detects these artifacts in the text after speech recognition. Smoothed text not only effectively improves readability after speech recognition but also reduces interference with subsequent natural language processing tasks.

[0004] Existing speech recognition methods do not take into account the uncontrollability of sequence generation algorithms during decoding, resulting in insufficient fidelity and fluency of the smoothed text. In addition, series-to-series models often generate a series of repeated fragments, causing the smoothed sentences to more or less change the original meaning of the spoken text, making the speech recognition results inaccurate or poorly readable, making it difficult to meet the needs of complex scenarios. Summary of the Invention

[0005] In view of the above analysis, the present invention aims to provide a speech recognition method, system and storage medium with smooth text; to solve the problem that the speech recognition method in the prior art cannot meet the needs of complex scenarios and the generated spoken transcription text has low fidelity and fluency.

[0006] The purpose of the present invention is mainly achieved through the following technical solutions:

[0007] In one aspect, the present invention provides a method for text-smooth speech recognition, comprising the following steps:

[0008] Get the speech to be recognized;

[0009] Convert the speech to be recognized into text to be smoothed;

[0010] The converted text to be smoothed is used to obtain the recognized smooth text using a text smoothing model; wherein, the text smoothing model is used to output the non-smooth ordered label of the current non-smooth text and the smooth text corresponding to the current non-smooth text; the non-smooth ordered label is predicted by the non-smooth label prediction task using the CRF method; the smooth text is searched by the smooth text generation task using the beam search method.

[0011] Furthermore, the speech to be recognized is converted into text to be smoothed, including using a parameter fine-tuning method based on the wav2vec2.0 model for speech recognition. The wav2vec2.0 model consists of three parts: a feature extraction layer, a context encoding layer, and an output layer. The feature extraction layer uses a multi-layer convolutional network to extract features at the speech frame level. The context encoding layer is used to extract correlation features between frames. The output layer uses a pre-trained language model to output the text to be smoothed after speech recognition.

[0012] Furthermore, the text smoothing model is trained using the T5-base pre-trained model as the initial model, including the input layer, encoder-decoder layer, and output layer;

[0013] The encoder uses T5's native self-attention mechanism-based encoding model; it obtains a latent vector with contextual representation of the non-smooth text based on the input non-smooth text;

[0014] The decoder is used to obtain the probability distribution of the target text words through attention calculation based on the encoder latent vector;

[0015] The output layer includes the non-smooth label prediction and the smooth text generation tasks; and is used to output non-smooth label sequences and smooth text.

[0016] Furthermore, the model training includes:

[0017] The samples are expanded by constructing non-smooth corpus pairs containing pseudo labels and constructing expanded corpus without labels to obtain the expanded training sample set;

[0018] The initial model is trained using the training sample set, and the loss function is used for gradient update to obtain a text smoothing model.

[0019] Furthermore, constructing non-smooth corpus pairs containing pseudo labels includes: constructing a corpus pair containing only the "Filter" category, constructing a corpus pair containing only the "Restart" category, and constructing a corpus pair containing both the "Filter" and "Restart" categories; wherein,

[0020] Use a set of modal particles and randomly insert them into any position of the original sentence to construct a corpus pair containing only the "Filter" category;

[0021] Randomly select a text segment from the original sentence and insert it into the adjacent position before the segment to construct a corpus pair containing only the "Restart" category;

[0022] For the same sentence, the process of constructing a corpus pair containing only the "Filter" category and the process of constructing a sample containing only the "Restart" category are executed in sequence to construct a corpus pair containing both the "Filter" and "Restart" categories.

[0023] Constructing the expanded corpus pair without labels includes inputting the corpus pair without labels in the original corpus into a similar text expansion model to obtain text similar to the original sentence, and constructing the expanded corpus pair without labels.

[0024] Furthermore, the loss function includes the loss function of label sequence recognition, the loss function of target text generation, and the damage function of the combination of the two, where;

[0025] The loss function for label sequence recognition is:

[0026]

[0027] S RealPath is the actual label path of the input text, is the total score of all tags at a certain moment, is the score of label i at a certain moment; The i-th word is marked as z i probability; Indicates that from label z i to z i+1 The transition probability; η is the CRF weight;

[0028] The loss function for target text generation is:

[0029]

[0030] λ is the balance parameter, δ is the weight parameter of Coverage, is the target word at time step t;

[0031] The loss function after combining the two is:

[0032] L(x,y,z|θ)=(1-β)L(x,y|θ)+βL(x,z|θ)

[0033] Among them, β is the weighting factor.

[0034] Furthermore, during the model training process, a coverage mechanism is introduced, including:

[0035] During the attention calculation process, the decoder sums the attention scores before the current time step to obtain the coverage feature vector c t ; c t As input to calculate attention,

[0036]

[0037] Among them, c t is the sum of the attention scores output by all decoding time steps before the current time step t, a t′ is the attention score at step t' during decoding;

[0038] The decoder uses the following formula to calculate the attention distribution with coverage mechanism:

[0039]

[0040] Among them, v, W h 、W s 、w c , b is a learnable parameter; is the covering feature vector.

[0041] Furthermore, during the model training process, a copy mechanism is introduced. The probability distribution of words generated by the decoder at each step and the probability of words copied from the original text are considered as a hybrid model. The attention score is used as the probability of the copied word. The probability distribution of the target text word is calculated using the following formula:

[0042]

[0043]

[0044]

[0045] Among them, P(w) is the final probability distribution of the target text words, p gen is the probability of copying the word of the input sequence, P vocab(w) is the probability distribution of words generated in the vocabulary, w is any word, For all words in the input sequence that are equal to w, the time step t corresponds to a t The probability value of the corresponding dimension, a t is the attention distribution, i.e. a t =softmax(e t ), e t is the attention distribution before normalization; is the context encoding vector of the encoder, h i is the encoder hidden state, s t is the decoder status, b ptr ,V′,V,b,b′ are learnable parameters.

[0046] On the other hand, a text smoothing system is provided, comprising a speech recognition device and a text smoothing device;

[0047] A speech recognition device for recognizing a speech data stream and generating a transcript of the original spoken language;

[0048] The text smoothing device includes a text smoothing system and an auxiliary annotation system. The text smoothing system is used to generate smooth text from the original spoken transcription text; the auxiliary annotation system is used to detect the non-smoothness of the speech transcription text to obtain a non-smooth label sequence.

[0049] In a third aspect, a computer-readable storage medium is also provided, wherein the storage medium stores computer instructions, and the instructions are used to be executed by a computer to implement the aforementioned text-smooth speech recognition method.

[0050] Beneficial effects of this technical solution:

[0051] The present invention introduces two tasks: the non-smooth sequence labeling task and the smooth text generation task, which can meet the complex scenario requirements of auxiliary labeling and smooth generation, further reducing the manpower and time costs of obtaining a large amount of labeled data, and the model has a strong text smoothing ability.

[0052] The present invention introduces a strategy for improving fidelity and fluency and a strategy for suppressing the generation of repeated segments, so that the generated smooth text is more readable without changing the original meaning.

[0053] The key technical points of the present invention are:

[0054] 1. By constructing training samples containing pseudo labels and constructing expanded training samples without labels, the original samples are expanded, which greatly enhances the training sample set.

[0055] 2. Introducing special symbols <rm / > It is used to fill the non-smooth gaps in the target sequence, which reduces the difficulty of learning the smooth model and allows the model to pay more attention to the contextual semantics of the sequence itself.

[0056] 3. The introduction of two tasks, namely the non-smooth sequence labeling task and the smooth text generation task, can meet the complex scenario requirements of auxiliary labeling and smooth generation, further reducing the manpower and time costs of obtaining large amounts of labeled data, and the model has relatively strong text smoothing capabilities.

[0057] 4. Introducing strategies to improve fidelity and fluency and suppressing the generation of repeated fragments, so that the generated smooth text is more readable without changing the original meaning.

[0058] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description or be understood by practicing the present invention. The purposes and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like parts throughout the drawings.

[0060] Figure 1 Flowchart of a text-smooth speech recognition method according to an embodiment of the present invention.

[0061] Figure 2 This is a structural diagram of a text smoothing model according to an embodiment of the present invention.

[0062] Figure 3 This is a diagram of the training architecture of an embodiment of the present invention.

[0063] Figure 4 This is a text-smooth speech recognition system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0065] The technical concept of the present invention: The present invention proposes a multi-task fine-tuning text smoothing model and training method based on the unsupervised pre-training model T5-base. In terms of sample construction, the sample diversity is increased by expanding the smooth and non-smooth sample sets; the multi-task neural network model realizes the two tasks of non-smooth sequence labeling and fluent text generation, which can obtain smooth text while detecting text non-smoothness; in the training stage, a multi-task joint learning method is used to fine-tune the parameters to obtain the weight of the text smoothing model. Among them, the mitigation mechanism Copy mechanism and Coverage mechanism are introduced to address the uncontrollable problem of fluent text generation, so as to improve the fidelity and fluency of the smoothed text. The model uses beam search to select the optimal smooth text sequence and CRF to select the optimal label sequence. It is verified on a real data set to assist in the sequence labeling task and achieve standard written text in the real scene of speech transcription.

[0066] In this embodiment, a text-smooth speech recognition method is provided. Figure 1 As shown, the following steps are included:

[0067] Step S1, obtaining the speech to be recognized; specifically, the speech to be recognized can be speech data in any language, and the speech recognition device can simultaneously realize language recognition and effective voice recognition.

[0068] Step S2: converting the speech to be recognized into text to be smoothed;

[0069] Specifically, a parameter fine-tuning method based on the wav2vec2.0 model can be used for speech recognition and output the original spoken transcription text; the model consists of three parts: feature extraction layer, context encoding layer and output layer; the feature extraction layer uses a multi-layer convolutional network to realize feature extraction at the speech frame level; the context encoding layer uses the transformer network based on the self-attention mechanism in the wav2vec2.0 model to extract the weights and features between each speech frame; finally, the output layer uses a pre-trained language model to improve the recognition accuracy, including adjusting the output layer and loss function of the wav2vec2.0 model based on language recognition and effective speech recognition tasks, using a fully connected network to transform the extracted weights and features to the label dimension of the corresponding task, and outputting the speech recognition results at the same time.

[0070] Step S3: using a text smoothness model for recognition, specifically including using a CRF method to predict non-smooth ordered labels, and using a beam search method to search for smooth text.

[0071] The text smoothing model uses the T5-base pre-trained model as the initial model for model training;

[0072] like Figure 2 As shown, the text smoothing model of this embodiment includes three parts: input layer, encoder-decoder layer and output layer; the encoder and decoder are both composed of 12 T5Blocks stacked together; in the encoder stage, each T5Block contains three parts: SelfAttention, LayerNorm and DenseReluDense; in the decoder stage, in addition to the existing modules in the encoder's T5Block, CrossAttention and LayerNorm modules are added to calculate similarity weights together with the encoder's hidden output to find potential associations with the input non-smooth text and enhance the feature representation capability of decoding. For the label prediction task, the decoded features obtained by the decoder are converted into a probability distribution from the input sequence to the label through full connection, and then the label of each input sequence is obtained through CRF label prediction. For the smooth text generation task, the above-mentioned decoding features are converted into a joint probability representation of the vocabulary and the copied original words through the Copy mechanism, and a beam search is used to obtain the final smooth text;

[0073] Specifically, the input layer receives the input of unsmooth text Source = <X1, X2... X L >, and uses E V×D to represent its features, where V and D are the vocabulary size of the unsmooth text and the feature dimension of the word embedding respectively.

[0074] The middle layer uses the structure of an encoder and a decoder to extract the context information of the text.

[0075] The output layer includes two parallel subtasks: the label prediction task and the smooth text generation task; the label prediction task uses a Conditional Random Field (CRF) to predict the label sequence, that is, Tag = <Z1, Z2... Z L >; the text generation task uses a language model to decode to obtain the final smooth text, that is, Target = <Y1, Y2... Y M [[ID=1​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​Unlike summary generation tasks, in actual speech recognition systems, the smoothed text is required to be faithful to the pre-smoothing text. To improve this fidelity, this solution introduces a copy mechanism in the decoder to improve the fidelity and fluency of the smoothed text.

[0083] The aforementioned copy mechanism treats the probability distribution of words generated at each step and the probability of words copied from the original text as a hybrid model, and uses the attention score as the probability of copying words:

[0084]

[0085]

[0086]

[0087] Among them, P(w) is the final probability distribution of the target text words, p gen is the probability of copying the word of the input sequence, P vocab(w) is the probability distribution of words generated in the vocabulary, w is any word, For all words in the input sequence that are equal to w, the time step t corresponds to a t The probability value of the corresponding dimension, a t is the attention distribution, i.e. a t =softmax(e t ), e t is the attention distribution before normalization; is the context encoding vector of the encoder, h i is the encoder hidden state, is the decoder state, b ptr ,V′,V,b,b′ are learnable parameters.

[0088] In addition, the sequence-to-sequence model usually generates sequences with repeated fragments. In order to suppress the generation of repeated fragments, the Coverage mechanism is introduced in the attention calculation process of the decoder. The purpose of this mechanism is mainly to suppress the generated words and prevent the generation of repeated words; the attention scores before the current time step are summed to obtain the coverage feature vector c t ; Use it as the input for calculating attention, which helps to avoid repeated words when calculating attention weights; c t As input to calculate attention:

[0089]

[0090] c t is the newly introduced coverage feature vector, which is the sum of the attention scores output by all decoding time steps before the current time step t; a t′is the attention score at step t' during decoding; the decoder uses the following formula to calculate the attention distribution with coverage mechanism:

[0091]

[0092] Table 1 is an example of generating repeated segments. After adopting the Coverage mechanism of this embodiment, this duplication phenomenon can be effectively suppressed.

[0093] Table 1 Example of repeated fragment generation

[0094]

[0095]

[0096] For the output layer, it includes label prediction tasks and smooth text generation tasks;

[0097] Label prediction task: To detect jerky positions, a conditional random field (CRF) is used to predict jerky label sequences. In the CRF, the input X is used as the observation sequence input, and the label sequence Z is the state sequence. The conditional probability distribution P(Z|X) of Z given X is modeled. The goal of the CRF is to find the optimal sequence that maximizes the objective function. During the training phase, the CRF learns the observation feature weights and state feature weights contained in P(Z|X). When using the trained CRF model for prediction (decoding), the dynamic programming Viterbi algorithm is used to solve the optimal path, that is, to find the most probable state sequence with the highest probability, and then obtain the output sequence Z.

[0098] Smooth text generation task: The commonly used smooth text generation method is the greedy method, which directly takes the maximum probability of each word in the decoder output as the final smooth text; the present invention uses the beam search method to search for the optimal target smooth text.

[0099] Beam search is a commonly used method for finding the best sequence result. Unlike greedy search, beam search maximizes the probability from the perspective of the entire word sequence. The beam search method has a beamwidth parameter, which refers to the number of words in each result y. i When , a beam width of candidate results will be considered. In this scheme, the beam width is set to 3. Specifically, the probability of the final target result using the beam search method is expressed as:

[0100]

[0101] Among them, t is the current time step of the result sequence output, T yis the length of the result sequence, divided by T y is to alleviate the tendency of beam search to favor shorter sentences, and α is to alleviate the tendency of dividing by T y The influence of , α is taken as 0.5.

[0102] Maximizing the above formula is expressed as

[0103] The purpose of the decoder in this scheme is to obtain the probability P(y1,y2…y n ) maximizes the corresponding result sequence y1,y2…y n .

[0104] Figure 3 This is a diagram of the training architecture of an embodiment of the present invention. Model training includes:

[0105] Training sample construction, that is, expanding the sample by constructing non-smooth corpus pairs containing pseudo labels and constructing expanded corpus pairs without labels to obtain the expanded training sample set;

[0106] The initial model is trained using the training sample set, and the loss function is used for gradient update to obtain a text smoothing model.

[0107] Common speech recognition text sluggishness phenomena mainly include "Filter" and "Restart"; among them, "Filter" mainly refers to words that have no actual meaning in the sentence, usually modal particles or echoing words, which do not contribute to the meaning of the sentence

[0108] "Restart" refers to the repetition, redundancy, correction or replacement and insertion of incomplete words in spoken language. This type of non-smooth phenomenon is complex and diverse, and it is impossible to list it all, which is the difficulty of the smooth task.

[0109] The aforementioned training sample construction method uses the SwitchBoard dataset as an example in this embodiment:

[0110] First, define W1 as a set of non-smooth text words in the "Filter" category. This set is obtained by counting all occurrences of SwitchBoard. <e / > The corresponding characters and words are obtained. <e / > A label to mark the position of the non-smooth sequence where the modal particle appears;

[0111] Define D as the original SwitchBoard dataset, D = D1 + D2, where D1 is the set of corpus pairs containing non-smooth labels, and D2 is the set of corpus pairs without labels;

[0112] Define the final training corpus set after expansion as D'=D1+D2+D1'+D2', where D1' is the set of non-smooth corpus pairs containing pseudo labels, and D2' is the set of expanded corpus pairs without labels;

[0113] Define L i is the length of the i-th sentence.

[0114] The training sample construction includes the following steps:

[0115] (1) Label definition: To simplify the label recognition difficulty of non-smooth text, the labels in the original SwitchBoard data were converted as follows to achieve the purpose of label optimization;

[0116] In the disfluency detection task, for the two types of disfluency sentences "Filter" and "Restart", SWBD uses " <e / > "and" <rm-n / > <rpendsub / > "," <rm-n / > <rpmid / > "," <rpendsub / > " to mark the non-smooth sequence position, use " <f / > " marks the smooth position. <rm-n>The N in the rm tag is used to point to the replaced text, indicating that the first N sequences starting from the current position are the replaced text. <rpmid / > Indicates the start or middle mark of the replacement text, <rpendsub / > Indicates the end marker of the replacement text.

[0117] For the model, the label structure of SWBD is complex and difficult to learn. In order to facilitate CRF training, this solution uses "B- <e / > ”、"I- <e / > ”、"B- <rm / > ”、"I- <rm / > ", "O" to indicate a non-smooth sequence, where "B- <e / > " indicates the starting position of the modal particle, "I- <e / > "Indicates the middle and end of the modal particle; "B- <rm / > " indicates the starting position of redundant words, "I- <rm / > "" indicates the middle and end positions of redundant words; "O" indicates the smooth position.

[0118] Non-smoothness detection is to predict the label sequences of the input text, and then detect the non-smooth text positions based on the labels and their positions, which will help to realize the auxiliary annotation work of the intelligent text processing system.

[0119] Table 2 shows the transformation results of the original SWBD labels;

[0120] Table 2 Comparison of samples before and after SWBD construction

[0121]

[0122] (2) Construct a set of non-smooth corpus pairs D1' containing pseudo labels:

[0123] First, a batch of fluent spoken texts is obtained, punctuation marks are removed from the texts, and then the texts are randomly divided into three sets according to the ratio of 3:3:4. The text sets are used as original construction samples containing only "Filter", only "Restart", and both "Filter" and "Restart". Through sample construction, the expanded sample set D containing the pseudo-label "Filter" category is obtained. filter , "Restart" category sample set D restart and the sample set D containing both "Filter" and "Restart" filter_restart ;

[0124] For the "Filter" type, the non-smooth part can usually be exhaustively enumerated. Use the limited set of words in W1 and randomly insert them into L. i Any position, get D filter ;

[0125] For the "Restart" type of non-smoothness: randomly select a text segment from the original sentence and insert it into the front adjacent position of the segment to obtain D restart ;

[0126] D filter_restart The dataset,by sequentially executing the above construction processes of "Filter" type and "Restart",gets a sample set containing both "Filter" and "Restart";

[0127] (3) Constructing an expanded corpus pair set D2' without labels: Input the original sentences in the corpus pair set D2 without labels in the SwitchBoard dataset into the similar text expansion model, and obtain multiple texts similar to the original sentences. These are added as new samples to the expanded corpus pair set D2' without labels.

[0128] (4) Training sample screening: The expanded sample set D' contains a large amount of dirty data. The present invention uses the trained Kenlm language model to obtain the fluency score of the target sentence. The score reflects whether the sentence belongs to a normal expression mode, and is used as a basis to screen out some non-fluent samples.

[0129] It should be noted that the original SWBD (SwitchBoard Dataset) has been divided into training, development, and test sets. During the training sample construction process, the training and development sets were expanded according to the sample construction process described above. The test set remained unchanged except for the label conversion. Tables 3 and 4 show the sample statistics before and after SWBD construction, respectively.

[0130] Table 3 Statistics of the number of samples and labels of the original SWBD dataset

[0131]

[0132] Table 4 Statistics of the number of training samples and labels after expansion

[0133]

[0134] Table 3 shows that most samples in the SWBD data do not contain non-smooth labels. For data without non-smooth labels, this example first uses English-Chinese and Chinese-English translation engines for data augmentation; then, the unlabeled data is constructed into data containing pseudo labels.

[0135] As a specific embodiment, the method for constructing data containing pseudo labels further includes:

[0136] Based on the SWBD dataset, we collected data from the "Spoken Modal Particles - Frequency" table, selected modal particles with a probability greater than 0.7, and randomly inserted them into the position after a segmentation in the original sentence.

[0137] Using the translation data in the training sample, the original sentence is segmented, and the segmented phrases are randomly selected and repeated, and inserted after the repeated words;

[0138] Since the original word is very similar to the word that can be modified, the word that needs to be replaced is simulated and inserted after the original word. Synonyms Replacement (SR) is performed using the EDA (Easy Data Augmentation) method to obtain a new non-smooth sentence.

[0139] As a result, the training data has more than doubled after processing, alleviating the overfitting and poor model accuracy issues caused by sample imbalance. Table 4 shows the statistics of the enhanced training samples.

[0140] like Figure 3 As shown, the text smoothing model of this embodiment is multi-task trained based on the T5-base pre-training model. After the training is completed, the text smoothing model weights are obtained.

[0141] This solution uses the T5-base pre-trained model as the initial model. Specifically, only the encoder weights of T5-base are initialized. The original text vocabulary of this solution uses the vocabulary of T5-base open source pre-trained model. The target text length of smooth text generation is usually different from the original text length, which makes model learning difficult. In order to reduce the learning difficulty and make the model pay more attention to the contextual semantics of the sequence itself, this solution introduces new words in the decoding vocabulary. <rm / > , the target sequence is appended to the end of the vocabulary <rm / > Special words are used as the vocabulary of the target text to fill the gaps in the target sequence that are not smooth.

[0142] The length of the vocabulary is a hyperparameter before model construction. Changing its length means retraining the model. Since the vocabulary of this solution uses the vocabulary in the T5 pre-trained model, which does not contain the tag <rm / > , for this purpose this plan will <rm / > Append to the end of the T5 vocabulary, and do not load the weights of the decoder output layer in the pre-trained model during training, so as to ensure <rm / > The weight space of all the previous words is not greatly affected, and it also ensures <rm / > During the smooth model training process, the weights of other words are adjusted together.

[0143] This paper builds a text smoothing multi-task model based on the pytorch open source framework; the source text vocabulary uses the vocabulary in T5-base, with a size of 32128; the target text vocabulary is 32129; throughout the training process, the following loss function is used for gradient update, with α taking the empirical value of 0.5; the training data is iterated 20 times, and training is stopped when the loss is around 0.05;

[0144] Define X, Y, and Z as the original sentence, target sentence, and original sentence label sequence respectively;

[0145] The loss function for the label prediction task is defined as:

[0146]

[0147] S RealPath is the actual label path of the input text, is the total score of all tags at a certain moment, is the score of label i at a certain moment; The i-th word is marked as z i probability; Indicates that from label z i to z i+1 The transition probability; η is the CRF parameter;

[0148] The loss function for the smooth text generation task is defined as:

[0149]

[0150] λ is a balance parameter, and in this embodiment, λ is set to 1; δ is the weight parameter of Coverage, is the target word at time step t. The benefit of this loss is that it penalizes the overlap between each attention distribution and the current coverage, which can suppress repeated attention;

[0151] The loss function of the entire text smoothing task is defined as:

[0152] L(x,y,z|θ)=(1-β)L(x,y|θ)+βL(x,z|θ);

[0153] A weighting factor β is introduced to balance the outputs of the two tasks. Since the smooth text generation task is more complex than the label prediction task, in order to give a higher weight to the more difficult task during the gradient update process using the loss function, β is set to 0.2 in this embodiment. This is more conducive to the two tasks converging in the same direction.

[0154] Table 5 shows an example of the smoothing results of the training corpus. After the disfluency detection and text smoothing tasks, the disfluency parts in the input text are detected, and then the label sequence and smooth text sequence are obtained by deletion.

[0155] Table 5 Training corpus examples

[0156]

[0157] The training results are compared in Table 6. It can be seen from the table that the label prediction results and the smoothed text effects obtained by using the T5-base+MultiTask multi-task smoothing solution of this solution have been improved to varying degrees.

[0158] Table 6 Comparison of training results of text smoothing device

[0159] Model Tag F1 BLEU of smoothed text Transformer+CRF 89.24％ 87.31 Transformer+DP * 90.70 T5-base+CRF 93.71％ 91.36 T5-base+DP * 96.0 T5-base+MultiTask 95％ 96.6

[0160] It's important to note that the pre-training task is to predict the words corresponding to the input text to obtain an enhanced representation of the spoken text information. This is used as the initial parameters to fine-tune the parameters of downstream tasks, resulting in better performance. In this solution, to accelerate model convergence and improve model recognition accuracy, the open-source T5 pre-trained model is used as the initial model. The linear word list layer of the original T5 output layer is not loaded during the model initialization phase, significantly improving training efficiency and text prediction accuracy.

[0161] Another embodiment of the present invention, as Figure 4 As shown, a speech recognition system is provided, including a speech recognition device and a text smoothing device;

[0162] A speech recognition device for recognizing a speech data stream and generating a transcript of the original spoken language;

[0163] The text smoothing device includes a text smoothing system and an auxiliary annotation system. The text smoothing system is used to generate smooth text from the original spoken transcription text; the auxiliary annotation system is used to detect the non-smoothness of the speech transcription text to obtain a non-smooth label sequence.

[0164] Table 7 shows the implementation results of the text-smooth speech recognition system. From the implementation results, it can be seen that the text-smooth speech recognition method of the present invention can well predict the non-smooth positions of the oral transcription recognition text, and the obtained smooth text is more fluent and more readable.

[0165] Table 7 Implementation results of the speech recognition system with smooth text

[0166]

[0167]

[0168] A third embodiment of the present invention further provides a computer-readable storage medium, in which computer instructions are stored. The stored computer instructions are used to be executed by a computer to implement the text-smooth speech recognition method of the aforementioned embodiment.

[0169] This solution introduces two tasks: the task of labeling non-fluent sequences and the task of generating fluent text. It can meet the complex scenario requirements of auxiliary labeling and smooth generation, further reducing the manpower and time costs of obtaining a large amount of labeled data, and the model has a relatively strong text smoothing ability. Therefore, the present invention is meaningful. Table 7 illustrates the implementation effect of this solution. It can be seen from the table that for English spoken data of different lengths, redundant and meaningless words can be effectively marked out, and the generated smooth text is more readable without changing the original meaning.

[0170] In summary, the present invention proposes a method, system and computer device for text-smooth speech recognition, which combines a multi-task learning method for disfluency detection and text smoothness. This method does not require manual feature extraction and direct input of spoken text data. It can detect the location of disfluencies while the text is smooth, and can meet the complex scenario requirements of auxiliary annotation and smooth generation; it further reduces the manpower and time costs of obtaining a large amount of labeled data, can meet the text smoothing tasks in actual scenarios, and saves time and space costs.

[0171] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0172] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A text-smooth speech recognition method, characterized in that: The following steps are involved: Get the speech to be recognized; Converting the speech to be recognized into text to be smoothed; The converted text to be smoothed is used to obtain a recognized smooth text using a text smoothing model; wherein the text smoothing model is used to output a non-smooth ordered label of the current non-smooth text and a smooth text corresponding to the current non-smooth text; the non-smooth ordered label is predicted by a non-smooth label prediction task using a CRF method; the smooth text is searched by a smooth text generation task using a beam search method; The text smoothing model uses the T5-base pre-trained model as the initial model for model training, including the input layer, encoder-decoder layer and output layer; the original text vocabulary of the text smoothing model uses the vocabulary of the T5-base open source pre-trained model, and special words are appended to the end of the vocabulary <rm / > As the vocabulary of the target text, the special words <rm / > Used to fill the gaps in the target sequence; the input layer uses E′ (V+1)×D Replace E V×D Represents word embedding features, where V and D are the vocabulary size of non-smooth text and the feature dimension of word embedding respectively; The model training includes: expanding samples by constructing non-smooth corpus pairs containing pseudo-labels and constructing expanded corpus pairs without labels to obtain an expanded training sample set; constructing non-smooth corpus pairs containing pseudo-labels includes: constructing corpus pairs containing only the "Filter" category, constructing corpus pairs containing only the "Restart" category, and constructing corpus pairs containing both the "Filter" and "Restart" categories; the method for constructing non-smooth corpus pairs containing pseudo-labels also includes: counting the frequency table of spoken modal particles based on the SWBD dataset, selecting modal particles with a probability greater than 0.7, and randomly inserting them into a position after a segmentation in the original sentence; using the translated corpus in the training sample, segmenting the original sentence, randomly selecting a phrase after the segmentation to repeat, and inserting it after the repeated word; The initial model is trained using the training sample set, and the gradient is updated using the loss function to obtain the text smoothing model; during the model training process, a Coverage mechanism is introduced, including: during the attention calculation process of the decoder, the attention scores before the current time step are summed to obtain the coverage feature vector c t ; c t As the input for calculating attention, the decoder uses the following formula to calculate the attention distribution with coverage mechanism: Among them, v, W h 、W s 、w c , b is a learnable parameter; is the covering feature vector, h i is the encoder hidden state, s t is the decoder status; During the model training process, a copy mechanism is also introduced. The word probability distribution generated by the decoder at each step and the probability of copying the original word are regarded as a hybrid model, and the attention score is used as the probability of copying the word.

2. The speech recognition method according to claim 1, wherein: The converting of the speech to be recognized into text to be smoothed includes: performing speech recognition using a parameter fine-tuning method based on a wav2vec2.0 model, wherein the wav2vec2.0 model includes three parts: a feature extraction layer, a context encoding layer, and an output layer; the feature extraction layer uses a multi-layer convolutional network to implement feature extraction at the speech frame level; the context encoding layer is used to extract correlation features between frames, and the output layer uses a pre-trained language model to output the text to be smoothed after speech recognition.

3. The speech recognition method according to claim 1 or 2, characterized in that: The encoder of the text smoothing model uses T5's native self-attention mechanism-based encoding model; based on the input non-smooth text, a latent vector with a contextual representation of the non-smooth text is obtained; The decoder is used to obtain the probability distribution of target text words through attention calculation based on the encoder latent vector; The output layer includes the non-smooth label prediction and the smooth text generation tasks; Used to output non-smooth label sequences and smooth text.

4. The speech recognition method according to claim 1, wherein: Using a set of modal particles, randomly inserting them into any position of the original sentence to construct the corpus pair containing only the "Filter" category; Randomly select a text segment from the original sentence and insert it into the adjacent position before the segment to construct the corpus pair containing only the "Restart" category; The steps of constructing a corpus pair containing only the "Filter" category and constructing a sample containing only the "Restart" category are sequentially executed on the same sentence to construct a corpus pair containing both the "Filter" and "Restart" categories. The constructing of the expanded corpus pair without labels includes inputting the corpus pair without labels in the original corpus into a similar text expansion model to obtain text similar to the original sentence, and constructing the expanded corpus pair without labels.

5. The speech recognition method according to claim 1, wherein: The loss function includes a loss function for label sequence recognition, a loss function for target text generation, and a damage function combining the two, wherein; The loss function for label sequence recognition is: S RealPath is the actual label path of the input text, is the total score of all tags at a certain moment, is the score of label i at a certain moment; The i-th word is marked as z i probability; Indicates that from label z i to z i+1 The transition probability of η is the CRF weight; The loss function for target text generation is: λ is the balance parameter, δ is the weight parameter of Coverage, is the target word at time step t; is the covering feature vector; The loss function after combining the two is: L(x,y,z|θ)=(1-β)L(x,y|θ)+βL(x,z|θ); Among them, β is the weighting factor.

6. The speech recognition method according to claim 3, wherein: The coverage feature vector c t Expressed as, Among them, c t is the sum of the attention scores output by all decoding time steps before the current time step t, a t ′ is the attention score at step t' during decoding.

7. The speech recognition method according to claim 3, wherein: The probability distribution of the target text words is calculated using the following formula: Among them, P(w) is the final probability distribution of the target text words, p gen is the probability of copying the word of the input sequence, P vocab(w) is the probability distribution of words generated in the vocabulary, w is any word, For all words in the input sequence that are equal to w, the time step t corresponds to a t The probability value of the corresponding dimension, a t is the attention distribution, i.e. a t =softmax(e t ), e t is the attention distribution before normalization; is the context encoding vector of the encoder, h i is the encoder hidden state, s t is the decoder status, b ptr ,V′,V,b,b′ are learnable parameters.

8. A text-smooth speech recognition system, characterized in that: Including voice recognition device and text smoothing device; The speech recognition device is used to recognize the speech data stream and generate the original spoken text transcription; The text smoothing device includes a text smoothing system and an auxiliary annotation system. The text smoothing system is used to generate smooth text from the original spoken transcription text to obtain a smooth text; the auxiliary annotation system is used to detect the non-smoothness of the speech transcription text to obtain a non-smooth label sequence; the text smoothing system and the auxiliary annotation system are both implemented based on a pre-trained text smoothing model; the text smoothing model uses the T5-base pre-training model as the initial model for model training, including an input layer, an encoder-decoder layer and an output layer; the original text vocabulary of the text smoothing model uses the vocabulary of the T5-base open source pre-training model, and special words are appended to the end of the vocabulary <rm / > As the vocabulary of the target text, the special words <rm / > Used to fill the gaps in the target sequence; the input layer uses E′ (V+1)×D Replace E V×D Represents word embedding features, where V and D are the vocabulary size of non-smooth text and the feature dimension of word embedding respectively; The model training includes: expanding samples by constructing non-smooth corpus pairs containing pseudo-labels and constructing expanded corpus pairs without labels to obtain an expanded training sample set; constructing non-smooth corpus pairs containing pseudo-labels includes: constructing corpus pairs containing only the "Filter" category, constructing corpus pairs containing only the "Restart" category, and constructing corpus pairs containing both the "Filter" and "Restart" categories; the method for constructing non-smooth corpus pairs containing pseudo-labels also includes: counting the frequency table of spoken modal particles based on the SWBD dataset, selecting modal particles with a probability greater than 0.7, and randomly inserting them into a position after a segmentation in the original sentence; using the translated corpus in the training sample, segmenting the original sentence, randomly selecting a phrase after the segmentation to repeat, and inserting it after the repeated word; The initial model is trained using the training sample set, and the gradient is updated using the loss function to obtain the text smoothing model; during the model training process, a Coverage mechanism is introduced, including: during the attention calculation process of the decoder, the attention scores before the current time step are summed to obtain the coverage feature vector c t ; c t As the input for calculating attention, the decoder uses the following formula to calculate the attention distribution with coverage mechanism: Among them, v, W h 、W s 、w c , b is a learnable parameter; is the covering feature vector; During the model training process, a copy mechanism is also introduced. The word probability distribution generated by the decoder at each step and the probability of copying the original word are regarded as a hybrid model, and the attention score is used as the probability of copying the word.

9. A computer-readable storage medium, characterized in that The storage medium stores computer instructions, and the instructions are used to be executed by a computer to implement the text-smooth speech recognition method according to any one of claims 1 to 7.

Images