Speech recognition method and device, electronic device, and storage medium

Abstract

The application relates to the technical field of voice processing, and discloses a voice recognition method, which comprises the following steps: recognizing voice information to obtain voice text; performing text feature extraction processing on the voice text to obtain text features; performing position feature extraction processing on the voice text to obtain position features; labeling at least one character in the voice text according to the text features and the position features to obtain label features corresponding to the voice text; and performing redundant character correction processing on the voice text according to the label features, the text features and the position features to obtain corrected voice text. The voice recognition text is more accurate in semantic expression because the text with redundant information can be corrected. The application also discloses a voice recognition device, an electronic device and a storage medium.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, such as a speech recognition method and apparatus, electronic device, and storage medium. Background Technology

[0002] In today's rapidly developing technological world, speech recognition technology has entered the lives of ordinary people. For example, the speech-to-text function in chat software, even with its high accuracy in recognizing individual words and phrases in spoken information, still cannot avoid the problem of redundant information in the converted sentences. For instance, the sentence "I want to go to Zhangjiajie in Hubei Province tomorrow, oh no, I mean Hunan Province" actually means "I want to go to Zhangjiajie in Hunan Province tomorrow." However, if the text were to perfectly capture the meaning of the speech, although each word would be correct, the overall sentence would be unclear. This problem is unavoidable with current speech recognition technology because, in many situations, the brain cannot keep up with the mouth when speaking, easily resulting in incorrect or reversed descriptions and other redundant information. Reversed descriptions include phrases like "Ah, no," "Wrong, wrong," and "Ah, wrong, it should be."

[0003] Existing speech recognition technology cannot correct text with redundant information, resulting in unclear semantics in the recognized text. Summary of the Invention

[0004] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0005] This disclosure provides a method, apparatus, electronic device, and storage medium for speech recognition, so that the semantics of the text obtained after speech recognition are more accurate.

[0006] In some embodiments, a speech recognition method includes: recognizing speech information to obtain speech text; performing text feature extraction processing on the speech text to obtain text features; performing position feature extraction processing on the speech text to obtain position features; labeling at least one character in the speech text according to the text features and the position features to obtain label features corresponding to the speech text; and performing redundant character correction processing on the speech text according to the label features, the text features, and the position features to obtain corrected speech text.

[0007] In some embodiments, a speech recognition apparatus includes: an acquisition module configured to recognize speech information to obtain speech text; a feature extraction module configured to perform text feature extraction processing on the speech text to obtain text features; and perform position feature extraction processing on the speech text to obtain position features; an annotation module configured to annotate at least one character in the speech text according to the text features and the position features to obtain tag features corresponding to the speech text; and a proofreading module configured to perform redundant character proofreading processing on the speech text according to the tag features, the text features, and the position features to obtain proofread speech text.

[0008] In some embodiments, the electronic device includes a processor and a memory storing program instructions, the processor being configured to perform the speech recognition method described above when the program instructions are executed.

[0009] In some embodiments, the storage medium stores program instructions that, when executed, perform the speech recognition method described above.

[0010] The speech recognition method, apparatus, electronic device, and storage medium provided in this disclosure can achieve the following technical effects: First, speech information is converted into speech text, and at least one character in the obtained speech text is marked to identify redundant information in the speech text; then, redundant character correction processing is performed on the obtained speech text to correct redundant information in the speech-converted text, thereby making the semantics of the speech-recognized text more accurate.

[0011] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0012] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein:

[0013] Figure 1 This is a schematic diagram of a speech recognition method provided in an embodiment of this disclosure;

[0014] Figure 2 This is a schematic diagram of the structure of a speech recognition model provided in an embodiment of this disclosure;

[0015] Figure 3 This is a schematic diagram of the structure of a redundancy information verification model provided in an embodiment of this disclosure;

[0016] Figure 4This is a schematic diagram of a method for training a redundancy information correction model provided in an embodiment of this disclosure;

[0017] Figure 5 This is a schematic diagram of another method for training a redundancy information correction model provided in an embodiment of this disclosure;

[0018] Figure 6 This is a schematic diagram of another speech recognition method provided in an embodiment of this disclosure;

[0019] Figure 7 This is a schematic diagram of the structure of a speech recognition device provided in an embodiment of this disclosure;

[0020] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0021] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0022] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0023] Unless otherwise stated, the term "multiple" means two or more.

[0024] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0025] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0026] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0027] Combination Figure 1As shown, this disclosure provides a speech recognition method, including:

[0028] Step S101: Recognize the speech information to obtain the speech text;

[0029] Step S102: Perform text feature extraction processing on the speech text to obtain text features; perform position feature extraction processing on the speech text to obtain position features;

[0030] Step S103: Label at least one character in the speech text according to the text features and position features to obtain the label features corresponding to the speech text;

[0031] Step S104: Perform redundant character proofreading on the speech text based on label features, text features, and location features to obtain the proofread speech text.

[0032] The speech recognition method provided in this embodiment first converts speech information into speech text and marks at least one character in the obtained speech text to identify redundant information in the speech text; then, redundant character correction processing is performed on the obtained speech text to correct redundant information in the speech-converted text, thereby making the semantics of the text recognized by speech more accurate.

[0033] In some embodiments, user-input voice information is recognized and analyzed to obtain voice-text. In some embodiments, voice information is collected through an electronic device, and the collected voice information is recognized and analyzed to obtain voice-text. Optionally, the electronic device includes a voice recorder, smartphone, tablet, etc.

[0034] Optionally, the speech information is recognized to obtain speech text, including: performing convolution processing on the speech information to obtain a speech information tensor; encoding the speech information tensor to obtain a first encoding vector; sequentially obtaining candidate characters at each first preset position based on the first encoding vector and a preset first start symbol; wherein the first preset position is the position corresponding to each character contained in the speech text; and concatenating the candidate characters at each first preset position in the order of their acquisition time to obtain the speech text. In this way, speech information can be recognized as corresponding text information. Subsequent redundant character proofreading of the recognized text information facilitates the removal of redundant information in the speech, thereby obtaining semantically correct speech text.

[0035] Optionally, based on the first encoding vector and the preset first start symbol, candidate characters at each first preset position are obtained sequentially, including: decoding the first encoding vector and the preset first start symbol to obtain the probability distribution of each candidate character in the preset first dictionary at each first preset position; and performing the following processing for each first preset position sequentially: selecting the candidate character with the highest probability from the probability distribution corresponding to the first preset position as the candidate character at the first preset position, thereby obtaining each character contained in the text information corresponding to the speech information.

[0036] Optionally, a pre-trained speech recognition model is used to recognize speech information in order to obtain speech text. The speech information is input into the pre-trained speech recognition model; the speech recognition model performs two-dimensional convolution processing on the speech information to obtain a speech information tensor; the encoder of the speech recognition model encodes the speech information tensor to obtain a first encoded vector. The decoder of the speech recognition model sequentially obtains candidate characters at each of the first preset positions based on the first encoding vector and the preset first start symbol. Specifically, the decoder performs matrix calculations on the first encoding vector and the preset first start symbol to obtain the first character representation vector, and then performs a softmax calculation on the first character representation vector to obtain the probability distribution of each candidate character in the preset first dictionary at the first position after the first start symbol. Candidate characters are then selected from the first dictionary based on this probability distribution. The decoder then repeats the following operations until candidate characters are selected at each of the first preset positions: performing matrix calculations on the first encoding vector and the first concatenated string to obtain the first character representation vector, and then performing a softmax calculation on the first character representation vector to obtain the probability distribution of each candidate character in the preset first dictionary at the first position after the first concatenated string. Candidate characters are then selected from the first dictionary based on this probability distribution. The speech recognition model then concatenates the selected candidate characters at each of the first preset positions in the order they were obtained to obtain the speech text. In this way, the speech recognition model can obtain speech text that most closely matches the meaning of the user's speech information.

[0037] Optionally, if the candidate character selected from the first dictionary is a preset first end symbol, then candidate characters are selected for each first preset position; wherein, the first position after the first start symbol to the last position before the first end symbol are each first preset position. Therefore, the positions from the first position after the first start symbol to the last position before the first end symbol are the positions of each character in the spoken text from left to right.

[0038] Optionally, the first concatenation string is obtained by concatenating the selected candidate characters sequentially after the first start symbol according to the order in which they were obtained, thus forming the first concatenation string.

[0039] Optionally, candidate characters are selected from the first dictionary based on the probability distribution, including: selecting the candidate character corresponding to the highest probability in the probability distribution from the first dictionary.

[0040] Taking a voice text containing n characters as an example, the position of each character in the voice text is the first preset position mentioned above. For example, when the voice text is "I miss you", n=3, where "I" is the first character in the voice text and is located in the first position from left to right (i.e., the first first preset position); "miss" is the second character in the voice text and is located in the second position from left to right (i.e., the second first preset position); and "you" is the third character in the voice text and is located in the third position from left to right (i.e., the third first preset position).

[0041] Thus, the execution process of the decoder of the above speech recognition model may specifically include: (1) performing matrix calculation on the first encoded vector and a preset first start symbol to obtain a first character representation vector corresponding to the first character "我", and performing softmax calculation on the first character representation vector corresponding to the first character "我" to obtain the probability distribution of each alternative character in a preset first dictionary at the first position after the first start symbol (i.e., the first first preset position above), and selecting an alternative character from the first dictionary according to this probability distribution as the first character included in the speech text, that is, the character "我"; (2) concatenating the selected first character "我" after the first start symbol to form a first concatenated string corresponding to the first character "我", performing matrix calculation on the first encoded vector and the first concatenated string corresponding to the first character "我" to obtain a first character representation vector corresponding to the second character "想", and performing softmax calculation on the first character representation vector corresponding to the second character "想" to obtain the probability distribution of each alternative character in the preset first dictionary at the first position after the first concatenated string of the character "我" (i.e., the second first preset position above), and selecting an alternative character from the first dictionary according to this probability distribution as the second character included in the speech text, that is, the character "想"; (3) concatenating the selected second character "想" after the first concatenated string corresponding to the first character "我" to form a first concatenated string corresponding to the second character "想", performing matrix calculation on the first encoded vector and the first concatenated string corresponding to the second character "想" to obtain a first character representation vector corresponding to the third character "你", and performing softmax calculation on the first character representation vector corresponding to the third character "你" to obtain the probability distribution of each alternative character in the preset first dictionary at the first position after the first concatenated string of the character "想" (i.e., the third first preset position above), and selecting an alternative character from the first dictionary according to this probability distribution as the third character included in the speech text, that is, the character "你", so as to recognize the speech text "我想你". By analogy, the speech recognition model can recognize and generate speech texts with n being any positive integer.

[0042] In some embodiments, the preset first start symbol is <cls>For a symbol, the decoder of the speech recognition model fuses the first start symbol with the first encoded vector transmitted from the encoder of the speech recognition model. After matrix operations of multiple decoding blocks of the decoder and the final softmax calculation, the probability distribution of each alternative character in the first dictionary is output. Among them, is the probability distribution of each alternative character in the first dictionary at the position corresponding to the t-th character in the speech text. For example, if the first dictionary is {one, already, with, according to, meaning, easy}, and t is 1, then the probability distributions of each alternative character "one", "already", "with", "according to", "meaning", "easy" in the first dictionary at the position corresponding to the first character in the speech text are [0.01, 0.95, 0.01, 0.01, 0.01, 0.01]; and the alternative character at the position corresponding to the maximum value of its probability is output. For example, if the maximum value of this probability distribution is 0.95 and it appears in the second position, then the alternative character in the second position of the first dictionary is output, that is, "already"; among them, the length of the first dictionary is the same as the length of the probability distribution. The output alternative characters are sequentially concatenated to the first start symbol in the order of selection from front to back. <cls>Then, a first concatenated string is formed. This first concatenated string and the first encoded vector are input again into the decoder of the speech recognition model for further decoding to obtain candidate characters. The decoder of the speech recognition model repeatedly executes the above decoding process. When the decoded candidate character is a preset first end symbol, for example... <end>If all candidate characters are selected at each of the first preset positions, the decoding process ends. All the decoded candidate characters are then concatenated in chronological order of their acquisition time to obtain the speech text.

[0043] Optionally, before recognizing speech information using a pre-trained speech recognition model, the method further includes training the speech recognition model, wherein training the speech recognition model includes: inputting sample speech information and its corresponding sample text information into the ASR (Automatic Speech Recognition) model for training to obtain the speech recognition model.

[0044] Combination Figure 2 As shown, optionally, the ASR model includes an encoder and a decoder. Optionally, the sample speech information X and its corresponding sample text information Y are input into the ASR model. The ASR model uses a 2D-Conv Subsampling operation to quantize the sample speech information X into a sample speech information tensor Xs. An embedding operation and a position embedding operation are then applied to the sample speech information tensor Xs to map it into a vector that can be computed by a multi-head attention mechanism. The encoder of the ASR model encodes the mapped sample speech information tensor Xs and the positional encoding, i.e., performs matrix calculations to obtain the first encoded vector He. The decoder of the ASR model encodes the sample text information Y, and maps the encoded sample text information Y to a sample text vector using word embedding and position embedding. The decoder of the ASR model then processes the sample text vector and the positional encoding. The encoding and the first sample encoding vector He are used to perform matrix calculations to obtain the first sample character representation vector. This first sample character representation vector is then subjected to softmax calculation to obtain the sample speech text output by the ASR model. The similarity between the sample speech text output by the ASR model and the input sample text information Y is calculated using cross-entropy. This similarity is determined as the third loss value. Based on the third loss value, the parameters of the ASR model are continuously adjusted through gradient backpropagation. Training ends when the third loss value is less than or equal to a set threshold, thus obtaining the speech recognition model. Optionally, training ends when the third loss value is greater than the set threshold, but a preset number of training iterations has been reached, thus obtaining the speech recognition model.

[0045] Optionally, by calculating the loss function L asr = -log p(y|x) yields the third loss value; where, L asr y is the third loss value, x is the input sample speech information, and y is the output sample speech text.

[0046] Optionally, the encoder of the ASR model has a Transformer structure. Optionally, the encoder of the ASR model includes several encoding blocks, each of which has the same structure. In the computation of each encoding block of the encoder, a multi-head attention is first performed, followed by a residual connection and normalization operation (Add & Norm), then a feedforward neural network operation, and finally an Add & Norm operation to output a representation vector.

[0047] Optionally, the sample text vector is a vector that can be computed by Multi-Head Attention.

[0048] Optionally, the encoder of the ASR model encodes the mapped sample speech information tensor Xs and PositionalEncoding to obtain the first encoded vector He. This process includes: the encoder first performs a first encoding block calculation on the mapped sample speech information tensor Xs and PositionalEncoding and outputs a first representation vector; then performs a second encoding block calculation on the first representation vector and outputs a second representation vector; then performs the next encoding block calculation on the second representation vector, and so on, until the encoder has performed M encoding blocks, resulting in the first encoded vector He; where M is a positive integer, for example, M = 12. This first encoded vector He is an encoded form of the sample speech information X.

[0049] Optionally, the decoder of the ASR model has a Transformer structure. Optionally, the decoder of the ASR model includes several decoding blocks, each of which has an identical structure. In the computation of each decoding block, the process first involves a masked multi-head attention mechanism followed by an Add&Norm operation, then another multi-head attention mechanism followed by an Add&Norm operation, then a Feed Forward operation, and finally an Add&Norm operation to output a representation vector.

[0050] Optionally, the decoder of the ASR model performs matrix calculations on the sample text vector, Positional Encoding, and the first sample encoding vector He to obtain the first sample character representation vector. This includes: the decoder of the ASR model performs a first decoding block calculation on the sample text vector, Positional Encoding, and the first sample encoding vector He and outputs the first representation vector; it performs a second decoding block calculation on the first representation vector and outputs the second representation vector; it then performs the next decoding block calculation on the second representation vector, and so on, until after N blocks of calculation by the decoder, it outputs the Nth representation vector; the Nth representation vector is determined as the first sample character representation vector; where N is a positive integer, for example, N is 12.

[0051] Optionally, text feature extraction processing is performed on the speech text to obtain text features, including: performing the following processing for each character in the speech text: performing a table lookup operation on the character to obtain the character vector corresponding to the character; performing matrix operations on the character vector and a preset word embedding matrix to obtain the character tensor corresponding to the character;

[0052] The character tensors corresponding to each character in the speech text are fused to obtain the corresponding text tensor, which is then used as the text feature. This allows for the extraction of text features, facilitating redundant character proofreading based on these features and ultimately removing redundant information from the speech text.

[0053] Optionally, perform text feature extraction processing on the speech text using a pre-trained redundant information proofreading model to obtain text features. Input the speech text into the pre-trained redundant information proofreading model; the redundant information proofreading model performs a look-up table operation on each character in the speech text according to a preset character vocabulary, finds the position of each character in the preset character vocabulary, and obtains the character vector corresponding to each character, that is, each character vector is a vector with a length equal to the length of the character vocabulary, a position of 1 in the vocabulary, and all other positions being 0; a number of characters are stored in the preset character vocabulary; perform matrix operations on each character vector with the preset word embedding matrix respectively to obtain the character tensor corresponding to each character in the speech text; perform fusion processing on the character tensors corresponding to each character in the speech text to obtain the text tensor corresponding to the speech text, and determine the text tensor as the text feature. For example, the character length of the speech text is 10, and it contains the character "今". The position of this character "今" in the character vocabulary is the 5th position, then the obtained character vector corresponding to the character "今" has a length of [character vocabulary length], a position of 1 in the 5th position, and all other positions being 0. The word embedding matrix (word_embedding) is a matrix with a dimension of [character vocabulary length, a] (where a is a positive integer). Perform matrix operations on this character vector [1, character vocabulary length] and this word embedding matrix, then obtain the character tensor corresponding to this character, which is a tensor of [1, a]. Perform fusion processing on the character tensors corresponding to the 10 characters in this speech text respectively, then finally obtain a text tensor of [10, a].

[0054] Optionally, perform position feature extraction processing on the speech text to obtain position features, including: performing the following processing for the position corresponding to each character in the speech text: performing a look-up table operation on the position corresponding to the character to obtain the position vector corresponding to the position; performing matrix operations on the position vector with the preset position embedding matrix to obtain the position tensor corresponding to the position;

[0055] Perform fusion processing on the position tensors corresponding to the positions of each character in the speech text to obtain the position tensor corresponding to the speech text, and determine the position tensor as the position feature. In this way, the position features corresponding to the speech text can be extracted, which is convenient for performing redundant character proofreading processing on the speech text according to the position features, so as to remove the redundant information in the speech.

[0056] Optionally, a pre-trained redundant information proofreading model is used to perform position feature extraction processing on the speech text to obtain position features. The speech text is input into the pre-trained redundant information proofreading model; the redundant information proofreading model performs a look-up table operation on the positions corresponding to each character in the speech text according to a preset text character length to obtain position vectors corresponding to the positions of each character; the position vectors are respectively subjected to matrix operations with a preset position embedding matrix to obtain position tensors corresponding to the positions of each character; the position tensors corresponding to the positions of each character in the speech text are fused to obtain a position tensor corresponding to the speech text, and the position tensor is determined as the position feature. In some embodiments, in the redundant information proofreading model, the preset text character length is 512. For example, if the speech text is "The weather is nice today", and the position of the character "气" is the fourth, then the position vector corresponding to the position of the character "气" is of length

[512] , with a value of 1 at the fourth position and 0 at the remaining positions; this position vector is subjected to a matrix operation with the position embedding matrix (position_embedding) to obtain the position tensor corresponding to the position of this character. Among them, the dimension of the position embedding matrix is [512, a] (where a is a positive integer).

[0057] Optionally, at least one character in the speech text is labeled according to the text feature and the position feature to obtain a label feature corresponding to the speech text, including: performing the following processing on each character in at least one character in the speech text: extracting a text sub-feature corresponding to the character from the text feature, extracting a position sub-feature corresponding to the character from the position feature, and performing a tensor calculation on the text sub-feature and the position sub-feature to obtain a target tensor corresponding to the character;

[0058] The target tensors corresponding to each character in at least one character in the speech text are fused to obtain a label tensor corresponding to the speech text, and the label tensor is determined as the label feature corresponding to the speech text.

[0059] Optionally, the label feature includes a label feature indicating a character error, a label feature indicating a correct character, a label feature indicating a semantic inversion, and a label feature indicating an unchanged character.

[0060] In this way, the label feature corresponding to the speech text can be obtained, which is convenient for performing redundant character proofreading processing on the speech text according to the label feature, so as to remove redundant information in the speech.

[0061] Optionally, a pre-trained redundancy information correction model is used to perform redundancy information correction on the speech text to obtain the corrected speech text. Specifically, the pre-trained redundancy information correction model is used to perform text feature extraction and positional feature extraction on the speech text to obtain text features and positional features, respectively; the pre-trained redundancy information correction model is used to label at least one character in the speech text based on the text features and positional features to obtain the corresponding label features of the speech text; the pre-trained redundancy information correction model is used to perform redundant character correction on the speech text based on the label features, text features, and positional features to obtain the corrected speech text.

[0062] Optionally, the pre-trained redundancy information correction model includes: a pre-trained character sequence labeling model and a pre-trained text modification model. The pre-trained character sequence labeling model is used to label at least one character in the speech text based on text features and positional features to obtain the corresponding label features of the speech text. The pre-trained text modification model is used to perform redundancy character correction processing on the speech text based on label features, text features, and positional features to obtain the corrected speech text.

[0063] Optionally, a pre-trained character sequence labeling model is used to label at least one character in the speech text based on text features and positional features to obtain the label features corresponding to the speech text. The character sequence labeling model performs the following processing for each character in the speech text: extracting the text sub-features of the corresponding character from the text features, extracting the positional sub-features of the corresponding character from the positional features, and performing tensor calculations on the extracted text sub-features and positional sub-features to obtain the target tensor corresponding to the character. The character sequence labeling model fuses the target tensors corresponding to each character in the speech text to obtain the label tensor corresponding to the speech text, and determines the label tensor as the label feature corresponding to the speech text. In this way, by labeling different characters through the character sequence labeling model and obtaining the label features corresponding to the speech text, it is easier to locate which characters in the text are redundant and which are correct.

[0064] In some embodiments, at least one character in the spoken text may be a Chinese character, an English character, or the like.

[0065] Optionally, redundant character correction processing is performed on the speech text based on label features, text features, and positional features to obtain the corrected speech text. This includes: encoding the label features, text features, and positional features to obtain a second encoding vector; obtaining correction characters at each of the second preset positions sequentially based on the second encoding vector and a preset second start symbol; wherein the second preset position is the position corresponding to each character contained in the corrected speech text; and concatenating the correction characters at each of the second preset positions sequentially according to the order in which they were obtained to obtain the corrected speech text. In this way, redundant character correction of the speech text based on label features, text features, and positional features can correct text with redundant information, making the semantics of the recognized text clearer.

[0066] Optionally, based on the second encoding vector and the preset second start symbol, the correction characters at each preset position are obtained sequentially, including: decoding the second encoding vector and the preset second start symbol to obtain the probability distribution of each correction character in the preset second dictionary at each second preset position; and performing the following processing for each second preset position sequentially: selecting the correction character corresponding to the highest probability in the probability distribution corresponding to the second preset position as the correction character at the second preset position.

[0067] Optionally, a pre-trained text modification model is used to perform redundant character proofreading on the speech text based on label features, text features, and positional features to obtain the proofread speech text. Optionally, the text modification model includes an encoder and a decoder. The encoder of the text modification model adds the label features, text features, and positional features and then performs encoding processing to obtain a second encoded vector. The decoder of the text modification model sequentially obtains the correction characters at each of the second preset positions based on the second encoding vector and the preset second start symbol. Specifically, the decoder performs matrix calculations on the second encoding vector and the preset second start symbol to obtain the second character representation vector. After processing the second character representation vector through a preset neural network, a softmax operation is performed to obtain the probability distribution of each correction character in the preset second dictionary at the first position after the second start symbol. The decoder then selects a correction character from the second dictionary based on this probability distribution. The decoder then repeats the following operations until a correction character is selected at each of the second preset positions: performing matrix calculations on the second encoding vector and the second concatenated string to obtain the second character representation vector; processing the second character representation vector through a preset neural network, a softmax operation is performed to obtain the probability distribution of each correction character in the preset second dictionary at the first position after the second concatenated string; and the decoder selects a correction character from the second dictionary based on this probability distribution. The text modification model then concatenates the selected correction characters sequentially according to the order in which they were obtained to obtain the corrected speech text.

[0068] Optionally, if the correction character selected from the second dictionary is a preset second end symbol, then candidate characters are selected for each second preset position; wherein, the first position after the second start symbol to the last position before the second end symbol are each second preset position. Therefore, the positions from the first position after the second start symbol to the last position before the second end symbol are the positions of each character in the corrected speech text from left to right.

[0069] Optionally, the second concatenation string is obtained by concatenating the selected correction characters sequentially after the second start symbol according to the order in which they were obtained, thus forming the second concatenation string.

[0070] Optionally, a correction character is selected from the second dictionary based on the probability distribution, including selecting the correction character corresponding to the highest probability in the probability distribution from the second dictionary.

[0071] At least one character in the speech text is labeled by a pre-trained character sequence labeling model to obtain the label features corresponding to the speech text; wherein, the label features include label features representing character errors, label features representing correct characters, label features representing semantic inversion, and label features representing unchanged characters, and then the text modification model performs redundant character proofreading on the speech text according to the label features, text features, and position features.

[0072] In the embodiment of the present application, since the label features can label and identify redundant characters in the speech text, and not only use the text features and position features to represent the speech text during the correction process, but also use the label features to represent the speech text, it can make the second encoding vector that simultaneously fuses the label features, text features, and position features maximize the restoration of the speech text, thereby improving the upper limit of machine learning, improving the fitting ability of the model, and further making the model more accurate in correcting texts with redundant information.

[0073] Taking the example that the proofread speech text contains m characters, the position corresponding to each character in the proofread speech text is the above-mentioned second preset position. For example, when the proofread speech text is "Good morning", m = 3, where "早 (zǎo)" is the first character in the proofread speech text and is in the first position from left to right in the proofread speech text (i.e., the first second preset position); "上 (shàng)" is the second character in the proofread speech text and is in the second position from left to right in the proofread speech text (i.e., the second second preset position), and "好 (hǎo)" is the third character in the proofread speech text and is in the third position from left to right in the proofread speech text (i.e., the third second preset position).

[0074] Thus, the execution process of the decoder of the above text modification model can specifically include: (1) performing matrix calculation on the second encoded vector and the preset second start symbol to obtain a second character representation vector corresponding to the first character "zao", and performing softmax calculation on the second character representation vector corresponding to the first character "zao" to obtain the probability distribution of each corrected character in the preset second dictionary at the first position after the second start symbol (i.e., the above first second preset position), and selecting a corrected character from the second dictionary according to this probability distribution as the first character included in the corrected speech text, that is, the character "zao"; (2) concatenating the selected first character "zao" after the second start symbol to form a second concatenated string corresponding to the first character "zao", performing matrix calculation on the second encoded vector and the second concatenated string corresponding to the first character "zao" to obtain a second character representation vector corresponding to the second character "shang", and performing softmax calculation on the second character representation vector corresponding to the second character "shang" to obtain the probability distribution of each corrected character in the preset second dictionary at the first position after the second concatenated string of the character "zao" (i.e., the above second second preset position), and selecting a corrected character from the second dictionary according to this probability distribution as the second character included in the corrected speech text, that is, the character "shang"; (3) concatenating the selected second character "shang" after the second concatenated string corresponding to the first character "zao" to form a second concatenated string corresponding to the second character "shang", performing matrix calculation on the second encoded vector and the second concatenated string corresponding to the second character "shang" to obtain a second character representation vector corresponding to the third character "hao", and performing softmax calculation on the second character representation vector corresponding to the third character "hao" to obtain the probability distribution of each corrected character in the preset second dictionary at the first position after the second concatenated string of the character "shang" (i.e., the above third second preset position), and selecting a corrected character from the second dictionary according to this probability distribution as the third character included in the corrected speech text, that is, the character "hao", so as to obtain the corrected speech text "good morning". By analogy, the text modification model can proofread and generate a corrected speech text with m being any positive integer.

[0075] In some embodiments, the text modification model performs text mapping on the speech text to obtain a text tensor (i.e., the text features mentioned above); it performs character position mapping on the speech text to obtain a position tensor (i.e., the position features mentioned above); the text tensor, position tensor, and label features output by the character sequence labeling model are summed and then input into several blocks of the Encoder of the text modification model for calculation to obtain a second encoding vector; wherein, during the block calculation process, the dimension of the tensor is always maintained at 'a' (where a is a positive integer); the dimension of the second encoding vector is (speech text length, a). The Decoder of the text modification model combines the second encoding vector with a preset second start symbol (e.g., ...). <cls>The input is fed into the decoder of the text modification model. After matrix operations across multiple blocks and a final softmax calculation, the probability distribution of each corrected character in the second dictionary at the first position after the second start symbol is obtained. The corrected character at the position corresponding to the maximum probability is output. The length of the second dictionary is the same as the length of the probability distribution. The output corrected characters are then concatenated sequentially according to the order in which they were obtained to the second start symbol. <cls>Then, a second concatenated string is constructed. This second concatenated string and the second encoding vector are input again into the decoder of the text modification model for further decoding to extract the correction character. The decoder of the text modification model repeatedly executes the above decoding process. When the decoded correction character is a preset second end symbol, for example... <end>If all the corrected characters are selected at each of the second preset positions, the decoding process ends. All the decoded corrected characters are then concatenated in chronological order of acquisition to obtain the corrected speech text, i.e., the semantically accurate speech text. Optionally, if several speech texts are simultaneously input into the text modification model, the resulting second encoding vector dimension is (number of speech texts, maximum speech text length, a). In this way, the ambiguous speech text is corrected through the text modification model, resulting in unambiguous and semantically clear speech text.

[0076] In some embodiments, the preset second dictionary includes 5000 characters, for example, " <cls>Ah, ah, ya, ....., Zhang, Zui, ...., <end>For the second character representation vector output by the last block of the Decoder in the text modification model, it passes through a feedforward neural network of dimension (a, 5000) (where a is a positive integer), and then through softmax to obtain a probability distribution of dimension 5000. Then, the Chinese character with the largest probability distribution is selected as the output at this time.

[0077] Optionally, the pre-trained character sequence labeling model calculates the label features corresponding to the speech text using a preset neural network Linear, and finally performs a softmax operation to obtain the probability distribution of at least one character in the speech text corresponding to each label in the preset label dictionary. The label corresponding to the highest probability is selected from each probability distribution, and the selected label is output to obtain the label corresponding to at least one character in the speech text output by the character sequence labeling model.

[0078] Optionally, the labels include labels representing character errors, labels representing correct characters, labels representing semantic inversion, and labels representing unchanged characters. For example, if the input speech text to the character sequence labeling model is "I want to go to Zhangjiajie, Hubei Province, oh no, Hunan Province", then the corresponding output labels would be:

[0079] " <o> <o> <o> <wrong> <wrong> <wrong> <o> <o> <o> <turn> <turn><RIGH T> <right> <right>The character sequence labeling model identifies words and phrases in a user's speech text that are suspected of containing redundant errors. Specifically, "O" indicates a character that remains unchanged, "WRONG" indicates a character with errors, "TURN" indicates a semantic reversal, and "RIGHT" indicates a correct character.

[0080] Optionally, in the speech text, characters corresponding to labels indicating character errors and labels indicating semantic inversion are deleted to obtain semantically clear speech text. By labeling different Chinese characters in the speech text using a character sequence labeling model, redundant and correct characters can be identified. Deleting redundant characters yields speech text with clearer semantics. In some embodiments, speech information is recognized to obtain speech text; the speech text is input into a pre-trained redundancy information correction model to label each character, obtaining labels for each character; wherein, the character labels are one of the following: labels indicating character errors, labels indicating correct characters, labels indicating semantic inversion, and labels indicating unchanged characters; characters labeled with labels indicating character errors and characters labeled with labels indicating semantic inversion are deleted, and the word order of the speech text after character deletion is adjusted, thereby achieving redundant character correction of the speech text to obtain corrected speech text.

[0081] Optionally, before using the pre-trained redundancy information correction model to perform redundancy information correction on the speech text, the method further includes training the redundancy information correction model. This training includes: training a first BERT model and a second BERT model; determining a stopping condition during the training of the first and second BERT models; completing the training of the first BERT model to obtain a character sequence labeling model when the stopping condition is met; and completing the training of the second BERT model to obtain a text modification model when the stopping condition is met. In this way, by simultaneously training the first and second BERT models and completing their respective training under the same stopping condition, a redundancy information correction model is obtained. This allows for redundancy character correction of the speech text using the redundancy information correction model, resulting in clearer semantics in the corrected speech text.

[0082] Optionally, training the first BERT model includes: training the first BERT model using pre-set sample speech text and sample labels corresponding to each sample character in the sample speech text. Optionally, the sample labels include labels representing character errors, labels representing correct characters, labels representing semantic inversion, and labels representing unchanged characters.

[0083] Combination Figure 3 As shown, optionally, the structure of the first BERT model is a Transformer structure. Optionally, the first BERT model is trained using preset sample speech text and sample labels corresponding to each sample character in the sample speech text, including: inputting the preset sample speech text and sample labels corresponding to at least one sample character in the sample speech text into the first BERT model; the first BERT model performs a lookup operation on each sample character in the sample speech text to obtain a sample character vector corresponding to each sample character; performs matrix operations on each sample character vector with a preset word embedding matrix to obtain a sample character tensor corresponding to each sample character; fuses the sample character tensors to obtain a sample text tensor corresponding to the sample speech text, i.e., sample text features; the first BERT model performs a lookup operation on the position corresponding to each sample character to obtain a sample position vector corresponding to each sample character; performs matrix operations on each sample position vector with a preset position embedding matrix to obtain a sample position tensor corresponding to each sample character; fuses the sample position tensors to obtain a sample position tensor corresponding to the sample speech text, i.e., sample position features. The first BERT model extracts sample text sub-features corresponding to sample characters with sample labels from sample text features, and extracts sample position sub-features corresponding to sample characters with sample labels from sample position features. Using several Transformer Blocks of the first BERT model, it performs tensor computation and fusion processing on the extracted sample text sub-features and sample position sub-features to obtain the sample label tensor corresponding to the sample speech text, i.e., the sample label features. The first BERT model calculates the sample label features using a pre-defined neural network Linear, and finally performs a softmax operation to obtain the probability distribution of each sample character in the sample speech text corresponding to each label in a pre-defined label dictionary. It selects the label corresponding to the highest probability from each probability distribution and outputs the selected label, obtaining the output label corresponding to each sample character of the first BERT model. It calculates the similarity between the output label corresponding to each sample character of the first BERT model and the sample label corresponding to each sample character in the input sample speech text, and adjusts the parameters of the first BERT model through backpropagation until the first BERT model training meets the stopping condition.

[0084] Optionally, the word embedding matrix and the position embedding matrix are initialized using a random initialization operation, that is, the elements in the word embedding matrix and the elements in the position embedding matrix are randomly initialized. During the subsequent training process, the elements of the word embedding matrix and the elements of the position embedding matrix are updated through backpropagation.

[0085] Optionally, the first BERT model includes several Transformer Blocks, each with the same structure. In the computation of each Transformer Block, it first goes through a multi-head attention mechanism, then through a residual connection and normalization operation (Add&Norm), then through a feedforward neural network operation, and finally through an Add&Norm operation to output a representation vector.

[0086] Optionally, the extracted sample text sub-features and sample location sub-features are processed by tensor calculation and fusion using several Transformer Blocks of the first BERT model to obtain the sample label features corresponding to the sample speech text. This includes: inputting the extracted sample text sub-features and sample location sub-features into the first Transformer Block of the first BERT model for calculation and outputting the first representation vector; inputting the first representation vector into the second Transformer Block of the first BERT model for calculation and outputting the second representation vector; inputting the second representation vector into the next Transformer Block for calculation, and so on, until the calculation is performed by N' Transformer Blocks of the first BERT model to obtain the sample label features corresponding to the sample speech text; where N' is a positive integer, for example, N' is 12.

[0087] In some embodiments, the preset label dictionary is [O, WRONG, RIGHT, TURN], where "O" is the label representing unchanged characters, "WRONG" is the label representing incorrect characters, "RIGHT" is the label representing correct characters, and "TURN" is the label representing semantic inversion. First, the first BERT model maps the sample speech text into a computable sample text tensor through the embedding method, maps the positions of each sample character in the sample speech text into a sample position tensor, and first calculates the sample text tensor and the sample position tensor through the first Transformer Block to output the first representation vector. During the calculation process of the Transformer Block, the dimension of the tensor always remains at a (where a is a positive integer); then, this first representation vector is input into the next Transformer Block and the calculation is repeated. After the calculation of N' Transformer Blocks, a tensor with a dimension of (length of speech text, a) is obtained, that is, the sample label feature corresponding to the sample speech text. A neural network with a dimension of (a * 4) is used for this sample label feature to transform it into a tensor with a dimension of (length of speech text, 4), and finally, a softmax operation is performed to obtain the probability distribution of each character in the sample speech text corresponding to each label in the label dictionary; the label corresponding to the maximum probability is selected from each probability distribution, and the selected label is output to obtain the output label of each character in the speech text output by the first BERT. For example: the sample speech text is "我是个好人", and the sample label feature obtained after inputting it into the first BERT model is:

[0088] [0.98, 0.01, 0.005, 0.005 0.97, 0.01, 0.01, 0.01 0.96, 0.02, 0.01, 0.01 0.95, 0.01, 0.03, 0.01 0.94, 0.02, 0.02, 0.02]

[0089] Among them, the probability distribution of the character "我" is the numbers in the first row, and 0.98 is the largest. Therefore, the label corresponding to the character "我" is the label corresponding to the position of 0.98, which is "O". The same applies to other characters, and the final output label of "我是个好人" is " <o> <o> <o> <o> <o>".

[0090] Optionally, training the second BERT model includes: training the second BERT model using preset sample speech text, the proofreading text corresponding to the sample speech text, and the sample label features corresponding to the sample speech text output by the first BERT model.

[0091] Combination Figure 3 As shown, optionally, the second BERT model includes an Encoder and a Decoder. The second BERT model is trained using preset sample speech text, corresponding proofreading text, and sample label features output by the first BERT model. This includes: inputting the preset sample speech text, corresponding proofreading text, and sample label features output by the first BERT model into the second BERT model; performing a lookup operation on each sample character in the sample speech text to obtain a sample character vector; performing matrix operations on each sample character vector with a preset word embedding matrix to obtain a sample character tensor; fusing the sample character tensors to obtain a sample text tensor corresponding to the sample speech text, i.e., sample text features; performing a lookup operation on the position corresponding to each sample character to obtain a sample position vector; performing matrix operations on each sample position vector with a preset position embedding matrix to obtain a sample position tensor corresponding to each sample character; and fusing the sample position tensors to obtain a sample position tensor corresponding to the sample speech text, i.e., sample position features. The encoder of the second BERT model adds the sample text features (word_embedding), sample position features (position_embedding), and sample tag features (tags_embedding) before encoding them to obtain the sample second encoding vector He'. The second BERT model maps the proofreading text corresponding to the sample speech text to obtain the sample proofreading text vector. The decoder of the second BERT model performs matrix calculations on the sample proofreading text vector and the sample second encoding vector He' to obtain the second sample character representation vector. The second sample character representation vector is then processed by softmax to obtain the output text of the second BERT model. The similarity between the output text of the second BERT model and the proofreading text corresponding to the sample speech text is calculated using cross-entropy. The parameters of the second BERT model are continuously adjusted through gradient backpropagation until the training of the second BERT model meets the stopping condition.

[0092] Optionally, the encoder of the second BERT model has a Transformer structure, i.e., a Transformer-Encoder. Optionally, the encoder of the second BERT model includes several Transformer Blocks, and the structure of each Transformer Block of the encoder is exactly the same.

[0093] Optionally, the encoder of the second BERT model adds the sample text features (word_embedding), sample position features (position_embedding), and sample tag features (tags_embedding), and then inputs the sum into the first Transformer Block of the second BERT model encoder to calculate and output the first representation vector. The first representation vector is then input into the second Transformer Block of the encoder to calculate and output the second representation vector. The second representation vector is then input into the next Transformer Block for calculation, and so on, until the calculation is performed by M' Transformer Blocks of the encoder, to obtain the second encoded vector He' of the sample; where M' is a positive integer, for example, M' is 12.

[0094] Optionally, the decoder of the second BERT model has a Transformer structure, i.e., a Transformer-Decoder. Optionally, the decoder of the second BERT model includes several blocks, and the structure of each block of the decoder is exactly the same. In the computation of each block of the decoder, it first goes through a Masked Multi-head Attention mechanism, then through an Add&Norm operation, then through another Masked Multi-head Attention mechanism, then through an Add&Norm operation, then through a FeedForward neural network operation, and finally through an Add&Norm operation to output a representation vector.

[0095] Optionally, the decoder of the second BERT model performs matrix calculations on the sample proofreading text vector and the sample second encoding vector He' to obtain the second sample character representation vector. This includes: inputting the sample proofreading text vector and the sample second encoding vector He' into the first block of the decoder of the second BERT model for calculation and outputting the first representation vector; inputting the first representation vector into the second block of the decoder for calculation and outputting the second representation vector; inputting the second representation vector into the next block for calculation, until after N” blocks of the decoder, the N”th representation vector is obtained, and the N”th representation vector is determined as the second sample character representation vector; where N” is a positive integer, for example, N” is 12.

[0096] Optionally, the training stop condition is obtained by: obtaining the first loss value of the first BERT model and the second loss value of the second BERT model; and determining the training stop condition based on the first and second loss values.

[0097] Optionally, obtaining the first loss value of the first BERT model includes: determining the similarity between the output label of at least one character output by the first BERT model and the sample label corresponding to at least one character in the sample speech text as the first loss value.

[0098] Optionally, by calculation Obtain the first loss value; where L TRC-ST The first loss value is given by Q, where Q is the number of sentences in the sample speech text, cross-entropy is the cross-entropy loss function, and label is the first loss value. i The sample label corresponding to at least one character in the i-th sentence of the sample speech text, f (TRC-ST) (x';θ) i The output label is obtained by inputting the i-th sample speech text x' into the first BERT model; θ is the parameter of the first BERT model.

[0099] Optionally, obtaining the second loss value of the second BERT model includes: determining the similarity between the output text of the second BERT model and the proofread text corresponding to the sample speech text as the second loss value.

[0100] Optionally, by calculation Obtain the second loss value; where L TRC-CS The second loss value is represented by Q, where Q is the number of sentences in the sample speech text, cross-entropy is the cross-entropy loss function, and S is the number of sentences in the sample speech text. i The proofreading text corresponding to the i-th sample speech text, f (TRC-CS) (x';θ') i The output text obtained after inputting the i-th sample speech text x' into the second BERT model; θ' is the parameter of the second BERT model.

[0101] Optionally, determining the training stop condition based on the first loss value and the second loss value includes: calculating the total loss value using the first loss value and the second loss value through a preset algorithm; determining the training stop condition as the total loss value being less than or equal to a preset loss threshold; or determining the training stop condition as the number of training iterations reaching a preset number when the total loss value is greater than the preset loss threshold.

[0102] Optionally, by calculating L TRC =L TRC-CS +λL TRC-ST Obtain the total loss value, where L TRC λ represents the total loss value, and λ is a coefficient between [0, 1]. λ is used to control the relationship between the first loss value and the second loss value. The closer λ is to 0, the less the redundancy information correction model uses the localization guidance information of the first BERT model when generating the model; the closer λ is to 1, the more the redundancy information correction model uses the localization guidance information of the first BERT model when generating the model.

[0103] Optionally, training the first BERT model includes: using the first loss value of the first BERT model as a first input parameter and inputting it into a preset first backpropagation algorithm; and using the first backpropagation algorithm with the first input parameter to adjust the parameters in the first BERT model.

[0104] Optionally, training the second BERT model includes: using the second loss value of the second BERT model as a second input parameter and inputting it into a preset second backpropagation algorithm; using the second backpropagation algorithm with the second input parameter to adjust the parameters in the second BERT model until the training stop condition is met.

[0105] Combination Figure 4 As shown, in some embodiments, a method for training a redundancy information correction model includes:

[0106] Step S401: Input the preset sample speech text and the sample label corresponding to at least one character in the sample speech text into the first BERT model, train the first BERT model, and output the sample label features corresponding to the sample speech text.

[0107] Step S402: Obtain the first loss value of the first BERT model.

[0108] Step S403: Input the preset sample speech text, the corresponding proofread text, and the sample label features output by the first BERT model into the second BERT model to train the second BERT model.

[0109] Step S404: Obtain the second loss value of the second BERT model.

[0110] Step S405: Determine the training stop condition based on the first loss value and the second loss value.

[0111] Step S406: Determine whether both the first BERT model and the second BERT model meet the stop training condition. If yes, proceed to step S407; otherwise, return to step S401.

[0112] Step S407: The first BERT model completes training and the second BERT model completes training.

[0113] In this way, the sample label features output by the first BERT model are used as the sample label features input by the second BERT model, so that the first BERT model and the second BERT model form a joint training mode. Under the condition of meeting the same stopping training condition, they complete their respective training, thereby obtaining a redundancy information correction model. This allows the redundant characters of the speech text to be corrected through the redundancy information correction model, making the semantics of the corrected speech text more accurate.

[0114] Optionally, training a second BERT model includes: training the second BERT model using preset sample speech text, the proofreading text corresponding to the sample speech text, and the sample label corresponding to at least one sample character in the sample speech text.

[0115] Optionally, the second BERT model is trained using preset sample speech text, corresponding proofreading text, and sample labels corresponding to at least one sample character in the sample speech text. This includes: inputting the preset sample speech text, corresponding proofreading text, and sample labels corresponding to at least one character in the sample speech text into the second BERT model; mapping each sample character in the sample speech text to obtain a sample text tensor, i.e., sample text features; mapping the positions corresponding to each sample character to obtain a sample position tensor, i.e., sample position features; and mapping the sample labels corresponding to at least one sample character in the sample speech text to obtain sample label features. The encoder of the second BERT model adds the sample text features (word_embedding), sample position features (position_embedding), and sample tag features (tags_embedding) before encoding them to obtain the sample second encoding vector He'. The second BERT model performs text mapping on the proofreading text corresponding to the sample speech text to obtain the sample proofreading text vector. The decoder of the second BERT model performs matrix calculation on the sample proofreading text vector and the sample second encoding vector He' to obtain the second sample character representation vector. The second sample character representation vector is then processed by softmax to obtain the output text of the second BERT model. The similarity between the output text of the second BERT model and the proofreading text corresponding to the sample speech text is calculated using cross-entropy. The parameters of the second BERT model are continuously adjusted through gradient backpropagation until the training of the second BERT model meets the stopping condition.

[0116] Combination Figure 5 As shown, in some embodiments, a method for training a redundancy information correction model includes:

[0117] First, steps S501 and S502 are executed. In step S501, the preset sample speech text and the sample label corresponding to at least one character in the sample speech text are input into the first BERT model to train the first BERT model. Then, step S503 is executed.

[0118] Step S502: Input the preset sample speech text, the sample label corresponding to at least one character in the sample speech text, and the proofreading text corresponding to the sample speech text into the second BERT model to train the second BERT model. Then execute step S504.

[0119] Step S503: Obtain the first loss value of the first BERT model. Then proceed to step S505.

[0120] Step S504: Obtain the second loss value of the second BERT model. Then proceed to step S505.

[0121] Step S505: Determine the training stop condition based on the first loss value and the second loss value. Then execute steps S506 and S507.

[0122] Step S506: Determine whether the first BERT model meets the stop training condition. If yes, proceed to step S508; otherwise, return to step S501.

[0123] Step S507: Determine whether the second BERT model meets the stop training condition. If yes, proceed to step S509; otherwise, return to step S502.

[0124] Step S508: The first BERT model has completed training.

[0125] Step S509: The second BERT model completes training.

[0126] In this way, by simultaneously training the first BERT model and the second BERT model, and completing their respective training under the same stopping condition, a redundancy information correction model is obtained. This model is used to correct redundant characters in speech text, making the semantics of the corrected speech text clearer.

[0127] Combination Figure 6 As shown, this disclosure provides another speech recognition method, including:

[0128] Step S601: Input the speech information into the pre-trained speech recognition model for speech recognition to obtain speech text;

[0129] Step S602: Input the speech text into the pre-trained redundancy information correction model to obtain the corrected speech text.

[0130] Optionally, the pre-trained redundancy information correction model includes a pre-trained character sequence labeling model and a pre-trained text modification model. The speech text is input into the pre-trained character sequence labeling model to obtain the corresponding label features; the speech text and its corresponding label features are then input into the pre-trained text modification model to perform redundancy character correction on the speech text, resulting in the corrected speech text.

[0131] In this way, the characters in the speech text are labeled by a pre-trained character sequence labeling model to obtain the label features corresponding to the speech text. Then, the speech text is modified by a text modification model to perform redundant character correction on the speech text. Since it can correct text with redundant information, the semantics of the text recognized by speech is more accurate.

[0132] Combination Figure 7 As shown, this disclosure provides a speech recognition apparatus, including: an acquisition module 701, a feature extraction module 702, an annotation module 703, and a proofreading module 704. The acquisition module 701 is configured to recognize speech information to obtain speech text; the feature extraction module 702 is configured to perform text feature extraction processing on the speech text to obtain text features; and to perform position feature extraction processing on the speech text to obtain position features; the annotation module 703 is configured to annotate at least one character in the speech text according to the text features and position features to obtain label features corresponding to the speech text; and the proofreading module 704 is configured to perform redundant character proofreading processing on the speech text according to the label features, text features, and position features to obtain proofread speech text.

[0133] The speech recognition apparatus provided in this embodiment can obtain speech text by analyzing speech information, perform text feature extraction and position feature extraction on the speech text to obtain text features and position features respectively, and annotate at least one character in the speech text to obtain the tag features corresponding to the speech text; then, based on the tag features, text features and position features, perform redundant character correction processing on the speech text. Since it can correct text with redundant information, the semantics of the speech-recognized text are more explicit.

[0134] Optionally, the acquisition module is configured to recognize the speech information and obtain the speech text by: performing convolution processing on the speech information to obtain a speech information tensor; performing encoding processing on the speech information tensor to obtain a first encoding vector; obtaining candidate characters at each first preset position according to the first encoding vector and a preset first start symbol; wherein, the first preset position is the position corresponding to each character contained in the speech text; and concatenating the candidate characters at each first preset position in the order of acquisition time to obtain the speech text.

[0135] Optionally, the acquisition module is configured to sequentially obtain candidate characters at each first preset position based on the first encoding vector and the preset first start symbol in the following manner: decode the first encoding vector and the preset first start symbol to sequentially obtain the probability distribution of each candidate character in the preset first dictionary at each first preset position; and sequentially perform the following processing for each first preset position: select the candidate character corresponding to the highest probability in the probability distribution corresponding to the first preset position as the candidate character at the first preset position.

[0136] Optionally, feature extraction is configured to perform text feature extraction processing on the speech text in the following manner to obtain text features: for each character in the speech text, perform the following processing: perform a table lookup operation on the character to obtain the character vector corresponding to the character; perform matrix operation on the character vector and a preset word embedding matrix to obtain the character tensor corresponding to the character; fuse the character tensors corresponding to each character in the speech text to obtain the text tensor corresponding to the speech text, and determine the text tensor as the text feature.

[0137] Optionally, feature extraction is configured to perform positional feature extraction processing on the speech text in the following manner to obtain positional features: for the position corresponding to each character in the speech text, perform the following processing: perform a lookup operation on the position to obtain the position vector corresponding to the position; perform matrix operation on the position vector and a preset position embedding matrix to obtain the position tensor corresponding to the position; fuse the position tensors corresponding to the positions of each character in the speech text to obtain the position tensor corresponding to the speech text, and determine the position tensor as the positional feature.

[0138] Optionally, the annotation module is configured to annotate at least one character in the speech text based on text features and position features to obtain the label features corresponding to the speech text: For each character in the at least one character, perform the following processing: extract the text sub-features of the corresponding character from the text features, extract the position sub-features of the corresponding character from the position features, and perform tensor calculation on the text sub-features and position sub-features to obtain the target tensor corresponding to the character; perform fusion processing on the target tensors corresponding to each character in the at least one character to obtain the label tensor corresponding to the speech text, and determine the label tensor as the label feature corresponding to the speech text.

[0139] Optionally, the proofreading module is configured to perform redundant character proofreading on the speech text based on label features, text features, and position features in the following manner to obtain the proofread speech text: Encoding the label features, text features, and position features to obtain a second encoding vector; obtaining the correction characters at each of the second preset positions sequentially based on the second encoding vector and a preset second start symbol; wherein the second preset position is the position corresponding to each character contained in the proofread speech text; and concatenating the correction characters at each of the second preset positions sequentially according to the order in which they were obtained to obtain the proofread speech text.

[0140] Optionally, the proofreading module is configured to obtain the correction characters at each preset position sequentially based on the second encoding vector and the preset second start symbol in the following manner: the second encoding vector and the preset second start symbol are decoded to obtain the probability distribution of each correction character in the preset second dictionary at each second preset position; the following processing is performed sequentially for each second preset position: the correction character corresponding to the highest probability in the probability distribution corresponding to the second preset position is selected as the correction character at the second preset position.

[0141] Combination Figure 8 As shown, this disclosure provides an electronic device including a processor 800 and a memory 801 storing program instructions. Optionally, the electronic device may further include a communication interface 802 and a bus 803. The processor 800, communication interface 802, and memory 801 can communicate with each other via the bus 803. The communication interface 802 can be used for information transmission. The processor 800 can call logical instructions in the memory 801 to execute the speech recognition method of the above embodiment.

[0142] Furthermore, the logic instructions in the aforementioned memory 801 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0143] The memory 801, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor 800 executes functional applications and data processing by running the program instructions / modules stored in the memory 801, thereby implementing the speech recognition method in the above embodiments.

[0144] The memory 801 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 801 may include high-speed random access memory and may also include non-volatile memory.

[0145] The electronic device provided in this embodiment can obtain speech text by analyzing speech information, perform text feature extraction and position feature extraction on the speech text to obtain text features and position features respectively, and mark at least one character in the speech text to obtain the tag features corresponding to the speech text; then, based on the tag features, text features and position features, the speech text is processed to correct redundant characters. Since it can correct text with redundant information, the semantics of the text recognized by speech is more accurate.

[0146] Alternatively, the electronic device may be a computer, server, etc.

[0147] This disclosure provides a storage medium storing program instructions, which, when executed, perform the aforementioned speech recognition method.

[0148] This disclosure provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions that, when executed by a computer, cause the computer to perform the aforementioned speech recognition method.

[0149] The aforementioned computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0150] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media capable of storing program code; it can also be a transient storage medium.

[0151] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms "a," "an," and "the" are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of one or more of the associated listed elements. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a…" does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0152] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0153] The methods and products (including but not limited to devices and equipment) disclosed in the embodiments herein can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed units may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to implement this embodiment according to actual needs. Furthermore, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0154] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.< / o> < / o> < / o> < / o> < / o> < / right> < / right> < / turn> < / turn> < / o> < / o> < / o> < / wrong> < / wrong> < / wrong> < / o> < / o> < / o> < / end> < / cls> < / end> < / cls> < / cls> < / end> < / cls> < / cls>

Claims

1. A method of speech recognition, characterized by, The method comprises: recognizing voice information to obtain voice text; performing text feature extraction processing on the voice text to obtain text features; performing position feature extraction processing on the voice text to obtain position features; annotating at least one character in the voice text according to the text features and the position features to obtain label features corresponding to the voice text; wherein the label features comprise at least one of the following: label features representing character errors, label features representing correct characters, label features representing semantic reversals, and label features representing unchanged characters; performing redundant character correction processing on the voice text according to the label features, the text features, and the position features to obtain a corrected voice text.

2. The method of claim 1, wherein, The method of recognizing voice information to obtain voice text comprises: performing convolution processing on the voice information to obtain a voice information tensor; performing encoding processing on the voice information tensor to obtain a first encoding vector; obtaining candidate characters at each first preset position in turn according to the first encoding vector and a preset first starting symbol; wherein the first preset positions are positions corresponding to each character included in the voice text; splicing the candidate characters at each first preset position in turn according to the order of obtaining time to obtain the voice text.

3. The method of claim 2, wherein, The method of obtaining candidate characters at each first preset position in turn according to the first encoding vector and a preset first starting symbol comprises: performing decoding processing on the first encoding vector and the preset first starting symbol to obtain the probability distribution of each candidate character in the preset first dictionary at each first preset position in turn; performing the following processing in turn for each first preset position: selecting the candidate character corresponding to the maximum probability in the probability distribution corresponding to the first preset position as the candidate character at the first preset position.

4. The method of claim 1, wherein, The method of performing text feature extraction processing on the voice text to obtain text features comprises: performing the following processing for each character in the voice text: performing a lookup table operation on the character to obtain a character vector corresponding to the character; and performing matrix operation on the character vector and a preset word embedding matrix to obtain a character tensor corresponding to the character; performing fusion processing on the character tensor corresponding to each character in the voice text to obtain a text tensor corresponding to the voice text, and determining the text tensor as the text features.

5. The method of claim 1, wherein, The method of performing position feature extraction processing on the voice text to obtain position features comprises: performing the following processing for the position corresponding to each character in the voice text: performing a lookup table operation on the position to obtain a position vector corresponding to the position; and performing matrix operation on the position vector and a preset position embedding matrix to obtain a position tensor corresponding to the position; performing fusion processing on the position tensor of the position corresponding to each character in the voice text to obtain a position tensor corresponding to the voice text, and determining the position tensor as the position features.

6. The method of claim 1, wherein, The labeling of at least one character in the speech text according to the text feature and the position feature to obtain a label feature corresponding to the speech text comprises: For each of the at least one character, the following processing is performed: extracting a text sub-feature corresponding to the character from the text feature, extracting a position sub-feature corresponding to the character from the position feature, and performing tensor calculation on the text sub-feature and the position sub-feature to obtain a target tensor corresponding to the character; The target tensors corresponding to each of the at least one character are fused to obtain a label tensor corresponding to the speech text, and the label tensor is determined as the label feature corresponding to the speech text.

7. The method of claim 1, wherein, The redundant character proofreading processing of the speech text according to the label feature, the text feature and the position feature to obtain a proofread speech text comprises: Encoding the label feature, the text feature and the position feature to obtain a second encoding vector; According to the second encoding vector and a preset second starting symbol, a corrected character at each second preset position is obtained in turn; wherein the second preset position is a position corresponding to each character contained in the proofread speech text; The corrected characters at each second preset position are spliced in turn according to the order of obtaining time to obtain the proofread speech text.

8. The method of claim 7, wherein, The obtaining of the corrected character at each preset position according to the second encoding vector and a preset second starting symbol comprises: Decoding the second encoding vector and the preset second starting symbol to obtain the probability distribution of each corrected character in the preset second dictionary at each second preset position in turn; For each second preset position, the following processing is performed in turn: Selecting a corrected character corresponding to the maximum probability in the probability distribution corresponding to the second preset position as the corrected character at the second preset position.

9. An apparatus for speech recognition, characterized by It comprises: The acquisition module is configured to recognize the speech information to obtain a speech text; The feature extraction module is configured to perform text feature extraction processing on the speech text to obtain a text feature; Performing position feature extraction processing on the speech text to obtain a position feature; The labeling module is configured to label at least one character in the speech text according to the text feature and the position feature to obtain a label feature corresponding to the speech text; wherein the label feature comprises at least one of the following: a label feature representing character error, a label feature representing character correctness, a label feature representing semantic inversion, and a label feature representing character invariance; The proofreading module is configured to perform redundant character proofreading processing on the speech text according to the label feature, the text feature and the position feature to obtain a proofread speech text.

10. An electronic device comprising a processor and a memory having stored program instructions, wherein the program instructions, when executed by the processor, cause the electronic device to perform the method of any one of claims 1-9. The processor is configured to execute the speech recognition method of any one of claims 1 to 8 when running the program instructions.

11. A storage medium storing program instructions, characterized in that, The program instructions, when running, execute the speech recognition method of any one of claims 1 to 8.

Images