Speech text correction method and device, equipment and storage medium

Abstract

The application discloses a speech text error correction method, a speech text error correction device, a speech text error correction equipment and a storage medium. The method comprises the following steps: obtaining speech information, obtaining a Chinese character sequence in the speech information and a pinyin sequence corresponding to the Chinese character sequence, generating a speech perception sequence according to the combination of the pinyin sequence and the Chinese character sequence, introducing an encoder with a separation mask, forming a coding sequence according to the speech perception sequence and the segmentation mask, establishing a mapping model according to the dependency relationship between an error text and a correct text, and obtaining a correct Chinese text corresponding to the speech information according to the coding sequence and the mapping model. The method can more accurately locate the part that may have errors, can better handle the complexity of the language structure, and thus improves the accuracy of error correction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech recognition, and in particular to a speech text error correction method and device, equipment and a storage medium. BACKGROUND

[0002] In the existing automatic speech recognition field, with the development of speech technology, speech transcription technology from speech to corresponding text has gradually popularized in daily life. The speech transcription technology mainly processes speech to text for long speech, so as to obtain the text content of the speech. At present, the speech transcription technology has been widely applied to various application scenarios including content recognition of a recording pen, subtitle generation and the like.

[0003] Therefore, the automatic speech recognition (ASR) text error correction technology plays an important role. For example, in the current insurance business scenario, the customer can input information through speech, and the ASR text error correction technology can automatically identify and correct the wrong words or phrases, so that the customer can input information more smoothly, thereby improving the customer experience and work efficiency. Although the existing technology can introduce speech information (Chinese pinyin), they usually combine speech representation with character representation, which often weakens the representation effect of normal text.

[0004] The errors introduced by automatic speech recognition (ASR) usually affect the execution of downstream tasks such as speech search and speech translation. The current error correction technology has been used to optimize the output sentence of the automatic speech recognition (ASR) model, and to achieve a lower word error rate (WER) than the original ASR output. There are mainly two methods to integrate speech features into the model in the existing technology. First, the pinyin of Chinese characters is encoded using a gate mechanism and integrated into the Chinese character representation. Second, a pronunciation prediction target is introduced to simulate the relationship between characters with similar phonetic systems. Although the performance of these methods has been greatly improved, they have two potential problems. First, during the training process, due to the entanglement between pinyin and text representation, the pinyin information may be ignored or dominated by the text information. Second, the introduction of pinyin features may weaken the representation of normal text. Intuitively, Chinese pinyin should be a supplement to Chinese text information, not a replacement, for two reasons. First, there is a one-to-many relationship between pinyin and Chinese characters, and it is more difficult to recover the correct Chinese characters from pinyin than from Chinese characters. Second, in the existing language model, the pinyin representation is not pre-trained as the text representation. Therefore, the model should avoid over-reliance on pinyin to avoid overfitting. SUMMARY

[0005] The main purpose of the present application is to provide a speech text error correction method, device, equipment and storage medium, which aims to solve the technical problem of low speech text error correction accuracy of the existing speech text error correction technology.

[0006] To achieve the above object, the application provides a speech text error correction method, which comprises the following steps:

[0007] obtaining a pinyin sequence and a Chinese character sequence in speech information, forming a corresponding speech perception sequence according to the pinyin sequence and the Chinese character sequence;

[0008] forming an encoding sequence according to the speech perception sequence and a separation mask;

[0009] establishing a mapping model of error text and correct text, and obtaining correct Chinese character text corresponding to the speech information according to the encoding sequence and the mapping model.

[0010] Optionally, the step of forming a corresponding speech perception sequence according to the pinyin sequence and the Chinese character sequence comprises:

[0011] encoding each Chinese character in the Chinese character sequence as the sum of a word embedding, a position embedding and a segment embedding;

[0012] encoding each pinyin in the pinyin sequence as the sum of an initial embedding, a final embedding, a position embedding and a segment embedding, wherein each Chinese character and its corresponding pinyin have the same position embedding after encoding.

[0013] Optionally, the step of forming an encoding sequence according to the speech perception sequence and a separation mask comprises:

[0014] the separation mask is a mask matrix M∈R 2n×2n :

[0015]

[0016] applying the separation mask to the encoding of the speech perception sequence, and outputting O l The formula is:

[0017]

[0018] O l = A l V l .

[0019] In the formula, A is a trainable parameter, H l-1 is the output of the previous layer, d is the dimension size, and M is a mask matrix.

[0020] Optionally, the step of establishing a mapping model of error text and correct text comprises:

[0021] obtaining a training data set of the error text and the correct text, and mapping the error text and the correct text into the formula:

[0022] H q = W q H e + b e

[0023] H k = W k H c + b c ;

[0024] The incorrect text is represented as H e , the correct text is represented as H c , H q is the hidden representation of the incorrect text H e after mapping, and H k is the hidden representation of the correct text H c after mapping;

[0025] The interaction between H q and H k is simulated by using a bi-affine attention, and the formula is:

[0026]

[0027] In the formula, U∈R |H|x|H|x|L| , |H| and |L| represent the hidden size and the label set size;

[0028] The cross-entropy is minimized to obtain the mapping model.

[0029] Optionally, before the step of obtaining the mapping model, the method further comprises:

[0030] Obtaining a training sample, and generating a prediction loss function of a text part according to the training sample, and the formula is:

[0031] In the formula, X is a character sequence in the training sample, Y is a correction sequence of X in the training sample, and p is an output distribution;

[0032] Obtaining a pinyin sequence Z corresponding to the correction sequence, and generating a prediction loss function of a pinyin part, and the formula is:

[0033] According to the prediction loss function of the text part, the prediction loss function of the pinyin part, and the cross-entropy The training model is trained.

[0034] Optionally, according to the prediction loss function of the text part, the prediction loss function of the pinyin part, and the cross-entropy The step of training the model comprises:

[0035] The prediction loss function of the text part is combined with the prediction loss function of the pinyin part and cross entropy A total loss function is established, and the formula is:

[0036] In the formula, alpha and beta are adjustable hyperparameters, and the model is trained by minimizing the total loss function L.

[0037] Optionally, the step of obtaining the pinyin sequence and the Chinese character sequence in the voice information comprises:

[0038] Separating an original human voice signal from the voice information;

[0039] Performing voice endpoint detection on the original human voice signal to split the original human voice signal into at least one segment of human voice signal according to the detected voice endpoint;

[0040] Performing voice recognition on each segment of human voice signal to obtain the pinyin sequence and the Chinese character sequence corresponding to each segment of human voice signal.

[0041] Further, in order to achieve the above-mentioned purpose, the present application also provides a voice text error correction device, which comprises:

[0042] A voice perception module obtains a pinyin sequence and a Chinese character sequence in voice information, and forms a corresponding voice perception sequence according to the pinyin sequence and the Chinese character sequence;

[0043] A voice coding module forms a coding sequence according to the voice perception sequence and a separation mask;

[0044] A voice mapping module establishes a mapping model of incorrect text and correct text, and obtains correct Chinese text corresponding to the voice information according to the coding sequence and the mapping model.

[0045] Further, in order to achieve the above-mentioned purpose, the present application also provides a voice text error correction device, which comprises a memory, a processor, and a voice text error correction program stored in the memory and executable on the processor, and the voice text error correction program implements the steps of the voice text error correction method when executed by the processor.

[0046] Further, in order to achieve the above-mentioned purpose, the present application also provides a storage medium, which stores a voice text error correction program, and the voice text error correction program implements the steps of the voice text error correction method when executed by a processor.

[0047] The application obtains the phonetic information, obtains the Chinese character sequence in the phonetic information and the pinyin sequence corresponding to the Chinese character sequence, generates a phonetic perception sequence according to the combination of the pinyin sequence and the Chinese character sequence, introduces an encoder with a separation mask, forms an encoding sequence according to the phonetic perception sequence and the segmentation mask, establishes a mapping model according to the dependency relationship between the error text and the correct text, and obtains the correct Chinese text corresponding to the phonetic information according to the encoding sequence and the mapping model, so that the part that may exist errors can be more accurately positioned, the complexity of the language structure can be better handled, and the accuracy of error correction is improved. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 The structure schematic diagram of the device hardware running environment related to the embodiment scheme of the speech text error correction device of the application is shown.

[0049] Figure 2 The flowchart of the first embodiment of the speech text error correction method of the application is shown.

[0050] Figure 3 The function module schematic diagram of the preferred embodiment of the speech text error correction device of the application is shown.

[0051] The implementation, functional features and advantages of the application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are merely intended to explain the application and not to limit the application.

[0053] It should be noted that in the existing automatic speech recognition field, with the development of speech technology, speech transcription technology from speech to corresponding text has gradually popularized in daily life. The speech transcription technology mainly processes speech to text for long speech, so as to obtain the text content of the speech. At present, the speech transcription technology has been widely applied to various application scenarios including content recognition of a recording pen, subtitle generation and the like.

[0054] Therefore, the automatic speech recognition (ASR) text error correction technology plays an important role. For example, in the current insurance business scenario, customers can input information through speech, and the ASR text error correction technology can automatically identify and correct error words or phrases, so that customers can input information more smoothly, thereby improving customer experience and work efficiency. Although the existing technology can introduce phonetic information (Chinese pinyin), they usually combine phonetic representation and character representation, which often weakens the representation effect of normal text.

[0055] Errors introduced by automatic speech recognition (ASR) often affect the performance of downstream tasks such as speech search, speech translation, etc. Current error correction techniques have been used to optimize the output sentences of automatic speech recognition (ASR) models and achieve lower word error rate (WER) than the original ASR output. There are mainly two methods to integrate phonetic features into the model. First, use the gate mechanism to encode the pinyin of Chinese characters and integrate it into the Chinese character representation. Second, introduce a pronunciation prediction target to simulate the relationship between characters with similar phonetic systems. Although the performance of these methods has been greatly improved, they have two potential problems. First, during the training process, due to the entanglement between pinyin and text representation, pinyin information may be ignored or dominated by text information. Second, the introduction of pinyin features may weaken the representation of normal text. Intuitively, Chinese pinyin should be a supplement to Chinese text information, not a replacement, for two reasons. First, there is a one-to-many relationship between pinyin and Chinese characters, and it is more difficult to recover the correct Chinese characters from pinyin than from Chinese characters. Second, in the existing language model, the pinyin representation is not pre-trained as a text representation. Therefore, the model should avoid over-reliance on pinyin to avoid overfitting.

[0056] Based on the above defects, the present application provides a speech text error correction device, referring to Figure 1 , Figure 1 The structure diagram of the hardware operating environment of the device involved in the embodiment scheme of the speech text error correction device of the present application.

[0057] As Figure 1 shown, the speech text error correction device can include a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 can include a display screen (Display), an input unit such as a keyboard (Keyboard), and an optional user interface 1003 can also include a standard wired interface, a wireless interface. The network interface 1004 can optionally include a standard wired interface, a wireless interface (such as a WI-FI interface). The memory 1005 can be a high-speed RAM memory, or a stable memory (non-volatile memory) such as a magnetic disk memory. The memory 1005 can also be an independent storage device from the aforementioned processor 1001.

[0058] Those skilled in the art can understand that Figure 1 the hardware structure of the speech text error correction device shown in the foregoing embodiments does not constitute a limitation on the speech text error correction device, and can include more or fewer components than shown, or combine certain components, or different component arrangements.

[0059] AsFigure 1 As shown, the memory 1005 as a storage medium can include an operating system, a network communication module, a user interface module, and a speech text correction program. Among them, the operating system is a program that manages and controls the speech text correction device and software resources, supports the running of the network communication module, the user interface module, the speech text correction program and other programs or software; the network communication module is used to manage and control the network interface 1004; the user interface module is used to manage and control the user interface 1003.

[0060] In Figure 1 In the hardware structure of the speech text correction device shown, the network interface 1004 is mainly used to connect the background server and communicate data with the background server; the user interface 1003 is mainly used to connect the client and communicate data with the client; the processor 1001 can call the speech text correction program stored in the memory 1005 and perform the following operations:

[0061] Obtain the pinyin sequence and the Chinese character sequence in the speech information, and form a corresponding speech perception sequence according to the pinyin sequence and the Chinese character sequence;

[0062] Form an encoding sequence according to the speech perception sequence and a separation mask;

[0063] Establish a mapping model of incorrect text and correct text, and obtain the correct Chinese character text corresponding to the speech information according to the encoding sequence and the mapping model.

[0064] Further, the step of forming a corresponding speech perception sequence according to the pinyin sequence and the Chinese character sequence comprises:

[0065] Encoding each Chinese character in the Chinese character sequence as the sum of a word embedding, a position embedding and a segment embedding;

[0066] Encoding each pinyin in the pinyin sequence as the sum of an initial embedding, a final embedding, a position embedding and a segment embedding, wherein each Chinese character and its corresponding pinyin have the same position embedding after encoding.

[0067] Further, the step of forming an encoding sequence according to the speech perception sequence and a separation mask comprises:

[0068] The separation mask is a mask matrix M ∈ R 2n×2n :

[0069]

[0070] The separation mask is used for encoding the speech perception sequence, and an output O l The formula is:

[0071]

[0072] O l =A l V l .

[0073] formula is a trainable parameter, H l-1 is the output of the previous layer, d is the dimension size, and M is a mask matrix.

[0074] Further, the step of establishing the mapping model of the error text and the correct text comprises:

[0075] Obtaining a training data set of the error text and the correct text, and mapping the error text and the correct text as a formula:

[0076] H q =W q H e +b e

[0077] H k =W k H c +b c ;

[0078] In the formula, the representation of the error text is H e , the representation of the correct text is H c , H q is the hidden representation of the error text H e after mapping, and H k is the hidden representation of the correct text H c after mapping.

[0079] The double-affine attention is used to simulate the interaction between H q and H k , and the formula is:

[0080]

[0081] In the formula, U∈R |H|x|H|x|L| , |H| and |L| represent the hidden size and the label set size.

[0082] Training until the cross-entropy is minimized to obtain the mapping model.

[0083] Further, before the step of obtaining the mapping model, the method further comprises:

[0084] Obtaining a training sample, and generating a prediction loss function of a text part according to the training sample, and the formula is:

[0085] In the formula, X is a character sequence in the training sample, Y is a correction sequence of X in the training sample, and p is an output distribution.

[0086] A pinyin sequence Z corresponding to the correction sequence is obtained, and a prediction loss function of the pinyin part is generated, with the formula being:

[0087] The prediction loss function of the text part and the prediction loss function of the pinyin part are combined according to a cross-entropy The training model is trained.

[0088] Further, the prediction loss function of the text part and the prediction loss function of the pinyin part are combined according to a cross-entropy The step of training the model comprises:

[0089] The prediction loss function of the text part and the prediction loss function of the pinyin part are combined according to a cross-entropy A total loss function is established, with the formula being:

[0090] In the formula, α and β are adjustable hyperparameters, and the model is trained by minimizing the total loss function L.

[0091] Further, the step of obtaining the pinyin sequence and the Chinese character sequence in the voice information comprises:

[0092] An original human voice signal is separated from the voice information.

[0093] Voice endpoint detection is performed on the original human voice signal, so as to split the original human voice signal into at least one segment of human voice signal according to the detected voice endpoint.

[0094] Voice recognition is performed on each segment of human voice signal, so as to obtain the pinyin sequence and the Chinese character sequence corresponding to each segment of human voice signal.

[0095] The specific implementation of the speech text correction device of the present application is basically the same as each embodiment of the speech text correction method described below, and will not be repeated here.

[0096] The present application also provides a speech text correction method based on the above-mentioned speech text correction device.

[0097] Reference Figure 2 , Figure 2 The flowchart of the first embodiment of the speech text correction method of the present application.

[0098] Embodiments of the present application provide embodiments of a speech text correction method. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in an order different from that shown.

[0099] In various embodiments of the speech text correction method, the execution subject is a speech text correction device.

[0100] The speech text correction method is applied to a speech text correction device, and the speech text correction method comprises:

[0101] Step S10, obtaining a pinyin sequence and a Chinese character sequence in the speech information, and forming a corresponding speech perception sequence according to the pinyin sequence and the Chinese character sequence;

[0102] In the existing automatic speech recognition field, with the development of speech technology, speech transcription technology from speech to corresponding text has gradually popularized in daily life. Speech transcription technology mainly aims at long speech to process speech to text, so as to obtain the text content of the speech. At present, speech transcription technology has been widely applied to various application scenarios including content recognition of a recording pen, subtitle generation, etc.

[0103] Therefore, the automatic speech recognition (ASR) text correction technology plays an important role. For example, in the current insurance business scenario, customers can input information through speech, and the ASR text correction technology can automatically identify and correct incorrect words or phrases, so that customers can input information more smoothly, thereby improving customer experience and work efficiency. Although the existing technology can introduce speech information (Chinese pinyin), it usually combines speech representation and character representation, which often weakens the representation effect of normal text.

[0104] Errors introduced by automatic speech recognition (ASR) often affect the performance of downstream tasks such as speech search, speech translation, etc. Current error correction techniques have been used to optimize the output sentences of automatic speech recognition (ASR) models and achieve lower word error rates (WER) than the original ASR output. There are mainly two methods to integrate phonetic features into the model in the prior art. First, use a gating mechanism to encode the pinyin and integrate it into the Chinese character representation. Second, introduce a pronunciation prediction target to simulate the relationship between characters with similar phonetic systems. Although the performance of these methods has been greatly improved, they have two potential problems. First, during the training process, due to the entanglement between pinyin and text representation, pinyin information may be ignored or dominated by text information. Second, the introduction of pinyin features may weaken the representation of normal text. Intuitively, Chinese pinyin should be a supplement to Chinese text information, not a replacement, for two reasons. First, there is a one-to-many relationship between pinyin and Chinese characters, and it is more difficult to recover the correct Chinese characters from pinyin than from Chinese characters. Second, in the existing language model, the pinyin representation is not pre-trained as the text representation. Therefore, the model should avoid over-reliance on pinyin to avoid overfitting.

[0105] To solve the technical problem of low accuracy of existing speech text error correction technology, a speech text error correction method is proposed in the embodiment of the present application, which aims to obtain Chinese character sequences and corresponding pinyin sequences in the speech information by obtaining speech information, then generate a phonetic-aware sequence according to the combination of the pinyin sequence and the Chinese character sequence, and then introduce an encoder with a separate mask to form a coding sequence according to the phonetic-aware sequence and the segmentation mask, and at the same time, establish a mapping model according to the dependency relationship between the error text and the correct text, then obtain the correct Chinese text corresponding to the speech information according to the coding sequence and the mapping model, which can more accurately locate the part that may have errors, can better handle the complexity of the language structure, and thus improve the accuracy of error correction.

[0106] The speech text error correction method in the embodiment is applicable to devices with speech text error correction function.

[0107] In the embodiment, first, the speech information is obtained, which can be a segment of speech containing human voice, or a segment of human voice and the corresponding Chinese character sequence converted into text, wherein the Chinese character sequence may have some errors and needs to be corrected to the true meaning of the human voice in the speech information. Similarly, the speech information can also include the pinyin sequence recognized from the human voice.

[0108] The speech perception sequence is constructed by appending the pinyin sequence to the Chinese character text. The pinyin of each Chinese character is a sequence of Latin letters, consisting of an initial, a final, and a tone, to represent the pronunciation. Taking the character sequence of the error text as the input X, its pinyin sequence is represented as R = {(init1, final1), (init2, final2), …, (initn, finaln)}, where initii and finali are the initial and final of the ith character xi respectively. Then, after appending R to X, a speech perception sequence S is obtained as the sequence input to the model. A stack of BERT and 12 Transformer blocks can be used as the encoder. Each Chinese character is encoded as the sum of a word embedding, a position embedding, and a segment embedding. Similarly, the pinyin of each Chinese character is encoded as the sum of an initial embedding, a final embedding, a position embedding, and a segment embedding, where the position embedding is the same as that of the Chinese character. For example: the input error text character sequence X: "我真是户秃", the corresponding pinyin sequence R is: "wo zhen shi hu tu", the correct text sequence is Y: "我真是糊涂", and the corresponding pinyin sequence Z is: "wo zhen shi hu tu".

[0109] Step S20, forming an encoded sequence according to the speech perception sequence and the separation mask;

[0110] In this embodiment, after obtaining the speech perception sequence, the utilization method of speech features can be improved, and through the separation mask, the attention is not allowed to transfer from pinyin to text to avoid information leakage. The context representation of each token is updated by aggregating information from other tokens through a multi-head attention network (MHA). At the l-th layer, the output O of each attention head l is calculated as:

[0111]

[0112] O l = A l V l .

[0113] where is a trainable parameter, H l-1 is the output of the previous layer, d is the dimension size, and M is the mask matrix. Specifically, we apply a separation mask that allows attention from the text representation to the speech representation, rather than the opposite. Therefore, we define the mask matrix M ∈ R 2n×2n as:

[0114]

[0115] The separation mask ensures that when M ij = -∞, the pinyin representation cannot collect information from the text characters. Then, the O of all headsl are concatenated and then processed through a linear transformation network and a normalization network. Afterwards, the resulting representation is fed into a feedforward network followed by another normalization network to generate H l , which is the final encoding sequence.

[0116] Step S30, a mapping model of the error text and the correct text is established, and a correct Chinese character text corresponding to the voice information is obtained according to the encoding sequence and the mapping model.

[0117] In this embodiment, the dependency relationship between the error text and the correct text is explicitly modeled, and the consistency between the error text and the correct text can provide high information. An interaction matrix can be used to jointly model the alignment, and a bilinear classifier is used to simulate the multiplicative interaction between the error text and the correct text. Assuming that the representation of the error text is H e , and the representation of the correct text is H c . First, two independent feedforward networks are used to map the hidden representation to an error hidden representation H q and a correct hidden representation H k :

[0118] H q = W q H e + b e

[0119] H k = W k H c + b c

[0120] Then a bilinear attention is used to simulate the interaction between H q and H k , where U∈R |H|x|H|x|L| , and |H|, |L| represent the hidden size and the label set size:

[0121]

[0122] The training objective is to minimize the cross-entropy of the given label:

[0123]

[0124] Finally, the correct Chinese character text corresponding to the voice information is obtained through the trained model.

[0125] ​The present invention obtains voice information, obtains a Chinese character sequence in the voice information and a corresponding pinyin sequence, then generates a voice perception sequence according to the combination of the pinyin sequence and the Chinese character sequence, and then introduces an encoder with a separation mask to form a coding sequence according to the voice perception sequence and the segmentation mask. At the same time, a mapping model is established according to the dependency relationship between the incorrect text and the correct text, so that the correct Chinese text corresponding to the voice information is obtained according to the coding sequence and the mapping model, which can more accurately locate the parts that may be incorrect, better handle the complexity of the language structure, and thus improve the accuracy of error correction.

[0126] Further, a second embodiment of the voice text error correction method of the present invention is proposed. In the above step S10, the step of forming a corresponding voice perception sequence according to the pinyin sequence and the Chinese character sequence includes:

[0127] Step a1, encoding each Chinese character in the Chinese character sequence as the sum of a word embedding, a position embedding, and a segment embedding;

[0128] Step a2, encoding each pinyin in the pinyin sequence as the sum of an initial embedding, a final embedding, a position embedding, and a segment embedding, where the position embeddings of each Chinese character and its corresponding pinyin are the same after encoding.

[0129] In this embodiment, the voice perception sequence is constructed by appending the pinyin sequence to the Chinese character text. The pinyin of each Chinese character is a sequence of Latin letters, consisting of an initial, a final, and a tone, to represent pronunciation.

[0130] Taking the character sequence of the incorrect text as the input X, its pinyin sequence is represented as R = {(init1, final1), (init2, final2),..., (initn, finaln)}, where initii and finali are the initial and final of the i-th character xi respectively. Then, after appending R to X, a voice perception sequence S is obtained as the sequence of the input model. A stack of BERT and 12 Transformer blocks can be used as the encoder. Each Chinese character is encoded as the sum of a word embedding, a position embedding, and a segment embedding. Similarly, the pinyin of each Chinese character is encoded as the sum of an initial embedding, a final embedding, a position embedding, and a segment embedding, where the position embedding is the same as that of the Chinese character. For example: the input incorrect text character sequence X: "我真是户秃", the corresponding pinyin sequence R is: "wo zhen shi hu tu", the correct text sequence is Y: "我真是糊涂", and the corresponding pinyin sequence Z is: "wozhen shi hu tu".

[0131] The embodiment can ensure that pinyin information is not ignored or dominated by text information, so that the accuracy of speech text correction is greatly improved.

[0132] Further, the third embodiment of the speech text correction method is provided.

[0133] In step b1, the separation mask is a mask matrix M∈R 2n×2n :

[0134]

[0135] The separation mask is used for encoding the speech perception sequence, and output O l The formula is:

[0136]

[0137] O l =A l V l .

[0138] In the formula is a trainable parameter, H l-1 is the output of the previous layer, d is the dimension size, and M is the mask matrix.

[0139] In the embodiment, after obtaining the speech perception sequence, the utilization mode of the speech feature can be improved, and the separation mask is used to prevent attention from being transferred from pinyin to text to avoid information leakage. The context representation of each token is updated by aggregating information from other tokens through multi-head attention network (MHA). In the lth layer, the output O l of each attention head is calculated as:

[0140]

[0141] O l =A l V l .

[0142] wherein is a trainable parameter, H l-1 is the output of the previous layer, d is the dimension size, and M is the mask matrix. Specifically, a separation mask is applied to allow attention from text representation to speech representation, instead of the opposite. Therefore, the mask matrix M∈R 2n×2n is defined as:

[0143]

[0144] The separator mask guarantees that in M ij =-∞ indicates that pinyin cannot collect information from text characters. Then, all the O's at the beginning are... l The data are connected and then processed through a linear transformation network and a normalization network. The resulting representation is then fed into a feedforward network, followed by another normalization network to generate H. l Finally, the encoded sequence is obtained.

[0145] This embodiment utilizes the above-mentioned segmentation annotation to calculate the cross-entropy loss, minimizes the cross-entropy loss, and then updates the network parameters to obtain an accurate image generation and segmentation network, thereby improving the accuracy of speech-text error correction.

[0146] Furthermore, a fourth embodiment of the speech-text error correction method of the present invention is proposed, wherein the step of establishing a mapping model between erroneous text and correct text in step b1 above includes:

[0147] Step c1: Obtain the training dataset of the erroneous text and the correct text, and map the erroneous text and the correct text to the formula:

[0148] H q =W q H e +b e

[0149] H k =W k H c +b c ;

[0150] Error text in the formula is represented as H. e The correct text is represented as H. c H q To convert the error text H e The incorrect hidden representation after mapping, H k To correctly extract the text H c The correct hidden representation after mapping;

[0151] Using dual affine attention to simulate H q and H k The interaction between them is expressed by the following formula:

[0152]

[0153] In the formula U∈R |H|x|H|x|L| |H| and |L| represent the hidden size and the tag set size, respectively.

[0154] Training until cross-entropy Minimize to obtain the mapping model.

[0155] 1. In this embodiment, the dependency between the error text and the correct text is explicitly modeled: the consistency between the error text and the correct text can provide high information. An interaction matrix can be utilized to jointly model the alignment, and a bilinear classifier is used to simulate the multiplicative interaction between the error text and the correct text. Suppose the hidden representation of the error text is H e , and the hidden representation of the correct text is H c . First, two independent feedforward networks are used to map the hidden representation to an error hidden representation H q and a correct hidden representation H k :

[0156] H q =W q H e +b e

[0157] H k =W k H c +b c

[0158] Then a bilinear attention is used to simulate the interaction between H q and H k , where U∈

[0159] R |H|x|H|x|L| , where |H|, |L| represent the hidden size and the label set size:

[0160]

[0161] The training objective is to minimize the cross-entropy of H given the label:

[0162]

[0163] Finally, the correct Chinese character text corresponding to the speech information is obtained through the trained model.

[0164] In this embodiment, the mapping model is established according to the dependency between the error text and the correct text, and then the correct Chinese character text corresponding to the speech information is obtained according to the encoded sequence and the mapping model, which can more accurately locate the part that may exist errors, can better handle the complexity of the language structure, and thus improve the accuracy of error correction.

[0165] Further, the fifth embodiment of the speech text error correction method is provided, and before the step of obtaining the mapping model in the above-mentioned step c1, the method further comprises:

[0166] Step d1, obtaining a training sample, generating a prediction loss function of a text part according to the training sample, and the formula is:

[0167] In the formula, X is a character sequence in the training sample, Y is a correction sequence of X in the training sample, and p is an output distribution.

[0168] Step d2, obtaining a pinyin sequence Z corresponding to the correction sequence, and generating a prediction loss function of a pinyin part, and the formula is:

[0169] Step d3, according to the prediction loss function of the text part and the prediction loss function of the pinyin part and cross entropy The training model.

[0170] In this embodiment, the prediction loss function of the text part is established: the context representation H of the final correct text is generated by the last layer hidden state of the encoder. The cross entropy loss function is used as the loss function of the speech text correction model. A training sample (X, Y) is obtained, wherein X is a character sequence in the training sample, Y is a correction sequence of X in the training sample, and p is an output distribution. The prediction loss of the text part S is calculated as:

[0171]

[0172] At the same time, the prediction loss function of the pinyin part is established: in order to assist the pinyin to character task, Z is used as the prediction label of S, wherein Z is the pinyin sequence corresponding to Y. The prediction loss of the pinyin part in S is defined as:

[0173]

[0174] Finally, according to the prediction loss function of the text part and the prediction loss function of the pinyin part and cross entropy The correction model is trained to obtain a speech text correction model, and automatic correction of speech text is realized.

[0175] In this embodiment, the mapping model is established according to the dependency relationship between the error text and the correct text, and then the correct Chinese character text corresponding to the speech information is obtained according to the encoding sequence and the mapping model, which can more accurately locate the part that may have errors, can better handle the complexity of the language structure, and thus improves the accuracy of correction.

[0176] Further, the sixth embodiment of the speech text correction method is provided, and the step d3 includes:

[0177] Step e1, the predicted loss function of the text part and the predicted loss function of the pinyin part and cross entropy The total loss function is established, and the formula is:

[0178] In the formula, alpha and beta are adjustable hyperparameters, and the model is trained by minimizing the total loss function L.

[0179] In this embodiment, the predicted loss function of the text part and the predicted loss function of the pinyin part and cross entropy The total loss function is established, and the formula is:

[0180] In the formula, alpha and beta are adjustable hyperparameters, and the model is trained by minimizing the total loss function L, to obtain a speech text correction model and realize a speech text correction function.

[0181] This embodiment decouples Chinese characters and pinyin representation, and the text and speech interaction is more direct and in-depth. In this way, it can be ensured that the pinyin information is not ignored or dominated by the text information. The design of the isolation mask effectively avoids information leakage between pinyin and text, and ensures that attention is focused on the data interaction itself. The improved method ensures that Chinese pinyin is a supplement to Chinese text information, rather than a replacement. At the same time, it avoids excessive dependence on pinyin, so as to avoid overfitting. By modeling the relationship between correct sentences and error sentences, the part where the error may exist can be more accurately located. Since the interaction matrix can capture the complex relationship between correct sentences and error sentences, it can better handle the complexity of the language structure, thereby improving the accuracy of the correction

[0182] Further, the seventh embodiment of the speech text correction method is provided, and the step of obtaining the pinyin sequence and the Chinese character sequence in the speech information in the step S10 comprises

[0183] Step f1, separating the original human voice signal from the speech information;

[0184] Step f2, performing speech endpoint detection on the original human voice signal to split the original human voice signal into at least one segment of human voice signal according to the detected speech endpoint;

[0185] Step f3, performing speech recognition on each segment of human voice signal to obtain the pinyin sequence and the Chinese character sequence corresponding to each segment of human voice signal.

[0186] In the embodiment, the original voice signal can include an original human voice signal and a background sound signal, and the voice transcription technology needs to transcribe the voice text embodied by the original human voice signal from the original voice signal. In the transcription process, the background sound signal will affect the voice transcription. Taking the application scene of generating film and television drama subtitles as an example, the voice of the film and television drama can include the human voice of the role and the background sound effect; taking the application scene of identifying the recording content of a recording pen as an example, in addition to the speech of the participants, the audio recorded by the recording pen in the conference site also includes the noise in the conference site.

[0187] By performing voice endpoint detection on the original human voice signal, the voice endpoints in the voice signal, that is, at least one group of voice start points and voice end points, can be detected. By the at least one group of voice start points and voice end points, the original human voice signal can be split into at least one piece of human voice signal, so as to remove the mute part in the original human voice signal. In specific implementation, the libfvad algorithm can be used to perform voice endpoint detection on the original human voice signal in the wav format, find the mute part therein, and split the original human voice signal into at least one piece of shorter human voice signal according to the detected voice endpoints.

[0188] A voice recognition model with high recognition accuracy can be trained in advance using 10,000+ hours of Chinese voice data. Then, the voice recognition model is used to perform voice recognition on each piece of human voice signal obtained after voice endpoint detection, to obtain an initial voice text corresponding to each piece of human voice signal.

[0189] The embodiment separates the original human voice signal from the original voice signal, and performs voice endpoint detection on the original human voice signal to split the original human voice signal into at least one piece of human voice signal according to the detected voice endpoints, thereby improving the accuracy of voice text error correction.

[0190] The application also provides a voice text error correction device.

[0191] Reference Figure 3 , Figure 3 It is a functional module schematic diagram of the first embodiment of the voice text error correction device of the application. The voice text error correction device comprises:

[0192] A determination module determines a picture generation network, and obtains a picture generation segmentation network according to the picture generation network and a labeled picture.

[0193] A voice perception module acquires a pinyin sequence and a Chinese character sequence in voice information, and forms a corresponding voice perception sequence according to the pinyin sequence and the Chinese character sequence.

[0194] A voice coding module forms a coding sequence according to the voice perception sequence and a separation mask.

[0195] The voice mapping module establishes a mapping model of incorrect text and correct text, and obtains correct Chinese character text corresponding to the voice information according to the coding sequence and the mapping model.

[0196] In addition, the embodiment of the present application further provides a storage medium.

[0197] The storage medium stores a voice text correction program, and the voice text correction program, when executed by the processor, implements the steps of the voice text correction method.

[0198] The storage medium of the present application has the same embodiment as the voice text correction method described above, and will not be repeated here.

[0199] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-mentioned specific embodiments, and the above-mentioned specific embodiments are only illustrative, but not limiting, and those skilled in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims, and the equivalent structures or equivalent flow changes made by using the content of the present application and the drawings, or directly or indirectly applied in other related technical fields, all belong to the protection of the present application.

[0200] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are only illustrative, for example, the flowcharts and block diagrams in the drawings show the possible implementation architecture, function and operation of the devices, methods and computer program products according to the embodiments of the present application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order from that noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for executing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0201] In addition, each of the functional modules in the various embodiments of the present application can be integrated together to form an independent part, or each of the modules can exist independently, or two or more modules can be integrated to form an independent part. When the functions are realized in the form of software functional modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0202] It should be noted that, in this paper, relational terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0203] The above is only various embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of speech text correction, characterized by, The speech text correction comprises the following steps: obtaining a pinyin sequence and a Chinese character sequence in speech information, encoding each Chinese character in the Chinese character sequence into a sum of a word embedding, a position embedding and a segment embedding, and encoding each pinyin in the pinyin sequence into a sum of an initial embedding, a final embedding, a position embedding and a segment embedding, wherein each Chinese character and its corresponding pinyin have the same position embedding after encoding, to form a corresponding speech perception sequence; forming an encoded sequence according to the speech perception sequence and a separate mask; establishing a mapping model of incorrect text and correct text, and obtaining correct Chinese character text corresponding to the speech information according to the encoded sequence and the mapping model.

2. The method of voice text correction as claimed in claim 1, wherein, The step of forming an encoded sequence according to the speech perception sequence and a separate mask comprises: The separation mask is a mask matrix M e : ； using the separation mask for encoding of the speech perceptual sequence, output The formula is: In the formula is a trainable parameter, is the output of the previous layer, d is the dimension size, and M is a mask matrix.

3. The method of voice text correction as claimed in claim 2, wherein, The step of establishing a mapping model of incorrect text and correct text comprises: obtaining a training data set of the incorrect text and the correct text, and mapping the incorrect text and the correct text into a formula: ； The erroneous text is represented in the formula as , and the correct text is represented as , is the hidden representation of the erroneous text after mapping, and is the hidden representation of the correct text after mapping. Simulating interactions between the two affine attentions using a formula of: and between the two affine attentions. ； In the formula, U∈ |H|, |L| represent the size of the hidden and label set size; training to minimize cross-entropy minimization, resulting in the mapping model.

4. The method of voice text correction as claimed in claim 3, wherein, Before the step of obtaining the mapping model, the following steps are further included: Obtaining training samples, generating a prediction loss function of a text part according to the training samples, and the formula is: ; In the formula, X is a character sequence in the training sample, Y is a correction sequence of X in the training sample, and p is an output distribution; A pinyin sequence Z corresponding to the correction sequence is obtained, and a prediction loss function of the pinyin part is generated, and the formula is: ; According to the prediction loss function of the text part and the prediction loss function of the pinyin part and cross entropy The training model.

5. The method of voice text correction as claimed in claim 4, wherein, The predicted loss function according to the text part and the predicted loss function and cross entropy of the pinyin part The step of training the model comprises: According to the prediction loss function of the text part and the prediction loss function of the pinyin part and cross entropy The total loss function is established, and the formula is: L= +α ; in the formula α, is a tunable hyperparameter, and the model is trained by minimizing the total loss function L.

6. The method of voice text correction as claimed in claim 1, wherein, The step of obtaining a pinyin sequence and a Chinese character sequence in speech information comprises: separating an original human voice signal from the speech information; performing speech endpoint detection on the original human voice signal to split the original human voice signal into at least one segment of human voice signal according to the detected speech endpoint; performing speech recognition on each segment of human voice signal to obtain the pinyin sequence and the Chinese character sequence corresponding to each segment of human voice signal.

7. A speech text correction device, characterized by The speech text correction device comprises: a speech perception module, which obtains a pinyin sequence and a Chinese character sequence in speech information, encodes each Chinese character in the Chinese character sequence into a sum of a word embedding, a position embedding and a segment embedding, and encodes each pinyin in the pinyin sequence into a sum of an initial embedding, a final embedding, a position embedding and a segment embedding, wherein each Chinese character and its corresponding pinyin have the same position embedding after encoding, to form a corresponding speech perception sequence; a speech encoding module, which forms an encoded sequence according to the speech perception sequence and a separate mask; a speech mapping module, which establishes a mapping model of incorrect text and correct text, and obtains correct Chinese character text corresponding to the speech information according to the encoded sequence and the mapping model.

8. A speech text correction device, characterized by The speech text correction device comprises a memory, a processor and a speech text correction program stored on the memory and executable on the processor, and the speech text correction program, when executed by the processor, implements the steps of the speech text correction method according to any one of claims 1-6.

9. A storage medium, characterized by The storage medium stores a speech text correction program, and the speech text correction program, when executed by the processor, implements the steps of the speech text correction method according to any one of claims 1-6.

Images