Speech recognition based sentence correction method and apparatus, device, and storage medium

Abstract

The present application relates to the technical field of sentence correction, and particularly relates to a sentence correction method and device based on speech recognition, equipment and a storage medium, wherein a comprehensive bundle search technology is adopted to recognize speech data, and a first candidate sentence with the highest bundle search score and a plurality of candidate sentences are extracted in terms of semantic features and pinyin features, and feature fusion is performed; the first candidate sentence is corrected by using the fused features after feature fusion, thereby reducing the negative influence caused by pronunciation problems of users and multi-pronunciation character problems of texts, and improving the accuracy and efficiency of sentence correction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sentence correction, and in particular relates to a sentence correction method and device based on speech recognition, equipment and a storage medium. BACKGROUND

[0002] In recent years, speech recognition technology has developed rapidly, and its application field is continuously expanding. Various speech recognition system products appear on the market. After speech recognition is converted into text output by using a speech recognition conversion tool, it is widely used in model training, media retrieval, subtitle generation, speech identification and the like.

[0003] However, actual user types are various, and the pronunciation of many people is far from standard pronunciation. Therefore, when speech recognition is converted into text by using a speech recognition conversion tool, the speech recognition conversion tool cannot accurately recognize this part of speech data due to the pronunciation problem of some users or the polyphonic character problem of text, and does not have a certain error correction capability, resulting in that the text generated by the speech recognition conversion tool is inconsistent with the correct text content, and the actual application effect is poor. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a sentence correction method and device based on speech recognition, equipment and a storage medium. A comprehensive bundle search technology is used to recognize speech data, and the first candidate sentence with the highest bundle search score and a plurality of candidate sentences are extracted for semantic features and pinyin features, and feature fusion is performed. The first candidate sentence is corrected by using the fusion features after feature fusion, the negative influence caused by the pronunciation problem of the user and the polyphonic character problem of the text is reduced, and the accuracy and efficiency of the sentence correction are improved. The technical scheme is as follows:

[0005] In a first aspect, the present application provides a sentence correction method based on speech recognition, comprising the following steps:

[0006] Obtain speech data, use a bundle search technology to recognize the speech data, obtain a first candidate sentence and a candidate sentence set, wherein the candidate sentence set includes a plurality of candidate sentences, the first candidate sentence is the candidate sentence with the highest bundle search score, and the first candidate sentence and the plurality of candidate sentences each include a plurality of characters;

[0007] Align the first candidate sentence with the plurality of candidate sentences in the candidate sentence set respectively to obtain a first aligned sentence and a plurality of candidate aligned sentences, wherein the first aligned sentence is an aligned sentence obtained by aligning the first candidate sentence, and the candidate aligned sentence is an aligned sentence obtained by aligning the candidate sentence.

[0008] pinyin conversion processing is performed on the characters corresponding to the position indexes of the first alignment sentence and the plurality of candidate alignment sentences, and pinyin information corresponding to the position indexes is obtained;

[0009] The pinyin information corresponding to the position indexes of the first alignment sentence and the plurality of candidate alignment sentences is input into a preset sentence correction model, semantic features corresponding to the position indexes of the first alignment sentence are obtained according to a semantic feature extraction module in the sentence correction model and the first alignment sentence, and pinyin features corresponding to the position indexes are obtained according to a pinyin feature extraction module in the sentence correction model and the pinyin concatenation information corresponding to the position indexes;

[0010] The semantic features corresponding to the position indexes of the first alignment sentence and the pinyin features corresponding to the position indexes are input into a feature fusion module in the sentence correction model, the semantic features and the pinyin features corresponding to the same position index are fused, and fusion information features corresponding to the position indexes are obtained;

[0011] The fusion information features corresponding to the position indexes are input into an output module in the sentence correction model, correction characters corresponding to the position indexes are obtained, and the first alignment sentence is subjected to sentence correction processing according to the correction characters corresponding to the position indexes, and a corrected sentence corresponding to the voice data is obtained.

[0012] In a second aspect, an embodiment of the present application provides a sentence correction device based on voice recognition, comprising:

[0013] A voice data recognition module is configured to obtain voice data, recognize the voice data by using a beam search technology, and obtain a first candidate sentence and a candidate sentence set, wherein the candidate sentence set includes a plurality of candidate sentences, the first candidate sentence is a candidate sentence with the highest beam search score, and the first candidate sentence and the plurality of candidate sentences each include a plurality of characters.

[0014] A sentence alignment module is configured to align the first candidate sentence with the plurality of candidate sentences in the candidate sentence set respectively, and obtain a first alignment sentence and a plurality of candidate alignment sentences, wherein the first alignment sentence is an alignment sentence obtained by aligning the first candidate sentence, and the candidate alignment sentence is an alignment sentence obtained by aligning the candidate sentence.

[0015] A pinyin conversion processing module is configured to perform pinyin conversion processing on characters corresponding to position indexes of the first alignment sentence and the plurality of candidate alignment sentences, and obtain pinyin information corresponding to the position indexes.

[0016] a feature extraction module configured to input the first aligned sentence and the pinyin information corresponding to the plurality of position indexes into a preset sentence correction model, obtain semantic features corresponding to the plurality of position indexes of the first aligned sentence according to a semantic feature extraction module in the sentence correction model, and obtain pinyin features corresponding to the plurality of position indexes according to a pinyin feature extraction module in the sentence correction model;

[0017] a feature fusion module configured to input the semantic features corresponding to the plurality of position indexes of the first aligned sentence and the pinyin features corresponding to the plurality of position indexes into a feature fusion module in the sentence correction model, fuse the semantic features and the pinyin features corresponding to the same position index, and obtain fusion information features corresponding to the plurality of position indexes;

[0018] a sentence correction module configured to input the fusion information features corresponding to the plurality of position indexes into an output module in the sentence correction model, obtain corrected characters corresponding to the plurality of position indexes, and perform sentence correction processing on the first aligned sentence according to the corrected characters corresponding to the plurality of position indexes to obtain a corrected sentence corresponding to the voice data.

[0019] In a third aspect, an apparatus is provided, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor; when the computer program is executed by the processor, the steps of the method for sentence correction based on voice recognition according to the first aspect are implemented.

[0020] In a fourth aspect, a storage medium is provided, which stores a computer program; when the computer program is executed by a processor, the steps of the method for sentence correction based on voice recognition according to the first aspect are implemented.

[0021] In the embodiments of the present application, the comprehensive beam search technology is adopted to recognize voice data, and the first candidate sentence and the plurality of candidate sentences with the highest beam search score are subjected to semantic feature extraction and pinyin feature extraction, and feature fusion is performed; the first candidate sentence is corrected by using the fusion features after feature fusion, the negative influence caused by the pronunciation problem of the user and the multi-pronunciation character problem of the text is reduced, and the accuracy and efficiency of sentence correction are improved.

[0022] For better understanding and implementation, the present application is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1A flowchart of a speech recognition based sentence correction method is provided for an embodiment of the present application;

[0024] Figure 2 A flowchart of S2 of a speech recognition based sentence correction method is provided for an embodiment of the present application;

[0025] Figure 3 A flowchart of S23 of a speech recognition based sentence correction method is provided for an embodiment of the present application;

[0026] Figure 4 A flowchart of S3 of a speech recognition based sentence correction method is provided for an embodiment of the present application;

[0027] Figure 5 A flowchart of a speech recognition based sentence correction method is provided for another embodiment of the present application;

[0028] Figure 6 A structural diagram of a speech recognition based sentence correction apparatus is provided for a fourth embodiment of the present application;

[0029] Figure 7 A structural diagram of an apparatus is provided for a fifth embodiment of the present application. DETAILED DESCRIPTION

[0030] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The description of the exemplary embodiments is intended to apply to various alternative embodiments as well. The following description is not limited to the exemplary embodiments, but rather, is applicable to any apparatus and method consistent with the present application as set forth in the claims.

[0031] The terminology used in the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in the description of the embodiments and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It also will be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0032] It should be understood that, although the terms first, second, third, etc. can be employed in this application to describe various information, the information should not be limited to these terms. These terms are only used to differentiate one piece of information from another piece of information of the same type. For example, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information, without departing from the scope of the application. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon determination" or "in response to a determination".

[0033] Referring to Figure 1 , Figure 1 A flowchart of a speech recognition-based sentence correction method provided for an embodiment of the present application is shown in FIG. 1. The method includes the following steps:

[0034] S1: Obtain speech data, and identify the speech data using a beam-search technique to obtain a first candidate sentence and a candidate sentence set.

[0035] The execution subject of the speech recognition-based sentence correction method is a speech recognition-based sentence correction device (hereinafter referred to as a correction device). In an optional embodiment, the correction device can be a computer device, which can be a server or a server cluster formed by a plurality of computer devices, for establishing a network connection with a client.

[0036] The beam-search technique is a search method that balances effect and speed. In the speech recognition result, the score path of each word is concatenated, and the top N results are obtained. Specifically, during the audio recognition process, the top N candidate sentences with the highest scores are retained when identifying each position word, where N is a candidate sentence number set artificially in advance.

[0037] In this embodiment, the correction device can obtain speech data input by a user or obtain speech data by connecting to a database. The speech data is identified using a beam-search technique to obtain a first candidate sentence and a candidate sentence set. The candidate sentence set includes a plurality of candidate sentences, the first candidate sentence is the candidate sentence with the highest beam-search score, and the first candidate sentence and the plurality of candidate sentences each include a plurality of characters. The non-Chinese characters include English characters, space characters, and special characters.

[0038] S2: Align the first candidate sentence with the plurality of candidate sentences in the candidate sentence set respectively to obtain a first aligned sentence and a plurality of candidate aligned sentences.

[0039] Since the first candidate sentence and the several candidate sentences in the candidate sentence set are obtained by using the cluster search technology, the lengths of the sentences can be different, and the recognized words or characters can be different.

[0040] In this embodiment, the correction device aligns the first candidate sentence with the several candidate sentences in the candidate sentence set respectively, to obtain a first aligned sentence and several candidate aligned sentences, wherein the first aligned sentence is an aligned sentence obtained by aligning the first candidate sentence, and the candidate aligned sentences are aligned sentences obtained by aligning the candidate sentences.

[0041] Since the aligned sentences can have positions that are empty after alignment, an empty character [Mask] is set at the positions that are empty after alignment, that is, the character corresponding to the position index of the sentence is the empty character [Mask].

[0042] Different sentences have the feature that the pronunciations of the characters at the same position are similar. In order to comprehensively consider the pinyin and Chinese corresponding to the characters of the two aligned sentences, the most reasonable one-to-one correspondence of each character in the two aligned sentences in position is obtained. Please refer to Figure 2 , Figure 2 The flowchart of S2 in the speech recognition-based sentence correction method provided by an embodiment of the present application includes steps S21-S23, and the details are as follows.

[0043] S21: Aligning the first candidate sentence with the several candidate sentences in the candidate sentence set to obtain several to-be-screened aligned sentences.

[0044] In this embodiment, the correction device aligns the first candidate sentence with the several candidate sentences in the candidate sentence set to obtain several to-be-screened aligned sentences.

[0045] Specifically, the correction device extracts one candidate sentence from the candidate sentence set in a non-replacement manner, aligns the first candidate sentence with the extracted candidate sentence for the first time to obtain a to-be-screened aligned sentence aligned for the first time, and then extracts one candidate sentence from the remaining candidate sentences in the candidate sentence set in a non-replacement manner, aligns the to-be-screened aligned sentence aligned for the last time with the extracted candidate sentence for the next time, until all the candidate sentences in the candidate sentence set are aligned, to obtain the several to-be-screened aligned sentences.

[0046] S22: Obtain the Chinese editing matrix of the plurality of to-be-screened alignment sentences, and accumulate the Chinese editing distance data of the characters corresponding to each position index of the same to-be-screened alignment sentence according to the Chinese editing matrix of the plurality of to-be-screened alignment sentences to obtain total Chinese editing distance data of the plurality of to-be-screened alignment sentences.

[0047] The Chinese editing matrix includes Chinese editing distance data of characters corresponding to a plurality of position indexes. In this embodiment, the correction device calculates the Chinese editing matrix of the plurality of to-be-screened alignment sentences by analyzing the characters of the plurality of to-be-screened alignment sentences using an editing distance method, wherein the Chinese editing matrix includes Chinese editing distance data of characters corresponding to a plurality of position indexes. The Chinese editing distance data of the characters corresponding to each position index of the same to-be-screened alignment sentence is accumulated according to the Chinese editing matrix of the plurality of to-be-screened alignment sentences to obtain total Chinese editing distance data of the plurality of to-be-screened alignment sentences. This can comprehensively consider the Chinese corresponding to the characters of the sentences processed by alignment to obtain the most reasonable one-to-one correspondence of each character in the two sentences processed by alignment in terms of position, thereby improving the accuracy of sentence alignment.

[0048] S23: Based on the total Chinese editing distance data, obtain the to-be-screened alignment sentence corresponding to the minimum total Chinese editing distance data from the plurality of to-be-screened alignment sentences as the first alignment sentence, and perform alignment processing on the first alignment sentence and a plurality of candidate sentences in the candidate sentence set to obtain a plurality of candidate alignment sentences.

[0049] In this embodiment, the correction device obtains the to-be-screened alignment sentence corresponding to the minimum total Chinese editing distance data from the plurality of to-be-screened alignment sentences as the first alignment sentence based on the total Chinese editing distance data, and performs alignment processing on the first alignment sentence and a plurality of candidate sentences in the candidate sentence set to obtain a plurality of candidate alignment sentences. The alignment processing can refer to the specific steps of S21.

[0050] Please refer to Figure 3 , Figure 3 The flowchart of S23 in the speech recognition-based sentence correction method provided by an embodiment of the present application includes steps S231-S234, which are as follows:

[0051] S231: Compare the plurality of to-be-screened alignment sentences with each other to obtain the to-be-screened alignment sentence corresponding to the minimum total Chinese editing distance data, and count the number of to-be-screened alignment sentences corresponding to the minimum total Chinese editing distance data.

[0052] In the embodiment, the correction device compares the plurality of to-be-screened aligned sentences with each other, obtains comparison results, obtains the to-be-screened aligned sentence corresponding to the minimum total Chinese edit distance data according to the comparison results, and counts the number of the to-be-screened aligned sentences corresponding to the minimum total Chinese edit distance data.

[0053] S232: If the number of the to-be-screened aligned sentences corresponding to the minimum total Chinese edit distance data is one, the to-be-screened aligned sentence corresponding to the minimum total Chinese edit distance data is taken as the first aligned sentence.

[0054] In the embodiment, if the number of the to-be-screened aligned sentences corresponding to the minimum total Chinese edit distance data is one, the to-be-screened aligned sentence corresponding to the minimum total Chinese edit distance data is taken as the first aligned sentence.

[0055] S233: If the number of the to-be-screened aligned sentences corresponding to the minimum total Chinese edit distance data is multiple, a pinyin edit matrix of the to-be-screened aligned sentences corresponding to the multiple minimum total Chinese edit distance data is calculated, the pinyin edit matrix includes pinyin edit distance data of characters corresponding to a plurality of position indexes; and the pinyin edit distance data of the characters corresponding to the position indexes of the same to-be-screened aligned sentence corresponding to the minimum total Chinese edit distance data is accumulated to obtain total pinyin edit distance data of the to-be-screened aligned sentences corresponding to the multiple minimum total Chinese edit distance data.

[0056] In the embodiment, if the number of the to-be-screened aligned sentences corresponding to the minimum total Chinese edit distance data is multiple, a pinyin edit matrix of the to-be-screened aligned sentences corresponding to the multiple minimum total Chinese edit distance data is calculated, the pinyin edit matrix includes pinyin edit distance data of characters corresponding to a plurality of position indexes.

[0057] The pinyin edit distance data of the characters corresponding to the position indexes of the same to-be-screened aligned sentence corresponding to the minimum total Chinese edit distance data is accumulated to obtain total pinyin edit distance data of the to-be-screened aligned sentences corresponding to the multiple minimum total Chinese edit distance data.

[0058] S234: Based on the total pinyin edit distance data, the to-be-screened aligned sentence corresponding to the minimum total pinyin edit distance data is obtained from the to-be-screened aligned sentences corresponding to the multiple minimum total Chinese edit distance data as the first aligned sentence.

[0059] In this embodiment, the correction device obtains, based on the total pinyin edit distance data, the alignment sentence to be screened corresponding to the minimum total pinyin edit distance data from the alignment sentences to be screened corresponding to the minimum total Chinese edit distance data, as the first alignment sentence. The pinyin corresponding to the characters of the sentence subjected to alignment processing can be comprehensively considered to obtain the most reasonable one-to-one correspondence in position of each character in the two sentences subjected to alignment processing, thereby improving the accuracy of sentence alignment.

[0060] S3: Perform pinyin conversion processing on the characters corresponding to the position indexes of the first alignment sentence and the candidate alignment sentences, and obtain pinyin information corresponding to the position indexes.

[0061] In this embodiment, the correction device performs pinyin conversion processing on the characters corresponding to the position indexes of the first alignment sentence and the candidate alignment sentences, and obtains pinyin information corresponding to the position indexes, so as to reduce the negative effects caused by pronunciation problems of the user and multi-phonetic character problems of the text.

[0062] Specifically, in the pinyin conversion process, since the character corresponding to the current position index can be a non-Chinese character, if the non-Chinese character is an English character, the pinyin information of the English character is the English character itself, if the non-Chinese character is a null character, the pinyin information of the null character is [Mask], and if the non-Chinese character is a special character, the pinyin information of the special character is [UNK].

[0063] Please refer to Figure 4 , Figure 4 The flowchart of S3 in the speech recognition-based sentence correction method provided by an embodiment of the present application is shown in FIG. 3, which includes steps S31-S33, and the details are as follows.

[0064] S31: Obtain a character set corresponding to a plurality of position indexes according to the first alignment sentence and the candidate alignment sentences.

[0065] In this embodiment, the correction device traverses the characters corresponding to the position indexes of the first alignment sentence and the candidate alignment sentences, and constructs a character set corresponding to the position indexes, wherein the character set includes the characters corresponding to the position indexes of the first alignment sentence and the candidate alignment sentences.

[0066] S32: Perform mapping processing on a plurality of characters in the character set corresponding to the position indexes according to a preset pinyin mapping table, and obtain a pinyin mapping vector set corresponding to the position indexes.

[0067] The pinyin mapping table includes a plurality of pinyin mapping vectors corresponding to a plurality of characters. In this embodiment, the correction device maps a plurality of characters corresponding to a plurality of position indexes according to a preset pinyin mapping table to obtain a plurality of pinyin mapping vectors corresponding to the plurality of position indexes. The plurality of pinyin mapping vectors include a plurality of pinyin mapping vectors corresponding to a plurality of characters corresponding to a plurality of position indexes.

[0068] S33: According to the plurality of pinyin mapping vectors corresponding to the plurality of position indexes, the pinyin mapping vectors corresponding to the same position index are spliced to obtain a plurality of pinyin splicing vectors corresponding to the plurality of position indexes. The plurality of pinyin splicing vectors corresponding to the plurality of position indexes are input into a preset full splicing network to obtain pinyin information corresponding to the plurality of position indexes.

[0069] In this embodiment, the correction device splices the pinyin mapping vectors corresponding to the same position index according to the plurality of pinyin mapping vectors corresponding to the plurality of position indexes to obtain a plurality of pinyin splicing vectors corresponding to the plurality of position indexes. The number of dimensions of the pinyin splicing vector is an integer multiple of the number of dimensions of a single pinyin mapping vector.

[0070] The plurality of pinyin splicing vectors corresponding to the plurality of position indexes are input into a preset full splicing network to obtain pinyin information corresponding to the plurality of position indexes by fusing the pinyin mapping vectors corresponding to a plurality of characters corresponding to each character position index.

[0071] S4: The first aligned sentence and the pinyin information corresponding to the plurality of position indexes are input into a preset sentence correction model. According to the first aligned sentence and a semantic feature extraction module in the sentence correction model, a plurality of semantic features corresponding to a plurality of position indexes of the first aligned sentence are obtained. According to the plurality of pinyin splicing information corresponding to the plurality of position indexes and a pinyin feature extraction module in the sentence correction model, a plurality of pinyin features corresponding to the plurality of position indexes are obtained.

[0072] The sentence correction model is a pre-trained language model, which can be a bert model, an elmo model, an lstm model, or a transformer model. The sentence correction model includes a semantic feature extraction module and a pinyin feature extraction module. The semantic feature extraction module can convert characters into corresponding semantic features, and the pinyin feature extraction module can convert pinyin information corresponding to characters into corresponding pinyin features.

[0073] In the embodiment, the correction device inputs the first alignment sentence and the pinyin information corresponding to the plurality of position indexes into a preset sentence correction model, obtains semantic features corresponding to the plurality of position indexes of the first alignment sentence according to a semantic feature extraction module in the sentence correction model, and obtains pinyin features corresponding to the plurality of position indexes according to a pinyin feature extraction module in the sentence correction model.

[0074] S5: The semantic features corresponding to the plurality of position indexes of the first alignment sentence and the pinyin features corresponding to the plurality of position indexes are input into a feature fusion module in the sentence correction model, the semantic features and the pinyin features corresponding to the same position index are fused, and fusion information features corresponding to the plurality of position indexes are obtained.

[0075] The sentence correction model further includes a feature fusion module. In the embodiment, the correction device inputs the semantic features corresponding to the plurality of position indexes of the first alignment sentence and the pinyin features corresponding to the plurality of position indexes into the feature fusion module in the sentence correction model, fuses the semantic features and the pinyin features corresponding to the same position index, and obtains fusion information features corresponding to the plurality of position indexes. Specifically, since the dimensions of the semantic features and the pinyin features are the same, the fusion mode can adopt a direct addition mode or a splicing mode.

[0076] S6: The fusion information features corresponding to the plurality of position indexes are input into an output module in the sentence correction model, correction character probabilities corresponding to the plurality of position indexes are obtained, and the first alignment sentence is subjected to sentence correction processing according to the correction characters corresponding to the plurality of position indexes, to obtain a corrected sentence corresponding to the voice data.

[0077] In the embodiment, the correction device inputs the fusion information features corresponding to the plurality of position indexes into the output module in the sentence correction model to obtain correction character probability vectors corresponding to the plurality of position indexes, and obtains correction characters corresponding to the plurality of position indexes according to the correction character probability vectors corresponding to the plurality of position indexes.

[0078] The correction character probability vector is a multi-dimensional vector, the multi-dimensional vector includes a plurality of numerical values, the number of the numerical values is consistent with the number of Chinese characters included in a preset dictionary, and each numerical value in the multi-dimensional vector represents a probability corresponding to each Chinese character in the dictionary. Specifically, the correction device obtains the correction character corresponding to the highest numerical value in the correction character probability vector according to the correction character probability vectors corresponding to the plurality of position indexes.

[0079] According to the corresponding correction character of the position index, the first alignment sentence is subjected to sentence correction processing, and a corrected sentence corresponding to the voice data is obtained.

[0080] Specifically, when the corresponding correction character of the position index is a Chinese character, the corresponding character in the first alignment sentence is replaced by the correction character; if the corresponding correction character of the position index is a blank character [Mask] or a non-Chinese character, the corresponding character in the first alignment sentence is emptied.

[0081] Please refer to Figure 5 , Figure 5 The flowchart of the sentence correction method based on voice recognition provided by another embodiment of the application also includes training the sentence correction model, and the training of the sentence correction model includes step S7, which is before step S4, and is specifically as follows:

[0082] S7: Obtain a sample sentence and a plurality of sample alignment sentences corresponding to the sample sentence, input the sample sentence and the plurality of sample alignment sentences corresponding to the sample sentence into a neural network model to be trained, adjust the model parameters of the neural network model to be trained by using a back propagation method, obtain a target neural network model as the sentence correction model.

[0083] In this embodiment, the correction device can use a random replacement and deletion method to construct a plurality of sample alignment sentences corresponding to a sample sentence input by a user, input the sample sentence and the plurality of sample alignment sentences corresponding to the sample sentence into a neural network model to be trained, adjust the model parameters of the neural network model to be trained by using a back propagation method, and obtain a target neural network model as the sentence correction model.

[0084] In an optional embodiment, the plurality of sample alignment sentences include Chinese characters and non-Chinese characters, wherein the non-Chinese characters include English characters, blank characters, and special characters, so that the trained sentence correction model can identify characters that are English characters, blank characters, and special characters, thereby improving the correction ability of the sentence.

[0085] Please refer to Figure 6 , Figure 6 The structure diagram of the sentence correction device based on voice recognition provided by one embodiment of the application, which can realize all or part of the sentence correction device based on voice recognition by software, hardware, or a combination of the two. The device 6 includes:

[0086] The voice data recognition module 61 is configured to obtain voice data, recognize the voice data by using a beam search technology, and obtain a first candidate sentence and a candidate sentence set, wherein the candidate sentence set includes a plurality of candidate sentences, the first candidate sentence is a candidate sentence with the highest beam search score, and the first candidate sentence and the plurality of candidate sentences each include a plurality of characters.

[0087] The sentence alignment module 62 is configured to perform alignment processing on the first candidate sentence and the plurality of candidate sentences in the candidate sentence set respectively, and obtain a first aligned sentence and a plurality of candidate aligned sentences, wherein the first aligned sentence is an aligned sentence obtained by performing alignment processing on the first candidate sentence, and the candidate aligned sentences are aligned sentences obtained by performing alignment processing on the candidate sentences.

[0088] The pinyin conversion processing module 63 is configured to perform pinyin conversion processing on the characters corresponding to a plurality of position indexes of the first aligned sentence and the plurality of candidate aligned sentences, and obtain pinyin information corresponding to the plurality of position indexes.

[0089] The feature extraction module 64 is configured to input the first aligned sentence and the pinyin information corresponding to the plurality of position indexes into a preset sentence correction model, obtain semantic features corresponding to the plurality of position indexes of the first aligned sentence according to a semantic feature extraction module in the sentence correction model, and obtain pinyin features corresponding to the plurality of position indexes according to pinyin concatenation information corresponding to the plurality of position indexes and a pinyin feature extraction module in the sentence correction model.

[0090] The feature fusion module 65 is configured to input the semantic features corresponding to the plurality of position indexes of the first aligned sentence and the pinyin features corresponding to the plurality of position indexes into a feature fusion module in the sentence correction model, fuse the semantic features and the pinyin features corresponding to the same position index, and obtain fusion information features corresponding to the plurality of position indexes.

[0091] The sentence correction module 66 is configured to input the fusion information features corresponding to the plurality of position indexes into an output module in the sentence correction model, obtain corrected characters corresponding to the plurality of position indexes, and perform sentence correction processing on the first aligned sentence according to the corrected characters corresponding to the plurality of position indexes, to obtain a corrected sentence corresponding to the voice data.

[0092] In the embodiment of the present application, the voice data is obtained through a voice data recognition module, the voice data is recognized using a beam search technology to obtain a first candidate sentence and a candidate sentence set, wherein the candidate sentence set includes a plurality of candidate sentences, the first candidate sentence is the candidate sentence with the highest beam search score, and the first candidate sentence and the plurality of candidate sentences each include a plurality of characters. The first candidate sentence and the plurality of candidate sentences in the candidate sentence set are aligned through a sentence alignment module to obtain a first aligned sentence and a plurality of candidate aligned sentences, wherein the first aligned sentence is the aligned sentence obtained by aligning the first candidate sentence, and the candidate aligned sentence is the aligned sentence obtained by aligning the candidate sentence. The characters corresponding to a plurality of position indexes of the first aligned sentence and the plurality of candidate aligned sentences are converted into pinyin through a pinyin conversion processing module to obtain pinyin information corresponding to the plurality of position indexes. The first aligned sentence and the pinyin information corresponding to the plurality of position indexes are input into a preset sentence correction model through a feature extraction module, semantic features corresponding to the plurality of position indexes of the first aligned sentence are obtained according to a semantic feature extraction module in the sentence correction model, and pinyin features corresponding to the plurality of position indexes are obtained according to pinyin concatenation information corresponding to the plurality of position indexes and a pinyin feature extraction module in the sentence correction model. The semantic features corresponding to the plurality of position indexes of the first aligned sentence and the pinyin features corresponding to the plurality of position indexes are input into a feature fusion module in the sentence correction model through a feature fusion module, the semantic features and the pinyin features corresponding to the same position index are fused to obtain fusion information features corresponding to the plurality of position indexes. The fusion information features corresponding to the plurality of position indexes are input into an output module in the sentence correction model through a sentence correction module, corrected characters corresponding to the plurality of position indexes are obtained, and the first aligned sentence is corrected according to the corrected characters corresponding to the plurality of position indexes to obtain a corrected sentence corresponding to the voice data. The voice data is recognized using the comprehensive beam search technology, the first candidate sentence with the highest beam search score and the plurality of candidate sentences are subjected to semantic feature extraction and pinyin feature extraction, feature fusion is performed, the fusion features after the feature fusion are used to correct the first candidate sentence, the negative effects caused by the pronunciation problems of the user and the multi-pronunciation character problems of the text are reduced, and the accuracy and efficiency of the sentence correction are improved.

[0093] Please refer to Figure 7 , Figure 7The structural schematic diagram of the device provided by the fifth embodiment of the present application is shown in the figure. The device 7 comprises a processor 71, a memory 72, and a computer program 73 stored in the memory 72 and executable on the processor 71. The computer device can store a plurality of instructions, which are suitable for being loaded by the processor 71 and executing the above-mentioned Figures 1 to 5 The method steps of the embodiment shown are not described in detail here, as they can be implemented as described in Embodiments 1 to 3.

[0094] The processor 71 can include one or more processing cores. The processor 71 connects various parts within the server through various interfaces and lines, executes various functions and processes data of the speech recognition-based sentence correction device 6 by running or executing instructions, programs, code sets or instruction sets stored in the memory 72, and calling data in the memory 72. Optionally, the processor 71 can be implemented in at least one of the hardware forms of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programble Logic Array (PLA). The processor 71 can be integrated with one or a combination of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem, etc. Among them, the CPU mainly processes the operating system, user interface, and application programs, etc.; the GPU is responsible for rendering and drawing the content to be displayed on the touch display screen; and the modem is used for processing wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 71, but can be realized by a separate chip.

[0095] The memory 72 can include a Random Access Memory (RAM) and a Read-Only Memory (ROM). Optionally, the memory 72 includes a non-transitory computer-readable storage medium. The memory 72 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 72 can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function (such as touch instructions, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area can store data involved in the above-mentioned various method embodiments, etc. The memory 72 can optionally be at least one storage device located away from the aforementioned processor 71.

[0096] The storage medium can store a plurality of instructions, which are suitable for being loaded by a processor and executing the method steps of the above-mentioned embodiments 1 to 3. For specific execution processes, refer to the specific descriptions of the embodiments 1 to 3, which are not described here.

[0097] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit. In addition, the specific names of the functional units and modules are only for convenient distinction, and do not limit the protection scope of the application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0098] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0099] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0100] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / terminal device and method can be implemented by other ways. For example, the above-mentioned apparatus / terminal device embodiments are only schematic, and the division of the modules or units is only a logical function division, and there can be another division way in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection between the units can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0101] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e., may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0102] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0103] The integrated module / unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer-readable storage medium. When the processor executes the computer program, the steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc.

[0104] The present application is not limited to the above-described embodiments, and various modifications or changes can be made to the present application without departing from the spirit and scope of the present application. If the modifications and changes belong to the scope of the claims of the present application and the equivalent technical scope, the present application also intends to include these modifications and changes.

Claims

1. A speech correction method based on speech recognition, characterized in that, Includes the following steps: Voice data is obtained, and the voice data is recognized using beam search technology to obtain a first candidate statement and a set of candidate statements. The set of candidate statements includes several candidate statements, the first candidate statement is the candidate statement with the highest beam search score, and the first candidate statement and several candidate statements each include characters corresponding to several position indices. Align the first candidate statement with several candidate statements in the candidate statement set to obtain several alignment statements to be filtered. Obtain the Chinese editing matrix of the plurality of alignment statements to be filtered. The Chinese editing matrix includes Chinese editing distance data of characters corresponding to several position indices. Based on the Chinese editing matrix of the plurality of alignment statements to be filtered, the Chinese editing distance data of characters corresponding to each position index of the same alignment statement to be filtered are accumulated to obtain the total Chinese editing distance data of the plurality of alignment statements to be filtered. Based on the total Chinese editing distance data, the several alignment statements to be filtered are compared with each other to obtain the alignment statement to be filtered corresponding to the minimum total Chinese editing distance data, and the number of alignment statements to be filtered corresponding to the minimum total Chinese editing distance data is counted. If the number of alignment statements to be filtered corresponding to the minimum total Chinese editing distance data is one, then the alignment statement to be filtered corresponding to the minimum total Chinese editing distance data is taken as the first alignment statement; If there are multiple pairs of alignment statements corresponding to the minimum total Chinese editing distance data, calculate the pinyin editing matrix of the alignment statements corresponding to the multiple minimum total Chinese editing distance data. The pinyin editing matrix includes the pinyin editing distance data of characters corresponding to several position indices. Accumulate the pinyin editing distance data of characters corresponding to each position index of the same alignment statement corresponding to the minimum total Chinese editing distance data to obtain the total pinyin editing distance data of the alignment statements corresponding to the multiple minimum total Chinese editing distance data. Based on the total pinyin editing distance data, the alignment statement corresponding to the minimum total pinyin editing distance data is obtained from the multiple alignment statements to be filtered corresponding to the minimum total Chinese editing distance data, and is used as the first alignment statement; The first alignment statement is aligned with several candidate statements in the candidate statement set to obtain several candidate alignment statements, wherein the first alignment statement is the alignment statement after aligning the first candidate statement, and the candidate alignment statement is the alignment statement after aligning the candidate statement. The characters corresponding to several position indices of the first alignment statement and several candidate alignment statements are processed by pinyin conversion to obtain pinyin information corresponding to several position indices; The first alignment statement and the pinyin information corresponding to the plurality of position indices are input into a preset statement correction model. Based on the first alignment statement and the semantic feature extraction module in the statement correction model, the semantic features corresponding to the plurality of position indices of the first alignment statement are obtained. Based on the pinyin splicing information corresponding to the plurality of position indices and the pinyin feature extraction module in the statement correction model, the pinyin features corresponding to the plurality of position indices are obtained. The semantic features corresponding to several position indices of the first alignment statement and the pinyin features corresponding to several position indices are input into the feature fusion module in the statement correction model. The semantic features and pinyin features corresponding to the same position index are fused to obtain the fused information features corresponding to the several position indices. The fusion information features corresponding to the plurality of position indices are input into the output module of the sentence correction model to obtain the correction characters corresponding to the plurality of position indices. Based on the correction characters corresponding to the plurality of position indices, the first aligned sentence is processed to obtain the corrected sentence corresponding to the speech data.

2. The speech correction method based on speech recognition according to claim 1, characterized in that, The characters in the first alignment statement and several candidate alignment statements include Chinese characters and non-Chinese characters, wherein the non-Chinese characters include English characters, null characters and special characters.

3. The speech correction method based on speech recognition according to claim 1, characterized in that, The step of performing pinyin conversion on the characters corresponding to the first alignment statement and several position indices of several candidate alignment statements to obtain pinyin information corresponding to several position indices includes the following steps: Based on the first alignment statement and several candidate alignment statements, obtain a character set corresponding to several position indices, wherein the character set includes the characters corresponding to the first alignment statement and several candidate alignment statements at the corresponding position indices; According to the preset pinyin mapping table, several characters in the character set corresponding to the several position indices are mapped to obtain the pinyin mapping vector set corresponding to the several position indices, wherein the pinyin mapping vector set includes the pinyin mapping vectors corresponding to the characters corresponding to the several position indices; Based on the set of pinyin mapping vectors corresponding to the plurality of position indices, the pinyin mapping vectors corresponding to the same position index are concatenated to obtain pinyin concatenation vectors corresponding to the plurality of position indices. The pinyin concatenation vectors corresponding to the plurality of position indices are then input into a preset full concatenation network to obtain the pinyin information corresponding to the plurality of position indices.

4. The speech correction method based on speech recognition according to claim 1, characterized in that, Before inputting the first alignment statement and the pinyin information corresponding to the plurality of position indices into the preset statement correction model, the step includes training the statement correction model, which includes the following steps: Obtain a sample statement and several corresponding sample alignment statements. Input the sample statement and several corresponding sample alignment statements into the neural network model to be trained. Use the backpropagation method to adjust the model parameters of the neural network model to be trained, and obtain the target neural network model as the statement correction model.

5. The speech correction method based on speech recognition according to claim 4, characterized in that: The sample alignment statements include Chinese characters and non-Chinese characters, wherein the non-Chinese characters include English characters, null characters, and special characters.

6. A speech correction device based on speech recognition, characterized in that, include: The speech data recognition module is used to obtain speech data, and to recognize the speech data using beam search technology to obtain a first candidate statement and a set of candidate statements. The set of candidate statements includes several candidate statements, the first candidate statement is the candidate statement with the highest beam search score, and the first candidate statement and several candidate statements each include several characters. The statement alignment module is used to align the first candidate statement with several candidate statements in the candidate statement set to obtain several alignment statements to be filtered. Obtain the Chinese editing matrix of the plurality of alignment statements to be filtered. The Chinese editing matrix includes Chinese editing distance data of characters corresponding to several position indices. Based on the Chinese editing matrix of the plurality of alignment statements to be filtered, the Chinese editing distance data of characters corresponding to each position index of the same alignment statement to be filtered are accumulated to obtain the total Chinese editing distance data of the plurality of alignment statements to be filtered. Based on the total Chinese editing distance data, the several alignment statements to be filtered are compared with each other to obtain the alignment statement to be filtered corresponding to the minimum total Chinese editing distance data, and the number of alignment statements to be filtered corresponding to the minimum total Chinese editing distance data is counted. If the number of alignment statements to be filtered corresponding to the minimum total Chinese editing distance data is one, then the alignment statement to be filtered corresponding to the minimum total Chinese editing distance data is taken as the first alignment statement; If there are multiple pairs of alignment statements corresponding to the minimum total Chinese editing distance data, calculate the pinyin editing matrix of the alignment statements corresponding to the multiple minimum total Chinese editing distance data. The pinyin editing matrix includes the pinyin editing distance data of characters corresponding to several position indices. Accumulate the pinyin editing distance data of characters corresponding to each position index of the same alignment statement corresponding to the minimum total Chinese editing distance data to obtain the total pinyin editing distance data of the alignment statements corresponding to the multiple minimum total Chinese editing distance data. Based on the total pinyin editing distance data, the alignment statement corresponding to the minimum total pinyin editing distance data is obtained from the multiple alignment statements to be filtered corresponding to the minimum total Chinese editing distance data, and is used as the first alignment statement; The first alignment statement is aligned with several candidate statements in the candidate statement set to obtain several candidate alignment statements, wherein the first alignment statement is the alignment statement after aligning the first candidate statement, and the candidate alignment statement is the alignment statement after aligning the candidate statement. The pinyin conversion processing module is used to perform pinyin conversion processing on the characters corresponding to several position indices of the first alignment statement and several candidate alignment statements, and obtain pinyin information corresponding to several position indices; The feature extraction module is used to input the first alignment statement and the pinyin information corresponding to the plurality of position indices into a preset statement correction model, and obtain the semantic features corresponding to the plurality of position indices of the first alignment statement according to the first alignment statement and the semantic feature extraction module in the statement correction model, and obtain the pinyin features corresponding to the plurality of position indices according to the pinyin splicing information corresponding to the plurality of position indices and the pinyin feature extraction module in the statement correction model. The feature fusion module is used to input the semantic features corresponding to several position indices of the first alignment statement and the pinyin features corresponding to the several position indices into the feature fusion module in the statement correction model, and to fuse the semantic features and pinyin features corresponding to the same position index to obtain the fused information features corresponding to the several position indices. The sentence correction module is used to input the fusion information features corresponding to the plurality of position indices into the output module of the sentence correction model, obtain the correction characters corresponding to the plurality of position indices, and perform sentence correction processing on the first aligned sentence based on the correction characters corresponding to the plurality of position indices to obtain the corrected sentence corresponding to the speech data.

7. A device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor; the computer program, when executed by the processor, implements the steps of the speech recognition-based speech correction method as described in any one of claims 1 to 6.

8. A storage medium, characterized in that: The storage medium stores a computer program that, when executed by a processor, implements the steps of the speech recognition-based speech correction method as described in any one of claims 1 to 6.

Images