Text error correction method and related device, equipment and storage medium

Abstract

The present application discloses a text error correction method and related devices, equipment and storage medium, the method comprising: segmenting the text units in the sentence to be corrected to obtain a number of character slices constituting the text units; obtaining a slice feature representation of each character slice; wherein the slice feature representation at least includes the first feature information of the characters contained in the character slice itself; performing spelling detection based on the slice feature representation of the character slices constituting each text unit to obtain the detection results of each text unit; based on the detection results of each text unit in the sentence to be corrected, obtaining a corrected sentence corresponding to the sentence to be corrected. The above scheme can realize automatic spelling error correction and improve the accuracy of spelling error correction.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a text error correction method and related devices, equipment and storage media. Background Art

[0002] At present, the traditional education field is gradually exploring the upgrading of education informatization. However, in real-life scenarios, the cultivation of basic skills such as word spelling still mainly depends on teachers' corrections, which brings great teaching pressure to teachers.

[0003] Existing error correction schemes generally adopt two approaches. One is based on the rich experience of experts in the field of linguistics, compiles grammatical knowledge into a large number of rules, and mainly uses template matching to correct spelling errors. The other uses neural networks for end-to-end learning, directly learns the feature forms of sample pairs of incorrect spellings and correct spellings in the training corpus, and relies on contextual information to correct spelling errors during use. For the former, it relies on the experience and knowledge of experienced experts, which not only requires huge investment, but also makes it difficult to cover situations such as long-tail errors, and is difficult to maintain in the later stage. For the latter, due to excessive reliance on sentence context information for error correction, once the context information is inaccurate, such as an increase in the number of incorrectly spelled words in a sentence, the accuracy of its error correction also drops sharply. In view of this, how to achieve automatic spelling correction and improve the accuracy of spelling correction has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a text correction method and related devices, equipment and storage media, which can realize automatic spelling correction and improve the accuracy of spelling correction.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a text correction method, including: segmenting the text units in the sentence to be corrected to obtain a number of character slices constituting the text units; obtaining a slice feature representation of each character slice; wherein the slice feature representation includes at least the first feature information of the characters contained in the character slice itself; performing spelling detection based on the slice feature representation of the character slices constituting each text unit to obtain the detection results of each text unit; and obtaining a corrected sentence corresponding to the sentence to be corrected based on the detection results of each text unit in the sentence to be corrected.

[0006] In order to solve the above-mentioned technical problems, the second aspect of the present application provides a text correction device, including: a text segmentation module, a feature acquisition module, a spelling detection module and a sentence correction module; the text segmentation module is used to segment the text units in the sentence to be corrected to obtain a number of character slices constituting the text units; the feature acquisition module is used to obtain a slice feature representation of each character slice; wherein the slice feature representation at least includes the first feature information of the characters contained in the character slice itself; the spelling detection module is used to perform spelling detection based on the slice feature representation of the character slices constituting each text unit to obtain the detection results of each text unit; the sentence correction module is used to obtain a corrected sentence corresponding to the sentence to be corrected based on the detection results of each text unit in the sentence to be corrected.

[0007] In order to solve the above technical problems, the third aspect of the present application provides an electronic device, including a memory and a processor coupled to each other, wherein program instructions are stored in the memory, and the processor is used to execute the program instructions to implement the above text error correction method.

[0008] In order to solve the above technical problems, the fourth aspect of the present application provides a computer-readable storage medium, which stores program instructions that can be executed by a processor, and the program instructions are used to implement the above text error correction method.

[0009] The above scheme divides the text unit in the sentence to be corrected to obtain a number of character slices and slice feature representations of the character slices, and performs spelling detection on the slice feature representations of the character slices that constitute each text unit to obtain a detection result. Since the detection results of each text unit are obtained by performing spelling detection based on the slice feature representations of the character slices that constitute each text unit, the detection results of each text unit in the sentence to be corrected pay more attention to the slice feature information of the character slices, which can reduce the influence of erroneous information in the context on the accuracy of the detection result, thereby improving the accuracy of the corrected sentence. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flowchart of an embodiment of a text error correction method provided by the present application;

[0011] Figure 2 yes Figure 1 The flowchart of step S11 is shown as an embodiment;

[0012] Figure 3 It is a structural schematic diagram of an embodiment of a character slice category model provided by the present application;

[0013] Figure 4 It is a structural diagram of an embodiment of an error correction network provided by the present application;

[0014] Figure 5This is a flowchart of an embodiment of obtaining a first sample sentence provided by the present application;

[0015] Figure 6 yes Figure 1 The flowchart of step S14 is shown as an embodiment;

[0016] Figure 7 It is a flowchart of an embodiment of determining a corrected sentence based on the perplexity of a candidate sentence provided by the present application;

[0017] Figure 8 It is a schematic diagram of the framework of an embodiment of a text error correction device provided by the present application;

[0018] Fig. 9 It is a schematic diagram of a framework of an embodiment of an electronic device provided by the present application;

[0019] Fig.10 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0020] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0021] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0022] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0023] See also Figure 1 , Figure 1 It is a flow chart of an embodiment of the text error correction method provided by the present application. It should be noted that if there are substantially the same results, this embodiment does not Figure 1 The process sequence shown is limited. Figure 1 As shown, this embodiment includes:

[0024] Step S11: segment the text unit in the sentence to be corrected to obtain a number of character slices constituting the text unit.

[0025] The method of this embodiment is used to process the sentence to be corrected to obtain the corresponding corrected sentence. In one embodiment, the sentence to be corrected can be any sentence that needs to be corrected, which can be obtained from local storage or cloud storage. It can be understood that in other embodiments, the sentence collected by the input device in real time can also be used as the sentence to be corrected, which is not specifically limited here. For example, the sentence to be corrected is "there is an alppe", "there is an apples in the tree", etc. Of course, the language of the sentence to be corrected may not be limited to English, but may also include but not limited to: French, Russian, Spanish and other alphabetic languages, which are not limited here. For ease of understanding, English is used as an example below.

[0026] It should be noted that in the disclosed embodiment, the text unit may include but is not limited to: words, phrases, etc., which are not limited here. For each text unit in the sentence to be corrected, if it is further divided into several character slices that constitute the text unit, the subsequent model can focus on these character slices, and then combine the context information to obtain the corrected sentence corresponding to the sentence to be corrected, and at the same time, it can also suppress the trouble caused by the wrong text unit in the context when correcting the sentence to be corrected. Therefore, in this embodiment, the text unit in the sentence to be corrected is segmented to obtain several character slices that constitute the text unit. For example, the text unit in the sentence to be corrected "there is an alppe" is segmented, and the two character slices of the text unit "there" are "th", "ere", the 1 character slice of the text unit "is" is "is", the 1 character slice of the text unit "an" is "an", and the 4 character slices of the text unit "alppe" are "a", "l", "pp" and "e".

[0027] In one embodiment, character slices can be obtained based on word-forming morphemes of a text unit, which may include but are not limited to: roots, prefixes, suffixes, etc., which are not limited here. For example, if the text unit "there" contains the suffix "ere", then based on the suffix, the character slices "th" and "ere" of the text unit can be obtained. Other situations can be deduced by analogy, and no examples are given here one by one.

[0028] In one embodiment, the text units in the sentence to be corrected are segmented to obtain a number of characters constituting the corresponding text units, and the characters can be combined into a number of character slices of the corresponding text unit according to certain preset rules. Among them, the preset rules and the model architecture for segmenting the text units of the sentence to be corrected are not limited, and can be specifically set according to actual use needs. For example, according to the slice categories of each character corresponding to each text unit, a number of character slices constituting the corresponding text unit are obtained. Specifically, Figure 2 As shown, Figure 2 yes Figure 1 The flowchart of step S11 of an embodiment shown in FIG. 1 is to combine the characters according to the slice categories of the characters corresponding to the text units to obtain a plurality of character slices constituting the corresponding text units, and specifically includes the following sub-steps:

[0029] Step S111: classify the characters contained in the text unit in the sentence to be corrected to obtain the slice category of each character.

[0030] In this embodiment, the characters contained in the text units in the sentence to be corrected are classified to obtain the slice category of each character. The slice category of the character is any one of the slice header, slice middle, slice tail, and separate slice. That is to say, in this embodiment, the characters contained in all the text units in the sentence to be corrected are classified to obtain the slice category of several characters constituting the corresponding text units, that is, to determine whether each character in the sentence to be corrected is a slice header, slice middle, slice tail, or separate slice.

[0031] Specifically, Figure 3 As shown, Figure 3 It is a structural diagram of an embodiment of a character slice category model provided by the present application. First, the sentence to be corrected is input into the vectorization module of the character slice category model, and the vectorization module first vectorizes each character in the sentence to be corrected; then, each vectorized character in the sentence to be corrected is input into the bidirectional long short word memory neural network (Bi-directional Long Short-Term Memory, BiLSTM) of the character slice category model, so that each character combines its own semantic information and the semantic information of the forward character to obtain the first semantics, and each character combines its own semantic information and the semantic information of the backward character to obtain the second semantics; finally, the first semantics and the second semantics are input into the classification module of the character slice category model to classify the characters contained in the text unit in the sentence to be corrected, that is, each character in the sentence to be corrected, and obtain the slice category of each character. It can be understood that in other embodiments, other character slice category model architectures can also be used to classify the characters contained in the text unit in the sentence to be corrected, which is not specifically limited here.

[0032] For example, taking the sentence to be corrected "there is an alppe" as an example, each character "t", "h", "e", "r", "e", "[]" (i.e., space), "i", "s", "[]", "a", "n", "[]", "a", "l", "p", "p" and "e" in the sentence to be corrected is input into the vectorization module of the character slice category model to vectorize each character; each vectorized character is input into the BiLSTM neural network to obtain the first semantics and the second semantics of each character; the first semantics and the second semantics corresponding to each character are input into a classification module to classify each character to obtain the slice category of each character, specifically: the slice category of the character "t" is the slice head, the slice category of the character "h" is the slice head, and the slice category of the character "h" is the slice head. The slice category is slice tail, the slice category of character "e" is slice head, the slice category of character "r" is slice middle, the slice category of character "e" is slice tail, the slice category of character "[]" is space, the slice category of character "i" is slice head, the slice category of character "s" is slice tail, the slice category of character "[]" is space, the slice category of character "a" is slice head, the slice category of character "n" is slice tail, the slice category of character "[]" is space, the slice category of character "a" is single slice, the slice category of character "l" is single slice, the slice category of character "p" is slice head, the slice category of character "p" is slice tail, and the slice category of character "e" is single slice. Among them, the characters corresponding to the slice category are consistent with the order of each character determined in the sentence to be corrected.

[0033] Step S112: Based on the slice category of each character, a number of character slices constituting the text unit are obtained.

[0034] In this embodiment, according to the slice category of each character, several character slices constituting the text unit can be obtained. Specifically, for the characters in the sentence to be corrected, traverse from left to right to find the first slice head, and continue to traverse backward until the first slice tail is found, then the first character slice is obtained; and so on, until the last character of the last text unit in the sentence to be corrected is traversed.

[0035] For example, if Figure 3As shown, for the characters in the sentence to be corrected "there is an alppe", traverse from the left to find the first slice head "t", and continue to traverse backwards until the first slice tail "h" is found, then the first character slice "th" is obtained; continue to traverse from left to right, find the second slice head "e", and continue to traverse backwards until the first slice tail "e" is found, then the second character slice "ere" is obtained; and so on, until the last character of the last text unit in the sentence to be corrected is traversed. Therefore, the character slices for the sentence to be corrected "there is an alppe" include: "th", "ere", "is", "an", "a", "l", "pp", "e".

[0036] Step S12: Obtain slice feature representation of each character slice.

[0037] The slice feature representation includes at least the first feature information of the characters contained in the character slice itself. In this embodiment, the slice feature representation of each character slice is obtained to facilitate subsequent spelling detection based on the slice feature representation of the character slices constituting each text unit, thereby obtaining the detection result of each text unit.

[0038] In one embodiment, the slice feature representation may also include a second feature information of the confidence of the character slice, that is, the slice feature representation includes the first feature information of the characters contained in the character slice itself and the second feature information of the confidence of the character slice, wherein the confidence represents the possibility of the character slice being misspelled. Since the proportion of misspellings of different character slices is different, it is considered that the possibility of misspellings of different character slices is different. For example, the character slice "ing" is basically not misspelled, so the confidence of the character slice "ing" will be relatively high. By setting the slice feature representation to include the confidence of the character slice, the confidence of each character slice can be determined, that is, the possibility of misspelling of each character slice can be determined, so that the subsequent guidance model will focus on the character slices with low confidence, and thus can greatly suppress the influence of the character slices with a high possibility of misspelling on the spelling correction. It can be understood that in other embodiments, the slice feature representation may also include other feature information, which can be set according to actual use needs and is not specifically limited here.

[0039] In one embodiment, the second feature information is obtained by feature extraction of reference information related to the confidence of the character slice, and the reference information includes at least one of the following: the length of the character slice, the commonness of the character slice, the commonness of the text unit to which the character slice belongs, and the part of speech of the text unit to which the character slice belongs. That is, the second feature information includes feature extraction of at least one of the length of the character slice, the commonness of the character slice, the commonness of the text unit to which the character slice belongs, and the part of speech of the text unit to which the character slice belongs. In theory, the longer the length of a character slice or the more common the character slice is, the inherent writing habits have been formed in the daily writing process and it is not easy to make spelling mistakes, indicating that the possibility of spelling mistakes in the character slice is lower, that is, the higher the confidence of the character slice. The more common the text unit to which a character slice belongs, the lower the possibility of the text unit being misspelled, that is, the higher the confidence of the character slice. The part-of-speech tagging model is used to process the sentence to be corrected to obtain the part-of-speech features of each text unit in the sentence to be corrected. The part-of-speech tagging model can determine whether each word belongs to a noun, a verb or an adjective according to the context content, which plays an important role in determining whether the current text unit is misspelled. For example, taking the sentence to be corrected "I have a carry" as an example, the part-of-speech tagging model determines that the text unit "carry" should be a noun according to the context, but the actual text unit "carry" is a verb, so the text unit "carry" may be misspelled. Among them, the second feature information is not limited to the above-mentioned ones, and can be specifically set according to actual use needs.

[0040] In a specific implementation, the reference information includes the length of the character slice, the commonness of the character slice, the commonness of the text unit to which the character slice belongs, and the part of speech of the text unit to which the character slice belongs. Assuming that the length of a character slice is the number n of characters it contains, the word embedding layer is used to extract the features corresponding to the length of the character slice. The specific formula is as follows:

[0041] e1=Emb1(n)

[0042] Wherein, e1 is the feature corresponding to the length of the character slice; n is the length of the character slice. It should be noted that the word embedding layer can be implemented by word vector tools such as word2vec, which is not limited here.

[0043] The commonness of a character slice can be understood as a function of the frequency of the character slice appearing in the external resource library, as follows:

[0044] p1=log 10 (1+count1)

[0045] Wherein, p1 is the commonness of the character slice, and count1 is the frequency of the character slice appearing in the external resource library. It should be noted that the external resource library may include but is not limited to: e-mails, chat records, student homework, etc., which are not limited here.

[0046] Furthermore, the common degree p1 of the character slice is rounded off, and the word embedding layer is used to extract the features corresponding to the common degree of the character slice. The specific formula is as follows:

[0047] e2=Emb2(round(p1))

[0048] Among them, e2 is the feature corresponding to the commonness of the character slice; p1 is the commonness of the character slice.

[0049] The commonness of the text unit to which the character slice belongs can be understood as a function of the frequency of the text unit to which the character slice belongs in the external resource library, as follows:

[0050] p2=log 10 (1+count2)

[0051] Among them, p2 is the commonness of the text unit to which the character slice belongs; count2 is the frequency of the text unit to which the character slice belongs in the external resource library.

[0052] Furthermore, the common degree p2 of the text unit to which the character slice belongs is rounded off, and the word embedding layer is used to extract the features corresponding to the common degree of the text unit to which the character slice belongs. The specific formula is as follows:

[0053] e3=Emb3(round(p2))

[0054] Among them, e3 is the feature corresponding to the commonness of the text unit to which the character slice belongs; p2 is the commonness of the text unit to which the character slice belongs.

[0055] The part-of-speech tagging model is used to obtain the part-of-speech of the text unit to which the character slice belongs, and the word embedding layer is used to extract the features corresponding to the part-of-speech of the text unit to which the character slice belongs. The specific formula is as follows:

[0056] e4=Emb4(POS(word))

[0057] Among them, e4 is the feature corresponding to the part of speech of the text unit to which the character slice belongs; POS is the part of speech tagging model; word is the text unit to which the character slice belongs. It should be noted that the part of speech tagging model may include but is not limited to: Hidden Markov Model (HMM), Conditional Random Fields (CRFs), etc., which are not limited here.

[0058] In addition, the word embedding layer is used to extract features of the character slice to obtain first feature information of the characters contained in the character slice itself. For example, it is assumed that the feature corresponding to the characters contained in the character slice itself is e0.

[0059] Therefore, the slice feature of each character slice is expressed as e=e0+e1+e2+e3+e4.

[0060] Step S13: Perform spelling detection based on the slice feature representation of the character slices constituting each text unit to obtain the detection result of each text unit.

[0061] In this embodiment, spelling detection is performed based on the slice feature representation of the character slices that constitute each text unit to obtain the detection results of each text unit. Among them, the content contained in the detection results of each text unit obtained is not specifically limited, and can be specifically set according to actual use needs. Since the detection results of each text unit are obtained by performing spelling detection based on the slice feature representation of the character slices that constitute each text unit, the detection results of each text unit in the error correction sentence are more concerned with the slice characteristic information of the character slice, thereby reducing the influence of erroneous information in the context on the accuracy of the detection results, thereby improving the accuracy of the corrected sentences obtained later.

[0062] Optionally, in one embodiment, the detection result includes a predicted probability value that the text unit should be corrected into a plurality of preset text units. The number of preset text units is not limited and can be specifically set according to actual use needs. For example, the preset text units are 1, 2, 3, 4, 5, 10 or 15, etc.

[0063] In a specific embodiment, the detection result includes the predicted probability value that each text unit should be corrected into two preset text units. For example, taking the text unit "carry" as an example, after performing spelling detection on the slice feature representation of the character slices constituting the text unit "carry", it is obtained that the text unit "carry" should be corrected into two preset text units and the corresponding predicted probability values ​​are: the predicted probability value corresponding to the preset text unit "car" is 98%, and the predicted probability value corresponding to the preset text unit "cat" is 78%.

[0064] Optionally, the network model for detecting the slice feature representation of the character slice of each text unit is not limited and can be specifically set according to actual use needs. For example, the detection result is obtained by using an error correction network. Specifically, Figure 4 As shown, Figure 4 It is a structural schematic diagram of an embodiment of an error correction network provided by the present application, in which the character slices constituting each text unit and their corresponding slice feature representations are input into the CNN network model in the error correction network, and after passing through the BiLSTM neural network and the classification module in the error correction network, the detection results of each text unit are output. For example, when the detection results include the predicted probability values ​​that each text unit should be respectively corrected as a number of preset text units, the character slices constituting each text unit and their corresponding slice feature representations are passed through the error correction network, and the output is the predicted probability values ​​that each text unit should be respectively corrected as a number of preset text units.

[0065] In a specific embodiment, the error correction network is pre-trained based on the first sample sentence and then trained based on the second sample sentence. The first sample sentence is constructed based on the sample corpus, the second sample sentence is collected from reality, and both the first sample sentence and the second sample sentence contain incorrectly spelled text units, and both the first sample sentence and the second sample sentence are annotated with correctly spelled text units corresponding to the incorrectly spelled text units. In other words, the use of the first sample sentence constructed based on the sample corpus and the second sample sentence collected from reality can construct a large amount of sample sentence data to train the error correction network. Compared with the error correction network trained using only the sample sentence data collected from reality, the error correction network trained using the sample sentence data obtained by artificial design and the sample sentence data obtained from reality in turn has higher accuracy. In addition, the sample corpus can specifically include candidate sample sentences, on this basis, the candidate sample sentences can be processed based on rules or models to obtain the first sample sentence. Exemplarily, a text generation model (e.g., an end-to-end network based on an encoder-decoder structure) can be pre-trained so that candidate sample sentences are input into the text generation model to generate a first sample sentence; alternatively, construction rules can be pre-defined, and candidate sample sentences can be constructed based on the construction rules to obtain a first sample sentence.

[0066] Further, taking the use of the first sample sentence to train the error correction network as an example, the sample slice feature representation of the sample character slices contained in each sample text unit in the first sample sentence can be obtained through the above-mentioned related description, and the sample slice feature representation of the sample character slices constituting each sample text unit is input into the error correction network to obtain the prediction probability value of each sample text unit being corrected as a number of preset text units, so that for each sample text unit, the sub-prediction loss of the corresponding sample text unit can be calculated based on whether the sample text unit is marked as a misspelled text unit, the corresponding correctly spelled text unit, and the prediction probability value of the above-mentioned several preset text units, and then the sub-prediction loss corresponding to each sample text unit in the first sample sentence is calculated to obtain the total loss value, so that the network parameters of the error correction network can be adjusted based on the total loss value through optimization methods such as gradient descent, and the above process is repeated until the error correction network is trained to convergence using the first sample sentence. On this basis, the error correction network can continue to be trained using the second sample sentence. The specific process can refer to the above-mentioned description of the training of the error correction network using the first sample sentence, which will not be repeated here.

[0067] In one embodiment, if Figure 5 As shown, Figure 5 : is a flowchart of an embodiment of obtaining a first sample sentence provided by the present application. The obtaining of the first sample sentence specifically includes the following sub-steps:

[0068] Step S51: Counting the statistical probability values ​​of spelling errors in sample text units in the candidate sample sentences.

[0069] In this embodiment, the statistical probability of errors in spelling of sample text units in candidate sample sentences is counted. That is, in this embodiment, the statistical probability values ​​of errors in spelling of sample text units in sample sentences are first counted, so that it can be determined which text units to edit based on the statistical probability values. The specific formula for the statistical probability value is:

[0070] W=N w / N a

[0071] Where W is the statistical probability value of a spelling error in a sample text unit; N w N is the number of spelling errors in the sample text unit; a is the total number of times the sample text unit appears.

[0072] Step S52: selecting at least one sample text unit based on the statistical probability value as the text unit to be edited.

[0073] In this embodiment, at least one sample text unit is selected as the text unit to be edited according to the statistical probability value. That is, at least one sample text unit is selected from the candidate sample sentences as the text unit to be edited according to the statistical probability value of the spelling errors of each sample text unit in the candidate sample sentences, so that it can be edited as a text unit with spelling errors later. Among them, the number of selected sample text units is not limited, and can be specifically set according to actual use needs.

[0074] In one embodiment, the sample text unit corresponding to the maximum statistical probability value in the candidate sample sentence is used as the text unit to be edited. The larger the statistical probability value, the greater the probability that the sample text unit in the candidate sample sentence is spelled incorrectly, so it is used as the text unit to be edited, so that it is subsequently edited as a text unit with spelling errors, which is more credible and more practical.

[0075] In other embodiments, the sample text units corresponding to the first preset number of statistical probability values ​​in the candidate sample sentences are used as the text units to be edited. The number of the first preset number of statistical probability values ​​is not limited and can be specifically set according to actual use needs.

[0076] Step S53: re-editing the text unit to be edited into a spelling error to obtain a first sample sentence.

[0077] In this embodiment, the text unit to be edited is re-edited to a spelling error to obtain a first sample sentence. Optionally, the re-editing of the text unit to be edited may be to add characters to the text unit to be edited, to modify the first preset characters of the text unit to be edited, or to delete the second preset characters of the text unit to be edited. The number of the first preset characters and the second preset characters is not limited and can be specifically set according to actual use needs.

[0078] Step S14: based on the detection results of each text unit in the sentence to be corrected, a correction sentence corresponding to the sentence to be corrected is obtained.

[0079] In this embodiment, based on the detection results of each text unit in the sentence to be corrected, a corrected sentence corresponding to the sentence to be corrected is obtained. Optionally, in one embodiment, the detection results include predicted probability values ​​that the text units should be corrected into several preset text units respectively. Among them, the number of preset text units is not limited, and can be specifically set according to actual use needs. For example, the preset text units are 1, 2, 3, 4, 5, 10 or 15, etc. Since the detection results of each text unit in the sentence to be corrected are more concerned with the slice characteristic information of the character slice, it can reduce the influence of the erroneous information in the context on the accuracy of the detection result, thereby improving the accuracy of the corrected sentence obtained.

[0080] For example, taking the sentence to be corrected "there is an alppe" as an example, spelling detection is performed based on the slice feature representation of the character slices that constitute each text unit to obtain the detection results of each text unit, and then according to the detection results, the corrected sentence corresponding to the sentence to be corrected "there is an alppe" is obtained as "there is an apple".

[0081] In the above implementation, the text unit in the sentence to be corrected is segmented to obtain a number of character slices and slice feature representations of the character slices, and the slice feature representations of the character slices constituting each text unit are subjected to spelling detection to obtain a detection result. Since the detection result of each text unit is obtained by performing spelling detection on the slice feature representations of the character slices constituting each text unit, the detection result of each text unit in the sentence to be corrected pays more attention to the slice feature information of the character slices, which can reduce the influence of the wrong information in the context on the accuracy of the detection result, thereby improving the accuracy of the corrected sentence.

[0082] See also Figure 6 , Figure 6 yes Figure 1 It should be noted that if there is substantially the same result, this embodiment does not use Figure 6 The process sequence shown is limited. Figure 6 As shown, this embodiment includes:

[0083] Step S141: In response to the text unit being inconsistent with the reference text unit of the text unit, the text unit is used as a text unit to be corrected.

[0084] Among them, the reference text unit is a preset text unit with the largest prediction probability value among several preset text units that the text unit should be corrected to. In this embodiment, in response to the inconsistency between the text unit and the reference text unit of the text unit, the text unit is used as a text unit to be corrected. That is to say, in this embodiment, the text unit is compared with the preset text unit with the largest prediction probability among several preset text units, that is, the reference text unit. If the text unit is consistent with the preset text unit with the largest prediction probability among several preset text units, it is considered that this text unit has no spelling errors and does not need to be corrected; if the text unit is inconsistent with the preset text unit with the largest prediction probability among several preset text units, it is considered that this text unit is spelled incorrectly and needs to be corrected as a text unit to be corrected.

[0085] For example, take the sentence to be corrected "there is an alppe", the text unit "alppe", and the five preset text units corresponding to the text unit, namely "A", "B", "C", "D" and "E"; since the text unit "alppe" is inconsistent with the preset text unit with the largest prediction probability value among its five preset text units, namely the reference text unit "A", the text unit "alppe" is used as the text unit to be corrected.

[0086] Step S142: using a plurality of preset text units corresponding to the text unit to be corrected as candidate text units to replace the text unit to be corrected, to obtain a plurality of candidate sentences.

[0087] In this embodiment, several preset text units corresponding to the text unit to be corrected are used as candidate text units to replace the text unit to be corrected, thereby obtaining several candidate sentences. For example, take the sentence to be corrected "there is analppe", the text unit to be corrected "alppe", and the five preset text units corresponding to the text unit to be corrected are "A", "B", "C", "D", and "D" as an example; the five preset text units corresponding to the text unit to be corrected "alppe" are used as candidate text units to replace the text unit to be corrected, and five candidate sentences are obtained, specifically: "there is an A", "there isan B", "there is an C", "there is an D", and "there is an E".

[0088] Step S143: Based on each candidate sentence, obtain a correction sentence corresponding to the error correction sentence.

[0089] In this embodiment, the correction sentence corresponding to the error correction sentence can be obtained according to each candidate sentence. In one embodiment, the perplexity (ppl) of each candidate sentence can be used to determine which candidate sentence is used as the correction sentence corresponding to the error correction sentence. In the field of natural language processing, perplexity is an indicator used to measure the quality of a language model. It mainly estimates the probability of the candidate sentence appearing based on each text unit. The higher the probability of the candidate sentence appearing, the higher the fluency of the candidate sentence, and the greater the possibility that it is the correction sentence corresponding to the error correction sentence. It can be understood that in other embodiments, other methods can also be used to determine which candidate sentence among the candidate sentences is the correction sentence corresponding to the error correction sentence.

[0090] In one embodiment, see Figure 7 , Figure 7 This is a flow chart of an embodiment of determining a corrected sentence based on the perplexity of a candidate sentence provided by the present application. Determining a corrected sentence based on the perplexity of a candidate sentence specifically includes the following sub-steps:

[0091] Step S71: Obtain the perplexity of each candidate sentence.

[0092] In this implementation, the perplexity of each candidate sentence is first obtained, so that whether to use the candidate sentence as a correction sentence is subsequently determined based on the perplexity of each candidate sentence.

[0093] Step S72: Based on the perplexity, select a candidate sentence as the correction sentence.

[0094] In this embodiment, a candidate sentence is selected as a correction sentence according to the perplexity of each candidate sentence. The smaller the perplexity, the greater the probability value of the corresponding candidate sentence, the greater the fluency of the candidate sentence, that is, the higher the possibility of the candidate sentence. Therefore, the candidate sentence corresponding to the smallest perplexity among the candidate sentences is used as the correction sentence.

[0095] See also Figure 8 , Figure 8 It is a schematic diagram of the framework of an embodiment of a text correction device provided by the present application. The text correction device 80 includes a text segmentation module 81, a feature acquisition module 82, a spelling detection module 83 and a sentence correction module 84. The text segmentation module 81 is used to segment the text units in the sentence to be corrected to obtain a number of character slices constituting the text units; the feature acquisition module 82 is used to obtain the slice feature representation of each character slice; wherein the slice feature representation at least includes the first feature information of the characters contained in the character slice itself; the spelling detection module 83 is used to perform spelling detection based on the slice feature representation of the character slices constituting each text unit to obtain the detection results of each text unit; the sentence correction module 84 is used to obtain the corrected sentence corresponding to the sentence to be corrected based on the detection results of each text unit in the sentence to be corrected.

[0096] The above-mentioned slice feature representation also includes second feature information of the confidence of the character slice, and the confidence represents the possibility of spelling errors in the character slice.

[0097] Among them, the above-mentioned second feature information is obtained by feature extraction of reference information related to the confidence of the character slice, and the reference information includes at least one of the following: the length of the character slice, the commonness of the character slice, the commonness of the text unit to which the character slice belongs, and the part of speech of the text unit to which the character slice belongs.

[0098] The above detection results include predicted probability values ​​that the text unit should be corrected into several preset text units respectively.

[0099] Among them, the sentence correction module 84 is used to obtain a corrected sentence corresponding to the sentence to be corrected based on the detection results of each text unit in the sentence to be corrected, specifically including: in response to the inconsistency between the text unit and the reference text unit of the text unit, taking the text unit as the text unit to be corrected; wherein the reference text unit is a preset text unit with the largest predicted probability value among several preset text units that the text unit should be corrected to; several preset text units corresponding to the text unit to be corrected are respectively used as candidate text units to replace the text unit to be corrected, to obtain several candidate sentences; based on each candidate sentence, obtaining a corrected sentence corresponding to the correction sentence.

[0100] The sentence correction module 84 is used to obtain a correction sentence corresponding to the error correction sentence based on each candidate sentence, specifically including: obtaining the perplexity of each candidate sentence; and selecting the candidate sentence as the correction sentence based on the perplexity.

[0101] Among them, the above detection results are obtained by using an error correction network, and the error correction network is pre-trained based on the first sample sentence and then trained based on the second sample sentence; wherein the first sample sentence is constructed based on a sample corpus, and the second sample sentence is collected from reality, and the first sample sentence and the second sample sentence both contain incorrectly spelled text units, and the first sample sentence and the second sample sentence are both annotated with correctly spelled text units corresponding to the incorrectly spelled text units.

[0102] The sample corpus includes candidate sample sentences, and the step of obtaining the first sample sentence includes: calculating the statistical probability value of spelling errors in sample text units in the candidate sample sentences; selecting at least one sample text unit as a text unit to be edited based on the statistical probability value; and re-editing the text unit to be edited into a spelling error to obtain the first sample sentence.

[0103] Among them, the text segmentation module 81 is used to segment the text unit in the sentence to be corrected to obtain a number of character slices that constitute the text unit, specifically including: classifying the characters contained in the text unit in the sentence to be corrected to obtain the slice category of each character; wherein the slice category is any one of the slice head, slice middle, slice tail, and single slice; based on the slice category of each character, obtain a number of character slices that constitute the text unit.

[0104] See also Fig. 9 , Fig. 9 1 is a schematic diagram of a framework of an embodiment of an electronic device provided by the present application. The electronic device 90 includes a memory 91 and a processor 92 coupled to each other, the memory 91 stores program instructions, and the processor 92 is used to execute the program instructions to implement the steps in any of the above-mentioned text error correction method embodiments. Specifically, the electronic device 90 may include but is not limited to: a desktop computer, a laptop computer, a server, a mobile phone, a tablet computer, etc., which are not limited here.

[0105] Specifically, the processor 92 is used to control itself and the memory 91 to implement the steps in any of the above-mentioned text error correction method embodiments. The processor 92 can also be called a CPU (Central Processing Unit). The processor 92 may be an integrated circuit chip with signal processing capabilities. The processor 92 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 92 can be implemented by an integrated circuit chip.

[0106] See also Fig.10 , Fig.10 The computer-readable storage medium 100 stores program instructions 101 that can be executed by a processor, and the program instructions 101 are used to implement the steps in any of the above text error correction method embodiments.

[0107] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0108] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0109] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0110] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0111] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0112] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0113] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A text error correction method, characterized in that: include: Segmenting the text unit in the sentence to be corrected to obtain a number of character slices constituting the text unit; Obtaining a slice feature representation of each of the character slices; wherein the slice feature representation includes first feature information of characters contained in the character slice itself and second feature information of the confidence of the character slice, wherein the confidence represents the possibility of the character slice being misspelled; Perform spelling detection based on slice feature representations of character slices constituting each of the text units to obtain detection results for each of the text units; Based on the detection results of each of the text units in the sentence to be corrected, obtaining a correction sentence corresponding to the sentence to be corrected; The step of segmenting the text unit in the sentence to be corrected to obtain a plurality of character slices constituting the text unit includes: Classifying the characters contained in the text unit in the sentence to be corrected to obtain a slice category of each character; wherein the slice category is any one of a slice head, a slice middle, a slice tail, and a single slice; Based on the slice categories of the respective characters, a plurality of character slices constituting the text unit are obtained.

2. The method according to claim 1, characterized in that The second feature information is obtained by feature extraction of reference information related to the confidence of the character slice, and the reference information includes at least one of the following: the length of the character slice, the commonness of the character slice, the commonness of the text unit to which the character slice belongs, and the part of speech of the text unit to which the character slice belongs.

3. The method according to any one of claims 1 to 2, characterized in that: The detection result includes predicted probability values ​​that the text unit should be respectively corrected into a plurality of preset text units.

4. The method according to claim 3, characterized in that The obtaining, based on the detection results of each of the text units in the sentence to be corrected, a correction sentence corresponding to the sentence to be corrected, comprises: In response to the text unit being inconsistent with a reference text unit of the text unit, the text unit is used as a text unit to be corrected; wherein the reference text unit is a preset text unit of the plurality of preset text units to which the text unit should be corrected respectively and which has the largest predicted probability value; Using several preset text units corresponding to the text unit to be corrected as candidate text units to replace the text unit to be corrected, to obtain several candidate sentences; Based on each of the candidate sentences, a correction sentence corresponding to the error correction sentence is obtained.

5. The method according to claim 4, characterized in that The obtaining, based on each of the candidate sentences, a correction sentence corresponding to the error correction sentence comprises: Obtaining the perplexity of each of the candidate sentences; Based on the perplexity, the candidate sentence is selected as the correction sentence.

6. The method according to claim 1, characterized in that The detection result is obtained by using an error correction network, and the error correction network is pre-trained based on the first sample sentence and then trained based on the second sample sentence; Among them, the first sample sentence is constructed based on a sample corpus, the second sample sentence is obtained by actual collection, and the first sample sentence and the second sample sentence both contain misspelled text units, and the first sample sentence and the second sample sentence are both annotated with the correctly spelled text units corresponding to the misspelled text units.

7. The method according to claim 6, characterized in that The sample corpus includes candidate sample sentences, and the step of obtaining the first sample sentence includes: Counting the statistical probability values ​​of errors in spelling of sample text units in the candidate sample sentences; Selecting at least one of the sample text units based on the statistical probability value as a text unit to be edited; The to-be-edited text unit is re-edited into a spelling error to obtain the first sample sentence.

8. A text error correction device, characterized in that: include: A text segmentation module is used to segment the text unit in the sentence to be corrected to obtain a number of character slices constituting the text unit; A feature acquisition module, used to acquire a slice feature representation of each of the character slices; wherein the slice feature representation includes first feature information of characters contained in the character slice itself and second feature information of the confidence of the character slice, wherein the confidence represents the possibility of the character slice being misspelled; A spelling detection module, used for performing spelling detection based on the slice feature representation of the character slices constituting each of the text units, and obtaining a detection result of each of the text units; A sentence correction module, used for obtaining a correction sentence corresponding to the sentence to be corrected based on the detection results of each of the text units in the sentence to be corrected; The step of segmenting the text unit in the sentence to be corrected to obtain a plurality of character slices constituting the text unit includes: Classifying the characters contained in the text unit in the sentence to be corrected to obtain a slice category of each character; wherein the slice category is any one of a slice head, a slice middle, a slice tail, and a single slice; Based on the slice categories of the respective characters, a plurality of character slices constituting the text unit are obtained.

9. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the text error correction method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the text error correction method according to any one of claims 1 to 7.

Images