Text error detection method, device, electronic device and storage medium

By training text translation, calculating text similarity and confusion, word alignment operations and other steps, training semantic unknown detection model is solved, and the problem of difficult to detect text semantic unknown errors in the prior art is achieved, and higher accuracy text detection and error correction are achieved.

CN113822052BActive Publication Date: 2025-04-18SHANGHAI LIULISHUO INFORMATION TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202010558705.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-06-18
Publication Date
2025-04-18
Estimated Expiration
2040-06-18

AI Technical Summary

Technical Problem

The prior art is difficult to effectively detect semantic errors in text, resulting in inaccurate error correction methods and inability to understand the user's true intention to express it.

Method used

By obtaining the training text of the first language, determine its confusion and grammatical error information, and translate it into the pivot language text of the second language, and then translate it back to the target text of the first language. The text similarity and confusion are calculated, and word alignment operations are performed to obtain alignment evaluation information. Based on these feature information, the initial model is trained to obtain a semantic unknown detection model, which is used to detect semantic unknown errors in the text of the statement to be detected.

Benefits of technology

It realizes accurate detection of semantic errors in text, improves the accuracy of text detection, can better understand the user's true intentions and provide meaningful feedback.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113822052B_ABST
    Figure CN113822052B_ABST
Patent Text Reader

Abstract

The present application discloses a text error detection method, which includes obtaining a training text in a first language, and determining the perplexity and grammatical error information of the training text; translating the training text into a pivot language text in a second language, and translating the pivot language text into a target text in a first language; calculating the text similarity between the training text and the target text, and determining the perplexity of the target text; performing word alignment operations on the training text and the target text with the pivot language text respectively, and obtaining alignment evaluation information of the training text and the target text; training an initial model to obtain a semantic ambiguity detection model, and performing a text error detection operation on the sentence text to be detected through the semantic ambiguity detection model. The present application can detect semantic ambiguity errors in text and improve the accuracy of text detection. The present application also discloses a text error detection device, an electronic device, and a storage medium, which have the above beneficial effects.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of text detection, and in particular to a text error detection method, device, electronic device and storage medium. Background Art

[0002] In interactive grammar feedback teaching, error detection of user input text can improve teaching quality. Semantic ambiguity errors in text are errors that cause ambiguity in the text. In related technologies, when the user input is a semantically unclear sentence, it is directly judged as a grammatical error and grammatical correction is performed. This correction method often results in the error being changed into an expression that is still incorrect. The above method cannot understand the user's true expression intention and provide meaningful feedback.

[0003] Therefore, how to detect semantically ambiguous errors in text and improve the accuracy of text detection is a technical problem that technical personnel in this field currently need to solve. Summary of the invention

[0004] The purpose of this application is to provide a text error detection method, device, electronic device and storage medium, which can detect semantically ambiguous errors in text and improve the accuracy of text detection.

[0005] In order to solve the above technical problems, the present application provides a text error detection method, which includes:

[0006] Obtaining a training text in a first language and determining perplexity and grammatical error information of the training text;

[0007] Translating the training text into a pivot language text in a second language, and translating the pivot language text into a target text in the first language;

[0008] Calculating the text similarity between the training text and the target text, and determining the perplexity of the target text;

[0009] Performing word alignment operations on the training text and the target text with the pivot language text respectively to obtain alignment evaluation information of the training text and the target text;

[0010] An initial model is trained according to the perplexity of the training text, the grammatical error information of the training text, the text similarity, the perplexity of the target text, the perplexity ratio of the target text to the training text, and the alignment evaluation information to obtain a semantically ambiguous detection model;

[0011] The semantic ambiguity detection model is used to perform text error detection operations on the sentence text to be detected.

[0012] Optionally, the grammatical error information includes an average probability of a grammatical error in each word in the training text, and the number of substitution errors for words with different roots.

[0013] Optionally, performing a word alignment operation on the training text and the target text with the pivot language text respectively to obtain alignment evaluation information of the training text and the target text includes:

[0014] Performing a word alignment operation on the training text and the pivot language text to obtain a first alignment result;

[0015] Performing a word alignment operation on the target text and the pivot language text to obtain a second alignment result;

[0016] Alignment evaluation information of the training text and the target text is determined according to the first alignment result and the second alignment result.

[0017] Optionally, the alignment evaluation information includes:

[0018] The ratio of the number of aligned content words in the training text to the number of all content words in the sentence;

[0019] The ratio of the number of aligned content words in the target text to the number of all content words in the sentence;

[0020] Alignment ratio; wherein the process of determining the alignment ratio includes: connecting the words in the training text and the target text that correspond to the pivot language text at the same time, and taking the ratio of the number of crossings of the connection and the total alignment as the alignment ratio;

[0021] The probability of the first word alignment probability and the second word alignment ratio; wherein the first word alignment probability is the probability of the training text and the pivot language text word alignment, and the second word alignment ratio is the probability of the training text and the pivot language text word alignment.

[0022] Optionally, performing a text error detection operation on the sentence text to be detected by using the semantic ambiguity detection model includes:

[0023] Determine the sentence text to be detected;

[0024] The sentence text to be detected is input into the semantic ambiguity detection model, and it is determined whether there is a text error in the sentence text to be detected according to the detection result output by the semantic ambiguity detection model.

[0025] Optionally, determining the sentence text to be detected includes:

[0026] If voice information is received, the voice information is converted into a sentence text to be detected in the first language.

[0027] Optionally, after performing a text error detection operation on the sentence text to be detected by using the semantic ambiguity detection model, the method further includes:

[0028] Marking text content with errors in the sentence text to be detected, and generating a corrected text in the first language based on the text content with errors.

[0029] The present application also provides a text error detection device, the text error detection device comprising:

[0030] A training text processing module, used for obtaining a training text in a first language and determining perplexity and grammatical error information of the training text;

[0031] A language translation module, configured to translate the training text into a pivot language text in a second language, and to translate the pivot language text into a target text in the first language;

[0032] A target text processing module, used to calculate the text similarity between the training text and the target text, and determine the perplexity of the target text;

[0033] A word alignment module, used to perform word alignment operations on the training text and the target text with the pivot language text respectively, to obtain alignment evaluation information of the training text and the target text;

[0034] A model training module, used to train an initial model to obtain a semantic ambiguity detection model according to the perplexity of the training text, the grammatical error information of the training text, the text similarity, the perplexity of the target text, the perplexity ratio of the target text to the training text, and the alignment evaluation information;

[0035] The detection module is used to perform text error detection operations on the sentence text to be detected through the semantic ambiguity detection model.

[0036] The present application also provides a storage medium on which a computer program is stored. When the computer program is executed, the steps of the above-mentioned text error detection method are implemented.

[0037] The present application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned text error detection method when calling the computer program in the memory.

[0038] The present application provides a text error detection method, comprising obtaining a training text in a first language, and determining the perplexity and grammatical error information of the training text; translating the training text into a pivot language text in a second language, and translating the pivot language text into a target text in the first language; calculating the text similarity between the training text and the target text, and determining the perplexity of the target text; performing word alignment operations on the training text and the target text with the pivot language text respectively, to obtain alignment evaluation information of the training text and the target text; training an initial model according to the perplexity of the training text, the grammatical error information of the training text, the text similarity, the perplexity of the target text, the perplexity ratio of the target text to the training text, and the alignment evaluation information to obtain a semantic ambiguity detection model; and performing a text error detection operation on a sentence text to be detected by using the semantic ambiguity detection model.

[0039] After obtaining the training text in the first language, the present application determines the perplexity and grammatical crop information of the training text. After translating the training text into the second language, it is translated into the target text in the first language, and then the perplexity of the target text and the text similarity between the target text and the training text are determined. The present application also performs word alignment operations on the training text and the target text with the pivot language text respectively to obtain alignment evaluation information, and then uses the above-mentioned feature information about the training text and the target text to train the initial model to obtain a semantic ambiguity detection model, and uses the semantic ambiguity detection model to detect semantic ambiguity errors in the sentence text to be detected. It can be seen that the present application can detect semantic ambiguity errors in text and improve the accuracy of text detection. The present application also provides a text error detection device, an electronic device and a storage medium, which have the above-mentioned beneficial effects and are not repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 A flowchart of a text error detection method provided in an embodiment of the present application;

[0042] Figure 2 A schematic diagram of a determination principle of an alignment ratio provided in an embodiment of the present application;

[0043] Figure 3 A schematic diagram of the structure of a text error detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0045] See below Figure 1 , Figure 1 A flowchart of a text error detection method provided in an embodiment of the present application.

[0046] Specific steps may include:

[0047] S101: Obtain a training text in a first language, and determine perplexity and grammatical error information of the training text;

[0048] The first language and the second language mentioned in this embodiment are any two different languages, which can be selected according to the specific application scenario. For example, the first language can be English and the second language can be Chinese. In this step, the training text of the first language is first obtained, and the training text can be any number. The training text can be a sentence text input by the user during the language teaching process, that is, the training text can contain various errors input by the user, including semantically unclear errors.

[0049] After obtaining the training text in the first language, the perplexity of each sentence of the training text can be calculated in units of sentences. As a feasible implementation, this embodiment can define the perplexity as the exponential form of the cross entropy, and the formula for calculating the perplexity is as follows:

[0050]

[0051] In the above formula, H(W) represents the entropy of a sentence, W represents a sentence, and w 1 w 2 ... N represents the N words in this sentence, where N represents the number of words, and P(w 1 w 2 ... N ) represents the joint probability of this sentence, and P(wi|w1...wi-1) represents the conditional probability of wi given w1...wi-1. The degree of deviation of the training text from the normal sentence can be determined based on the perplexity.

[0052] This step can also determine the grammatical error information of each sentence of the training text. Specifically, the grammatical error information of the training text can be determined using a grammatical error detection model. The grammatical error information mentioned in this embodiment may include the average probability of a grammatical error in each word in the training text, and the number of word replacement errors with different roots. Word replacement errors with different roots are the same grammatical errors that occur after replacing the root. For example, the root of writing and writes is write, reading and writing belong to word replacements with different roots, and "I like to reading" and "I like to writing" belong to word replacement errors with different roots.

[0053] S102: translating the training text into a pivot language text in a second language, and translating the pivot language text into a target text in a first language;

[0054] In this step, a translator can be used to translate the training text in the first language into the pivot language text in the second language, and then translate the pivot language text into the target text in the first language. The above process is equivalent to back-translating the training text to obtain the target text. The pivot language text is the conversion bridge between the training text and the target text.

[0055] S103: Calculate the text similarity between the training text and the target text, and determine the perplexity of the target text;

[0056] Among them, based on the training text and the target text, this step calculates the text similarity between the training text and the target text. As a feasible implementation method, this embodiment can calculate the text similarity between the training text and the target text based on the Word Mover's Distance metric. This embodiment can also use the method for calculating the perplexity of the training text in the above text to calculate the perplexity of the target text.

[0057] S104: performing word alignment operations on the training text and the target text with the pivot language text respectively, to obtain alignment evaluation information of the training text and the target text;

[0058] In this step, the training text can be aligned with the pivot language text, and the target text can be aligned with the pivot language text to obtain alignment evaluation information of the training text and the target text. In this embodiment, the training text sentences and the target text can be word-aligned with the pivot language text respectively by the fast_align (a fast word alignment toolkit) method.

[0059] Specifically, the process of obtaining the alignment evaluation information of the training text and the target text may include: performing a word alignment operation on the training text and the pivot language text to obtain a first alignment result; performing a word alignment operation on the target text and the pivot language text to obtain a second alignment result; and determining the alignment evaluation information of the training text and the target text according to the first alignment result and the second alignment result. The above alignment evaluation information includes: the ratio of the number of aligned content words in the training text to the number of all content words in the sentence, the ratio of the number of aligned content words in the target text to the number of all content words in the sentence, the alignment number ratio, the probability of the first word alignment probability and the second word alignment ratio. The process of determining the alignment number ratio in the above text includes: connecting the words in the training text and the target text that correspond to the pivot language text at the same time, taking the ratio of the number of crossings of the connection and the total alignment as the alignment number ratio, the first word alignment probability is the probability of the word alignment of the training text and the pivot language text, and the second word alignment ratio is the probability of the word alignment of the training text and the pivot language text. Content words may include verbs, nouns, adjectives and adverbs.

[0060] See also Figure 2 , Figure 2 A schematic diagram of the principle of determining the alignment ratio provided in an embodiment of the present application is provided, wherein the training text is "That apple was mother buy for me", the pivot language text is "That apple was bought for me by my mother", and the target text obtained by back translation is "That apple was bought for me by my mother". The words in the training text and the target text that correspond to the pivot language text at the same time are "That" and "That", "apple" and "apple", "was" and "was", "mother" and "mother", "buy" and "bought", "for" and "for", "me" and "me". The connection results are as follows: Figure 2 As shown, there are three line crossings, and the ratio of the number of line crossings to the total alignment (ie, the alignment number ratio) is 3 / 7.

[0061] S105: training an initial model according to the perplexity of the training text, the grammatical error information of the training text, the text similarity, the perplexity of the target text, the perplexity ratio of the target text to the training text, and the alignment evaluation information to obtain a semantically ambiguous detection model;

[0062] This step is based on executing the related operations of S101 to S104, and uses the perplexity of the training text, the grammatical error information of the training text, the text similarity, the perplexity of the target text, the perplexity ratio of the target text to the training text, and the alignment evaluation information as feature information for training the initial model, thereby obtaining a semantically ambiguous detection model capable of detecting semantically ambiguous texts.

[0063] S106: Perform text error detection operation on the sentence text to be detected through the semantic ambiguity detection model.

[0064] Among them, on the basis of obtaining the semantic ambiguity detection model, the text of the sentence to be detected can be input into the semantic ambiguity detection model, and it is judged whether there is semantic ambiguity text error in the sentence to be detected according to the output result of the semantic ambiguity detection model.

[0065] After obtaining the training text in the first language, this embodiment determines the perplexity and grammatical information of the training text. After translating the training text into the second language, it is translated into the target text in the first language, and then the perplexity of the target text and the text similarity between the target text and the training text are determined. This embodiment also performs word alignment operations on the training text and the target text with the pivot language text to obtain alignment evaluation information, and then uses the above-mentioned feature information about the training text and the target text to train the initial model to obtain a semantic ambiguity detection model, and uses the semantic ambiguity detection model to detect semantic ambiguity errors in the sentence text to be detected. It can be seen that this embodiment can detect semantic ambiguity errors in the text and improve the accuracy of text detection.

[0066] As for Figure 1 Further introduction to the corresponding embodiment, the process of performing text error detection operation through the semantic ambiguity detection model in S106 may include: determining the sentence text to be detected; inputting the sentence text to be detected into the semantic ambiguity detection model, and judging whether there is a text error in the sentence text to be detected based on the detection result output by the semantic ambiguity detection model.

[0067] In a specific implementation scenario, this embodiment can realize spoken language detection. If voice information is received, the voice information is converted into a sentence text to be detected in the first language, so that a semantically ambiguous detection model can be used to perform text error detection operations on the sentence text to be detected.

[0068] Furthermore, after performing a text error detection operation on the sentence text to be detected through the semantic ambiguity detection model, erroneous text content can also be marked in the sentence text to be detected, and a corrected text in the first language can be generated based on the erroneous text content.

[0069] The following is an example of an actual application to illustrate the process described in the above example. This example provides a method for identifying semantically unclear sentences input by second language learner users. The method is used to determine whether sentences in English writing have unclear semantics due to missing sentence structure, Chinese expression habits, misspellings or misremembering of words or phrases. This example constructs effective text features through a variety of tools and strategies, and trains a semantic ambiguity detection model based on these features, so that it can identify whether a sentence is semantically unclear and provide timely feedback to the user to meet the user's needs for correct expression. This example may include the following steps:

[0070] Step 1: Extract text features;

[0071] Specifically, this embodiment can extract a variety of different text features based on the training text, for example, including:

[0072] (1) The perplexity of each training text;

[0073] (2) the number of grammatical errors in each training text;

[0074] The number of grammatical errors in each training text may include the average probability of occurrence of grammatical errors for each word and the number of substitution errors for words with different roots.

[0075] (3) text similarity, perplexity of the target text, and perplexity ratio of the target text to the training text;

[0076] In this embodiment, the training text can be used as an input sentence, translated into English and then back-translated to obtain the target text. The text similarity between the training text and the target text is calculated based on the Word Mover's Distance metric. The perplexity of the target text is calculated, and the ratio of the perplexity of the target text to the perplexity of the training text is determined.

[0077] (4) Alignment evaluation information of training text and target text;

[0078] Among them, the training text and the target text are word-aligned with the pivot language (Chinese) sentence respectively by the fast_align method: the ratio of the content words aligned in the training text to the number of all content words in the training text is obtained, the ratio of the content words aligned in the target text to the number of all content words in the target text is calculated, the words of the training text and the target text that can correspond to the same Chinese word at the same time are connected, the ratio of the number of crossovers of the connection and the total number of alignments is calculated; the ratio of the word alignment scores of the training text and the target text is calculated. The alignment evaluation information may include: the ratio of the content words aligned in the training text to the number of all content words in the sentence, the ratio of the content words aligned in the target text to the number of all content words in the sentence, the ratio of the alignment numbers, the probability of the first word alignment probability and the probability of the second word alignment ratio (i.e., the ratio of the word alignment scores of the training text and the target text is calculated).

[0079] Step 2: Use a nonlinear classifier to fit the text features of the training samples and save the model. The sample label is whether the semantics is unclear.

[0080] The specific fitting process may include: performing maximum and minimum normalization on the text features in step 1 to convert the text features to the range of [0,1], dividing the text evenly into N parts, taking N-1 parts as training sets each time, and using the remaining 1 part as a validation set to perform N fittings using a binary classifier to obtain 5 models.

[0081] Step 3: Use the trained nonlinear classifier to predict new samples;

[0082] The specific prediction process may include: extracting the text features in step 1 for the new sample, normalizing the text features extracted from the new sample using the maximum and minimum values ​​in the training set, using the trained N binary classification models to predict the normalized new sample to obtain N prediction values, and using the prediction results of most models among the N prediction values ​​as the prediction results of the new sample.

[0083] The above embodiment provides a general solution for predicting semantically ambiguous sentences, which can solve the problem of unclear semantic expressions of English learners in writing or oral assessments, and can train an efficient model based on multiple text features to achieve reasonable results in identifying semantically ambiguous problems.

[0084] See also Figure 3 , Figure 3 A schematic diagram of the structure of a text error detection device provided in an embodiment of the present application;

[0085] The device may include:

[0086] A training text processing module 100 is used to obtain a training text in a first language and determine the perplexity and grammatical error information of the training text;

[0087] A language translation module 200, for translating the training text into a pivot language text in a second language, and translating the pivot language text into a target text in the first language;

[0088] A target text processing module 300 is used to calculate the text similarity between the training text and the target text, and determine the perplexity of the target text;

[0089] A word alignment module 400 is used to perform word alignment operations on the training text and the target text with the pivot language text respectively, to obtain alignment evaluation information of the training text and the target text;

[0090] A model training module 500 is used to train an initial model to obtain a semantic ambiguity detection model according to the perplexity of the training text, the grammatical error information of the training text, the text similarity, the perplexity of the target text, the perplexity ratio of the target text to the training text, and the alignment evaluation information;

[0091] The detection module 600 is used to perform text error detection operations on the sentence text to be detected through the semantic ambiguity detection model.

[0092] After obtaining the training text in the first language, this embodiment determines the perplexity and grammatical information of the training text. After translating the training text into the second language, it is translated into the target text in the first language, and then the perplexity of the target text and the text similarity between the target text and the training text are determined. This embodiment also performs word alignment operations on the training text and the target text with the pivot language text to obtain alignment evaluation information, and then uses the above-mentioned feature information about the training text and the target text to train the initial model to obtain a semantic ambiguity detection model, and uses the semantic ambiguity detection model to detect semantic ambiguity errors in the sentence text to be detected. It can be seen that this embodiment can detect semantic ambiguity errors in the text and improve the accuracy of text detection.

[0093] Furthermore, the grammatical error information includes an average probability of a grammatical error in each word in the training text, and the number of substitution errors of words with different roots.

[0094] Furthermore, the word alignment module 400 is used to perform a word alignment operation on the training text and the pivot language text to obtain a first alignment result; to perform a word alignment operation on the target text and the pivot language text to obtain a second alignment result; and to determine alignment evaluation information of the training text and the target text based on the first alignment result and the second alignment result.

[0095] Furthermore, the alignment evaluation information includes:

[0096] The ratio of the number of aligned content words in the training text to the number of all content words in the sentence;

[0097] The ratio of the number of aligned content words in the target text to the number of all content words in the sentence;

[0098] Alignment ratio; wherein the process of determining the alignment ratio includes: connecting the words in the training text and the target text that correspond to the pivot language text at the same time, and taking the ratio of the number of crossings of the connection and the total alignment as the alignment ratio;

[0099] The probability of the first word alignment probability and the second word alignment ratio; wherein the first word alignment probability is the probability of the training text and the pivot language text word alignment, and the second word alignment ratio is the probability of the training text and the pivot language text word alignment.

[0100] Furthermore, the detection module 600 includes:

[0101] A text determination unit, used to determine the text of the sentence to be detected;

[0102] The detection unit is used to input the sentence text to be detected into the semantic ambiguity detection model, and determine whether there is a text error in the sentence text to be detected according to the detection result output by the semantic ambiguity detection model.

[0103] Furthermore, the text detection unit is configured to convert the voice information into a sentence text to be detected in the first language if voice information is received.

[0104] Furthermore, it also includes:

[0105] The error correction module is used to mark erroneous text content in the sentence text to be detected after performing a text error detection operation on the sentence text to be detected through the semantic ambiguity detection model, and generate a corrected text in the first language based on the erroneous text content.

[0106] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0107] The present application also provides a storage medium on which a computer program is stored, and when the computer program is executed, the steps provided in the above embodiment can be implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0108] The present application also provides an electronic device, which may include a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps provided in the above embodiment may be implemented. Of course, the electronic device may also include various network interfaces, power supplies and other components.

[0109] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

[0110] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A text error detection method, characterized in that: include: Obtaining a training text in a first language and determining perplexity and grammatical error information of the training text; Translating the training text into a pivot language text in a second language, and translating the pivot language text into a target text in the first language; Calculating the text similarity between the training text and the target text, and determining the perplexity of the target text; Performing word alignment operations on the training text and the target text with the pivot language text respectively to obtain alignment evaluation information of the training text and the target text; An initial model is trained according to the perplexity of the training text, the grammatical error information of the training text, the text similarity, the perplexity of the target text, the perplexity ratio of the target text to the training text, and the alignment evaluation information to obtain a semantically ambiguous detection model; The semantic ambiguity detection model is used to perform text error detection operations on the sentence text to be detected.

2. The text error detection method according to claim 1, characterized in that: The grammatical error information includes the average probability of a grammatical error in each word in the training text and the number of substitution errors of words with different roots.

3. The text error detection method according to claim 1, characterized in that: Performing word alignment operations on the training text and the target text with the pivot language text respectively to obtain alignment evaluation information of the training text and the target text, including: Performing a word alignment operation on the training text and the pivot language text to obtain a first alignment result; Performing a word alignment operation on the target text and the pivot language text to obtain a second alignment result; Alignment evaluation information of the training text and the target text is determined according to the first alignment result and the second alignment result.

4. The text error detection method according to claim 3, characterized in that: The alignment evaluation information includes: The ratio of the number of aligned content words in the training text to the number of all content words in the sentence; The ratio of the number of aligned content words in the target text to the number of all content words in the sentence; Alignment ratio; wherein the process of determining the alignment ratio includes: connecting the words in the training text and the target text that correspond to the pivot language text at the same time, and taking the ratio of the number of crossings of the connection and the total alignment as the alignment ratio; A first word alignment probability and a second word alignment probability; wherein the first word alignment probability is the probability of the training text being aligned with the pivot language text word, and the second word alignment probability is the probability of the target text being aligned with the pivot language text word.

5. The text error detection method according to claim 1, characterized in that: Performing a text error detection operation on the sentence text to be detected by the semantic ambiguity detection model includes: Determine the sentence text to be detected; The sentence text to be detected is input into the semantic ambiguity detection model, and it is determined whether there is a text error in the sentence text to be detected according to the detection result output by the semantic ambiguity detection model.

6. The text error detection method according to claim 5, characterized in that: Determining the sentence text to be detected includes: If voice information is received, the voice information is converted into a sentence text to be detected in the first language.

7. The text error detection method according to claim 1, characterized in that: After performing text error detection operation on the sentence text to be detected by the semantic ambiguity detection model, the method further includes: Marking text content with errors in the sentence text to be detected, and generating a corrected text in the first language based on the text content with errors.

8. A text error detection device, characterized in that: include: A training text processing module, used for obtaining a training text in a first language and determining perplexity and grammatical error information of the training text; A language translation module, configured to translate the training text into a pivot language text in a second language, and to translate the pivot language text into a target text in the first language; A target text processing module, used to calculate the text similarity between the training text and the target text, and determine the perplexity of the target text; A word alignment module, used to perform word alignment operations on the training text and the target text with the pivot language text respectively, to obtain alignment evaluation information of the training text and the target text; A model training module, used to train an initial model to obtain a semantic ambiguity detection model according to the perplexity of the training text, the grammatical error information of the training text, the text similarity, the perplexity of the target text, the perplexity ratio of the target text to the training text, and the alignment evaluation information; The detection module is used to perform text error detection operations on the sentence text to be detected through the semantic ambiguity detection model.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps of the text error detection method according to any one of claims 1 to 7 are implemented.

10. A storage medium, characterized in that: The storage medium stores computer executable instructions, and when the computer executable instructions are loaded and executed by the processor, the steps of the text error detection method as described in any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Machine processing and text error correction method and device, computing device and storage medium

    CN109753636A

  • Error prediction in spoken dialog systems

    US20060149544A1