Text information processing method and device, and storage medium

Abstract

The application provides a text information processing method and device and a storage medium, wherein the method comprises: obtaining a to-be-processed text, and determining each to-be-identified sentence containing an easily-confused word in the to-be-processed text; performing sentence probability calculation on the to-be-identified sentence by using a first pre-training model to obtain the sentence probability of the to-be-identified sentence; performing semantic probability calculation on the to-be-identified sentence by using a second pre-training model to obtain the semantic probability of the to-be-identified sentence; and performing calculation on the identification features of each to-be-identified sentence of the to-be-processed text by using an ensemble learning model to obtain the respective corresponding calculation results of each to-be-identified sentence; wherein the identification features of the to-be-identified sentence are multi-dimensional features composed of the sentence probability, the semantic probability, the sentence length and the index number of the easily-confused word in the to-be-identified sentence; and based on the calculation results, the sentence with an incorrectly used easily-confused word and the incorrectly used easily-confused word are determined from the to-be-identified sentence corresponding to the to-be-processed text.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular, to a text information processing method, an apparatus, and a storage medium. Background Art

[0002] In the publishing industry, compared with manual proofreading, using Chinese automatic proofreading tools for text proofreading greatly improves the efficiency of text proofreading.

[0003] Current Chinese automatic proofreading tools mainly adopt Chinese proofreading methods based on deep learning models and statistical language model algorithms to identify non-part-of-speech errors in text. Among them, non-part-of-speech errors refer to writing errors of a single character in a word, for example, "逆境" (adversity) is mistakenly written as "逆竟".

[0004] However, the Chinese proofreading method used by existing Chinese automatic proofreading tools cannot accurately identify incorrectly used confusing words and sentences. Among them, confusing words specifically refer to words with similar semantics that are easily misused in Chinese, such as "权力" (power) and "权利" (right), and the two easily confused words that are often misused correspond to each other and are each other's confusing word. Summary of the Invention

[0005] The present application provides a text information processing method, an apparatus, and a storage medium, so as to solve the problem that existing Chinese proofreading methods used by Chinese automatic proofreading tools cannot accurately identify and determine incorrectly used confusing words.

[0006] In a first aspect, the present application provides a text information processing method, comprising:

[0007] acquiring a text to be processed, and determining each to-be-identified statement containing a confusing word in the text to be processed;

[0008] performing statement probability calculation on the to-be-identified statement by using a first pre-trained model to obtain a statement probability of the to-be-identified statement; and performing semantic probability calculation on the to-be-identified statement by using a second pre-trained model to obtain a semantic probability of the to-be-identified statement; wherein the statement probability is a statistical similarity probability of word and sentence expression between the to-be-identified statement and a training sample; and the semantic probability is a statistical similarity probability between the context of the confusing word in the to-be-identified statement and the context of the confusing word in the training sample;

[0009] An ensemble learning model is used to calculate the discrimination features of each sentence to be identified in the text to be processed, and the calculation results corresponding to each sentence to be identified are obtained. The discrimination features of the sentences to be identified are multi-dimensional features composed of the sentence probability, semantic probability, sentence length and the index number of the easily confused words in the sentence to be identified in the easily confused word library.

[0010] Based on the calculation results, statements with incorrect use of easily confused words and easily confused words with incorrect use are identified from the statements to be identified corresponding to the text to be processed.

[0011] Optionally, the statement probability includes a first statement probability and a second statement probability, and the semantic probability includes a first semantic probability and a second semantic probability;

[0012] The step of using a first pre-trained model to calculate the sentence probability of the sentence to be identified, and using a second pre-trained model to calculate the semantic probability of the sentence to be identified, includes:

[0013] Replace the easily confused words in the statement to be identified with the corresponding confused words to obtain the confused word replacement statement corresponding to the statement to be identified;

[0014] The first pre-trained model is used to calculate the sentence probability of the sentence to be identified and its corresponding obfuscated word replacement sentence, respectively, to obtain the first sentence probability of the sentence to be identified and the second sentence probability of the obfuscated word replacement sentence; and,

[0015] The second pre-trained model is used to calculate the semantic probability of the statement to be identified and its corresponding obfuscated word replacement statement, respectively, to obtain the first semantic probability of the statement to be identified and the second semantic probability of the obfuscated word replacement statement.

[0016] Optionally, the discrimination feature of the statement to be discriminated is a multi-dimensional feature composed of the first statement probability, the second statement probability, the first semantic probability, the second semantic probability, the statement length, and the index number of the easily confused words in the statement to be discriminated in the easily confused word library;

[0017] An ensemble learning model is used to calculate the discrimination features of each sentence to be identified in the text to be processed, and the calculation results corresponding to each sentence to be identified are obtained, including:

[0018] The Lightweight Gradient Boosting Machine (lightGBM) model is used to calculate the discrimination features of each sentence to be identified in the text to be processed, and the calculation results corresponding to each sentence to be identified are obtained.

[0019] Optionally, the calculation result is a numerical calculation result;

[0020] Based on the calculation results, the steps of identifying statements with incorrect use of easily confused words and the incorrectly used easily confused words from the statements to be identified corresponding to the text to be processed include:

[0021] The calculation results corresponding to each statement to be identified are compared with a preset threshold. Statements to be identified with calculation results less than the preset threshold are identified as statements with incorrect use of easily confused words. Furthermore, easily confused words with calculation results less than the preset threshold are identified as easily confused words with incorrect use.

[0022] Optionally, the first pre-trained model is a statistical language model, and the second pre-trained model is a masked language model;

[0023] The step of using a first pre-trained model to calculate the sentence probability of the sentence to be identified, and using a second pre-trained model to calculate the semantic probability of the sentence to be identified, includes:

[0024] A statistical language model is used to calculate the statement probability of the statement to be identified, and a masked language model is used to calculate the semantic probability of the statement to be identified.

[0025] Optionally, after determining, based on the calculation results, the statements with incorrect use of easily confused words and the incorrectly used easily confused words from the statements to be identified corresponding to the text to be processed, the method further includes:

[0026] The text to be processed is marked with erroneous or easily confused words.

[0027] Optionally, the training samples used for training the first pre-trained model, the second pre-trained model, and the ensemble learning model are obtained in the following way:

[0028] Specify the target raw corpus from the pre-calibrated corpus, and identify the training sentences containing easily confused words from the target raw corpus;

[0029] From multiple training statements corresponding to the target original corpus, a target training statement is specified, and easily confused words in the target training statement are replaced with the confused words corresponding to the easily confused words to obtain the training replacement statement;

[0030] The training sample is composed of multiple training statements and multiple training replacement statements.

[0031] Secondly, this application provides a text information processing apparatus, comprising: a preprocessing module, a model processing module, and a postprocessing module; wherein,

[0032] The preprocessing module is used to acquire the text to be processed and determine the sentences to be identified in the text to be processed that contain easily confused words.

[0033] The model processing module is used to calculate the sentence probability of the sentence to be identified using a first pre-trained model to obtain the sentence probability of the sentence to be identified; and to calculate the semantic probability of the sentence to be identified using a second pre-trained model to obtain the semantic probability of the sentence to be identified; and to calculate the discrimination features of each sentence to be identified in the text to be processed using an ensemble learning model to obtain the calculation results corresponding to each sentence to be identified; wherein, the sentence probability is the statistical similarity probability of the word and sentence expression between the sentence to be identified and the training samples; the semantic probability is the statistical similarity probability between the context of the easily confused words in the sentence to be identified and the context of the easily confused words in the training samples; the discrimination features of the sentence to be identified are multi-dimensional features composed of the sentence probability, semantic probability, sentence length, and the index number of the easily confused words in the sentence to be identified in the easily confused word library;

[0034] The post-processing module is used to determine, based on the calculation results, the statements with incorrect use of easily confused words and the easily confused words with incorrect use from the statements to be identified corresponding to the text to be processed.

[0035] Thirdly, this application provides a text processing apparatus, comprising:

[0036] Processor and memory;

[0037] The memory stores executable instructions that the processor can execute;

[0038] The processor executes the executable instructions stored in the memory, causing the processor to perform the method described above.

[0039] Fourthly, this application provides a storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described above.

[0040] The text information processing method provided in this application calculates the sentence probability and semantic probability of each sentence to be identified in the text to be processed using a first pre-trained model and a second pre-trained model. Based on the multi-dimensional discrimination features composed of the sentence probability, semantic probability, sentence length, and index number of easily confused words, an ensemble learning model is used to calculate these multi-dimensional discrimination features, yielding the calculation results of the ensemble learning model. Based on these results, the method efficiently and accurately identifies sentences and easily confused words in the text to be processed that use incorrectly used easily confused words. This application solves the problem that existing Chinese automated proofreading tools cannot accurately identify and determine easily confused words that use incorrectly used words. Attached Figure Description

[0041] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0042] Figure 1 This is a diagram illustrating a text information processing scenario provided in an embodiment of this application.

[0043] Figure 2 A flowchart of a text information processing method provided in an embodiment of this application;

[0044] Figure 3 This is a structural diagram of a text information processing device provided in an embodiment of this application;

[0045] Figure 4 This is a structural diagram of a text processing device provided in an embodiment of this application.

[0046] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0048] Current automated Chinese proofreading tools primarily employ algorithms based on deep learning and statistical language models. These existing methods can quickly and accurately identify non-part-of-speech errors in text.

[0049] Because easily confused words are not only semantically similar but also often sound and look alike, they are frequently used incorrectly, leading to errors in the text's meaning. Furthermore, the misuse of easily confused words usually refers to their placement in sentences where their meaning does not match, even though the misused easily confused words themselves do not contain non-part-of-speech errors (such as spelling mistakes). Therefore, existing Chinese proofreading methods cannot identify these misused easily confused words. For example, text A contains the sentence "We should not only enjoy power, but also bear obligations." The easily confused word "power" is misused in this sentence, which should be identified during text proofreading and corrected to "rights" based on the identification result. However, since the sentence "We should not only enjoy power, but also bear obligations" does not contain any non-part-of-speech errors, even if the easily confused word "power" is misused, existing automated Chinese proofreading tools cannot identify or determine the misused easily confused word "power" and its related sentences.

[0050] To address this issue, this application proposes a text information processing method. This method processes sentences containing easily confused words to obtain their distinguishing features. Based on these features, an ensemble learning model is used to calculate and process the sentences, determining whether they contain incorrectly used easily confused words. Furthermore, the method identifies the incorrectly used easily confused words from among these incorrectly used sentences. The distinguishing features of the sentences are multi-dimensional features consisting of the sentence probability, semantic probability, sentence length, and the index number of the easily confused word in the easily confused word database. This proposed text information processing method solves the problem that existing Chinese automated proofreading tools cannot accurately identify and determine incorrectly used easily confused words.

[0051] The text information processing method provided in this application will be described below with reference to some embodiments.

[0052] Figure 1 This is a diagram illustrating a text information processing scenario provided in an embodiment of this application. For example... Figure 1As shown, the text information processing device 11 acquires the text to be processed. For example, the text information processing device 11 can acquire the text to be processed from other devices that store the text to be processed, or it can acquire the text to be processed directly entered on the text information processing device 11. This embodiment does not impose specific limitations here. After acquiring the text to be processed, the text information processing device 11 processes the acquired text as follows to obtain the text to be processed, as shown below. Figure 1 The text to be processed shown is marked with incorrect or easily confused words:

[0053] The text information processing device 11 identifies sentences containing easily confused words from the text to be processed. The device uses a first pre-trained model to calculate the sentence probability of the sentences to be identified, and a second pre-trained model to calculate the semantic probability of the sentences to be identified. The sentence probability is the statistical similarity probability between the sentences to be identified and the training samples; the semantic probability is the statistical similarity probability between the context of easily confused words in the sentences to be identified and the context of easily confused words in the training samples. For example, the device uses the first pre-trained model to perform a comprehensive statistical calculation of the probability of occurrence of the words constituting the sentences to be identified in the training samples and the probability of occurrence of the local arrangement of these words in the sentences to be identified, thus obtaining the sentence probability of the sentences to be identified. The device uses the second pre-trained model to perform a statistical calculation of the probability of occurrence of the combination of easily confused words and context in the sentences to be identified in the training samples, thus obtaining the semantic probability of the sentences to be identified. Then, the text information processing device 11 uses an ensemble learning model to calculate the discrimination features of each sentence to be identified in the text to be processed, obtaining the calculation results corresponding to each sentence. The discrimination features of each sentence are multi-dimensional features composed of the sentence probability, semantic probability, sentence length, and the index numbers of easily confused words in the easily confused word library. Finally, based on the calculation results corresponding to each sentence, the text information processing device 11 identifies the sentences with incorrect use of easily confused words and the incorrectly used easily confused words from the sentences to be identified in the text to be processed.

[0054] Optionally, after the text information processing device 11 determines that the text to be processed contains easily confused words with incorrect usage, it identifies and marks these words to obtain the text. Figure 1 The text to be processed is shown after incorrectly identified and marked with easily confused words.

[0055] The text information processing method provided in this application calculates the sentence probability and semantic probability of each sentence to be identified in the text to be processed using a first pre-trained model and a second pre-trained model. Based on the multi-dimensional discrimination features composed of the sentence length, index number of easily confused words, sentence probability, and semantic probability of each sentence to be identified, an ensemble learning model is used to calculate these multi-dimensional discrimination features, yielding the calculation results of the ensemble learning model. Based on these results, sentences and easily confused words with incorrect usage of easily confused words in the text to be processed are efficiently and accurately identified. The text information processing method provided in this application solves the problem that existing Chinese automated proofreading tools cannot accurately identify and determine easily confused words with incorrect usage.

[0056] The following is combined Figure 2 The text information processing method provided in this application is described in detail. Figure 2 A flowchart of a text information processing method provided in an embodiment of this application. Figure 2 The execution subject of the embodiment shown is Figure 1 The text information processing device 11 in the illustrated embodiment. For example... Figure 2 As shown, the method includes:

[0057] S201. Obtain the text to be processed and identify the sentences to be identified that contain easily confused words in the text to be processed.

[0058] Specifically, the text information processing device 11 acquires the text to be processed and identifies each sentence in the text to be processed that contains easily confused words.

[0059] For example, the text information processing device 11 determines the sentences to be identified in the text to be processed that contain easily confused words based on easily confused words in an easily confused word library. The easily confused word library can be a library stored on the text information processing device 11 or a library on other devices. Two corresponding easily confused words form an easily confused word pair. Each easily confused word pair in the easily confused word library has a unique index number across the entire library.

[0060] Optionally, after determining that each sentence in the text to be processed contains easily confused words, the text information processing device 11 obtains the sentence length L of each sentence to be identified and the index number W of each easily confused word in the easily confused word database. The sentence length L of the sentence to be identified can be the number of characters constituting that sentence.

[0061] S202. The first pre-trained model is used to calculate the sentence probability of the sentence to be identified, and the second pre-trained model is used to calculate the semantic probability of the sentence to be identified, and the semantic probability of the sentence to be identified is obtained.

[0062] Specifically, the text information processing device 11 uses a first pre-trained model to calculate the sentence probability of the sentence to be identified obtained in step S201, and uses a second pre-trained model to calculate the semantic probability of the sentence to be identified. The sentence probability is the statistical similarity probability of the word and sentence expressions between the sentence to be identified and the training samples; the semantic probability is the statistical similarity probability between the context of easily confused words in the sentence to be identified and the context of easily confused words in the training samples.

[0063] Furthermore, the statement probability includes a first statement probability and a second statement probability, and the semantic probability includes a first semantic probability and a second semantic probability. The text information processing device 11 replaces easily confused words in the statement to be identified with their corresponding confused words, obtaining the confused word-replaced statement. Then, the text information processing device 11 uses a first pre-trained model to calculate the statement probability of the statement to be identified and its corresponding confused word-replaced statement, obtaining the first statement probability of the statement to be identified and the second statement probability corresponding to the confused word-replaced statement; and uses a second pre-trained model to calculate the semantic probability of the statement to be identified and its corresponding confused word-replaced statement, obtaining the first semantic probability of the statement to be identified and the second semantic probability corresponding to the confused word-replaced statement.

[0064] Optionally, the first pre-trained model is a statistical language model. For example, the first pre-trained model is at least one statistical language model; the second pre-trained model is a masked language model, also for example, at least one masked language model. The statistical language model can be an N-gram language model, and the masked language model can be a BERT (Bidirectional Encoder Representation from Transformers) masked language model, or an LSTM (Longshort-term memory) language model, an LSTM model based on an attention mechanism, or other language models. The text information processing device 11 uses the statistical language model to calculate the sentence probability of the sentence to be identified, and uses the masked language model to calculate the semantic probability of the sentence to be identified, thus obtaining the semantic probability of the sentence to be identified.

[0065] For example, the text information processing device 11 uses an N-gram language model to calculate the probability of the sentence to be identified and its corresponding obfuscated word replacement sentence, respectively, to obtain the first probability of the sentence to be identified and the second probability of the sentence corresponding to the obfuscated word replacement sentence; and uses a BERT masked language model to calculate the semantic probability of the sentence to be identified and its corresponding obfuscated word replacement sentence, respectively, to obtain the first semantic probability of the sentence to be identified and the second semantic probability of the sentence corresponding to the obfuscated word replacement sentence. Here, N in the N-gram language model can be any value from 1, 2, 3, ..., n, where n is a natural number greater than zero.

[0066] The first sentence probability, second sentence probability, first semantic probability, and second semantic probability obtained in this step, together with the sentence length L of the sentence to be identified and the index number W of easily confused words in the sentence to be identified, constitute the multidimensional features (i.e., the distinguishing features) of the sentence to be identified. In other words, steps S201 and S202 convert each sentence to be identified into its corresponding multidimensional features so that the subsequent ensemble learning model can accurately and efficiently identify and determine whether the sentence to be identified contains easily confused words or incorrect statements.

[0067] S203. An ensemble learning model is used to calculate the discrimination features of each sentence to be identified in the text to be processed, and the calculation results corresponding to each sentence to be identified are obtained.

[0068] Specifically, the text information processing device 11 uses an ensemble learning model to calculate the discrimination features of each sentence to be identified in the text to be processed, and obtains the calculation results corresponding to each sentence. Among them, the discrimination features of the sentences to be identified are multi-dimensional features composed of the sentence probability, semantic probability, sentence length, and the index number of easily confused words in the sentence to be identified in the easily confused word library.

[0069] Furthermore, the distinguishing features of the sentences to be distinguished are multi-dimensional features composed of the first sentence probability, the second sentence probability, the first semantic probability, the second semantic probability, the sentence length, and the index numbers of easily confused words in the easily confused word library. The ensemble learning model is a classifier model. For example, the text information processing device 11 uses a Lightweight Gradient Boosting Machine (lightGBM) model to calculate the distinguishing features of each sentence to be distinguished in the text to be processed, and obtains the calculation results corresponding to each sentence. This classifier model is any classifier model with multi-feature ensemble processing capabilities, including but not limited to lightGBM.

[0070] S204: Determine, based on a calculation result, a sentence with a confusable word usage error and the incorrectly used confusable word from the candidate sentences corresponding to the text to be processed.

[0071] Specifically, the text information processing device 11 determines, based on the calculation result, a sentence with a confusable word usage error and the incorrectly used confusable word from the candidate sentences corresponding to the text to be processed.

[0072] For example, the calculation result is a numerical calculation result. The text information processing device 11 compares the calculation result corresponding to each candidate sentence with a preset threshold, determines that the candidate sentence corresponding to the calculation result smaller than the preset threshold is the sentence with a confusable word usage error, and determines that the confusable word corresponding to the calculation result smaller than the preset threshold is the incorrectly used confusable word.

[0073] Optionally, after determining, based on the calculation result, a sentence with a confusable word usage error and the incorrectly used confusable word from the candidate sentences corresponding to the text to be processed, the text information processing device 11 marks the incorrectly used confusable word in the text to be processed. For example, the text information processing device 11 marks the incorrectly used confusable word in the text to be processed as Figure 1 shown in the marking.

[0074] The text information processing method provided in this embodiment is described below with reference to specific examples. The examples are as follows:

[0075] Assume that the candidate sentence X0 contained in text A is "We should not only enjoy power, but also undertake obligations." After acquiring text A, the processing procedure of text A by the text information processing device 11 is specifically as follows:

[0076] (1) The text information processing device 11 determines, based on the confusable word "权力 (power)" in the confusable word bank, the candidate sentence X0 of text A, as well as the sentence length Lx of X0 and the index number Wx of the confusable word in X0.

[0077] (2) The text information processing device 11 replaces the confusable word "权力 (power)" in candidate sentence X0 with the confusable alternative word "权利 (right)" corresponding to "权力 (power)", to obtain a confusable word replacement sentence X1 corresponding to candidate sentence X0. Then, the text information processing device 11 uses an N-gram language model to perform sentence probability calculation on X0 and X1 respectively, to obtain a first sentence probability P corresponding to X0 N (X0) and a second sentence probability P corresponding to X1 N (X1); meanwhile, the text information processing device 11 uses a BERT masked language model to perform semantic probability calculation on X0 and X1 respectively, to obtain a first semantic probability P corresponding to X0 B (x0 | X0,mask ) and the second semantic probability P corresponding to X1 B (x1|X 1,mask ), wherein x0 and x1 are two groups of mutually distinct characters of statement X0 and statement X1 respectively, that is, x0 represents the character "力" in statement X0, and x1 represents the character "利" in statement X1. P B (x0|X 0,mask ) represents the probability obtained by calculation through the BERT masked language model after masking x0 in statement X0, similarly, P B (x1|X 1,mask ) represents the probability obtained by calculation through the BERT masked language model after masking x1 in statement X1.

[0078] (3) The text information processing device 11 takes the discrimination features of the to-be-discriminated statement X0, such as a six-dimensional feature [P N (X0), P N (X1), P B (x0|X 0,mask ), P B (x1|X 1,mask ), Lx, Wx], and inputs it into the lightGBM model, the six-dimensional features inputted are calculated and processed by the lightGBM model to obtain the probability value P of X0 X0 .

[0079] (4) The text information processing device 11 compares the calculation result of the to-be-discriminated statement X0, that is, the probability value P X0 with a preset threshold δ, if P X0 < δ, then it is determined that the to-be-discriminated statement X0 is a statement with incorrect use of easily confused words, and it is determined that the easily confused word "权力" is an incorrectly used easily confused word.

[0080] (5) The text information processing device 11 marks and labels the easily confused word "权力" in X0, so as to perform correction processing on the incorrectly used easily confused word.

[0081] Optionally, for the text information processing method provided by the present application, the first pre-trained model, the second pre-trained model and the ensemble learning model used therein all need to be trained with training samples before application, so as to ensure the accuracy of these models in application. In the text information processing method provided by the present application, the training samples for model training of the first pre-trained model, the second pre-trained model and the ensemble learning model can be obtained in the manner shown in the following steps I-III:

[0082] I. Specify the target raw corpus from the pre-proofed corpus, and identify the training sentences containing easily confused words from the target raw corpus. The pre-proofed corpus includes, but is not limited to, encyclopedic corpora, journal corpora, and newspaper corpora.

[0083] II. From the multiple training statements corresponding to the target original corpus, specify the target training statement, and replace the easily confused words in the target training statement with the corresponding confused words to obtain the training replacement statement.

[0084] III. Combine multiple training statements and multiple training replacement statements to form a training sample.

[0085] Therefore, the text information processing method provided in this application can use training data (i.e., training samples) for its model based on correct corpus through word pair replacement. Overall, the method provided in this application avoids a large amount of training data annotation work in terms of model training data preparation and model training, and can complete the cold start of the model under low resource conditions, making model training more efficient.

[0086] The technical effects of the text information processing method provided in this embodiment are explained below with specific examples. Specifically:

[0087] The basic information of the instance is shown in Table 1 below.

[0088] Table 1. Basic Information of the Example

[0089]

[0090] This example first uses the training data shown in Table 1 to train the N-gram model, LSTM model, LSTM (attention) model, BERT model, and the text information processing method provided in this application, respectively. Then, using the trained models, experiments are conducted to identify easily confused words that are incorrectly used in the texts to be processed, namely, text 1 and text 2, as shown in Table 1. The experimental results are shown in Table 2.

[0091] Table 2 Experimental Results

[0092]

[0093] The methods in Table 2 are explained below:

[0094] The single N-gram model method is a method that uses an N-gram language model alone to identify easily confused words in the text to be processed.

[0095] The single LSTM model method is a method that uses an LSTM language model alone to identify easily confused words in the text to be processed.

[0096] The single LSTM (attention) model method is a method that uses an LSTM language model based on the attention mechanism alone to identify easily confused words in the text to be processed.

[0097] Single BERT model approach: A method for identifying easily confused words in the text being processed using only the BERT language model.

[0098] The models provided in this application are as follows: the first pre-trained model is an N-gram language model; the second pre-trained model is a BERT masked language model; and the ensemble learning model is a lightGBM model.

[0099] In the N-gram language model shown in Table 2, N = 3.

[0100] As shown in Table 2, the accuracy of identifying easily confused words with incorrect usage in the samples is not as high as that of the text information processing method provided in this application when using N-gram language models, LSTM language models, LSTM models based on attention mechanisms, and BERT language models alone.

[0101] Therefore, the text information processing method provided in this application calculates the sentence probability and semantic probability of each sentence to be identified in the text to be processed using a first pre-trained model and a second pre-trained model. This yields the first sentence feature, second sentence feature, first semantic feature, and second sentence feature of each sentence to be identified. Combining the sentence length and the index number of easily confused words in each sentence forms its own six-dimensional features. Then, an ensemble learning model is used to calculate and process these six-dimensional features, thus efficiently and accurately identifying sentences and easily confused words in the text to be processed that use incorrect easily confused words. The incorrectly used easily confused words are then marked and labeled to facilitate correction. The text information processing method provided in this application solves the problem that existing Chinese automated proofreading tools cannot accurately identify and determine easily confused words using incorrectly used words. Furthermore, the method provided in this application avoids extensive training data labeling work in terms of model training data preparation and model training, enabling cold starts of the model under low-resource conditions and ensuring more efficient model training.

[0102] This application also provides a text information processing device in its embodiments. The following is in conjunction with... Figure 3 The text information processing apparatus provided in this application will be described. Figure 3 This is a structural diagram of a text information processing device provided in an embodiment of this application. Figure 3As shown, the text information processing device 3 includes: a preprocessing module 31, a model processing module 32, and a postprocessing module 33.

[0103] The preprocessing module 31 is used to acquire the text to be processed and identify the sentences to be identified that contain easily confused words in the text to be processed.

[0104] The model processing module 32 is used to calculate the sentence probability of the sentence to be identified using a first pre-trained model, and to calculate the semantic probability of the sentence to be identified using a second pre-trained model. The model processing module 32 is also used to calculate the discrimination features of each sentence to be identified in the text to be processed using an ensemble learning model, obtaining the calculation results corresponding to each sentence. Specifically, the sentence probability is the statistical similarity probability of word and sentence expression between the sentence to be identified and the training samples; the semantic probability is the statistical similarity probability between the context of easily confused words in the sentence to be identified and the context of easily confused words in the training samples; and the discrimination features of the sentence to be identified are multi-dimensional features composed of the sentence probability, semantic probability, sentence length, and the index number of easily confused words in the easily confused word library.

[0105] The post-processing module 33 is used to determine, based on the calculation results, the statements with incorrect use of easily confused words and the easily confused words with incorrect use from the statements to be identified corresponding to the text to be processed.

[0106] Optionally, the post-processing module 33 is also used to identify and label the text to be processed with incorrect or easily confused words.

[0107] Optionally, the model processing module 32 includes a first submodule 321, a second submodule 322, and a combination submodule 323. Wherein,

[0108] The first submodule 321 is used to calculate the sentence probability of the sentence to be identified using the first pre-trained model, and obtain the sentence probability of the sentence to be identified.

[0109] The second submodule 322 is used to calculate the semantic probability of the sentence to be identified using the second pre-trained model, and obtain the semantic probability of the sentence to be identified.

[0110] Combined with submodule 323, it is used to calculate the discrimination features of each sentence to be identified in the text to be processed using an ensemble learning model, and obtain the calculation results corresponding to each sentence to be identified.

[0111] The text information processing device provided in this application embodiment has the same implementation principle and technical effects as... Figure 2 The specific implementation principles and technical effects of the embodiments shown are similar, and will not be repeated here.

[0112] This application also provides a text processing device. Figure 4 This is a structural diagram of a text processing device provided in an embodiment of this application. Figure 4 As shown, the text processing device 4 includes a processor 41 and a memory 42. The memory 42 stores executable instructions of the processor 41, enabling the processor 41 to execute the technical solutions of the above-described method embodiments. The implementation principle and technical effects are similar, and will not be repeated here. It should be understood that the processor 41 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor. The memory 42 may include high-speed random access memory (RAM), and may also include non-volatile memory (NVM), such as at least one disk storage device, or a USB flash drive, external hard drive, read-only memory, disk, or optical disc, etc.

[0113] This application embodiment also provides a storage medium storing computer-executable instructions. When these computer-executable instructions are executed by a processor, they implement the aforementioned text information processing method. The storage medium can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible by a general-purpose or special-purpose computer.

[0114] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.

[0115] This application also provides a program product, such as a computer program, which, when executed by a processor, implements the text information processing method covered by this application.

[0116] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text information processing method, characterized in that, include: Obtain the text to be processed, and identify the sentences to be distinguished that contain easily confused words in the text to be processed; The first pre-trained model is used to calculate the sentence probability of the sentence to be identified, thereby obtaining the sentence probability of the sentence to be identified; The second pre-trained model is used to calculate the semantic probability of the statement to be identified, thereby obtaining the semantic probability of the statement to be identified; wherein, the statement probability is the statistical similarity probability of the word and sentence expression between the statement to be identified and the training samples; the semantic probability is the statistical similarity probability between the context of the easily confused word in the statement to be identified and the context of the easily confused word in the training samples. An ensemble learning model is used to calculate the discrimination features of each sentence to be identified in the text to be processed, and the calculation results corresponding to each sentence to be identified are obtained. The discrimination features of the sentences to be identified are multi-dimensional features composed of the sentence probability, semantic probability, sentence length and the index number of the easily confused words in the sentence to be identified in the easily confused word library. Based on the calculation results, statements with incorrect use of easily confused words and easily confused words with incorrect use are identified from the statements to be identified corresponding to the text to be processed.

2. The method according to claim 1, characterized in that, The statement probability includes a first statement probability and a second statement probability, and the semantic probability includes a first semantic probability and a second semantic probability; The first pre-trained model is used to calculate the sentence probability of the sentence to be identified, thereby obtaining the sentence probability of the sentence to be identified; The second pre-trained model is then used to calculate the semantic probability of the statement to be identified, including: Replace the easily confused words in the statement to be identified with the corresponding confused words to obtain the confused word replacement statement corresponding to the statement to be identified; The first pre-trained model is used to calculate the sentence probability of the sentence to be identified and its corresponding obfuscated word replacement sentence, respectively, to obtain the first sentence probability of the sentence to be identified and the second sentence probability of the obfuscated word replacement sentence; and, The second pre-trained model is used to calculate the semantic probability of the statement to be identified and its corresponding obfuscated word replacement statement, respectively, to obtain the first semantic probability of the statement to be identified and the second semantic probability of the obfuscated word replacement statement.

3. The method according to claim 2, characterized in that, The discrimination feature of the statement to be discriminated is a multi-dimensional feature consisting of the first statement probability, the second statement probability, the first semantic probability, the second semantic probability, the statement length, and the index number of the easily confused words in the statement to be discriminated in the easily confused word library. An ensemble learning model is used to calculate the discrimination features of each sentence to be identified in the text to be processed, and the calculation results corresponding to each sentence to be identified are obtained, including: The Lightweight Gradient Boosting Machine (lightGBM) model is used to calculate the discrimination features of each sentence to be identified in the text to be processed, and the calculation results corresponding to each sentence to be identified are obtained.

4. The method according to claim 1, characterized in that, The calculation results are in numerical form. Based on the calculation results, the steps of identifying statements with incorrect use of easily confused words and the incorrectly used easily confused words from the statements to be identified corresponding to the text to be processed include: The calculation results corresponding to each statement to be identified are compared with a preset threshold. Statements to be identified with calculation results less than the preset threshold are identified as statements with incorrect use of easily confused words. Furthermore, easily confused words with calculation results less than the preset threshold are identified as easily confused words with incorrect use.

5. The method according to any one of claims 1-4, characterized in that, The first pre-trained model is a statistical language model, and the second pre-trained model is a masked language model; The first pre-trained model is used to calculate the sentence probability of the sentence to be identified, thereby obtaining the sentence probability of the sentence to be identified; The second pre-trained model is then used to calculate the semantic probability of the statement to be identified, including: The sentence probability of the sentence to be identified is calculated by using a statistical language model. The semantic probability of the statement to be identified is calculated using a masked language model.

6. The method according to any one of claims 1-4, characterized in that, After determining, based on the calculation results, the statements with incorrect use of easily confused words and the incorrectly used easily confused words from the statements to be identified corresponding to the text to be processed, the method further includes: The text to be processed is marked with erroneous or easily confused words.

7. The method according to any one of claims 1-4, characterized in that, The training samples used for training the first pre-trained model, the second pre-trained model, and the ensemble learning model were obtained in the following way: Specify the target raw corpus from the pre-calibrated corpus, and identify the training sentences containing easily confused words from the target raw corpus; From multiple training statements corresponding to the target original corpus, a target training statement is specified, and easily confused words in the target training statement are replaced with the confused words corresponding to the easily confused words to obtain the training replacement statement; The training sample is composed of multiple training statements and multiple training replacement statements.

8. A text information processing device, characterized in that, include: The module consists of a preprocessing module, a model processing module, and a postprocessing module; among which, The preprocessing module is used to acquire the text to be processed and determine the sentences to be identified in the text to be processed that contain easily confused words. The model processing module is used to calculate the sentence probability of the sentence to be identified using a first pre-trained model to obtain the sentence probability of the sentence to be identified; and to calculate the semantic probability of the sentence to be identified using a second pre-trained model to obtain the semantic probability of the sentence to be identified; and to calculate the discrimination features of each sentence to be identified in the text to be processed using an ensemble learning model to obtain the calculation results corresponding to each sentence to be identified; wherein, the sentence probability is the statistical similarity probability of the word and sentence expression between the sentence to be identified and the training samples; the semantic probability is the statistical similarity probability between the context of the easily confused words in the sentence to be identified and the context of the easily confused words in the training samples; the discrimination features of the sentence to be identified are multi-dimensional features composed of the sentence probability, semantic probability, sentence length, and the index number of the easily confused words in the sentence to be identified in the easily confused word library; The post-processing module is used to determine, based on the calculation results, the statements with incorrect use of easily confused words and the easily confused words with incorrect use from the statements to be identified corresponding to the text to be processed.

9. A text processing device, characterized in that, include: Processor and memory; The memory stores executable instructions that the processor can execute; The processor executes the executable instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.

10. A storage medium, characterized in that, The storage medium stores computer execution instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

Images