Text correction method, system and storage medium

Abstract

The application discloses a kind of text error correction method, system and storage medium, and the application belongs to text error correction technical field.The present application comprises the following steps: obtaining confirmed error-free text data as training corpus, and adding segmentation mark by artificial marking, word pair segmentation is carried out on sample, and the word pair and its part of speech of each text are obtained.Then, the word pair segmentation model is trained using these data, so that it can output the segmented word pair and the corresponding part of speech.After inputting the text to be corrected into the model, the semantic consistency index is evaluated.Considering the word length and the part of speech, the complexity index of the word pair is calculated, and the error judgment coefficient is generated by combining the two items.By comparing with the set threshold, the word pair to be corrected is marked out.The word meaning similarity of the word pair to be corrected and the word pair in the candidate word pair library is calculated, and the word pair with the highest similarity is selected to replace to realize text error correction.This method effectively integrates word pair analysis and similarity calculation to improve the accuracy and fluency of the text.

Description

Technical Field

[0001] This invention relates to the field of text correction technology, specifically to a text correction method, system, and storage medium. Background Technology

[0002] In the field of modern Natural Language Processing (NLP), text correction is an important and complex task, widely used in various scenarios such as automatic translation, intelligent writing, and speech recognition. Traditional text correction methods mainly rely on rule and dictionary matching, which often fails to effectively handle complex language structures and semantic relationships. This leads to poor performance of these methods when faced with contextual changes, synonym usage, and sentence diversity. Therefore, improving the accuracy and intelligence of text correction has become a pressing technical challenge for researchers.

[0003] Traditional text correction methods mostly rely on dictionary and rule matching, identifying potential errors based on spell checking and linguistic rules. However, these methods typically focus only on a single error type, such as spelling errors or whitespace errors, lacking a deep understanding of the context. Therefore, rule-based error correction methods are prone to misjudgment and omission when dealing with natural language texts with complex contexts and diverse structures.

[0004] With the continuous development of deep learning and statistical models, text correction techniques based on statistical methods and machine learning have gradually become a research focus. These methods utilize large corpora for training and can automatically correct errors through probabilistic statistics, semantic analysis, and other means. In recent years, in particular, neural network-based models, such as recurrent neural networks (RNNs) and long short-term memory networks (LSTMs), have made significant progress in text correction. These models can capture more complex language patterns and enhance their understanding of long-distance dependencies and contextual information. However, existing deep learning methods still face the challenge of effectively balancing various factors in text correction (such as part-of-speech, semantics, and complexity) to improve the accuracy and efficiency of correction.

[0005] In the prior art, CN113515934A discloses a text correction method, apparatus, storage medium, and electronic device. This method involves: acquiring raw text; inputting the raw text into an error correction model to obtain an error correction label sequence output by the model; wherein the error correction model is trained using training samples, which include the text to be corrected and a text error correction label sequence serving as labels for the text to be corrected, and the error correction label sequence acting as a conversion condition for converting the raw text into the target text; and performing error correction processing on the raw text based on the error correction label sequence to obtain the target text. However, this method is based on labeled training data. Therefore, if the model only learns certain types of errors or optimizes only within certain domains, it may not be able to handle other types of errors or unseen errors effectively. For example, a model that handles spelling errors may perform poorly when handling grammatical or contextual errors. Furthermore, if the error correction model's understanding of context is insufficient, it may overlook certain context-dependent errors. For example, some words may have different meanings in different contexts, and the model may not be able to correctly determine this. This reduces the accuracy and effectiveness of the error correction results.

[0006] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] The purpose of this invention is to provide a text correction method, system, and storage medium to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A text correction method, the specific steps of which include:

[0010] We acquire several language text data that have been confirmed to be error-free, and use the acquired text data as sample training corpus. We use manual labeling to add segmentation tags to perform word pair segmentation on the sample training corpus, and obtain several word pairs for each text data in the sample training corpus. We then add corresponding parts of speech to the word pairs.

[0011] A word pair segmentation model is established. The language text data in the sample training corpus is used as the input data, and the added segmentation markers and the part-of-speech tags of the word pairs are used as labels to train the word pair segmentation model. The input of the word pair segmentation model is the language text data, and the output is several word pairs after the language text data is segmented and their corresponding parts of speech.

[0012] Input the text sentence to be corrected into the word pair segmentation model that has been trained. The word pair segmentation model outputs several word pairs after segmentation of the text sentence to be corrected. Calculate the joint probability of each segmented word pair in the sample training corpus. Calculate the semantic consistency index of the word pair through the joint probability.

[0013] Obtain the word length of each word pair. Based on the word length and part of speech of each word pair, calculate the complexity index of the corresponding word pair. Based on the obtained word pair complexity index, combine it with the semantic consistency index of the word pair to generate the error judgment coefficient of the word pair. Compare the error judgment coefficient of the word pair with the pre-set error judgment threshold. Mark the corresponding word pairs whose error judgment coefficient exceeds the error judgment threshold as word pairs to be corrected.

[0014] Based on the labeled word pairs to be corrected, the semantic similarity between the word pairs to be corrected and each word pair in the candidate word pair library is calculated. The word pair with the highest semantic similarity in the candidate word pair library is selected and replaced with the word pair to be corrected to complete the text correction.

[0015] Furthermore, using the acquired text data as sample training corpus, segmentation markers are added manually to segment the sample training corpus into word pairs. The specific segmentation method is as follows: taking a sentence as the standard unit, the complete sentences contained in the entire text data are determined, and the complete sentences are segmented according to the added segmentation markers. The logic for adding segmentation markers is as follows: segmentation markers are added to word pairs of different parts of speech in a sentence, and the sentence is segmented into several word pairs by adding segmentation markers. The parts of speech include nouns of people, verbs, adverbs, conjunctions, prepositions, adjectives, and nouns of objects.

[0016] Furthermore, a word pair segmentation model was established based on the LSTM (Long Short-Term Memory) network model. Activation functions and optimization algorithms were selected, with the Tanh function chosen as the activation function and Adam as the optimization algorithm for the LSTM model. The formula for the Tanh function is:

[0017]

[0018] In the formula, Represents the Tanh function, with the independent variable... This represents the weighted sum of the neuron's inputs, that is, the result of the weighted sum of the inputs received by the neuron from the previous layer;

[0019] Simultaneously, the hyperparameters of the LSTM model are set, including: number of network layers, number of iterations, learning rate, batch size, number of training iterations, number of batches, and number of hidden layer neurons.

[0020] The network is set to a four-layer structure, the number of iterations is set to 300, the learning rate is set to 0.001, the batch size is set to 32, the number of training iterations is set to 300, the batch size is set to 256, and the number of hidden layer neurons is set to 32.

[0021] Furthermore, based on the segmented word pairs, the joint probability of each word pair appearing in the sample training corpus is calculated. The formula used to calculate the joint probability of each word pair appearing in the sample training corpus is as follows:

[0022]

[0023] In the formula, This represents the j-th word pair after segmentation, where j is the index of the segmented word pair. ,in The number of word pairs generated after segmenting the text sentence to be corrected. Within the text sentence indicating the area to be corrected, word pairs are excluded. The i-th word pair outside the text sentence to be corrected, where i represents the number of word pairs excluded from the text sentence. Indexes of other word pairs, and ,in To remove word pairs from the text sentence to be corrected The number of other word pairs besides This indicates word pairs within the text sentence to be corrected. Word pair The joint probability of simultaneous occurrence This indicates that in the sample training corpus, word pairs and word pair The number of times they appear together in a sentence This indicates that in the sample training corpus, word pairs Total number of occurrences.

[0024] Furthermore, the semantic consistency index of word pairs is calculated using joint probability, and the specific formula used to calculate the semantic consistency index of word pairs is as follows:

[0025]

[0026] In the formula, This represents the semantic consistency index of the j-th word pair within the text sentence to be corrected. The word length of the text sentence to be corrected. This indicates word pairs within the text sentence to be corrected. Word pair The joint probability of simultaneous occurrence.

[0027] Furthermore, the word length and part-of-speech category of each word pair are obtained. Based on the word length and part-of-speech category of each word pair, the complexity index of the corresponding word pair is calculated. The formula used to calculate the complexity index of the word pair is as follows:

[0028]

[0029] In the formula, For word pairs The complexity index, For the corresponding words in the sentence The number of word pairs with the same part of speech For word pairs Word length, For word pairs The number of repeated characters within the string;

[0030] Based on the obtained word pair complexity index, combined with the word pair semantic consistency index, the error judgment coefficient of the word pair is generated. The formula used to calculate the error judgment coefficient of the word pair is as follows:

[0031]

[0032] In the formula, Indicator pairs The error judgment coefficient, and These are the weighting coefficients for the semantic consistency index and the complexity index of word pairs, respectively. and and All are greater than 0;

[0033] The error judgment coefficient of each word pair is compared with a pre-set error judgment threshold. Word pairs with error judgment coefficients exceeding the threshold are marked as word pairs to be corrected. The logic for determining word pairs to be corrected is as follows:

[0034] when If the word pair does not contain errors, no marking is required.

[0035] when When an error is found in a word pair, it is marked as a word pair to be corrected.

[0036] In the formula, This is the threshold for incorrect judgment.

[0037] Furthermore, based on the labeled word pairs to be corrected, the semantic similarity between the word pairs to be corrected and each word pair in the candidate word pair library is calculated. The logic for calculating the semantic similarity between the word pairs to be corrected and each word in the candidate word pair library is as follows: determine the part of speech of the word pairs to be corrected, and calculate the semantic similarity in the candidate word pair library with the same part of speech. The formula for calculating the semantic similarity is as follows:

[0038]

[0039] In the formula, This indicates that within the text sentence to be corrected, the x-th word pair to be corrected... Word pairs in the candidate word pair library semantic similarity For structural similarity, Indicator pairs The semantic consistency index, specifically, is the index that measures word pairs. Replacement word pairs Afterwards, word pair The semantic consistency index in the text sentences to be corrected. For word form similarity, , and These are structural similarity and word pair similarity, respectively. The weighting coefficients of semantic consistency index and word form similarity, where and , and All are greater than 0;

[0040] Among them, structural similarity The formula used for the calculation is:

[0041]

[0042] In the formula, and Each is a word pair and word pair The magnitude of word vectors, and Each is a word pair and word pair Word vectors;

[0043] Word form similarity The formula used for the calculation is:

[0044]

[0045] In the formula, Indicator pairs and word pair Character-level edit distance.

[0046] The present invention also provides a text correction system, which is used to perform the above-described text correction method, comprising:

[0047] The text segmentation module is used to acquire several language text data that have been determined to be error-free. The acquired text data is used as the sample training corpus. Segmentation tags are added using a manual labeling method to perform word pair segmentation on the sample training corpus, resulting in several word pairs for each text data in the sample training corpus, and corresponding part-of-speech tags are added to the word pairs.

[0048] The segmentation model training module is used to build a word pair segmentation model. It takes the language text data in the sample training corpus as input data and uses the added segmentation tags and the part-of-speech tags of the word pairs as labels to train the word pair segmentation model. The input of the word pair segmentation model is language text data, and the output is several word pairs after the language text data is segmented and their corresponding parts of speech.

[0049] The word pair semantic analysis module is used to input the text sentence to be corrected into the word pair segmentation model that has been trained. The word pair segmentation model outputs several word pairs after the text sentence to be corrected is segmented. The joint probability of each segmented word pair appears in the sample training corpus is calculated. The semantic consistency index of the word pair is calculated through the joint probability.

[0050] The text error detection module is used to obtain the word length of each word pair, calculate the complexity index of the corresponding word pair based on the word length and part of speech of each word pair, and generate the error judgment coefficient of the word pair by combining the obtained word pair complexity index with the semantic consistency index of the word pair. The error judgment coefficient of the word pair is compared with the pre-set error judgment threshold, and the corresponding word pairs with error judgment coefficients exceeding the error judgment threshold are marked as word pairs to be corrected.

[0051] The text correction and replacement module is used to calculate the semantic similarity between the labeled word pair to be corrected and each word pair in the candidate word pair library, select the word pair with the highest semantic similarity in the candidate word pair library, and replace the word pair to be corrected with the selected word pair to complete the text correction.

[0052] The present invention also provides a text correction storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the text correction method as described above.

[0053] Compared with the prior art, the beneficial effects of the present invention are:

[0054] First, this method manually labels error-free text data and trains it using a word pair segmentation model, forming a model capable of automatically identifying word pairs in text. This model not only accurately segments word pairs but also assigns appropriate part-of-speech tags to each pair, providing more semantic information support for subsequent error correction. This method overcomes the limitations of traditional text correction, which relies solely on single-word errors, and can more meticulously capture the semantic relationships between words and dependencies within sentences, making it particularly suitable for handling complex natural language contexts. By calculating the semantic consistency index and complexity index between word pairs, the error probability of each pair can be comprehensively evaluated. During the error correction process, the trained model first segments word pairs in the text. Then, based on the semantic consistency, part-of-speech, and complexity information of the word pairs, an error judgment coefficient is calculated and compared with a preset threshold, thus accurately marking the word pairs to be corrected. This process fully considers the multidimensional features of language, relying not only on simple spelling errors or grammatical rules but also comprehensively analyzing the semantic differences between word pairs, sentence dependencies, and lexical complexity, effectively reducing misjudgments and omissions. Furthermore, by calculating the semantic similarity between the word pair to be corrected and the word pairs in the candidate word pair library, the candidate word pairs with higher similarity are selected for replacement, thereby ensuring the semantic consistency and fluency of the text. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the overall method flow of the present invention;

[0056] Figure 2 This is a schematic diagram of the overall system structure of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0058] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0059] Example:

[0060] Please see Figure 1 The present invention provides a technical solution:

[0061] A text correction method, the specific steps of which include:

[0062] Step 1: Obtain several language text data that have been confirmed to be error-free. Use the obtained text data as sample training corpus. Add segmentation tags using manual labeling to segment the sample training corpus into word pairs, obtaining several word pairs for each text data in the sample training corpus, and add corresponding parts of speech to the word pairs.

[0063] The acquired text data was used as the sample training corpus. Segmentation tags were added manually to segment the sample training corpus into word pairs. The specific segmentation method was as follows: a sentence was used as the standard unit to determine the complete sentences contained in the entire text data. The complete sentences were then segmented according to the added segmentation tags. The logic for adding segmentation tags was as follows: segmentation tags were added to word pairs of different parts of speech in a sentence. The sentence was then segmented into several word pairs by adding segmentation tags. The parts of speech included nouns of people, verbs, adverbs, conjunctions, prepositions, adjectives, and nouns of objects.

[0064] The LabelImg tool is used to annotate word pairs with segmentation tags, and corresponding part-of-speech tags are added to the word pairs. The specific annotation steps are standard techniques and will not be elaborated here.

[0065] Step 2: Establish a word pair segmentation model. Use the language text data in the sample training corpus as input data, and use the added segmentation tags and the part-of-speech tags of the word pairs as labels to train the word pair segmentation model. The input of the word pair segmentation model is the language text data, and the output is several word pairs after the language text data is segmented and their corresponding parts of speech.

[0066] A word pair segmentation model was established based on the LSTM (Long Short-Term Memory) network model. Activation functions and optimization algorithms were selected, with the Tanh function chosen as the activation function and Adam as the optimization algorithm for the LSTM model. The formula for the Tanh function is:

[0067]

[0068] In the formula, Represents the Tanh function, with the independent variable... This represents the weighted sum of the neuron's inputs, that is, the result of the weighted sum of the inputs received by the neuron from the previous layer;

[0069] Simultaneously, the hyperparameters of the LSTM model are set, including: number of network layers, number of iterations, learning rate, batch size, number of training iterations, number of batches, and number of hidden layer neurons.

[0070] The network is set to a four-layer structure, the number of iterations is set to 300, the learning rate is set to 0.001, the batch size is set to 32, the number of training iterations is set to 300, the batch size is set to 256, and the number of hidden layer neurons is set to 32.

[0071] LSTM networks are particularly well-suited for processing and modeling long sequences of data. Compared to traditional RNNs (Recurrent Neural Networks), LSTMs effectively avoid the vanishing and exploding gradient problems, enabling the model to capture long-distance dependencies in text. This is especially important for text segmentation tasks, as relationships between words can span multiple word positions.

[0072] In natural language processing, the meaning and usage of words often depend on the overall meaning of the sentence. LSTM can update the state at each time step of the sequence, thereby better modeling the overall meaning information of the sentence and helping to identify the boundaries and semantic relationships of word pairs during segmentation.

[0073] Step 3: Input the text sentence to be corrected into the word pair segmentation model that has been trained. The word pair segmentation model outputs several word pairs after segmentation of the text sentence to be corrected. Calculate the joint probability of each segmented word pair in the sample training corpus. Calculate the semantic consistency index of the word pair through the joint probability.

[0074] Based on the segmented word pairs, the joint probability of each word pair appearing in the sample training corpus is calculated. The formula used to calculate the joint probability of each word pair appearing in the sample training corpus is as follows:

[0075]

[0076] In the formula, This represents the j-th word pair after segmentation, where j is the index of the segmented word pair. ,in The number of word pairs generated after segmenting the text sentence to be corrected. Within the text sentence indicating that errors need to be corrected, word pairs are excluded. The i-th word pair outside the text sentence to be corrected, where i represents the number of word pairs excluded from the text sentence. Indexes of other word pairs, and ,in To remove word pairs from the text sentence to be corrected The number of other word pairs besides This indicates word pairs within the text sentence to be corrected. Pair of words The joint probability of simultaneous occurrence This indicates that in the sample training corpus, word pairs and word pair The number of times they appear together in a sentence This indicates that in the sample training corpus, word pairs Total number of occurrences.

[0077] The semantic consistency index of word pairs is calculated using joint probability. The specific formula used to calculate the semantic consistency index of word pairs is as follows:

[0078]

[0079] In the formula, This represents the semantic consistency index of the j-th word pair within the text sentence to be corrected. The word length of the text sentence to be corrected. This indicates word pairs within the text sentence to be corrected. Word pair The joint probability of simultaneous occurrence.

[0080] Step 4: Obtain the word length of each word pair. Based on the word length and part of speech of each word pair, calculate the complexity index of the corresponding word pair. Based on the obtained word pair complexity index, combine it with the semantic consistency index of the word pair to generate the error judgment coefficient of the word pair. Compare the error judgment coefficient of the word pair with the pre-set error judgment threshold. Mark the corresponding word pairs whose error judgment coefficient exceeds the error judgment threshold as word pairs to be corrected.

[0081] Obtain the word length and part-of-speech category of each word pair. Based on the word length and part-of-speech category of each word pair, calculate the complexity index of the corresponding word pair. The formula used to calculate the complexity index of the word pair is as follows:

[0082]

[0083] In the formula, For word pairs The complexity index, For the corresponding words in the sentence The number of word pairs with the same part of speech For word pairs Word length, specifically word pair The length of the characters, For word pairs The number of repeated characters within the string;

[0084] in, Indicator pairs The number of repeated characters in a word pair. A higher value indicates a simpler internal structure and higher repetition rate. Repeated characters may signify lower distinguishability or higher repetition rate, which is often a hallmark feature in language complexity analysis. More repeated characters increase the probability of redundancy errors. Therefore... with complexity index The complexity is directly proportional to the number of repeated characters, which have a strong non-linear impact on complexity; therefore, an exponential function should be used. Amplify the contribution of high repetition to complexity.

[0085] Corresponding words in the sentence Number of word pairs with the same part of speech The more pairs of words with the same part of speech there are, the more the sentence deviates from its standard structure, and the greater the probability of a grammatical error. Complexity index Proportional, through Using this factor can reasonably reflect the impact of part-of-speech frequency on complexity. Using a logarithmic function can limit its growth to a reasonable range and avoid excessive values.

[0086] It refers to word pairs The total length of a word pair typically includes the sum of the character counts of both words. Longer word pairs mean that more characters may be needed to convey information and construct more complex language structures. The longer the words, the more verbose and complex the expression may be, thus increasing the complexity of the word pair. Using the square function... The purpose of weighting word pair lengths is to more clearly reflect the impact of word length on complexity, because the effect of increasing word length on language complexity is non-linear, and longer words usually mean higher expressive complexity.

[0087] Based on the obtained word pair complexity index, combined with the word pair semantic consistency index, the error judgment coefficient of the word pair is generated. The formula used to calculate the error judgment coefficient of the word pair is as follows:

[0088]

[0089] In the formula, Indicator pairs The error judgment coefficient, and These are the weighting coefficients for the semantic consistency index and the complexity index of word pairs, respectively. and and All are greater than 0;

[0090] Among word pairs Complexity index The larger the value, the greater the probability of redundant errors and syntax errors, hence the complexity index. Coefficient of error judgment The complexity of a word pair is generally proportional to its frequency of occurrence, morphological changes, and grammatical structure. A higher complexity index means the word pair is more difficult to process in practical applications and may lead to more misjudgments. For example, grammatically complex, spelled similar, or multi-meaning word pairs are more likely to cause errors. The square root form of the complexity index... This indicates that the impact of complexity on misjudgments is not linear. In other words, as complexity increases, the probability of a misjudgment increases, but the rate of increase gradually decreases. This aligns with the situation where, after a certain point, the impact of misjudgments no longer increases exponentially.

[0091] It measures the semantic closeness of two words. High semantic consistency between two word pairs indicates a stronger connection in the meaning of a sentence, thus suggesting a lower likelihood of misjudging them. Conversely, low semantic consistency suggests they might be misjudged or confused in some tasks. (Exponential form) Semantic consistency implies that its impact on misjudgments grows exponentially. That is, semantic consistency has a far greater influence on misjudgments than complexity, consistent with reality where semantic inconsistency between word pairs typically leads to a significantly higher probability of misjudgment. For example, "apple" and "orange" have significant semantic differences; even if they are similar in complexity, the probability of misjudgment increases significantly.

[0092] Among these, since the semantic consistency of word pairs has a significant impact on determining whether an error exists, therefore, the following settings are made: and and All are greater than 0.

[0093] The error judgment coefficient of each word pair is compared with a pre-set error judgment threshold. Word pairs with error judgment coefficients exceeding the threshold are marked as word pairs to be corrected. The logic for determining word pairs to be corrected is as follows:

[0094] when If the word pair does not contain any errors, no marking is required.

[0095] when When an error is found in a word pair, it is marked as a word pair to be corrected.

[0096] In the formula, This is the threshold for incorrect judgment.

[0097] Step 5: Based on the labeled word pairs to be corrected, calculate the semantic similarity between the word pairs to be corrected and each word pair in the candidate word pair library, select the word pair with the highest semantic similarity in the candidate word pair library, and replace the word pairs to be corrected with the selected word pairs to complete the text correction.

[0098] Based on the labeled word pairs to be corrected, the semantic similarity between the word pairs to be corrected and each word pair in the candidate word pair library is calculated. The logic for calculating the semantic similarity between the word pairs to be corrected and each word in the candidate word pair library is as follows: determine the part of speech of the word pairs to be corrected, and calculate the semantic similarity in the candidate word pair library with the same part of speech. The formula for calculating the semantic similarity is as follows:

[0099]

[0100] In the formula, This indicates that within the text sentence to be corrected, the x-th word pair to be corrected... Word pairs in the candidate word pair library semantic similarity For structural similarity, Indicator pairs The semantic consistency index, specifically, is the index that measures word pairs. Replacement word pairs Afterwards, word pair The semantic consistency index in the text sentences to be corrected. For word form similarity, , and These are structural similarity and word pair similarity, respectively. The weighting coefficients of semantic consistency index and word form similarity, where and , and All are greater than 0;

[0101] Through logarithm Smoothing processes prevent structural similarity from being overly biased towards extreme values ​​(such as two completely unrelated words), thus making it more adjustable.

[0102] pass The semantic consistency index is controlled within the range of [0,1] in the form of [formula], avoiding over-amplification of certain rarer co-occurrences. The semantic consistency index of the j-th word pair in the text sentence to be corrected is calculated using the method described above.

[0103] The morphological similarity between two words is measured by directly applying the reciprocal of the Levenshtein distance, i.e., word form similarity.

[0104] Among them, structural similarity has the greatest impact on semantic similarity, while word pair similarity... semantic consistency index Compared to word form similarity The impact on semantic similarity is slightly larger, therefore, it is set to... and , and All are greater than 0.

[0105] Among them, structural similarity The formula used for the calculation is:

[0106]

[0107] In the formula, and Each is a word pair and word pair The magnitude of word vectors, and Each is a word pair and word pair Word vectors;

[0108] One can directly find and obtain the word vectors for each word pair using word vector models (such as Word2Vec or GloVe).

[0109] Word form similarity The formula used for the calculation is:

[0110]

[0111] In the formula, Indicator pairs and word pair Character-level edit distance.

[0112] Character edit distance (VOD) is a metric used to measure the difference between two strings. It represents the minimum number of operations required to transform one string into another, typically involving the three basic operations: insertion, deletion, and replacement. Dynamic programming, also known as Levenshtein distance, can be used to efficiently calculate the VOD between two strings. Dynamic programming constructs a matrix to progressively calculate the minimum number of edit operations required to transform one string into another.

[0113] Please see Figure 2 The present invention also provides a text correction system, which is used to perform the above-described text correction method, including:

[0114] The text segmentation module is used to acquire several language text data that have been determined to be error-free. The acquired text data is used as the sample training corpus. Segmentation tags are added using a manual labeling method to perform word pair segmentation on the sample training corpus, resulting in several word pairs for each text data in the sample training corpus, and corresponding part-of-speech tags are added to the word pairs.

[0115] The segmentation model training module is used to build a word pair segmentation model. It takes the language text data in the sample training corpus as input data and uses the added segmentation tags and the part-of-speech tags of the word pairs as labels to train the word pair segmentation model. The input of the word pair segmentation model is language text data, and the output is several word pairs after the language text data is segmented and their corresponding parts of speech.

[0116] The word pair semantic analysis module is used to input the text sentence to be corrected into the word pair segmentation model that has been trained. The word pair segmentation model outputs several word pairs after the text sentence to be corrected is segmented. The joint probability of each segmented word pair appears in the sample training corpus is calculated. The semantic consistency index of the word pair is calculated through the joint probability.

[0117] The text error detection module is used to obtain the word length of each word pair, calculate the complexity index of the corresponding word pair based on the word length and part of speech of each word pair, and generate the error judgment coefficient of the word pair by combining the obtained word pair complexity index with the semantic consistency index of the word pair. The error judgment coefficient of the word pair is compared with the pre-set error judgment threshold, and the corresponding word pairs with error judgment coefficients exceeding the error judgment threshold are marked as word pairs to be corrected.

[0118] The text correction and replacement module is used to calculate the semantic similarity between the labeled word pair to be corrected and each word pair in the candidate word pair library, select the word pair with the highest semantic similarity in the candidate word pair library, and replace the word pair to be corrected with the selected word pair to complete the text correction.

[0119] The present invention also provides a text correction storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the text correction method as described above.

[0120] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0121] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0122] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0123] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A text error correction method, characterized in that, The specific steps include: We acquire several language text data that have been confirmed to be error-free, and use the acquired text data as sample training corpus. We use manual labeling to add segmentation tags to perform word pair segmentation on the sample training corpus, and obtain several word pairs for each text data in the sample training corpus. We then add corresponding parts of speech to the word pairs. A word pair segmentation model is established. The language text data in the sample training corpus is used as the input data, and the added segmentation markers and the part-of-speech tags of the word pairs are used as labels to train the word pair segmentation model. The input of the word pair segmentation model is the language text data, and the output is several word pairs after the language text data is segmented and their corresponding parts of speech. Input the text sentence to be corrected into the word pair segmentation model that has been trained. The word pair segmentation model outputs several word pairs after segmentation of the text sentence to be corrected. Calculate the joint probability of each segmented word pair in the sample training corpus. Calculate the semantic consistency index of the word pair through the joint probability. Obtain the word length of each word pair. Based on the word length and part of speech of each word pair, calculate the complexity index of the corresponding word pair. Based on the obtained word pair complexity index, combine it with the semantic consistency index of the word pair to generate the error judgment coefficient of the word pair. Compare the error judgment coefficient of the word pair with the pre-set error judgment threshold. Mark the corresponding word pairs whose error judgment coefficient exceeds the error judgment threshold as word pairs to be corrected. Based on the labeled word pairs to be corrected, the semantic similarity between the word pairs to be corrected and each word pair in the candidate word pair library is calculated. The word pair with the highest semantic similarity in the candidate word pair library is selected and replaced with the word pair to be corrected to complete the text correction.

2. The text correction method according to claim 1, characterized in that: The acquired text data was used as the sample training corpus. Segmentation tags were added manually to segment the sample training corpus into word pairs. The specific segmentation method was as follows: a sentence was used as the standard unit to determine the complete sentences contained in the entire text data. The complete sentences were then segmented according to the added segmentation tags. The logic for adding segmentation tags was as follows: segmentation tags were added to word pairs of different parts of speech in a sentence. The sentence was then segmented into several word pairs by adding segmentation tags. The parts of speech included nouns of people, verbs, adverbs, conjunctions, prepositions, adjectives, and nouns of objects.

3. The text correction method according to claim 1, characterized in that: A word pair segmentation model was established based on the LSTM (Long Short-Term Memory) network model. Activation functions and optimization algorithms were selected, with the Tanh function chosen as the activation function and Adam as the optimization algorithm for the LSTM model. The formula for the Tanh function is: ； In the formula, Represents the Tanh function, with the independent variable... This represents the weighted sum of the neuron's inputs, that is, the result of the weighted sum of the inputs received by the neuron from the previous layer; Simultaneously, the hyperparameters of the LSTM model are set, including: number of network layers, number of iterations, learning rate, batch size, number of training iterations, number of batches, and number of hidden layer neurons. The network is set to a four-layer structure, the number of iterations is set to 300, the learning rate is set to 0.001, the batch size is set to 32, the number of training iterations is set to 300, the batch size is set to 256, and the number of hidden layer neurons is set to 32.

4. The text correction method according to claim 1, characterized in that: Based on the segmented word pairs, the joint probability of each word pair appearing in the sample training corpus is calculated. The formula used to calculate the joint probability of each word pair appearing in the sample training corpus is as follows: ； In the formula, This represents the j-th word pair after segmentation, where j is the index of the segmented word pair. ,in The number of word pairs generated after segmenting the text sentence to be corrected. Within the text sentence indicating the area to be corrected, word pairs are excluded. The i-th word pair outside the text sentence to be corrected, where i represents the number of word pairs excluded from the text sentence. Indexes of other word pairs, and ,in To remove word pairs from the text sentence to be corrected The number of other word pairs besides This indicates word pairs within the text sentence to be corrected. Word pair The joint probability of simultaneous occurrence This indicates that in the sample training corpus, word pairs and word pair The number of times they appear together in a sentence This indicates that in the sample training corpus, word pairs Total number of occurrences.

5. A text correction method according to claim 4, characterized in that: The semantic consistency index of word pairs is calculated using joint probability. The specific formula used to calculate the semantic consistency index of word pairs is as follows: ； In the formula, This represents the semantic consistency index of the j-th word pair within the text sentence to be corrected. The word length of the text sentence to be corrected. This indicates word pairs within the text sentence to be corrected. Word pair The joint probability of simultaneous occurrence.

6. A text correction method according to claim 5, characterized in that: Obtain the word length and part-of-speech category of each word pair. Based on the word length and part-of-speech category of each word pair, calculate the complexity index of the corresponding word pair. The formula used to calculate the complexity index of the word pair is as follows: ； In the formula, For word pairs The complexity index, For the text sentences to be corrected, the word pairs The number of word pairs with the same part of speech For word pairs Word length, For word pairs The number of repeated characters within the string; Based on the obtained word pair complexity index, combined with the word pair semantic consistency index, the error judgment coefficient of the word pair is generated. The formula used to calculate the error judgment coefficient of the word pair is as follows: ； In the formula, Indicator pairs The error judgment coefficient, and These are the weighting coefficients for the semantic consistency index and the complexity index of word pairs, respectively. and and All are greater than 0; The error judgment coefficient of each word pair is compared with a pre-set error judgment threshold. Word pairs with error judgment coefficients exceeding the threshold are marked as word pairs to be corrected. The logic for determining word pairs to be corrected is as follows: when If the word pair does not contain errors, no marking is required. when When an error is found in a word pair, it is marked as a word pair to be corrected. In the formula, This is the threshold for incorrect judgment.

7. A text correction method according to claim 6, characterized in that: Based on the labeled word pairs to be corrected, the semantic similarity between the word pairs to be corrected and each word pair in the candidate word pair library is calculated. The logic for calculating the semantic similarity between the word pairs to be corrected and each word in the candidate word pair library is as follows: determine the part of speech of the word pairs to be corrected, and calculate the semantic similarity in the candidate word pair library with the same part of speech. The formula for calculating the semantic similarity is as follows: ； In the formula, This indicates that within the text sentence to be corrected, the x-th word pair to be corrected... Word pairs in the candidate word pair library semantic similarity For structural similarity, Indicator pairs The semantic consistency index, specifically, is the index that measures word pairs. Replacement word pairs Afterwards, word pair The semantic consistency index in the text sentences to be corrected. For word form similarity, , and These are structural similarity and word pair similarity, respectively. The weighting coefficients of semantic consistency index and word form similarity, where and , and All are greater than 0; Among them, structural similarity The formula used for the calculation is: ； In the formula, and Each is a word pair and word pair The magnitude of word vectors, and Each is a word pair and word pair Word vectors; Word form similarity The formula used for the calculation is: ； In the formula, Indicator pairs and word pair Character-level edit distance.

8. A text correction system, characterized in that: The text correction system is used to execute the text correction method according to any one of claims 1-7, including: The text segmentation module is used to acquire several language text data that have been determined to be error-free. The acquired text data is used as the sample training corpus. Segmentation tags are added using a manual labeling method to perform word pair segmentation on the sample training corpus, resulting in several word pairs for each text data in the sample training corpus, and corresponding part-of-speech tags are added to the word pairs. The segmentation model training module is used to build a word pair segmentation model. It takes the language text data in the sample training corpus as input data and uses the added segmentation tags and the part-of-speech tags of the word pairs as labels to train the word pair segmentation model. The input of the word pair segmentation model is language text data, and the output is several word pairs after the language text data is segmented and their corresponding parts of speech. The word pair semantic analysis module is used to input the text sentence to be corrected into the word pair segmentation model that has been trained. The word pair segmentation model outputs several word pairs after the text sentence to be corrected is segmented. The joint probability of each segmented word pair appears in the sample training corpus is calculated. The semantic consistency index of the word pair is calculated through the joint probability. The text error detection module is used to obtain the word length of each word pair, calculate the complexity index of the corresponding word pair based on the word length and part of speech of each word pair, and generate the error judgment coefficient of the word pair by combining the obtained word pair complexity index with the semantic consistency index of the word pair. The error judgment coefficient of the word pair is compared with the pre-set error judgment threshold, and the corresponding word pairs with error judgment coefficients exceeding the error judgment threshold are marked as word pairs to be corrected. The text correction and replacement module is used to calculate the semantic similarity between the labeled word pair to be corrected and each word pair in the candidate word pair library, select the word pair with the highest semantic similarity in the candidate word pair library, and replace the word pair to be corrected with the selected word pair to complete the text correction.

9. A text correction storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the text correction method as described in any one of claims 1-7.

Images