A Text Error Correction Method Based on External Knowledge Encoding Network

Abstract

This invention discloses a text correction method based on an external knowledge encoding network, comprising: acquiring text data to be corrected in a specific domain, inputting it into the BERT layer of a text correction network to obtain a sentence vector representation CLS of the text to be corrected; obtaining external knowledge vector representations through an external knowledge encoding network and storing them in a vector database D; calculating the similarity between the sentence vector representation CLS of the text to be corrected and all external knowledge vector representations in the vector database D, obtaining a similarity score of the sentence vector representation CLS relative to all external knowledge vector representations; selecting the external knowledge vector representation with the highest similarity score and replacing the sentence vector representation CLS of the text to be corrected with it to obtain a new sentence vector representation CLS'; inputting the new sentence vector representation CLS' into other modules of the text correction network, which outputs the corrected text data. This invention can solve the problem that text correction models cannot utilize external knowledge for factual error correction, thereby improving the performance of text correction models.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology, and in particular relates to a text error correction method based on an external knowledge encoding network. Background Technology

[0002] With the rapid development of big data and deep learning technologies, text correction systems can utilize massive corpus data and efficient error correction algorithm models to automatically identify and correct pronunciation, spelling, grammatical, and semantic errors in text. However, ordinary error correction models are unable to identify certain established factual knowledge in text and require specific external knowledge to assist the model in identification and correction.

[0003] For text correction algorithms, traditional sequence labeling models can only learn the probability distribution between texts based on the dataset used during training. They cannot incorporate the latest knowledge information in real time and cannot identify and correct texts that have no grammatical or phonetic errors but contain factual errors. On the other hand, autoregressive generative algorithms give the model greater freedom to correct text. As the number of model parameters increases, the knowledge contained in the model also increases. To a certain extent, the model can identify and correct simple factual errors in the text based on its own contextual understanding. However, its autoregressive correction method has greater diversity and is prone to changing the semantics of the original text. This correction method is not advisable for some specific texts. Furthermore, increasing the number of model parameters will reduce the system's efficiency and increase the system's maintenance and operating costs.

[0004] Both sequence labeling algorithms and autoregressive generative algorithms have certain problems in judging and correcting real-time knowledge or fixed knowledge. The former requires the creation of a dataset with specific knowledge to enhance the model's capabilities, and the model lacks universality and cannot be directly applied to various fields. The latter has a large number of model parameters, high operating costs, and unreliable generated text, and direct application may cause greater errors.

[0005] Therefore, developing an efficient method to encode factual knowledge and assist error correction models in text correction is crucial for improving the accuracy of text correction algorithms and enhancing office efficiency. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a text correction method based on an external knowledge encoding network, which includes:

[0007] S1: Obtain the text data to be corrected in a specific domain, input it into the Bert layer in the text correction network with fixed parameters, and obtain the sentence vector representation CLS of the text to be corrected.

[0008] S2: Input the knowledge set of a specific domain into the trained external knowledge encoding network to obtain the external knowledge vector representation, and store it in the vector database D;

[0009] S3: Calculate the similarity between the sentence vector representation CLS of the text to be corrected and all external knowledge vector representations in the vector database D, and obtain the similarity score of the sentence vector representation CLS relative to all external knowledge vector representations.

[0010] S4: Select the external knowledge vector representation with the highest similarity score and replace the sentence vector representation CLS of the text to be corrected with it to obtain a new sentence vector representation CLS';

[0011] S5: Input the new sentence vector representation CLS' into other modules in the text correction network, which outputs the corrected text data.

[0012] Furthermore, the training process of the external knowledge encoding network is as follows:

[0013] S101: Obtain text samples from a specific domain, manually annotate them to obtain positive labels in the text sample set, and extract the sentence vector representation CLS of the text sample set;

[0014] S102: Input a text sample from a specific domain and a specific knowledge set into the BM25 model to calculate the correlation and obtain the negative label of the text sample.

[0015] S103: Input text samples from a specific domain and their positive and negative labels into the BGE network to obtain a knowledge sequence representation composed of each knowledge;

[0016] S104: The sentence vector representation CLS is fused with the knowledge sequence representation, and the fused representation is input into the local interactive attention network to obtain the interaction representation vectors between each knowledge corresponding to the positive and negative labels and the sentence vector representation CLS.

[0017] S105: Calculate the contrastive loss based on the interaction representation vector, and optimize the external knowledge encoding network through backpropagation using the contrastive loss.

[0018] Furthermore, the training process of the external knowledge encoding network also includes: aligning the sentence vector representation CLS with the knowledge sequence representation in length, adding a position decay factor α to the corresponding position of each sentence vector representation CLS to obtain a position decay text sequence vector, fusing it with the knowledge sequence representation, and inputting the fused representation into the local interactive attention network.

[0019] The beneficial effects of this invention are as follows: This invention adopts a local interactive attention network mechanism, which learns the representation of the text to be corrected in the text correction system through contrastive learning, and narrows the distance between it and related knowledge in the high-dimensional space. At the same time, by using text decay position vectors and spread masks, the proportion of negative samples in contrastive learning is increased, thereby improving the model performance. This makes the representation of related knowledge closer to the representation of the text to be corrected in the high-dimensional space, and finally integrates it into the error correction model, solving the problem that it cannot use external knowledge for factual error correction. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the steps of an embodiment of the present invention;

[0021] Figure 2 This is a flowchart illustrating an embodiment of the present invention;

[0022] Figure 3 This is a flowchart illustrating the steps of the external knowledge encoding network in an embodiment of the present invention;

[0023] Figure 4 This is a flowchart illustrating the external knowledge encoding network in an embodiment of the present invention.

[0024] Figure 5 This is a schematic diagram of a specific spread_mask in an embodiment of the present invention. Detailed Implementation

[0025] The terms “first,” “second,” and “third,” etc., used in the specification and claims of this application are used to distinguish different objects, rather than to describe a specific order of objects.

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] This invention proposes a text correction method based on an external knowledge encoding network, referring to... Figure 1 , Figure 2As shown, the method includes:

[0028] S1: Obtain the text data to be corrected in a specific domain, input it into the BERT layer in the text correction network, and obtain the sentence vector representation CLS of the text to be corrected;

[0029] S2: Input the knowledge set of a specific domain into the trained external knowledge encoding network to obtain the external knowledge vector representation, and store it in the vector database D;

[0030] S3: Calculate the similarity between the sentence vector representation CLS of the text to be corrected and all external knowledge vector representations in the vector database D, and obtain the similarity score of the sentence vector representation CLS relative to all external knowledge vector representations.

[0031] S4: Select the external knowledge vector representation with the highest similarity score and replace the sentence vector representation CLS of the text to be corrected with it to obtain a new sentence vector representation CLS';

[0032] S5: Input the new sentence vector representation CLS' into other modules in the text correction network, which outputs the corrected text data.

[0033] The text correction network in this embodiment of the invention is a BERT-based neural network, including BERT layers and other modules. The parameters of the text correction network are fixed.

[0034] In S1, a specific domain-specific text dataset to be corrected is selected based on the requirements. This text dataset includes text data without factual errors (also known as correct knowledge) and text data with factual errors (also known as irrelevant knowledge).

[0035] In some preferred embodiments, reference is made to Figure 3 , 4 As shown, the training process of the external knowledge encoding network is as follows:

[0036] S101: Obtain a text sample set for a specific domain, manually annotate it to obtain positive labels in the text sample set, and extract the sentence vector representation CLS of the text sample set;

[0037] S102: Input a text sample from a specific domain and a specific knowledge set into the BM25 model to calculate the correlation and obtain the negative label of the text sample.

[0038] S103: Input text samples from a specific domain and their positive and negative labels into the BGE network to obtain a knowledge sequence representation composed of each knowledge;

[0039] S104: The sentence vector representation CLS is fused with the knowledge sequence representation, and the fused representation is input into the local interactive attention network to obtain the interaction representation vectors between each knowledge corresponding to the positive and negative labels and the sentence vector representation CLS.

[0040] S105: Calculate the contrastive loss based on the interaction representation vector, and optimize the external knowledge encoding network through backpropagation using the contrastive loss.

[0041] Reference Figure 4 As shown in S101, the text samples in this embodiment of the invention use a text dataset in the geographic field. The text dataset is selected in a 1:1 ratio to include text data without knowledge errors and text data with knowledge errors. After manual data annotation, the text data without knowledge errors is used as positive labels, that is, the correct knowledge is used as positive labels.

[0042] In step S101, text samples and a specific knowledge set are input into the BM25 model for relevance calculation. This yields similarity scores for each piece of knowledge within the text sample. The top 30 pieces of erroneous knowledge (i.e., non-positive label knowledge) with the highest similarity scores are selected as the negative labels for the current sample to be corrected. This specific knowledge set is a set of knowledge from the same technical field as the training sample set.

[0043] For example, refer to Figure 4 As shown, a specific knowledge set in the geographical domain is selected as the external knowledge set. Within this set, external knowledge triples (h, r, t), i.e., factual statement triples, are extracted from a specific text dataset in the geographical domain. h represents the head entity, t represents the tail entity, and r represents the relationship between the head and tail entities. The external knowledge triples (h, r, t) are then converted into knowledge text according to a fixed format: subject knowledge - relation - object knowledge. For example, for text data without factual errors: the capital of China is Beijing, its triple is (China, capital, Beijing); for text data with factual errors: the capital of Sichuan is Chongqing, its triple is (Sichuan, capital, Chengdu).

[0044] BM25 is a classic information retrieval algorithm, an improved version of the Okapi TF-IDF algorithm designed to address some of its shortcomings. BM25 is widely used as a ranking function in information retrieval to estimate the relevance between document D and user query Q. BM25 is an improvement based on a probabilistic retrieval framework, particularly effective when handling long documents and short queries. The core idea of ​​BM25 is based on term frequency (TF) and inverse document frequency (IDF), while also incorporating document length information to calculate the relevance between document D and query Q. The widely used search engine Elasticsearch incorporates the BM25 algorithm for full-text search.

[0045] The formula for the BM25 model is:

[0046]

[0047] Where BM25(D,Q) represents the similarity score, D represents the text composed of a certain knowledge triple, Q represents the text to be corrected, n is the number of words in Q after jieba segmentation, and q i Let IDF(q) represent the i-th word in Q. i ) represents q i The inverse document frequency, f(q) i D) represents q i The frequency of text in text D, len(D) represents the length of text D, avg_len represents the average length of all texts in D, N represents the total number of texts in D, and n(q) represents the frequency of text in D. i ) indicates that it contains q i The number of texts, k i b and are adjustable parameters.

[0048] In some preferred embodiments, the relevance of a specific domain text sample set and a specific knowledge set is calculated using the BM25 model to obtain a similarity score (BM25(D,Q)). The top 30 knowledge items with the highest similarity scores (excluding positive labels) are selected as negative labels for the current text sample set. P represents the knowledge text with the positive label, and N... i This represents the i-th knowledge text among 30 negative labels.

[0049] By constructing negative labels in this way, the most similar and easily confused textual knowledge can be found at the text level. Through the contrastive learning method in the external knowledge encoding network, the high-dimensional distance between positive and negative labels is widened in the feature space, thereby achieving the purpose of separating easily confused knowledge.

[0050] Reference Figure 4 As shown, in S103, the sentence-level feature vector CLS of the text sample, its positively labeled knowledge text P, and its 30 negatively labeled knowledge text N are combined. i Input the BGE network to obtain the embedding representation. Then, arrange the embedding representation according to {[CLS],[CLS],[P],[CLS],[N1],[CLS][N2],...,[CLS][N...} 30 The sequence is concatenated in the form of ]} to obtain a knowledge sequence representation composed of each knowledge item.

[0051] In some preferred embodiments, the sentence vector representation CLS is length-aligned with the knowledge sequence representation, and a position decay factor α is added to the corresponding position of each sentence vector representation CLS to obtain a position decay text sequence vector, which is then fused with the knowledge sequence representation. The fused representation is then input into a local interactive attention network.

[0052] Specifically, the sentence vector CLS of the text to be corrected is expanded in the sequence dimension according to the knowledge sequence representation length, so that it is consistent with the knowledge sequence representation length, thereby achieving length alignment of the feature sequences of the two.

[0053] The specific formula for calculating the position attenuation factor is as follows:

[0054]

[0055] Where α represents the position decay factor, pos represents the position of the sentence-level feature vector CLS, e represents the natural logarithm, i represents the i-th dimension in the sentence-level feature vector CLS, dim represents the dimension size of the sentence-level feature vector CLS, sin represents the sine trigonometric function, and cos represents the cosine trigonometric function.

[0056] The position decay factor is an improvement on the position encoding of the original transformer model. In this scheme, since cls is an extension, it does not have position encoding information on the sequence. The position encoding of the transformer only ensures that the representation of each position is inconsistent. Here, we hope that the role of cls at each position will gradually weaken. Therefore, the original position encoding is improved so that the representation of the cls vector weakens continuously as the sequence length increases.

[0057] The position-decayed text sentence vectors are fused with the knowledge sequence representations and fed into a local interactive attention network to obtain the interactive representation vectors of each knowledge corresponding to the positive and negative labels and the sentence vector representation CLS.

[0058] The output of the local interactive attention network is specifically represented as follows:

[0059]

[0060] In the formula, Vec represents the output of the local interactive attention network, Softmax(·) represents the Softmax activation function, seq represents the vector obtained by adding the position-decayed text sequence vector and the knowledge sequence representation at corresponding positions, and W q (·), W k (·) and W v (·) represent different linear network layers, each responsible for performing different linear transformations on the original sequence. TThis indicates the matrix transpose, dim indicates the dimension of the seq matrix in the last dimension, spread_mask indicates the local attention mask matrix, and local_mask indicates the global knowledge mask.

[0061] The local attention mask matrix spread_mask is specifically represented as follows:

[0062]

[0063] In the formula, i represents the i-th row of the spread_mask matrix, j represents the j-th column of the spread_mask matrix, and P start P represents the starting position of a piece of knowledge in a knowledge sequence representation. end This indicates the end position of a knowledge in the knowledge sequence representation, seq_len indicates the length of the knowledge sequence representation, and others indicates other items.

[0064] The local attention mask matrix spread_mask ensures that each piece of knowledge can only focus on the CLS and itself.

[0065] The local_mask is specifically represented as:

[0066]

[0067] In the formula, i represents the i-th row of local_mask, j represents the j-th column of local_mask, and P start "Others" indicates the starting position of a knowledge in the knowledge sequence representation, and "others" indicates other items.

[0068] local_mask focuses on suppressing errors caused by operations between different CLSs within the same sequence.

[0069] Appendix Figure 5 This diagram illustrates a specific spread_mask in an embodiment of the present invention. Figure 5 In this context, C represents the sentence-level feature vector CLS, P represents the positive label, N represents the negative label, i represents the i-th row of the spread_mask matrix, j represents the j-th column of the spread_mask matrix, and P... start Indicates the starting position of the positive or negative label, P end Indicates the end position of the positive or negative label, 1 represents 1e -12 .

[0070] In some preferred embodiments, the formula for calculating the contrast loss is as follows:

[0071]

[0072] Where loss represents the loss function of the external knowledge encoding network, q represents the sentence vector representation CLS of the current text sample set, and k + k represents the vector representation of the positive label in the current text data to be corrected. i This represents the vector representation of the i-th negative label in the current text data to be corrected, t represents the adjustable temperature coefficient parameter, and score i This represents the similarity score corresponding to the i-th negative label after BM25 correlation calculation.

[0073] This contrastive loss combines retrieval results at the text level (BM25) and semantic level (Embedding) to fine-tune model parameters. It can effectively differentiate the vector representations of easily confused knowledge in high-dimensional space. In application, it can greatly improve the accuracy of recalling relevant knowledge and enhance the accuracy of incorporating external knowledge in the field of text correction.

[0074] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include ROM, RAM, disk, or optical disk, etc.

[0075] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A text error correction method based on an external knowledge encoding network, characterized in that, include: Obtain text data to be corrected in a specific domain, input it into the BERT layer in the text correction network, and obtain the sentence vector representation (CLS) of the text to be corrected. A knowledge set of a specific domain is input into a trained external knowledge encoding network to obtain external knowledge vector representations, which are then stored in a vector database D. Calculate the similarity between the sentence vector representation CLS of the text to be corrected and all external knowledge vector representations in the vector database D, and obtain the similarity score of the sentence vector representation CLS relative to all external knowledge vector representations. The external knowledge vector representation with the highest similarity score is selected and used to replace the sentence vector representation CLS of the text to be corrected, resulting in a new sentence vector representation CLS'. The new sentence vector representation CLS' is input into other modules in the text correction network, which outputs the corrected text data. The training process of the external knowledge encoding network is as follows: Obtain a text sample set for a specific domain, manually annotate it to obtain positive labels in the text sample set, and extract the sentence vector representation (CLS) of the text sample set. By inputting a text sample from a specific domain and a specific knowledge set into the BM25 model to calculate the correlation, the negative label of the text sample is obtained. By inputting text samples from a specific domain and their positive and negative labels into the BGE network, a knowledge sequence representation composed of each piece of knowledge is obtained. The sentence vector representation CLS is length-aligned with the knowledge sequence representation, and a position decay factor is added to the corresponding position of each sentence vector representation CLS. The position decay text sequence vector is obtained and fused with the knowledge sequence representation. The fused representation is then input into the local interactive attention network to obtain the interactive representation vector of each knowledge corresponding to the positive and negative labels and the sentence vector representation CLS. The contrastive loss is calculated based on the interaction representation vector, and the external knowledge encoding network is optimized through backpropagation using this contrastive loss.

2. The text correction method based on an external knowledge encoding network according to claim 1, characterized in that, The specific formula for calculating the position attenuation factor is as follows: in, Indicates the position attenuation factor. This indicates the position of the sentence-level feature vector CLS. Represents the natural logarithm. This represents the i-th dimension in the sentence-level feature vector CLS. The dimension of the sentence-level feature vector CLS is represented by sin, and cos represents the sine trigonometric function.

3. The text correction method based on an external knowledge encoding network according to claim 1, characterized in that, The output of the local interactive attention network is specifically represented as follows: In the formula, This represents the output of the local interactive attention network. This represents the Softmax activation function. This represents the vector obtained by adding the position-decayed text sequence vector and the knowledge sequence representation according to their corresponding positions. This represents the first linear network layer. Both represent the second linear network layer. Both represent the third linear network layer. Indicates matrix transpose. This indicates the size of the seq matrix in the last dimension. Local attention mask matrix, Represents a global knowledge mask.

4. The text correction method based on an external knowledge encoding network according to claim 1, characterized in that, The specific formula for calculating the contrast loss is as follows: in, The loss function of the external knowledge encoding network is represented. The sentence vector representation CLS of the current text sample. This represents the vector representation of the positive label in the current text data to be corrected. This represents the vector representation of the i-th knowledge point corresponding to the negative label in the current text data to be corrected. This indicates an adjustable temperature coefficient parameter. This represents the similarity score corresponding to the i-th negative label after BM25 correlation calculation.

5. The text correction method based on an external knowledge encoding network according to any one of claims 1-4, characterized in that, The positive labels represent correct knowledge from the text sample set.

6. The text correction method based on an external knowledge encoding network according to any one of claims 1-4, characterized in that, The relevance of text samples in a specific domain and a specific knowledge set is calculated to obtain a similarity score. The top 30 knowledge items with the highest similarity scores (excluding positive labels) are selected as negative labels for the current text sample set.

Images