Text error correction method based on external knowledge coding network

Abstract

The invention discloses a text error correction method based on an external knowledge coding network, and the method comprises the steps: obtaining to-be-corrected text data in a specific field, inputting the to-be-corrected text data into a Bert layer in a text error correction network, and obtaining a sentence vector representation CLS of a to-be-corrected text; obtaining external knowledge vector representation through an external knowledge coding network, and storing the external knowledge vector representation in a vector database D; calculating the similarity between the sentence vector representation CLS of the text to be corrected and all the external knowledge vector representations in the vector database D to obtain the similarity score of the sentence vector representation CLS relative to all the external knowledge vector representations; selecting the external knowledge vector representation with the highest similarity score, and replacing the sentence vector representation CLS of the to-be-corrected text with the external knowledge vector representation to obtain a new sentence vector representation CLS '; the new sentence vector representation CLS'is input into other modules in the text error correction network, and the text error correction network outputs the text data after error correction. According to the text error correction model and the text error correction method, the problem that the text error correction model cannot use external knowledge to carry out factual error correction can be solved, and the performance of the text error correction model is improved.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and in particular relates to a text error correction method based on an external knowledge encoding network. Background Art

[0002] With the rapid development of big data and deep learning technologies, text correction systems can use massive corpus data and efficient error correction algorithm models to automatically identify and correct phonetic and glyphic errors, grammatical errors, and semantic errors in texts. However, ordinary error correction models are unable to identify some established factual knowledge in the text, and specific external knowledge is required to assist the model in identification and correction.

[0003] For text error correction algorithms, for traditional sequence annotation algorithm models, they can only learn the probability distribution between texts based on the data set used during training. They cannot integrate the latest knowledge information in real time and cannot identify and correct some texts that have no grammatical, phonetic and glyphic errors but have factual expression errors. For autoregressive generative algorithm models, autoregression gives the model greater freedom to correct the text. At the same time, as the number of model parameters gradually increases, the knowledge contained in the model also increases synchronously. The model can, to a certain extent, identify and correct simple factual errors in the text based on its own contextual understanding ability, but its autoregressive correction method has greater diversity and is prone to changing the semantics of the original text. This correction method is not advisable for texts in some specific occasions. Secondly, the increase in the number of model parameters will reduce the work efficiency of the system and increase the maintenance and operation costs of the system.

[0004] Whether it is a sequence labeling algorithm or an autoregressive generative algorithm, there are certain problems in the identification and correction of real-time knowledge or fixed knowledge. The former is because it is necessary to specially create a data set of specified knowledge to enhance the model's capabilities. The model is not universal and cannot be directly applied to various fields. The latter is because the model has a large number of parameters, high operating costs, and unreliable generated text. Direct application may cause greater errors.

[0005] Therefore, developing an efficient method to encode factual knowledge and assist the error correction model in text correction is very important for improving the accuracy of text correction algorithms and improving office efficiency. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention proposes a text error correction method based on an external knowledge encoding network, the method comprising:

[0007] S1: Obtain the text data to be corrected in a specific field, input it into the Bert layer in the text correction network with fixed parameters, and obtain the sentence vector representation CLS of the text to be corrected;

[0008] S2: Input the knowledge set of a specific field into the trained external knowledge encoding network to obtain the external knowledge vector representation, which is stored in the vector database D;

[0009] S3: Calculate the similarity between the sentence vector representation CLS of the text to be corrected and all external knowledge vector representations in the vector database D, and obtain the similarity score of the sentence vector representation CLS relative to all external knowledge vector representations;

[0010] S4: Select the external knowledge vector representation with the highest similarity score and use it to replace the sentence vector representation CLS of the text to be corrected, to obtain a new sentence vector representation CLS';

[0011] S5: Input the new sentence vector representation CLS' into other modules in the text error correction network, and the text error correction network outputs the corrected text data.

[0012] Furthermore, the training process of the external knowledge encoding network is:

[0013] S101: Obtain text samples in a specific field, obtain positive labels in the text sample set after manual annotation, and extract sentence vector representation CLS of the text sample set;

[0014] S102: Input the text sample of the specific field and the specific knowledge set into the BM25 model to calculate the correlation, and obtain the negative label of the text sample;

[0015] S103: Input the text samples of the specific field and their positive and negative labels into the BGE network to obtain a knowledge sequence representation composed of each knowledge;

[0016] S104: Fusing the sentence vector representation CLS with the knowledge sequence representation, and inputting the fused representation into the local interactive attention network to obtain interactive representation vectors of each piece of knowledge corresponding to the positive and negative labels and the sentence vector representation CLS;

[0017] S105: Calculate the contrast loss according to the interaction representation vector, and optimize the external knowledge encoding network through back propagation of the contrast loss.

[0018] Furthermore, in the training process of the external knowledge encoding network, it also includes: aligning the length of the sentence vector representation CLS with the knowledge sequence representation, and adding a position attenuation factor α to the corresponding position of each sentence vector representation CLS, obtaining a position attenuated text sequence vector and fusing it with the knowledge sequence representation, and inputting the fused representation into the local interactive attention network.

[0019] Beneficial effects of the present invention: The present invention adopts a local interactive attention network mechanism, and learns the representation of the text to be corrected in the text correction system through contrastive learning, and shortens the distance between it and related knowledge in high-dimensional space. At the same time, through the text attenuation position vector and spread_mask, the negative sample ratio in contrastive learning is increased, and the model effect is improved, so that the final representation of the relevant knowledge is close to the representation of the text to be corrected in the high-dimensional space, and finally integrated into the correction model, solving the problem that it cannot use external knowledge for factual correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flowchart of steps of an embodiment of the present invention;

[0021] Figure 2 It is a schematic diagram of a flow chart of an embodiment of the present invention;

[0022] Figure 3 A flowchart of the steps of the external knowledge encoding network in an embodiment of the present invention;

[0023] Figure 4 Schematic diagram of the process of the external knowledge encoding network in an embodiment of the present invention;

[0024] Figure 5 FIG. 4 is a specific spread_mask schematic diagram in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The terms "first", "second", "third", etc. in the description and claims of this application are used to distinguish different objects rather than to describe a specific order of the objects.

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The embodiment of the present invention proposes a text error correction method based on an external knowledge encoding network, referring to Figure 1 , Figure 2As shown, the method includes:

[0028] S1: Obtain the text data to be corrected in a specific field, input it into the Bert layer in the text correction network, and obtain the sentence vector representation CLS of the text to be corrected;

[0029] S2: Input the knowledge set of a specific field into the trained external knowledge encoding network to obtain the external knowledge vector representation, which is stored in the vector database D;

[0030] S3: Calculate the similarity between the sentence vector representation CLS of the text to be corrected and all external knowledge vector representations in the vector database D, and obtain the similarity score of the sentence vector representation CLS relative to all external knowledge vector representations;

[0031] S4: Select the external knowledge vector representation with the highest similarity score and use it to replace the sentence vector representation CLS of the text to be corrected, to obtain a new sentence vector representation CLS';

[0032] S5: Input the new sentence vector representation CLS' into other modules in the text error correction network, and the text error correction network outputs the corrected text data.

[0033] The text error correction network in the embodiment of the present invention is a neural network based on Bert, including a Bert layer and other modules. The parameters of the text error correction network are fixed.

[0034] In S1, a text data set to be corrected in a specific field is selected according to demand, and the text data set to be corrected includes text data without knowledge errors (also called correct knowledge) and text data with knowledge errors (also called irrelevant knowledge).

[0035] In some preferred embodiments, reference Figure 3 , 4 As shown, the training process of the external knowledge encoding network is:

[0036] S101: Obtain a text sample set in a specific field, obtain positive labels in the text sample set after manual annotation, and extract the sentence vector representation CLS of the text sample set;

[0037] S102: Input the text sample of the specific field and the specific knowledge set into the BM25 model to calculate the correlation, and obtain the negative label of the text sample;

[0038] S103: Input the text samples of the specific field and their positive and negative labels into the BGE network to obtain a knowledge sequence representation composed of each knowledge;

[0039] S104: Fusing the sentence vector representation CLS with the knowledge sequence representation, and inputting the fused representation into the local interactive attention network to obtain interactive representation vectors of each piece of knowledge corresponding to the positive and negative labels and the sentence vector representation CLS;

[0040] S105: Calculate the contrast loss according to the interaction representation vector, and optimize the external knowledge encoding network through back propagation of the contrast loss.

[0041] Reference Figure 4 As shown, in S101, the text sample in the embodiment of the present invention adopts a text data set in the geographical field, and the text data set filters out text data without knowledge errors and text data with knowledge errors in a 1:1 ratio. After manual data annotation, the text data without knowledge errors is used as a positive label, that is, the correct knowledge is used as a positive label.

[0042] In S101, the text sample and the specific knowledge set are input into the BM25 model together, and the correlation is calculated to obtain the similarity score of each knowledge in the text sample, and the top 30 erroneous knowledge (i.e., knowledge with non-positive labels) with the highest similarity scores are selected as negative labels for the current sample to be corrected. The specific knowledge set is a knowledge set in the same technical field as the training sample set.

[0043] For example, refer to Figure 4 As shown, a specific knowledge set in the geographical field is selected as the external knowledge set. In the specific knowledge set, external knowledge triples (h, r, t), i.e., factual statement triples, are extracted based on a specific text data set in the geographical field. h represents the head entity, t represents the tail entity, and r represents the relationship between the head entity and the tail entity. The external knowledge triples (h, r, t) are converted into knowledge texts in a fixed format, and the format is: subject knowledge-relationship-object knowledge. For example, for text data without factual errors: the capital of China is Beijing, and its triple is (China, capital, Beijing); for text data with factual errors: the capital of Sichuan is Chongqing, and its triple is (Sichuan, capital, Chengdu).

[0044] BM25 is a classic information retrieval algorithm. It is an improved version of the Okapi TF-IDF algorithm, which aims to solve some of the shortcomings of the Okapi TF-IDF algorithm. BM25 is widely used as a ranking function in the field of information retrieval to estimate the relevance between document D and user query Q. BM25 is an improvement based on the probabilistic retrieval framework, especially when dealing with long documents and short queries. The core idea of ​​BM25 is to calculate the relevance between document D and query Q based on term frequency (TF) and inverse document frequency (IDF), while also introducing document length information. The widely used search engine ES has a built-in BM25 algorithm for full-text retrieval.

[0045] The formula for the BM25 model is:

[0046]

[0047] Among them, BM25(D,Q) represents the similarity score, D represents a text composed of a knowledge triple, Q represents the text to be corrected, n is the number of words in Q after Jieba segmentation, and q i represents the i-th word in Q, IDF(q i ) indicates q i The inverse document frequency, f(q i , D) represents q i The frequency in the text D, len(D) represents the length of the text D, avg_len represents the average length of all texts D, N represents the total number of texts in D, n(q i ) means including q i The number of texts, k i and b are adjustable parameters.

[0048] In some preferred embodiments, the BM25 model is used to calculate the correlation between the text sample set in a specific field and the specific knowledge set to obtain a similarity score (BM25 (D, Q)), and the top 30 knowledge with the highest similarity score excluding the positive label are selected as the negative label of the current text sample set. P represents the knowledge text of the positive label, N i represents the i-th knowledge text among the 30 negative labels.

[0049] By constructing negative labels in this way, we can find the most similar and most easily confused text knowledge at the text level. Through the contrastive learning method in the external knowledge encoding network, we can widen the high-dimensional distance between positive and negative labels in the feature space, thereby achieving the purpose of separating easily confused knowledge.

[0050] Reference Figure 4 As shown, in S103, the sentence-level feature vector CLS of the text sample and its knowledge text P with positive labels and 30 knowledge texts N with negative labels are combined. i Input the BGE network to obtain an embedding representation. The embedding representation is transformed into {[CLS],[CLS],[P],[CLS],[N1],[CLS][N2],...,[CLS][N 30 ]} are spliced ​​in the sequence dimension to obtain a knowledge sequence representation consisting of each knowledge component sequence.

[0051] In some preferred embodiments, the sentence vector representation CLS is length-aligned with the knowledge sequence representation, and a position attenuation factor α is added to the corresponding position of each sentence vector representation CLS to obtain a position-attenuated text sequence vector and fuse it with the knowledge sequence representation, and the fused representation is input into the local interactive attention network.

[0052] Specifically, the sentence vector CLS of the text to be corrected is expanded in the sequence dimension according to the knowledge sequence representation length, so that it is consistent with the knowledge sequence representation length, and the length of the feature sequences of the two are aligned.

[0053] The calculation formula of the position attenuation factor is specifically:

[0054]

[0055] Among them, α represents the position attenuation factor, pos represents the position of the sentence-level feature vector CLS, e represents the natural logarithm, i represents the i-th dimension in the sentence-level feature vector CLS, dim represents the dimension size of the sentence-level feature vector CLS, sin represents the sine trigonometric function, and cos represents the cosine trigonometric function.

[0056] The position attenuation factor is an improvement on the position encoding of the original transformer model. In this scheme, since CLS is expanded and has no position encoding information in the sequence, the position encoding of the transformer only ensures that the representation of each position is inconsistent. Here, we hope that the role of CLS at each position will gradually weaken, so the original position encoding is improved so that as the sequence length continues to increase, the representation of the CLS vector continues to weaken.

[0057] The position-attenuated text sentence vector is fused with the knowledge sequence representation and sent into the local interactive attention network to obtain the interactive representation vector of each piece of knowledge corresponding to the positive and negative labels and the sentence vector representation CLS.

[0058] The output of the local interactive attention network is specifically expressed as:

[0059]

[0060] Where Vec represents the output of the local interactive attention network, Softmax(·) represents the Softmax activation function, seq represents the vector obtained by adding the position-attenuated text sequence vector and the knowledge sequence representation according to the corresponding position, and W q (·),W k (·) and W v (·) represent different linear network layers, which are responsible for performing different linear transformations on the original sequence. (·) TRepresents matrix transpose, dim represents the dimension size of the seq matrix on the last dimension, spread_mask represents the local attention mask matrix, and local_mask represents the global knowledge mask.

[0061] The local attention mask matrix spread_mask is specifically expressed as:

[0062]

[0063] Where i represents the i-th row of the spread_mask matrix, j represents the j-th column of the spread_mask matrix, and P start Indicates the starting position of a piece of knowledge in the knowledge sequence representation, P end It indicates the end position of a piece of knowledge in the knowledge sequence representation, seq_len indicates the length of the knowledge sequence representation, and others indicates others.

[0064] The local attention mask matrix spread_mask can ensure that each knowledge can only focus on CLS and itself.

[0065] The local_mask is specifically expressed as:

[0066]

[0067] Where i represents the i-th row of local_mask, j represents the j-th column of local_mask, and P start It indicates the starting position of a piece of knowledge in the knowledge sequence representation, and others indicates others.

[0068] local_mask focuses on suppressing the errors caused by operations between different CLS in the same sequence.

[0069] Attached Figure 5 A specific spread_mask schematic diagram in an embodiment of the present invention is shown. Figure 5 In the above example, C represents the sentence-level feature vector CLS, P represents the positive label, N represents the negative label, i represents the i-th row of the spread_mask matrix, j represents the j-th column of the spread_mask matrix, and ... start Indicates the starting position of the positive or negative label, P end Indicates the end position of the positive or negative label, 1 means 1e -12 .

[0070] In some preferred embodiments, the calculation formula of contrast loss is specifically:

[0071]

[0072] Among them, loss represents the loss function of the external knowledge encoding network, q represents the sentence vector representation CLS of the current text sample set, and k + represents the vector representation corresponding to the positive label in the current text data to be corrected, k i represents the vector representation corresponding to the i-th negative label in the current text data to be corrected, t represents the adjustable temperature coefficient parameter, and score i Represents the similarity score corresponding to the i-th negative label after BM25 correlation calculation.

[0073] This contrast loss combines the retrieval results at the text level (BM25) and the semantic level (Embedding) to tune the model parameters, which can effectively separate the vector representation of easily confused knowledge in high-dimensional space. In application, it can greatly improve the accuracy of recalling relevant knowledge and enhance the accuracy of integrating external knowledge in the field of text error correction.

[0074] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which can include: ROM, RAM, disk or CD, etc.

[0075] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A text error correction method based on an external knowledge encoding network, characterized in that: include: Obtain the text data to be corrected in a specific field, input it into the Bert layer in the text correction network, and obtain the sentence vector representation CLS of the text to be corrected; Input the knowledge set of a specific field into the trained external knowledge encoding network to obtain the external knowledge vector representation, which is stored in the vector database D; Calculate the similarity between the sentence vector representation CLS of the text to be corrected and all external knowledge vector representations in the vector database D, and obtain the similarity score of the sentence vector representation CLS relative to all external knowledge vector representations; Select the external knowledge vector representation with the highest similarity score and use it to replace the sentence vector representation CLS of the text to be corrected, so as to obtain a new sentence vector representation CLS'; The new sentence vector representation CLS' is input into other modules in the text correction network, and the text correction network outputs the corrected text data.

2. The text error correction method based on the external knowledge encoding network according to claim 1 is characterized in that: The training process of the external knowledge encoding network is: Obtain a text sample set in a specific field, obtain the positive labels in the text sample set after manual annotation, and extract the sentence vector representation CLS of the text sample set; Input the text sample of a specific field and the specific knowledge set into the BM25 model to calculate the correlation and obtain the negative label of the text sample; The text samples of a specific field and their positive and negative labels are input into the BGE network to obtain a knowledge sequence representation composed of each piece of knowledge; The sentence vector representation CLS is fused with the knowledge sequence representation, and the fused representation is input into the local interactive attention network to obtain the interactive representation vector of each piece of knowledge corresponding to the positive and negative labels and the sentence vector representation CLS; The contrast loss is calculated based on the interaction representation vector, and the external knowledge encoding network is optimized by backpropagation through the contrast loss.

3. The text error correction method based on the external knowledge encoding network according to claim 2 is characterized in that: During the training process of the external knowledge encoding network, it also includes: aligning the length of the sentence vector representation CLS with the knowledge sequence representation, adding a position attenuation factor α to the corresponding position of each sentence vector representation CLS, obtaining a position attenuated text sequence vector and fusing it with the knowledge sequence representation, and inputting the fused representation into the local interactive attention network.

4. The text error correction method based on the external knowledge encoding network according to claim 3 is characterized in that: The calculation formula of the position attenuation factor is specifically: Among them, α represents the position attenuation factor, pos represents the position of the sentence-level feature vector CLS, e represents the natural logarithm, i represents the i-th dimension in the sentence-level feature vector CLS, dim represents the dimension size of the sentence-level feature vector CLS, sin represents the sine trigonometric function, and cos represents the cosine trigonometric function.

5. The text error correction method based on the external knowledge encoding network according to claim 1 is characterized in that: The output of the local interactive attention network is specifically expressed as: Where Vec represents the output of the local interactive attention network, Softmax(·) represents the Softmax activation function, seq represents the vector obtained by adding the position-attenuated text sequence vector and the knowledge sequence representation according to the corresponding position, and W q (·) represents the first linear network layer, W k (·) denotes the second linear network layer, W v (·) indicates the third linear network layer, (·) T Represents matrix transpose, dim represents the dimension size of the seq matrix on the last dimension, spread_mask represents the local attention mask matrix, and local_mask represents the global knowledge mask.

6. The text error correction method based on the external knowledge encoding network according to claim 1 is characterized in that: The calculation formula of contrast loss is as follows: Among them, loss represents the loss function of the external knowledge encoding network, q represents the sentence vector representation CLS of the current text sample, and k + represents the vector representation corresponding to the positive label in the current text data to be corrected, k i It represents the vector representation of the negative label corresponding to the i-th knowledge in the current text data to be corrected, t represents the adjustable temperature coefficient parameter, and score i Represents the similarity score corresponding to the i-th negative label after BM25 correlation calculation.

7. The text error correction method based on the external knowledge encoding network according to any one of claims 1 to 6, characterized in that: The positive label is the correct knowledge in the text sample set.

8. The text error correction method based on the external knowledge encoding network according to any one of claims 1 to 6, characterized in that: The correlation between the text samples in a specific field and the specific knowledge set is calculated to obtain a similarity score, and the top 30 knowledge with the highest similarity scores excluding the positive labels are selected as negative labels for the current text sample set.

Images