A government affair text error correction method and system

Abstract

The application relates to a government affair text error correction method and system, which comprises the following steps: constructing and training a government affair text error correction model, including: obtaining original government long text, and performing data preprocessing to obtain a government short sentence data set DS1; performing government affair text confusion processing on the DS1 to obtain a confusion-correct text pair data set DS2; performing wrong character confusion and phonetic symbol acquisition on the text of the DS2 to obtain a character shape and phonetic symbol data set DS3; integrating the DS2 and the DS3 to obtain an overall error correction data set DS4; training a language feature embedding model Model emb by using the DS3; training a government affair text error correction model Model emb by using the DS4 and taking the Model fix as an embedding model; performing sentence preprocessing on to-be-detected government text to obtain multiple to-be-detected government short sentences, inputting the to-be-detected government short sentences into the government affair text error correction model, obtaining an error correction result, replacing the erroneous government short sentences, and forming government text after error correction. The method and system can effectively detect and correct text errors in government text, and have high detection efficiency and accuracy.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method and system for correcting errors in government documents. Background Technology

[0002] With the rapid advancement of internet technology, my country's government information platforms are increasingly becoming digitalized and transparent, greatly facilitating public access to relevant information. As bridges between the government and the public, government websites and integrated media platforms must ensure the accuracy and compliance of the information they publish. Traditional methods of rigorous manual review of information content are costly and inefficient, failing to meet practical needs. Therefore, there is an urgent need for an automated and intelligent government information review system to guarantee content security.

[0003] The main research focus in text correction is on improving model learning capabilities and training data quality. Existing deep learning-based text correction algorithms often use publicly available datasets for training and testing. However, these datasets are often small and insufficient for training models in real-world production environments. Furthermore, the significant overhead of data annotation when using large-scale web text crawling as datasets remains a challenge. Automated dataset construction methods tailored to the characteristics of government documents meet practical needs while ensuring data quality.

[0004] In deep learning-based text correction research, most methods incorporate natural language feature embedding learning into the overall model training. However, due to the discrepancy between the dataset and real-world data, the model is more prone to overfitting. Training the language feature embedding model through a reasonable self-supervised pre-training task can make the overall model more robust and also significantly improve data utilization.

[0005] The remarkable success of large-scale AI models has not only garnered widespread attention in the field of natural language processing but also highlighted the superior performance and unique advantages of GPT-like architecture models. These models, based on the Transformer architecture, are able to deeply understand the complexity and nuances of language through pre-training on massive amounts of text data. The generative capabilities of GPT-like architecture models further enhance the quality and readability of text. Using GPT-like architecture models for text correction tasks can yield even better results.

[0006] To address this, an invention patent with publication number CN118643798A discloses a text correction method and system based on federated learning. The method includes: constructing corpora corresponding to each participant to form corresponding training sets; fine-tuning a pre-trained model to obtain fine-tuned initial model parameters, resulting in an initial text correction model; encrypting the fine-tuned initial model parameters to obtain encrypted initial model parameters; calculating the similarity of the corpora or scenario tasks of each participant, filtering model parameters to be aggregated, performing secure aggregation, and obtaining corresponding aggregated model parameters; decrypting the aggregated model parameters to further update the initial text correction model; and performing multiple rounds of federated learning to obtain the optimal text correction model for correcting the text to be corrected.

[0007] Another invention patent, CN118468861A, discloses a method, system, device, and medium for text correction based on rule matching. The method includes: acquiring the original text data to be corrected, a general dictionary, and a confusion word dictionary; preprocessing the original text data to be corrected to obtain input text; performing error detection processing on the input text using the general dictionary and the confusion word dictionary to obtain a set of suspected errors; and iterating and matching the suspected error set based on the confusion word dictionary; in response to errors in the suspected error set existing in the confusion word dictionary, directly replacing the errors with correct values ​​from the confusion word dictionary; in response to errors in the suspected error set not existing in the confusion word dictionary, constructing a candidate set based on the errors, calculating the language perplexity of all replacement words in the candidate set using a preset language model, and selecting the replacement word with the lowest perplexity to replace the error.

[0008] Another invention patent, CN118627495A, discloses a text intelligent error correction method, device, electronic device, and storage medium. The method includes: embedding the original medical consultation text into a preset matrix layer to obtain converted medical text; using an encoder in a trained text error correction model to compile the converted medical text to obtain model-readable text; using a judge in the text error correction model to identify abnormal characters in the model-readable text to obtain abnormal text; using an error corrector in the text error correction model to correct the abnormal text to obtain corrected text; randomly sorting the words and phrases in the corrected text to obtain a sequence of words and phrases, and extracting the preferred sequence of words and phrases from the sequence to obtain preferred text; and using a decoder in the trained text error correction model to decode the preferred text to obtain the target medical text.

[0009] Another invention patent, CN118643825A, discloses a Chinese text error correction method, system, storage medium, and electronic device, including: acquiring a large language model, which is used to realize text error detection and text error correction of Chinese text; acquiring a first input instruction text, which is used to indicate whether the input text has a text error; inputting the first input instruction text and the Chinese text to be detected into the large language model to obtain the text error detection result of the Chinese text to be detected; when the text error detection result indicates that a text error exists, acquiring a second input instruction text, which is used to indicate that the text error is corrected; inputting the second input instruction text and the Chinese text to be detected into the large language model to obtain the corrected Chinese text corresponding to the Chinese text to be detected.

[0010] Another invention patent, CN118194854A, discloses a method for correcting Chinese text based on whole-word masking and dependency masking, including the following steps: Step S1: Collect and construct a dataset; Step S2: Construct a text correction model, which includes a spelling correction model and a grammar correction model; Step S3: Import erroneous sentences into the spelling correction model in Step S2 to obtain spell-corrected sentences, and simultaneously train the spelling correction model; Step S4: Import the spell-corrected sentences into the detection network of the grammar correction model in Step S2 to obtain the grammar error types of the spell-corrected sentences, and train the grammar correction model; Step S5: Import sentences with grammar error types into the correction network of the grammar correction model, perform grammar correction on the sentences with obtained grammar error types, and obtain grammar-corrected sentences.

[0011] As can be seen from the above, existing methods for training text correction models typically employ deep neural network structures. Based on a large amount of training data, they use learning objectives with relevant loss functions for training and learning, ultimately learning network weights that can understand text semantics and identify and correct text errors for text correction. However, during the actual training of a government text correction model, the inventors realized that existing technologies have at least the following problems: 1. Existing training methods for Chinese natural language models use a single-perspective text embedding method, which does not fully utilize the features of Chinese characters themselves, potentially resulting in insufficient text embedding feature representation capabilities; 2. Existing general-purpose text correction models use datasets obtained from a wide range of networks or existing small-scale datasets for model training. There is no cost-effective and efficient method to obtain high-quality correction datasets, leading to insufficient model training or poor performance in real-world production environments. Summary of the Invention

[0012] The purpose of this invention is to provide a method and system for correcting errors in government documents. This method and system can effectively detect and correct text errors in government documents, and has high detection efficiency and accuracy.

[0013] To achieve the above objectives, the technical solution adopted by this invention is: a method for correcting errors in government documents, comprising:

[0014] Building and training a government document error correction model includes the following steps:

[0015] Step A1: Obtain relevant government documents as the original long government document;

[0016] Step A2: Preprocess the original long government documents to obtain the short government documents dataset DS1;

[0017] Step A3: Perform government affairs text obfuscation on the government affairs short sentence dataset DS1 to obtain the obfuscated-correct text pair dataset DS2;

[0018] Step A4: Perform misspelling and pronunciation acquisition on the text in the DS2 dataset of confused-correct text pairs, and calculate the glyph and pronunciation matching features of the samples before and after confusion to obtain the glyph and pronunciation dataset DS3;

[0019] Step A5: Integrate the obfuscated-correct text pair dataset DS2 with the glyph and phonetic dataset DS3 to obtain the overall error correction dataset DS4;

[0020] Step A6: Train the language feature embedding model using the DS3 glyph and phonology dataset. emb ;

[0021] Step A7: Transfer the model emb The embedded model used as the error correction model for government documents is trained using the overall error correction dataset DS4. fix ;

[0022] The government text to be detected is preprocessed by segmenting sentences to obtain multiple short sentences to be detected. These sentences are then input into the government text error correction model one by one to obtain the error correction results. Finally, the obtained error correction results are used to replace the erroneous short sentences to form the corrected government text.

[0023] Furthermore, step A2 specifically includes:

[0024] Step A21: Using symbols indicating sentence breaks as delimiters, the acquired original long government documents are divided into multiple short sentences, resulting in a short sentence set {s1, s2, ..., s...}. n}, where n is the number of samples in the short sentence set;

[0025] Step A22: In the short sentence set {s1, s2, ..., s...} n In the code, each short sentence is prefixed with a sentence identifier and terminating with a sentence identifier, as shown below:

[0026] S i =concat([BOS], s i , [EOS])

[0027] Among them, S i This is a sample of concatenated short sentences, where concat(·) is the concatenation function, and s i Let [BOS] be the i-th short sentence sample in the short sentence set, where [BOS] is the sentence beginning identifier, indicating the start of the sentence, and [EOS] is the sentence ending identifier, indicating the end of the sentence.

[0028] This yields the dataset of government affairs phrases required for training: DS1 = {S1, S2, ..., S} n}

[0029] Furthermore, step A3 specifically includes:

[0030] Step A31: Set the set of rules for obfuscating government documents: Rule = {R1, R2, ..., R...} m The obfuscation rules are expressed as follows:

[0031] R j =Fake j →Real j

[0032] Among them, R j For the j-th obfuscation rule in the set of government document obfuscation rules, Real j This is a fragment of a real government document; Fake j In actual writing, it is often used in conjunction with Real. j The obfuscated government text fragments, j = 1, 2, ..., m, where m is the number of samples in the set of government text obfuscation rules;

[0033] Step A32: Obfuscate the real government short sentences in the government short sentence dataset DS1 according to the obfuscation rules in the government text obfuscation rule set Rule, forming the initial obfuscated government short sentence dataset {S′1, S′2, ..., S′ l}; Among them, confusing government affairs phrases are represented as follows:

[0034] S′ i =S i .replace(Real t Fake t )

[0035] Among them, S′ iTo generate confusing government policy phrases, S i Real government short sentences from the DS1 dataset are real government short sentences. t and Fake t The obfuscation rules are R respectively. t The dataset contains real and obfuscated government text fragments. x.replace(y, z) is a replacement function that replaces fragment y in string x with fragment z; i = 1, 2, ..., l, where l is the number of samples in the initial obfuscated government short sentence dataset.

[0036] Step A33: Combine the generated obfuscated government phrases with their corresponding real government phrases to form the initial obfuscated-correct text pair dataset ds2 = {C1, C2, ..., C...} l The obfuscated-correct text pair is represented as follows:

[0037] C i =(S′) i S i )

[0038] Among them, C i To obfuscate-correct text pairs, S′ i To confuse short government phrases, S i S′ i Corresponding real government policy phrases;

[0039] Step A34: To enable the model to learn the ability to distinguish error-free government documents during training, add error-free text pairs to the initial confused-correct text pair dataset ds2, resulting in the confused-correct text pair dataset DS2 = {C1, C2, ..., C...} k The added error-free text pairs are represented as follows:

[0040] C q =(S′) q S q )

[0041] Among them, C q For error-free text pairs, S′ q S is a collection of obfuscated government phrases within an error-free text pair. q For error-free text pairs S′ q The corresponding real government affairs phrases, and S′ q =S q That is, in an error-free text pair, the obfuscated government phrase is equivalent to the corresponding real government phrase, q = l+1, l+2, ..., k, where k is the number of samples in the obfuscated-correct text pair dataset; the obfuscated-correct text pair dataset DS2 = {C1, C2, ..., C...} k The corresponding dataset is a collection of obfuscated government policy phrases {S′1, S′2, ..., S′}. l S′l+1 ,…,S′ k}

[0042] Further, step A4 specifically includes:

[0043] Step A41: Extract all characters from the obfuscated-correct text pair dataset DS2 and remove duplicates, then construct a character vocabulary. l sym For character vocabulary V sym Length;

[0044] Step A42: Based on the character dictionary V sym Constructing a character phonetic dictionary The key-value pairs are represented as follows:

[0045] kv h =Sym h →Py h

[0046] Among them, kv h Sym is the h-th key-value pair in the character phonetic dictionary. h For character vocabulary V sym Characters in Py h For the character Sym h The corresponding pronunciation; the method for obtaining the pronunciation of Chinese characters is as follows:

[0047] Py h = cnchar.spell(Sym h )

[0048] Among them, Sym h For Chinese characters, cnchar.spell(·) is a Pinyin function provided by the cnchar library, which returns a Pinyin string as its parameter.

[0049] The pronunciation of non-Chinese characters is replaced by the character itself, as shown below:

[0050] Py h =Sym h

[0051] Step A43: Extract the obfuscated government phrase dataset {S′1, S′2, ..., S′} from the obfuscated-correct text pair dataset DS2. k Each confused government affairs phrase in the dataset is processed to remove typos, resulting in a general confused phrase dataset {S″1, S″2, ..., S″}. k}; where, the general phrase for confusion is represented as follows:

[0052] S″ i =confuse(S′ i)

[0053] Among them, S″ i To confuse common short phrases, S′ i To obfuscate short government sentences, `confuse(·)` is a typo obfuscation function. This function simulates common typo generation scenarios in real-world situations, including three cases: similar-looking characters, homophones, and random input. The specific implementation method is as follows:

[0054] S′=[w1,w2,…w s ]

[0055] S″=[w′1,w′2,…w′ s ]=confuse(S′)={wordConfuse(w i )|i=1,2,…,s}

[0056] Where S′ is the original character sequence consisting of s characters, which is the obfuscated government affairs phrase; S″ is the character sequence obtained after obfuscating each character in the original character sequence, which is the obfuscated general phrase; and wordConfuse(·) is the character obfuscation function, which is implemented as follows:

[0057] r = random()

[0058]

[0059] Among them, random() is a random number function that returns a floating-point number with a value range of (0, 1); symConfuse(·) is a similar-looking character confusion function that returns a similar-looking character when the input character is a Chinese character, and randomly returns a non-Chinese character from the character dictionary when the input character is a non-Chinese character; pyConffuse(·) is a homophone confusion function that returns a homophone when the input character is a Chinese character, and returns the original character when the input character is a non-Chinese character; and randConfuse(·) is a random character confusion function that randomly returns a character from the character dictionary.

[0060] Step A44: Use the character phonetic dictionary S2P to parse the obfuscated government affairs short sentence dataset {S′1, S′2, ..., S′...} k} is mapped to a confused government phonetic dataset {P′1, P′2, ..., P′ k} will obfuscate the general short sentence dataset {S″1, S″2, ..., S″}. k} is mapped to a confused general phonetic dataset {P″1, P″2, ..., P″} k};

[0061] Step A45: Compare the character sequences S′=[w1, W2, … w1] before and after the typo obfuscation process. s] and S″=[w′1, w′2,…w′ s The corresponding glyph matching degree sequence M is calculated. sym =[m sym,1 m sym,2 , ...m sym，s ]; where m sym，s Represents the character w s With the character w′ s The glyph matching degree is calculated using the following formula:

[0062] When w i With w′ i When all characters are Chinese characters:

[0063]

[0064] Among them, getStrokes(·) is the stroke acquisition function, which returns the stroke sequence of Chinese characters; match(x, y) is the matching element calculation function, which returns the number of common elements in sequence x and sequence y; and len(·) is the sequence length calculation function, which returns the number of elements in the input sequence.

[0065] When w i With w′ i When there is only one Chinese character in the m sym，i =0;

[0066] When w i With w′ i When neither of them are Chinese characters, m sym,i =0.8;

[0067] Step A46: Compare the pronunciation sequences P′ = [p1, p2, ... p1] before and after the typo confusion treatment. s ] and P″=[p′1,p′2,…p′ s The corresponding phonetic matching degree sequence M is calculated. py =[m py,1 m py,2 , ...m py,s ]; where p i Represents the character w i The corresponding pronunciation is p′ i Represents the character w′ i The corresponding pronunciation is m py，s Indicates the pronunciation p s With the pronunciation p′ s The formula for calculating the phonetic matching degree is as follows:

[0068] When w i With w′ i When all characters are Chinese characters:

[0069]

[0070] When w i With w′ i When not all characters are Chinese characters, m py，i =0;

[0071] Step A47: Combine the character sequence S′ and phonetic sequence P′ before typo obfuscation, the character sequence S″ and phonetic sequence P″ after typo obfuscation, and the character shape matching degree sequence M. sym The sequence M of word pronunciation matching degree py The merged glyph and phonetic data sample Q is represented as follows:

[0072] Q = [(S′,S″,M] sym ), (P′, P″, M py )]

[0073] Integrate all data samples to form the character shape and pronunciation dataset DS3 = {Q1, Q2, ..., Q...} k}

[0074] Further, in step A5, the obfuscated-correct text pair dataset DS2 and the glyph and phonetic dataset DS3 are integrated to form a comprehensive error correction dataset DS4 = {C′1, C′2, ..., C′}. k The overall error correction dataset DS4 has the same sample format as DS2, which is the confusion-correct text pair format C′. i =(S″′ i S i ), where S i To obfuscate the true government phrases in the DS2 dataset, S″′ i The mixed obfuscated government phrases are defined as follows: the mixed obfuscated government phrases are 50% likely to be the obfuscated government phrases S′ in the obfuscated-correct text pair dataset DS2. i There is a 50% probability that it corresponds to the confused general phrase S″ in the glyph and phonetic dataset DS3. i The overall error correction dataset DS4 covers data on normal sentences, confused government short sentences, and confused general short sentences in government text error correction scenarios.

[0075] Furthermore, step A6 specifically includes:

[0076] Step A61: Construct the initial language feature embedding model The model comprises a glyph feature embedding layer, a phonetic feature embedding layer, a language feature extraction layer, a glyph matching degree calculation layer, a phonetic matching degree calculation layer, and a language feature reconstruction layer; the computable parameters of each layer are as follows:

[0077] The glyph feature embedding layer contains parameters l sym For character vocabulary V sym Length, d m The number of embedding feature dimensions;

[0078] The phonetic feature embedding layer contains parameters

[0079] The language feature extraction layer contains parameters

[0080] The glyph matching degree calculation layer contains parameters

[0081] The phonetic matching calculation layer contains parameters.

[0082] The language feature reconstruction layer contains parameters

[0083] Step A62: Take samples Q = [(S′, S″, M] from dataset DS3. sym ), (P′, P″, M py )], input S′ and S″ into the glyph feature embedding layer to obtain the true glyph feature Feature′ sym With input glyph features sym , means as follows:

[0084] OH S′ =onehot(Index(V) sym ,S′),l sym )

[0085] OH S″ =onehot(Index(V) sym ,S″),l sym )

[0086]

[0087] Where i is the current training iteration number of the model, and let s be the number of elements in both S′ and S″. The function Index(x, y) returns an integer sequence of the same length as the sequence y, where each element is the index of the element at the same position in y in the sequence x. The function onehot(x, l) returns the one-hot matrix obtained by one-hot encoding the integer sequence x.

[0088] Step A63: Input the phonetic feature embedding layer with P′ and P″ to obtain the true phonetic feature Feature′. py Input phonetic features py , means as follows:

[0089] OH P′ =onehot(Index(V) py ,P′),l py )

[0090] OH P″ =onehot(Index(V) py ,P″),l py )

[0091]

[0092] Let the number of elements in P′ and P″ both be s, then

[0093] Step A64: Select Feature″ sym With Feature py The concatenated input embedding feature Feature′ is input into the language feature extraction layer to obtain the abstract language feature Feature″, as follows:

[0094] Feature′=concat(Feature″ sym Feature″ py )

[0095]

[0096] Att = softmax(score)

[0097] Feature″=V×Att

[0098] in, The function softmax(s) is a normalization function that maps the input matrix to the exponential domain and then performs column normalization to obtain matrix s′, as shown below:

[0099]

[0100] Step A65: Input the Feature″ into the glyph matching degree calculation layer to obtain the model glyph matching degree sequence M′. sym , means as follows:

[0101]

[0102] M′ sym =sigmoid(MF sym )

[0103] in, The function sigmoid(M) is a normalization function that maps each element in the input sequence to the range (0, 1) to obtain the sequence M′, as shown below:

[0104]

[0105] Step A66: Input the abstract language feature "Feature″" into the phonetic matching degree calculation layer to obtain the model phonetic matching degree sequence M′. py , means as follows:

[0106]

[0107] M′ py =sigmoid(MF py )

[0108] in,

[0109] Step A67: Input glyph features. sym Model glyph matching degree sequence M′ sym Input phonetic features py Matching sequence M′ with the model pronunciation py The input language feature reconstruction layer yields the reconstructed language features, which are represented as follows:

[0110] Feature″′ sym =scale(Feature") sym M′ sym )

[0111] Feature″′ py =scale(Feature") py M′ py )

[0112] IF = concat(Feature″′) sym Feature″′ py )

[0113]

[0114] Att′ = softmax(score′)

[0115] Feature″′=V′×Att′

[0116] in, The function scale(x, y) is a scaling function that scales the elements in x. j Scaling to yj To obtain x′ by multiplying by 1, the formula is as follows:

[0117] x′ j =x j *y j

[0118] Step A68: Based on the model glyph matching degree sequence M′ sym Model phonetic matching sequence M′p y、 Reconstructing the language features (Feature″′) and the character shape matching sequence M sym Phonetic matching sequence M py Features of real glyphs sym Authentic pronunciation features py The model loss L is calculated by iterating the model parameters using gradient descent, as shown below:

[0119] Feature = concat(Feature') sym Feature′ py )

[0120] L m_s =MSE(M sym M′ sym )

[0121] L m_p =MSE(M py M′ py )

[0122] L f =MSE(Feature,Feature″′)

[0123] L = L m_s +L m_p +L f

[0124] W (i+1) =backwardByLoss(L, W) (i) )

[0125] Among them, L m_s To calculate the loss for font matching, L m_p To calculate the loss for the phonetic matching degree, L f For the embedding feature reconstruction loss, W (i) For all updatable parameters after the i-th iteration of the model, the function MSE(X,Y) returns the root mean square error of the input object, and the function backwardByLoss(L,W) is the backward update function, which calculates the parameters of the new iteration based on the gradient of the loss function L and chain propagation.

[0126] Step A69: Repeat steps A62-A68 until the model converges or the set number of iterations is reached. The model training is complete, and the trained model is the language feature embedding model. emb .

[0127] Furthermore, step A7 specifically includes:

[0128] Step A71: Construct an initial government document error correction model The model includes a text feature embedding layer, an encoder, and a decoder; wherein, the text feature embedding layer is the language feature embedding model trained in step A6. emb The encoder is used to convert text sequence data into a computer-computable feature vector sequence. The encoder is composed of multiple layers of the same structure and is used to extract high-level features of the input text. The decoder is composed of multiple layers of the same structure and is connected to a classification network. It is used to autoregressively output the sequence bit by bit by combining the high-level features given by the encoder and the information of the output sequence.

[0129] Step A72: Take a sample C′ = (S″′, S) from the overall error correction dataset DS4, where S″′ is the input text and S is the correct text corresponding to S″′; input the input text S″′ into the text feature embedding layer to obtain the input text embedding feature. input , means as follows:

[0130] P″′=transByDict(S2P,S″′)

[0131] Feature input =Model emb (S″′,P″′)

[0132] Here, S2P is the constructed character phonetic dictionary, the function transByDict(S2P, S″′) is the dictionary conversion function, which maps the elements in the input text S″′ one by one according to the key-value relationships in the dictionary S2P, and P″′ is the converted input phonetic sequence. s is the length of the input text S″′, d m is the dimension of the embedding vector;

[0133] Step A73: Embed the input text with features input Input the encoder to obtain high-level text features. en ;

[0134] Step A74: Take the beginnings of the input text S″′ of different lengths as the output sequence, and input the output sequence into the text feature embedding layer as in step A72 to obtain the output text embedding feature. outputAnd then with Feature en The data are input together into the decoder to obtain the model's distribution prediction sequence result for the output sequence, as shown below:

[0135] S″′ ，i =S″′[:i]

[0136] P″′ ，i =transByDict(S2P,S″′) ，i )

[0137] Feature outpu t = Model emb (P″′ ，i ,S″′ ，i )

[0138] Feature de =Decoder(Feature) en Feature output )

[0139] result i+1 =Classifier(Feature) de,i )

[0140] Where S″′[:i] represents the first i elements of the input text S″′. Let $\frac{i}{i+1}$ be the i-th element of the distribution prediction sequence $\frac{result}{i}$, representing the probability distribution of the modification result of the (i+1)-th element of the input text $S″′$. i+1 The j-th element This indicates that the original character will be modified to the character table V. sym The probability of finding the j-th character;

[0141] Step A75: Compare the distribution prediction sequence result with the true output sequence, i.e. the correct text S, element by element, calculate the cross-entropy loss of the classification task, and iterate the model parameters using the gradient descent method.

[0142] Step A76: Repeat steps A72-A75 until the model converges or the set number of iterations is reached. The model training is complete, and the trained model is the government document error correction model. fix .

[0143] The present invention also provides a government document error correction system, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-mentioned method steps.

[0144] Compared with existing technologies, the present invention has the following beneficial effects: The present invention provides a method and system for correcting errors in government documents. This method and system combine government document obfuscation rules with conventional text obfuscation rules, and automatically obtains government document error correction datasets by obfuscating publicly available government documents. This overcomes the current difficulties in text error correction research, such as insufficient dataset quantity and poor dataset quality. At the same time, the present invention proposes a unique language embedding model, which explicitly defines text embedding features from the perspectives of glyphs and phonetics, reducing the learning difficulty of Chinese character semantics for deep learning networks. Furthermore, based on the original dataset, an innovative self-supervised pre-training task is designed, which meets the training requirements of the embedding model while improving the utilization rate of data. Finally, an encoder-decoder model is built using a GPT-like model structure to complete the error correction task, thereby improving the accuracy of government document error correction tasks. Attached Figure Description

[0145] Figure 1 This is a flowchart illustrating the method implementation of an embodiment of the present invention;

[0146] Figure 2 This is a schematic diagram of the structure of the language feature embedding model in an embodiment of the present invention;

[0147] Figure 3 This is a schematic diagram of the structure of the government document error correction model in an embodiment of the present invention. Detailed Implementation

[0148] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0149] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0150] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0151] like Figure 1 As shown, this embodiment provides a method for correcting errors in government documents, including:

[0152] Building and training a government document error correction model includes the following steps:

[0153] Step A1: Obtain relevant government documents as the original long government document.

[0154] Step A2: Perform data preprocessing on the original long government affairs text to obtain the government affairs short sentence dataset DS1.

[0155] Step A3: Perform government affairs text obfuscation on the government affairs short sentence dataset DS1 to obtain the obfuscated-correct text pair dataset DS2.

[0156] Step A4: Perform spell confusion and phonetic extraction on the text in the confusion-correct text pair dataset DS2, and calculate the glyph and phonetic matching features of the samples before and after confusion to obtain the glyph and phonetic dataset DS3.

[0157] Step A5: Integrate the obfuscated-correct text pair dataset DS2 with the glyph and phonetic dataset DS3 to obtain the overall error correction dataset DS4.

[0158] Step A6: Train the language feature embedding model using the DS3 glyph and phonology dataset. emb .

[0159] Step A7: Embed language features into the model emb The embedded model used as the error correction model for government documents is trained using the overall error correction dataset DS4. fix The text returned by the government document error correction model is the final error correction result.

[0160] After training the government text error correction model, the government text to be detected is preprocessed by segmentation into sentences to obtain multiple short sentences to be detected. These sentences are then input into the government text error correction model one by one to obtain the error correction results. Finally, the obtained error correction results are used to replace the erroneous short sentences to form the corrected government text.

[0161] This embodiment provides a training method for a government text error correction model that automatically generates datasets and uses a special self-supervised task to strengthen the embedding model. The implementation is as follows: 1. Collect training samples from public websites; 2. Preprocess the training samples, including phonetic annotation, government rule obfuscation, and misspelling obfuscation; 3. Use the generated special dataset to perform self-supervised training on the language embedding model, with training objectives including glyph matching recognition, phonetic matching recognition, and language feature restoration; 4. Use the trained language embedding model as the embedding layer of the overall model and train the overall GPT-like model using the dataset. This application combines an automatic dataset generation method, a self-supervised training method for the embedding model, and a large language generation model architecture. It addresses the need for large-scale, high-quality training data for language models, performs language embedding from glyph and phonetic features, designs a specialized self-supervised training method to improve data utilization and learning depth, and finally uses a more powerful deep learning model architecture to build the overall model. The relevant content involved in the training process of the government text error correction model is further detailed below.

[0162] In step A1, government-related texts can be obtained from public websites at all levels of local and central government, including government texts on various topics.

[0163] Step A2 specifically includes:

[0164] Step A21: Using commas, periods, semicolons, line breaks, and other symbols that indicate sentence breaks as delimiters, divide the acquired original long government documents into multiple short sentences, obtaining a short sentence set {s1, s2, ..., s...}. n}, where n is the number of samples in the short sentence set.

[0165] Step A22: In the short sentence set {s1, s2, ..., s...} n In the code, each short sentence is prefixed with a sentence identifier and terminating with a sentence identifier, as shown below:

[0166] S i =concat([BOS], s i , [EOS])

[0167] Among them, S i This is a sample of concatenated short sentences, where concat(·) is the concatenation function, and s i Let [i] be the i-th short sentence sample in the short sentence set. [BOS] is the sentence beginning identifier, indicating the start of the sentence, and [EOS] is the sentence ending identifier, indicating the end of the sentence.

[0168] This yields the dataset of government affairs phrases required for training: DS1 = {S1, S2, ..., S} n}

[0169] Step A3 is an automatic obfuscated text generation method driven by the purpose of correcting government documents; the generated obfuscated texts will be paired one-to-one with the original correct government documents to form an obfuscated-correct text pair dataset DS2 = {C1, C2, ..., C...} k This data is used to subsequently generate other datasets and train error correction models; specifically, it includes:

[0170] Step A31: Set the set of rules for obfuscating government documents: Rule = {R1, R2, ..., R...} m The obfuscation rules are expressed as follows:

[0171] R j =Fake j →Real j

[0172] Among them, R j For the j-th obfuscation rule in the set of government document obfuscation rules, Real j This is a fragment of a real government document; Fake j In actual writing, it is often used in conjunction with Real. j The obfuscated government text fragments, j = 1, 2, ..., m, where m is the number of samples in the set of government text obfuscation rules.

[0173] Step A32: Obfuscate the real government short sentences in the government short sentence dataset DS1 according to the obfuscation rules in the government text obfuscation rule set Rule, forming the initial obfuscated government short sentence dataset {S′1, S′2, ..., S′ l}; Among them, confusing government affairs phrases are represented as follows:

[0174] S′ i =S i .replace(Real t Fake t )

[0175] Among them, S′ i To generate confusing government policy phrases, S i Real government short sentences from the DS1 dataset are real government short sentences. t and Fake t The obfuscation rules are R respectively. t The dataset contains real and obfuscated government text fragments. x.replace(y, z) is a replacement function that replaces fragment y in string x with fragment z; i = 1, 2, ..., l, where l is the number of samples in the initial obfuscated government short sentence dataset.

[0176] Step A33: Combine the generated obfuscated government phrases with their corresponding real government phrases to form the initial obfuscated-correct text pair dataset ds2 = {C1, C2, ..., C...} l The obfuscated-correct text pair is represented as follows:

[0177] C i =(S′) i S i )

[0178] Among them, C i To obfuscate-correct text pairs, S′ i To confuse short government phrases, S i S′ i Corresponding real government policy phrases.

[0179] Step A34: To enable the model to learn the ability to distinguish error-free government documents during training, add error-free text pairs to the initial confused-correct text pair dataset ds2, resulting in the confused-correct text pair dataset DS2 = {C1, C2, ..., C...} k The added error-free text pairs are represented as follows:

[0180] C q =(S′) q S q )

[0181] Among them, C q For error-free text pairs, S′ q S is a collection of obfuscated government phrases within an error-free text pair. q For error-free text pairs S′ q The corresponding real government affairs phrases, and S′ q =S q That is, in an error-free text pair, the obfuscated government phrase is equivalent to the corresponding real government phrase, q = l+1, l+2, ..., k, where k is the number of samples in the obfuscated-correct text pair dataset; the obfuscated-correct text pair dataset DS2 = {C1, C2, ..., C...} k The corresponding dataset is a collection of obfuscated government policy phrases {S′1, S′2, ..., S′}. l S′ l+1 ,…,S′ k}

[0182] Step A4 involves performing misspelling and pinyin annotation on the confused text in dataset DS2 to obtain general error-corrected samples; calculating the similarity features of the pronunciation and character shape of the samples before and after confusion; and organizing the samples to obtain the character shape and pronunciation dataset DS3 = {Q1, Q2, ..., Q...} k Specifically, it includes:

[0183] Step A41: Extract all characters from the obfuscated-correct text pair dataset DS2 and remove duplicates, then construct a character vocabulary. l sym For character vocabulary V sym The length.

[0184] Step A42: Using the cnchar library in JavaScript, based on the character dictionary V... sym Constructing a character phonetic dictionary The key-value pairs are represented as follows:

[0185] kv h =Sym h →Py h

[0186] Among them, kv h Sym is the h-th key-value pair in the character phonetic dictionary. h For character vocabulary V sym Characters in Py h For the character Sym h The corresponding pronunciation.

[0187] The method for obtaining the pronunciation of Chinese characters is as follows:

[0188] Py h = cnchar.spell(Sym h )

[0189] Among them, Sym h For Chinese characters, cnchar.spell(·) is a Pinyin function provided by the cnchar library, which returns a Pinyin string as its parameter.

[0190] The pronunciation of non-Chinese characters is replaced by the character itself, as shown below:

[0191] Py h =Sym h

[0192] Step A43: Extract the obfuscated government phrase dataset {S′1, S′2, ..., S′} from the obfuscated-correct text pair dataset DS2. k Each confused government affairs phrase in the dataset is processed to remove typos, resulting in a general confused phrase dataset {S″1, S″2, ..., S″}. k}; where, the general phrase for confusion is represented as follows:

[0193] S″ i =confuse(S′ i )

[0194] Among them, S″ iTo confuse common short phrases, S′ i To obfuscate short government sentences, `confuse(·)` is a typo obfuscation function. This function simulates common typo generation scenarios in real-world situations, including three cases: similar-looking characters, homophones, and random input. The specific implementation method is as follows:

[0195] S′=[w1,w2,…w s ]

[0196] S″=[w′1,w′2,…w′ s ]=confuse(S′)={wordConfuse(w i )|i=1,2,…,s}

[0197] Where S′ is the original character sequence consisting of s characters, which is the obfuscated government affairs phrase; S″ is the character sequence obtained after obfuscating each character in the original character sequence, which is the obfuscated general phrase; and wordConfuse(·) is the character obfuscation function, which is implemented as follows:

[0198] r = random()

[0199]

[0200] Among them, random() is a random number function that returns a floating-point number with a range of (0, 1); symConfuse(·) is a similar-looking character confusion function that returns a similar-looking character when the input character is a Chinese character, and randomly returns a non-Chinese character from the character dictionary when the input character is a non-Chinese character; pyConfuse(·) is a homophone confusion function that returns a homophone when the input character is a Chinese character, and returns the original character when the input character is a non-Chinese character; and randConfuse(·) is a random character confusion function that randomly returns a character from the character dictionary.

[0201] Step A44: Use the character phonetic dictionary S2P to parse the obfuscated government affairs short sentence dataset {S′1, S′2, ..., S′...} k} is mapped to a confused government phonetic dataset {P′1, P′2, ..., P′ k} will obfuscate the general short sentence dataset {S″1, S″2, ..., S″}. k} is mapped to a confused general phonetic dataset {P″1, P″2, ..., P″} k}

[0202] Step A45: Compare the character sequences S′=[w1, W2, … w1] before and after the typo obfuscation process. s ] and S″=[w′1, w′2,…w′ sThe corresponding glyph matching degree sequence M is calculated. sym =[m sym,1 m sym,2 , ...m sym，s ]; where m sym，s Represents the character w s With the character w′ s The glyph matching degree is calculated using the following formula:

[0203] When w i With w′ i When all characters are Chinese characters:

[0204]

[0205] Among them, getStrokes(·) is the stroke acquisition function, which returns the stroke sequence of Chinese characters; match(x, y) is the matching element calculation function, which returns the number of identical elements in sequence x and sequence y; and len(·) is the sequence length calculation function, which returns the number of elements in the input sequence.

[0206] When w i With w′ i When there is only one Chinese character in the m sym,i =0.

[0207] When w i With w′ i When neither of them are Chinese characters, m sym,i =0.8.

[0208] Step A46: Compare the pronunciation sequences P′ = [p1, p2, ... p1] before and after the typo confusion treatment. s ] and P″=[p′1,p′2,…p′ s The corresponding phonetic matching degree sequence M is calculated. py =[m py,1 m py,2 , ...m py,s ]; where p i Represents the character w i The corresponding pronunciation is p′ i Represents the character w′ i The corresponding pronunciation is m py，s Indicates the pronunciation p s With the pronunciation p′ s The formula for calculating the phonetic matching degree is as follows:

[0209] When w i With w′ i When all characters are Chinese characters:

[0210]

[0211] When w i With w′ i When not all characters are Chinese characters, m py，i =0.

[0212] Step A47: Combine the character sequence S′ and phonetic sequence P′ before typo obfuscation, the character sequence S″ and phonetic sequence P″ after typo obfuscation, and the character shape matching degree sequence M. sym The sequence M of word pronunciation matching degree py The merged glyph and phonetic data sample Q is represented as follows:

[0213] Q = [(S′,S″,M] sym ), (P′, P″, M py )]

[0214] Integrate all data samples to form the character shape and pronunciation dataset DS3 = {Q1, Q2, ..., Q...} k}

[0215] In step A5, the obfuscated-correct text pair dataset DS2 and the glyph and phonetic dataset DS3 are integrated to form a holistic error correction dataset DS4 = {C′1, C′2, ..., C′}. k The overall error correction dataset DS4 has the same sample format as DS2, which is the confusion-correct text pair format C′. i =(S″′ i S i ), where S i To obfuscate the true government phrases in the DS2 dataset, S″′ i The mixed obfuscated government phrases are defined as follows: the mixed obfuscated government phrases are 50% likely to be the obfuscated government phrases S′ in the obfuscated-correct text pair dataset DS2. i That is, the confused government short phrase S′ in the confused government short phrase dataset. i There is a 50% probability that it corresponds to the confused general phrase S″ in the glyph and phonetic dataset DS3. i The overall error correction dataset DS4 covers data on normal sentences, confused government short sentences, and confused general short sentences in government text error correction scenarios, which more realistically simulates the data distribution under actual conditions and is conducive to the error correction model to learn comprehensively.

[0216] Step A6 models the features of each text character from two dimensions: glyph and phonology. Simultaneously, it utilizes a single-character error correction pre-training task to enhance the model's learning of character features, thereby training and obtaining a language feature embedding model. emb Specifically, it includes:

[0217] Step A61: Construct the initial language feature embedding model The model includes a glyph feature embedding layer, a phonetic feature embedding layer, a language feature extraction layer, a glyph matching degree calculation layer, a phonetic matching degree calculation layer, and a language feature reconstruction layer; the computable parameters of each layer in the initial model are as follows:

[0218] The glyph feature embedding layer contains parameters l sym For character vocabulary V sym Length, d m The number of embedding feature dimensions;

[0219] The phonetic feature embedding layer contains parameters

[0220] The language feature extraction layer contains parameters

[0221] The glyph matching degree calculation layer contains parameters

[0222] The phonetic matching calculation layer contains parameters.

[0223] The language feature reconstruction layer contains parameters

[0224] Step A62: Take samples Q = [(S′, S″, M] from dataset DS3. sym ), (P′, P″, M py )], input S′ and S″ into the glyph feature embedding layer to obtain the true glyph feature Feature′ sym With input glyph features sym , means as follows:

[0225] OH S′ =onehot(Index(V) sym ,S′),l sym )

[0226] OH S′ =onehot(Index(V) sym ,S″),l sym )

[0227]

[0228] Where i is the current training iteration number of the model, and let s be the number of elements in both S′ and S″. The function Index(x, y) returns an integer sequence of the same length as the sequence y, where each element is the index of the element at the same position in y in the sequence x. The function onehot(x, l) returns the one-hot matrix obtained by one-hot encoding the integer sequence x.

[0229] Step A63: Input the phonetic feature embedding layer with P′ and P″ to obtain the true phonetic feature Feature′. py Input phonetic features py , means as follows:

[0230] OH P′ =onehot(Index(V) py ,P′),l py )

[0231] OH P″ =onehot(Index(V) py ,P″),l py )

[0232]

[0233] Let the number of elements in P′ and P″ both be s, then

[0234] Step A64: Select Feature″ sym With Feature py The concatenated input embedding feature Feature′ is input into the language feature extraction layer to obtain the abstract language feature Feature″, as follows:

[0235] Feature′=concat(Feature″ sym Feature″ py )

[0236]

[0237] Att = softmax(score)

[0238] Feature″=V×Att

[0239] in, The function softmax(s) is a normalization function that maps the input matrix to the exponential domain and then performs column normalization to obtain matrix s′, as shown below:

[0240]

[0241] Step A65: Input the Feature″ into the glyph matching degree calculation layer to obtain the model glyph matching degree sequence M′. sym , means as follows:

[0242]

[0243] M′ sym =sigmoid(MF sym )

[0244] in, The function sigmoid(M) is a normalization function that maps each element in the input sequence to the range (0, 1) to obtain the sequence M′, as shown below:

[0245]

[0246] Step A66: Input the abstract language feature "Feature″" into the phonetic matching degree calculation layer to obtain the model phonetic matching degree sequence M′. py , means as follows:

[0247]

[0248] M′ py =sigmoid(MF py )

[0249] in,

[0250] Step A67: Input glyph features. sym Model glyph matching degree sequence M′ sym Input phonetic features py Matching sequence M′ with the model pronunciation py The input language feature reconstruction layer yields the reconstructed language features, which are represented as follows:

[0251] Feature″′ sym =scale(Feature") sym M′ sym )

[0252] Feature″′ py =scale(Feature") py M′ py )

[0253] IF = concat(Feature″′) sym Feature″′ py )

[0254]

[0255] Att′ = softmax(score′)

[0256] Feature″′=V′×Att′

[0257] in, The function scale(x, y) is a scaling function that scales the elements in x. j Scaling to y j To obtain x′ by multiplying by 1, the formula is as follows:

[0258] x′ j =x j *y j

[0259] Step A68: Based on the model glyph matching degree sequence M′ sym Model phonetic matching sequence M′ py , Reconstructing language features (Feature″′), and character shape matching sequence M sym Phonetic matching sequence M py Features of real glyphs sym Authentic pronunciation features py The model loss L is calculated by iterating the model parameters using gradient descent, as shown below:

[0260] Feature = concat(Feature') sym Feature′ py )

[0261] L m_s =MSE(M sym M′ sym )

[0262] L m_p =MSE(M py M′ py )

[0263] L f =MSE(Feature,Feature″′)

[0264] L = L m_s +L m_p +L f

[0265] W (i+1) =backwardByLoss(L, W) (i) )

[0266] Among them, L m_s To calculate the loss for font matching, L m_p To calculate the loss for the phonetic matching degree, L f For the embedding feature reconstruction loss, W (i)For all updatable parameters of the model after the i-th iteration, the function MSE(X, Y) returns the root mean square error of the input object, where W (i) For all updatable parameters of the model after the i-th iteration, the function MSE(X, Y) returns the root mean square error of the input object, as shown below:

[0267]

[0268] The function backwardByLoss(L, W) is the backward update function, which calculates the parameters of the new iteration based on the gradient of the loss function L and chain propagation.

[0269] Step A69: Repeat steps A62-A68 until the model converges or the set number of iterations is reached. The model training is complete, and the trained model is the language feature embedding model. emb Its structure is as follows Figure 2 As shown.

[0270] Step A7 uses a GPT-like model structure to build an encoder-decoder model, embedding the pre-trained language features into the model. emb As the feature embedding layer in the model, the model's error correction capability for various types of government documents is trained using the overall error correction dataset DS4, ultimately yielding the government document error correction model. fix Specifically, it includes:

[0271] Step A71: Construct an initial government document error correction model The model includes a text feature embedding layer, an encoder, and a decoder; wherein, the text feature embedding layer is the language feature embedding model trained in step A6. emb The encoder is used to convert text sequence data into a computer-computable feature vector sequence. The encoder is composed of multiple layers of the same structure and is used to extract high-level features of the input text. The decoder is composed of multiple layers of the same structure and is connected to a classification network. It is used to autoregressively output the sequence bit by bit by combining the high-level features given by the encoder with the information of the output sequence.

[0272] Step A72: Take a sample C′ = (S″′, S) from the overall error correction dataset DS4, where S″′ is the input text and S is the correct text corresponding to S″′; input the input text S″′ into the text feature embedding layer to obtain the input text embedding feature. input , means as follows:

[0273] P″′=transByDict(S2P,S″′)

[0274] Feature input =Model emb(S″′,P″′)

[0275] Here, S2P is the constructed character phonetic dictionary, the function transByDict(S2P, S″′) is the dictionary conversion function, which maps the elements in the input text S″′ one by one according to the key-value relationships in the dictionary S2P, and P′″ is the converted input phonetic sequence. s is the length of the input text S″′, d m is the dimension of the embedding vector.

[0276] Step A73: Embed the input text with features input Input the encoder to obtain high-level text features. en .

[0277] Step A74: Take the beginnings of the input text S″′ of different lengths as the output sequence, and input the output sequence into the text feature embedding layer as in step A72 to obtain the output text embedding feature. output And then with Feature en The data are input together into the decoder to obtain the model's distribution prediction sequence result for the output sequence, as shown below:

[0278] S″′ ，i =S″′[:i]

[0279] P″′ ，i =transByDict(S2P,S″′) ，i )

[0280] Feature output =Model emb (P″′ ，i ,S″′ ，i )

[0281] Feature de =Decoder(Feature) en Feature output )

[0282] result i+1 =Classifier(Feature) de,i )

[0283] Where S″′[:i] represents the first i elements of the input text S″′. Let $\frac{i}{i+1}$ be the i-th element of the distribution prediction sequence $\frac{result}{i}$, representing the probability distribution of the modification result of the (i+1)-th element of the input text $S″′$. i+1 The j-th element This indicates that the original character will be modified to the character table V. sym The probability of finding the j-th character;

[0284] Step A75: Compare the distribution prediction sequence result with the true output sequence, i.e. the correct text S, element by element, calculate the cross-entropy loss of the classification task, and iterate the model parameters using the gradient descent method.

[0285] Step A76: Repeat steps A72-A75 until the model converges or the set number of iterations is reached. The model training is complete, and the trained model is the government document error correction model. fix Its structure is as follows Figure 3 As shown.

[0286] After training the government document error correction model, the text error correction is performed according to the following steps:

[0287] Step B1: The user uploads the text T to be detected. After preprocessing T by sentence segmentation, the sample set to be detected is obtained as {S0, S1, ..., S}. n The processing method is as described in step A2;

[0288] Step B2: Use the character phonetic dictionary S2P to map the phonetics of the samples, obtaining the sample phonetic sequence set {P0, P1, ..., P} corresponding to the sample set. n The specific formula is as follows:

[0289] P i =transByDict(S2P,S i )

[0290] The method for obtaining the character phonetic dictionary S2P is as described in step A42;

[0291] Step B3: Input a text sample S and its corresponding phonetic sequence sample P into the embedding layer of the government text error correction model. emb To obtain the text embedding feature input The formula is as follows:

[0292] Feature input =Model emb (S, P)

[0293] Step B4: Embed text features imput Input Model Encoder en Obtain high-level text features en The formula is as follows:

[0294] Feature en =Model en(Feature input )

[0295] Step B5: Construct the initial return sequence O (0) This sequence contains only one sentence-initial identifier [BOS], expressed as follows:

[0296] O (0) =[′[BOS]′]

[0297] Step B6: Same as steps B2-B3, for the returned sequence O (i) Perform phonetic mapping and feature embedding to obtain the output text embedding features. output ;

[0298] Step B7: Extract high-level text features en With output text embedding features output Input Model Decoder de The model's distribution prediction for the (i+1)th element of the output sequence is obtained. The character number with the highest generation probability is then added to the output sequence. The specific formula is as follows:

[0299] result i+1 =Model de (Feature en Feature output )

[0300]

[0301] O (i+1) =concat(O (i) , [o i+1 ])

[0302] Step B8: Repeat steps B6-B7 until the next character number o is generated. i+1 When the sentence-ending identifier [EOS] in the corresponding vocabulary or the output sequence reaches the upper limit of the model length, the output sequence O = O. (i) This is the error correction result of the model for sample S;

[0303] Step B9: Repeat steps B3-B8 for each sample in the set to be tested to obtain the error correction result set {O0, O1, ..., O...} n The error correction results are concatenated into a short text to obtain the error correction result T′ of the text T to be detected.

[0304] This embodiment also provides a government document error correction system, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-described method steps. Specifically, the government document error correction system includes:

[0305] Government Document Collection Module; used to collect original long government documents required for model training;

[0306] Data processing module: The original long text is segmented into sentences to obtain the government affairs short sentence dataset DS1; common government affairs text obfuscation processing is applied to dataset DS1 to obtain the obfuscated-correct text pair dataset DS2; general error correction obfuscation processing and feature annotation are performed on the text in dataset DS2 to obtain the phonetic and glyph dataset DS3; datasets DS2 and DS3 are integrated to obtain the overall error correction dataset DS4.

[0307] Model training module: Uses the DS3 dataset to train and obtain the language feature embedding model. emb The government document error correction model Model was trained using the overall error correction dataset DS4. fix ;

[0308] Dataset storage module: Stores all datasets saved and used during the training process;

[0309] Model training weight management module: manages and stores different training weights obtained from different training data, training strategies, model hyperparameters and other factors; loads the corresponding training weights for model training and model inference.

[0310] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0311] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0312] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0313] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0314] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for correcting errors in government documents, characterized in that, include: Building and training a government document error correction model includes the following steps: Step A1: Obtain relevant government documents as the original long government document; Step A2: Preprocess the original long government documents to obtain the short government documents dataset DS1; Step A3: Perform government affairs text obfuscation on the government affairs short sentence dataset DS1 to obtain the obfuscated-correct text pair dataset DS2; Step A4: Perform misspelling and pronunciation acquisition on the text in the DS2 dataset of confused-correct text pairs, and calculate the glyph and pronunciation matching features of the samples before and after confusion to obtain the glyph and pronunciation dataset DS3; Step A5: Integrate the obfuscated-correct text pair dataset DS2 with the glyph and phonetic dataset DS3 to obtain the overall error correction dataset DS4; Step A6: Train the language feature embedding model using the DS3 glyph and phonology dataset. emb ; Step A7: Transfer the model emb The embedded model used as the error correction model for government documents is trained using the overall error correction dataset DS4. fix ; The government text to be detected is preprocessed by segmentation into sentences to obtain multiple short sentences to be detected. These sentences are then input into the government text error correction model one by one to obtain the error correction results. Finally, the error correction results are used to replace the erroneous short sentences to form the corrected government text. Step A3 specifically includes: Step A31: Set the set of rules for obfuscating government documents: Rule = {R1, R2, ..., R} m The obfuscation rules are expressed as follows: R j =Fake j →Real j Among them, R j For the j-th obfuscation rule in the set of government document obfuscation rules, Real j This is a fragment of a real government document; Fake j In actual writing, it is often used in conjunction with Real. j The obfuscated government text fragments, j = 1, 2, ..., m, where m is the number of samples in the set of government text obfuscation rules; Step A32: Obfuscate the real government short sentences in the government short sentence dataset DS1 according to the obfuscation rules in the government text obfuscation rule set Rule, forming the initial obfuscated government short sentence dataset {S1′,S2′,…,S l ′}; Among them, confusing government affairs phrases are represented as follows: S′ i =S i .replace(Real t ,Fake t ) Among them, S′ i To generate confusing government policy phrases, S i Real government short sentences from the DS1 dataset are real government short sentences. t and Fake t The obfuscation rules are R respectively. t The dataset contains real and obfuscated government text fragments. x.replace(y,z) is a replacement function that replaces fragment y in string x with fragment z; i = 1, 2, ..., l, where l is the number of samples in the initial obfuscated government short sentence dataset. Step A33: Combine the generated obfuscated government policy phrases with their corresponding real government policy phrases to form the initial obfuscated-correct text pair dataset ds2={C1,C2,…,C l The obfuscated-correct text pair is represented as follows: C i =(S′ i ,S i ) Among them, C i To obfuscate-correct text pairs, S′ i To confuse short government phrases, S i S′ i Corresponding real government policy phrases; Step A34: To enable the model to learn the ability to distinguish error-free government documents during training, add error-free text pairs to the initial confused-correct text pair dataset ds2, resulting in the confused-correct text pair dataset DS2 = {C1, C2, ..., C...} k The added error-free text pairs are represented as follows: C q =(S′ q ,S q ) Among them, C q For error-free text pairs, S′ q S is a collection of obfuscated government phrases within an error-free text pair. q For error-free text pairs S′ q The corresponding real government affairs phrases, and S′ q =S q That is, in an error-free text pair, the obfuscated government phrase is equivalent to the corresponding real government phrase, q = l+1, l+2, ..., k, where k is the number of samples in the obfuscated-correct text pair dataset; the obfuscated-correct text pair dataset DS2 = {C1, C2, ..., C...} k The corresponding dataset is a collection of obfuscated government policy phrases {S′1,S′2,…,S′}. l ,S′ l+1 ,…,S′ k } 2. The method for correcting errors in government documents according to claim 1, characterized in that, Step A2 specifically includes: Step A21: Using symbols indicating sentence breaks as delimiters, the acquired original long government documents are divided into multiple short sentences, resulting in a short sentence set {s1, s2, ..., s...} n }, where n is the number of samples in the short sentence set; Step A22: In the short sentence set {s1,s2,…,s…} n In the code, each short sentence is prefixed with a sentence identifier and terminating with a sentence identifier, as shown below: S i =concat([BOS],s i ,[EOS]) Among them, S i This is a sample of concatenated short sentences, where concat(·) is the concatenation function, and s i Let [BOS] be the i-th short sentence sample in the short sentence set, where [BOS] is the sentence beginning identifier, indicating the start of the sentence, and [EOS] is the sentence ending identifier, indicating the end of the sentence. This yields the dataset of government policy phrases required for training: DS1 = {S1, S2, ..., S} n } 3. The method for correcting errors in government documents according to claim 1, characterized in that, Step A4 specifically includes: Step A41: Extract all characters from the obfuscated-correct text pair dataset DS2 and remove duplicates, then construct a character vocabulary. l sym For character vocabulary V sym Length; Step A42: Based on the character dictionary V sym Constructing a character phonetic dictionary The key-value pairs are represented as follows: kv h =Sym h →Py h Among them, kv h Sym is the h-th key-value pair in the character phonetic dictionary. h For character vocabulary V sym Characters in Py h For the character Sym h The corresponding pronunciation; the method for obtaining the pronunciation of Chinese characters is as follows: Py h =cnchar.spell(Sym h ) Among them, Sym h For Chinese characters, cnchar.spell(·) is a Pinyin function provided by the cnchar library, which returns a Pinyin string as its parameter. The pronunciation of non-Chinese characters is replaced by the character itself, as shown below: Py h =Sym h Step A43: Extract the obfuscated government phrase dataset {S′1,S′2,…,S′} from the obfuscated-correct text pair dataset DS2. k Each confused government affairs phrase in the dataset is processed to remove typos, resulting in a general confused phrase dataset {S″1,S″2,…,S″}. k }; where, the general phrase for confusion is represented as follows: S″ i =confuse(S′ i ) Among them, S″ i To confuse common short phrases, S′ i To obfuscate short government sentences, `confuse(·)` is a typo obfuscation function. This function simulates common typo generation scenarios in real-world situations, including three cases: similar-looking characters, homophones, and random input. The specific implementation method is as follows: S′=[w1,w2,…w s ] S″=[w′1,w′2,…w′ s ]=confuse(S′)={wordConfuse(w i )|i=1,2,…,s} Where S′ is the original character sequence consisting of s characters, which is the obfuscated government affairs phrase; S″ is the character sequence obtained after obfuscating each character in the original character sequence, which is the obfuscated general phrase; and wordConfuse(·) is the character obfuscation function, which is implemented as follows: r = random() Among them, random() is a random number function that returns a floating-point number with a range of (0,1); symConfuse(·) is a similar-looking character confusion function that returns a similar-looking character when the input character is a Chinese character, and randomly returns a non-Chinese character from the character dictionary when the input character is a non-Chinese character; pyConfuse(·) is a homophone confusion function that returns a homophone when the input character is a Chinese character, and returns the original character when the input character is a non-Chinese character; and randConfuse(·) is a random character confusion function that randomly returns a character from the character dictionary. Step A44: Use the character phonetic dictionary S2P to parse the obfuscated government affairs short sentence dataset {S′1,S′2,…,S′ k } is mapped to a confused government affairs pronunciation dataset {P′1,P′2,…,P′ k } will obfuscate the general short sentence dataset {S″1,S″2,…,S″}. k } is mapped to a confused general phonetic dataset {P″1,P″2,…,P″} k }; Step A45: Compare the character sequences S′=[w1,w2,…w1,w2,…w1,w2,…w2,w3,w4,w5,w6,w7,w8,w9,w1,w2,w9,w1,w2,w3,w4,w5,w6,w7,w8 ... s ] and S″=[w′1,w′2,…w′ s The corresponding glyph matching degree sequence M is calculated. sym =[m sym,1 ,m sym,2 ,…m sym,s ]; where m sym,s Represents the character w s With the character w′ s The glyph matching degree is calculated using the following formula: When w i With w′ i When all characters are Chinese characters: Among them, getStrokes(·) is the stroke acquisition function, which returns the stroke sequence of Chinese characters; match(x,y) is the matching element calculation function, which returns the number of common elements in sequence x and sequence y; and len(·) is the sequence length calculation function, which returns the number of elements in the input sequence. When w i With w′ i When there is only one Chinese character in the m sym,i =0; When w i With w′ i When neither of them are Chinese characters, m sym,i =0.8; Step A46: Compare the pronunciation sequences P′ = [p1, p2, ... p1] before and after the typo confusion treatment. s ] and P″=[p′1,p′2,…p′ s The corresponding phonetic matching degree sequence M is calculated. py =[m py,1 ,m py,2 ,…m py,s ]; where p i Represents the character w i The corresponding pronunciation is p′ i Represents the character w′ i The corresponding pronunciation is m py,s Indicates the pronunciation p s With the pronunciation p′ s The formula for calculating the phonetic matching degree is as follows: When w i With w′ i When all characters are Chinese characters: When w i With w′ i When not all characters are Chinese characters, m py,i =0; Step A47: Combine the character sequence S′ and phonetic sequence P′ before typo obfuscation, the character sequence S″ and phonetic sequence P″ after typo obfuscation, and the character shape matching degree sequence M. sym The sequence M of word pronunciation matching degree py The merged glyph and phonetic data sample Q is represented as follows: Q=[(S′,S″,M sym ),(P′,P″,M py )] Integrating all data samples, a character shape and pronunciation dataset DS3 = {Q1, Q2, ..., Q} is formed. k } 4. The method for correcting errors in government documents according to claim 3, characterized in that, In step A5, the obfuscated-correct text pair dataset DS2 and the glyph and phonetic dataset DS3 are integrated to form a holistic error correction dataset DS4 = {C′1, C′2, ..., C′}. k The overall error correction dataset DS4 has the same sample format as DS2, which is the confusion-correct text pair format C′. i =(S″′ i ,S i ), where S i To obfuscate the true government phrases in the DS2 dataset, S″′ i The mixed obfuscated government phrases are defined as follows: the mixed obfuscated government phrases are 50% likely to be the obfuscated government phrases S′ in the obfuscated-correct text pair dataset DS2. i There is a 50% probability that it corresponds to the confused general phrase S″ in the glyph and phonetic dataset DS3. i The overall error correction dataset DS4 covers data on normal sentences, confused government short sentences, and confused general short sentences in government text error correction scenarios.

5. The method for correcting errors in government documents according to claim 1, characterized in that, Step A6 specifically includes: Step A61: Construct the initial language feature embedding model The model comprises a glyph feature embedding layer, a phonetic feature embedding layer, a language feature extraction layer, a glyph matching degree calculation layer, a phonetic matching degree calculation layer, and a language feature reconstruction layer; the computable parameters of each layer are as follows: The glyph feature embedding layer contains parameters l sym For character vocabulary V sym Length, d m The number of embedding feature dimensions; The phonetic feature embedding layer contains parameters The language feature extraction layer contains parameters The glyph matching degree calculation layer contains parameters The phonetic matching calculation layer contains parameters. The language feature reconstruction layer contains parameters Step A62: Take sample Q = [(S′,S″,M] from dataset DS3 sym ),(P′,P″,M py )], input S′ and S″ into the glyph feature embedding layer to obtain the true glyph feature Feature′ sym With input glyph features sym , means as follows: OH S′ =onehot(Index(V sym ,S′),l sym ) OH S″ =onehot(Index(V sym ,S″),l sym ) Where i is the current training iteration number of the model, and let s be the number of elements in both S′ and S″. The function Index(x,y) returns an integer sequence of the same length as the sequence y, where each element is the index of the element at the same position in y in the sequence x. The function onehot(x,l) returns the one-hot matrix obtained by one-hot encoding the integer sequence x. Step A63: Input the phonetic feature embedding layer with P′ and P″ to obtain the true phonetic feature Feature′. py Input phonetic features py , means as follows: OH P′ =onehot(Index(V py ,P′),l py ) OH P″ =onehot(Index(V py ,P″),l py ) Let the number of elements in P′ and P″ both be s, then Step A64: Select Feature″ sym With Feature py The concatenated input embedding feature Feature′ is input into the language feature extraction layer to obtain the abstract language feature Feature″, as follows: Feature′=concat(Feature″ sym ,Feature″ py ) Att = softmax(score) Feature″=V×Att in, The function softmax(s) is a normalization function that maps the input matrix to the exponential domain and then performs column normalization to obtain matrix s′, as shown below: Step A65: Input the Feature″ into the glyph matching degree calculation layer to obtain the model glyph matching degree sequence M′. sym , means as follows: M′ sym =sigmoid(MF sym ) in, The function sigmoid(M) is a normalization function that maps each element in the input sequence to the range (0,1) to obtain the sequence M′, as shown below: Step A66: Input the abstract language feature "Feature″" into the phonetic matching degree calculation layer to obtain the model phonetic matching degree sequence M′. py , means as follows: M′ py =sigmoid(MF py ) in, Step A67: Input glyph features. sym Model glyph matching degree sequence M′ sym Input phonetic features py Matching sequence M′ with the model pronunciation py The input language feature reconstruction layer yields the reconstructed language features, which are represented as follows: Feature″′ sym =scale(Feature″′ sym ,M′ sym ) Feature″′ py =scale(Feature″ py ,M′ py ) IF=concat(Feature″′ sym ,Feature″′ py ) Feature″′=V′×Att′ in, The function scale(x,y) is a scaling function that scales the elements of x. j Scaling to y j To obtain x′ by multiplying by 1, the formula is as follows: x′ j =x j *y j Step A68: Based on the model glyph matching degree sequence M′ sym Model phonetic matching sequence M′ py , Reconstructing language features (Feature″′), and character shape matching sequence M sym Phonetic matching sequence M py Features of real glyphs sym Authentic pronunciation features py The model loss L is calculated by iterating the model parameters using gradient descent, as shown below: Feature=concat(Feature′ sym ,Feature′ py ) L m_s =MSE(M sym ,M′ sym ) L m_p =MSE(M py ,M′ py ) L f =MSE(Feature,Feature″′) L=L m_s +L m_p +L f IN (i+1) =backwardByLoss(L,W (i) ) Among them, L m_s To calculate the loss for font matching, L m_p To calculate the loss for the phonetic matching degree, L f For the embedding feature reconstruction loss, W (i) For all updatable parameters after the i-th iteration of the model, the function MSE(X,Y) returns the root mean square error of the input object, and the function backwardByLoss(L,W) is the backward update function, which calculates the parameters of the new iteration based on the gradient of the loss function L and chain propagation. Step A69: Repeat steps A62-A68 until the model converges or the set number of iterations is reached. The model training is complete, and the trained model is the language feature embedding model. emb .

6. The method for correcting errors in government documents according to claim 1, characterized in that, Step A7 specifically includes: Step A71: Construct an initial government document error correction model The model includes a text feature embedding layer, an encoder, and a decoder; wherein, the text feature embedding layer is the language feature embedding model trained in step A6. emb The encoder is used to convert text sequence data into a computer-computable feature vector sequence. The encoder is composed of multiple layers of the same structure and is used to extract high-level features of the input text. The decoder is composed of multiple layers of the same structure and is connected to a classification network. It is used to autoregressively output the sequence bit by bit by combining the high-level features given by the encoder and the information of the output sequence. Step A72: Take a sample C′ = (S″′, S) from the overall error correction dataset DS4, where S″′ is the input text and S is the correct text corresponding to S″′; input the input text S″′ into the text feature embedding layer to obtain the input text embedding feature. input , means as follows: P″′=transByDict(S2P,S″′) Feature input =Model emb (S″′,P″′) Here, S2P is the constructed character phonetic dictionary, the function transByDict(S2P, S″′) is the dictionary conversion function, which maps the elements in the input text S″′ one by one according to the key-value relationships in the dictionary S2P, and P″′ is the converted input phonetic sequence. s is the length of the input text S″′, d m is the dimension of the embedding vector; Step A73: Embed the input text with features input Input the encoder to obtain high-level text features. en ; Step A74: Take the beginnings of the input text S″′ of different lengths as the output sequence, and input the output sequence into the text feature embedding layer as in step A72 to obtain the output text embedding feature. output And then with Feature en The data are input together into the decoder to obtain the model's distribution prediction sequence result for the output sequence, as shown below: S″′, i =S″′[:i] P″′, i =transByDict(S2P,S″′, i ) Feature output =Model emb (P″′, i ,S″′, i ) Feature de =Decoder(Feature en ,Feature output ) result i+1 =Classifier(Feature de,i ) Where S″′[:i] represents the first i elements of the input text S″′. Let $\frac{i}{i+1}$ be the i-th element of the distribution prediction sequence $\frac{result}{i}$, representing the probability distribution of the modification result of the (i+1)-th element of the input text $S″′$. i+1 The j-th element This indicates that the original character will be modified to the character table V. sym The probability of finding the j-th character; Step A75: Compare the distribution prediction sequence result with the true output sequence, i.e. the correct text S, element by element, calculate the cross-entropy loss of the classification task, and iterate the model parameters using the gradient descent method. Step A76: Repeat steps A72-A75 until the model converges or the set number of iterations is reached. The model training is complete, and the trained model is the government document error correction model. fix .

7. A government document error correction system, characterized in that, It includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor, which, when executed by the processor, enable the implementation of the steps of the method as described in any one of claims 1-6.

Images