Government affair text error correction method and system

Abstract

The invention relates to a government affair text error correction method and system, and the method comprises the steps: constructing and training a government affair text error correction model: obtaining an original government affair long text, and carrying out the data preprocessing, and obtaining a government affair short sentence data set DS1; performing government affair text confusion processing on the DS1 to obtain a confusion-correct text pair data set DS2; carrying out wrongly written character confusion and character pronunciation acquisition on the text in the DS2 to obtain a font and character pronunciation data set DS3; integrating the DS2 and the DS3 to obtain an overall error correction data set DS4; a DS3 is adopted to train a language feature embedding model Modelemb; adopting DS4 to train a government affair text error correction model Modelfix which takes Modelemb as an embedded model; and performing sentence segmentation preprocessing on a to-be-detected government affair text to obtain multiple to-be-detected government affair short sentences, inputting the multiple to-be-detected government affair short sentences into the government affair text error correction model to obtain an error correction result, and replacing the wrong government affair short sentences to form an error-corrected government affair text. The method and the system can effectively detect and correct the text errors in the government affair text, and have relatively high detection efficiency and accuracy.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a government text error correction method and system. Background Art

[0002] With the rapid advancement of Internet technology, my country's government information platforms are becoming more and more digital and transparent, making it easier for the public to obtain relevant information. Government websites and government media serve as a bridge between the government and the public, and the information they publish must be accurate and compliant. The traditional method of strictly reviewing information content manually is costly and inefficient, and it is difficult to meet actual needs. Therefore, there is an urgent need for an automated and intelligent government information review system to ensure content security.

[0003] The main research direction of text error correction tasks is to improve the learning ability of models and the quality of training data. Existing text error correction algorithms based on deep learning mostly use relevant public data sets for model training and testing. Such data sets are often small in scale and difficult to meet the training requirements of models in actual production environments. At the same time, the huge overhead of data annotation is difficult to solve when crawling large-scale network texts as data sets. The automated data set construction method based on the characteristics of government texts not only meets actual needs but also ensures data quality.

[0004] In the text error correction research based on deep learning, most methods incorporate natural language feature embedding learning into the overall model training. Due to the deviation between the data set and the real data, the model is more likely to overfit. Training the language feature embedding model through reasonable self-supervised pre-training tasks can make the overall model more robust and also greatly improve the utilization of data.

[0005] The remarkable success of AI big models has not only attracted widespread attention in the field of natural language processing, but also highlighted the excellent performance and unique advantages of GPT-like structural models. This type of model, based on the Transformer architecture, can deeply understand the complexity and nuances of language through pre-training on massive text data. The generation ability of GPT-like structural models further improves the quality and readability of text. Using GPT-like structural models for text error correction tasks can bring better results.

[0006] To this end, an invention patent with a publication number of CN118643798A discloses a text error correction method and system based on federated learning. The method includes: constructing a corpus corresponding to each participant to form a corresponding training set; fine-tuning the pre-trained model to obtain the fine-tuned initial model parameters to obtain the initial text error correction model; encrypting the fine-tuned initial model parameters to obtain the encrypted initial model parameters; calculating the similarity of the corpus or scenario tasks of each participant, screening the model parameters to be aggregated, performing secure aggregation, and obtaining the corresponding aggregate model parameters; decrypting the aggregate model parameters, and further updating the initial text error correction model; performing multiple rounds of federated learning to obtain the optimal text error correction model to correct the text to be corrected.

[0007] Another invention patent with publication number CN118468861A discloses a method, system, device and medium for text error correction based on rule matching, the method comprising: obtaining original text data to be corrected, a general dictionary and a confusion word dictionary, and performing preprocessing operations on the original text data to be corrected to obtain input text; performing error detection processing on the input text through the general dictionary and the confusion word dictionary to obtain a suspected error set, and traversing and matching the suspected error set based on the confusion word dictionary; in response to the error in the suspected error set existing in the confusion word dictionary, directly replacing the error with the correct value in the confusion word dictionary; in response to the error in the suspected error set not existing in the confusion word dictionary, constructing a candidate set based on the error, calculating the language perplexity of all replacement words in the candidate set through a preset language model, and selecting the replacement word with the smallest perplexity to replace the error.

[0008] Another invention patent with publication number CN118627495A discloses a text intelligent error correction method, device, electronic device and storage medium, wherein the method includes: embedding the original medical consultation text into a preset matrix layer to obtain a converted medical text; using the encoder in the trained text error correction model to perform text compilation on the converted medical text to obtain a model-readable text; using the judger in the text error correction model to identify abnormal text in the model-readable text to obtain an abnormal text; using the corrector in the text error correction model to perform text correction on the abnormal text to obtain a corrected text; randomly sorting the words and sentences in the corrected text to obtain sequence words and sentences, and extracting the preferred sequence words and sentences in the sequence words and sentences to obtain a preferred text; using the decoder in the trained text error correction model to perform text decoding on the preferred text to obtain a target medical text.

[0009] Another invention patent with publication number CN118643825A discloses a Chinese text error correction method, system, storage medium and electronic device, including: obtaining a large language model, which is used to realize text error detection and text error correction of Chinese text; obtaining a first input indication text, which is used to indicate whether there is a text error in the input text; inputting the first input indication text and the Chinese text to be detected into the large language model to obtain a text error detection result of the Chinese text to be detected; when the text error detection result is that there is a text error, obtaining a second input indication text, which is used to indicate error correction for the text error; inputting the second input indication text and the Chinese text to be detected into the large language model to obtain the error correction Chinese text corresponding to the Chinese text to be detected.

[0010] Another invention patent with publication number CN118194854A discloses a Chinese text correction method based on full-word masking and dependency masking, comprising the following steps: step S1: collecting and constructing a data set; step S2: constructing a text correction model, the text correction model comprising a spelling correction model and a grammar correction model; step S3: importing the erroneous sentence into the spelling correction model in step S2, obtaining the sentence after spelling correction, and training the spelling correction model at the same time; step S4: importing the sentence after spelling correction into the detection network of the grammar correction model in step S2, obtaining the grammatical error type of the sentence after spelling correction, and training the grammar correction model; step S5: importing the sentence of grammatical error type into the correction network in the grammar correction model, performing grammatical correction on the sentence of the obtained grammatical error type, and obtaining the sentence after grammar correction.

[0011] As can be seen from the above, the existing training methods for text error correction models usually adopt the structure of a deep neural network, and then based on a large amount of training data, use the learning objectives of the relevant loss function for training and learning, and finally learn the network weights that can understand the text semantics and use this as a basis to identify and modify text errors for text error correction. However, in the process of actually implementing the training of the government text error correction model, the inventor realized that the existing technology has at least the following problems: 1. The existing training methods for Chinese natural language models use a single-angle text embedding method, which does not fully utilize the characteristics of Chinese characters themselves, which may cause the trained text embedding feature representation ability to be insufficient; 2. The existing general-purpose text error correction model uses a data set obtained from a large range of the network or an existing small-scale data set for model training. There is no low-cost and efficient method to obtain a high-quality error correction data set, resulting in insufficient model training or poor performance in an actual production environment. Summary of the invention

[0012] The object of the present invention is to provide a method and system for correcting government texts, which can effectively detect and correct text errors in government texts and have high detection efficiency and accuracy.

[0013] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a government text error correction method, comprising:

[0014] Building and training a government text error correction model includes the following steps:

[0015] Step A1: Obtaining government affairs related text as original government affairs chief text;

[0016] Step A2: preprocess the original government affairs text to obtain the government affairs short sentence dataset DS1;

[0017] Step A3: Perform government text obfuscation processing on the government sentence dataset DS1 to obtain the obfuscated-correct text pair dataset DS2;

[0018] Step A4: Obfuscate the text in the obfuscated-correct text pair dataset DS2 for incorrect characters and obtain the pronunciation, and calculate the glyph and pronunciation matching characteristics of the samples before and after obfuscation to obtain the glyph and pronunciation dataset DS3;

[0019] Step A5: Integrate the confusion-correct text pair dataset DS2 and the character shape and pronunciation dataset DS3 to obtain the overall error correction dataset DS4;

[0020] Step A6: Use the glyph and pronunciation dataset DS3 to train the language feature embedding model Model emb ;

[0021] Step A7: Model emb Used as an embedding model for the government text error correction model, the overall error correction dataset DS4 is used to train the government text error correction model Model fix ;

[0022] The government text to be tested is preprocessed by dividing it into sentences to obtain multiple government short sentences to be tested, and then they are input into the government text error correction model one by one to obtain error correction results; then, the error correction results are used to replace the erroneous government short sentences to form a corrected government text.

[0023] Furthermore, the step A2 specifically includes:

[0024] Step A21: The symbol indicating the interruption of the sentence is used as a separator to divide the original government minister text into multiple short sentences to obtain a short sentence set {s1, s2, ..., s n}, n is the number of samples in the short sentence set;

[0025] Step A22: In the short sentence set {s1, s2, ..., s n The beginning and end of each short sentence in} are concatenated with the beginning and end identifiers, respectively, as shown below:

[0026] S i =concat([BOS], s i , [EOS])

[0027] Among them, S i is the concatenated short sentence sample, concat(·) is the concatenation function, s i is the i-th short sentence sample in the short sentence set, [BOS] is the sentence start identifier, marking the beginning of the sentence, and [EOS] is the sentence end identifier, marking the end of the sentence;

[0028] Thus, we can obtain the government sentence dataset DS1 required for training = {S1, S2, ..., S n}.

[0029] Furthermore, the step A3 specifically includes:

[0030] Step A31: Set the government text obfuscation rule set Rule = {R1, R2, ..., R m}; The obfuscation rules are as follows:

[0031] R j =Fake j →Real j

[0032] Among them, R j is the jth obfuscation rule in the government text obfuscation rule set, Real j This is a real government text fragment, Fake j In actual writing, it is often used with Real j Obfuscated government text fragments, j = 1, 2, ..., m, where m is the number of samples in the government text obfuscation rule set;

[0033] Step A32: Obfuscate the real government sentences in the government sentence dataset DS1 according to the obfuscation rules in the government text obfuscation rule set Rule to form an initial obfuscated government sentence dataset {S′1, S′2, …, S′ l}; Among them, the confusing government phrases are expressed as follows:

[0034] S′ i =S i .replace(Real t ,Fake t )

[0035] Among them, S′ iis the generated confusing government sentence, S i is the real government sentence in the government sentence dataset DS1, t and Fake t They are respectively the confusion rules R t The real government text fragments and obfuscated government text fragments recorded in , x.replace(y, z) is a replacement function, which means replacing the fragment y in the string x with the fragment z; i = 1, 2, ..., l, l is the number of samples in the initial obfuscated government short sentence data set;

[0036] Step A33: Combine the generated obfuscated government short sentences with the corresponding real government short sentences to form an initial obfuscated-correct text pair data set ds2 = {C1, C2, ..., C l}; the obfuscated-correct text pairs are as follows:

[0037] C i =(S′ i , S i )

[0038] Among them, C i For the confusion-correct text pair, S′ i To confuse government short sentences, S i S′ i The corresponding real government short sentences;

[0039] Step A34: In order to enable the model to learn the ability to distinguish correct government texts during training, correct text pairs are added to the initial confusion-correct text pair data set ds2 to obtain the confusion-correct text pair data set DS2 = {C1, C2, ..., C k}; The added error-free text pairs are represented as follows:

[0040] C q =(S′ q , S q )

[0041] Among them, C q is an error-free text pair, S′ q For the confusing government sentences in the correct text pair, S q Align S′ to the correct text q The corresponding real government sentence, and S′ q =S q , that is, the confused government short sentences in the correct text pair are equivalent to the corresponding real government short sentences, q = l+1, l+2, ..., k, k is the number of samples in the confused-correct text pair data set; the confused-correct text pair data set DS2 = {C1, C2, ..., C k} corresponds to the confusing government sentence dataset {S′1, S′2, …, S′ l , S′l+1 ,…,S′ k}.

[0042] Furthermore, the step A4 specifically includes:

[0043] Step A41: Extract all characters in the obfuscated-correct text pair dataset DS2 and remove duplicates to build a character vocabulary l sym is the character vocabulary V sym Length;

[0044] Step A42: According to the character word list V sym Build a character phonetic dictionary The key-value pairs are as follows:

[0045] kv h =Sym h →Py h

[0046] Among them, kv h is the hth key-value pair in the character-phonetic dictionary, Sym h is the character vocabulary V sym Characters in Py h For the character Sym h The corresponding pronunciation; the method for obtaining the pronunciation of Chinese characters is as follows:

[0047] Py h =cnchar.spell(Sym h )

[0048] Among them, Sym h is a Chinese character, cnchar.spell(·) is a pinyin function provided by the cnchar library, and the function returns a pinyin string.

[0049] The pronunciation of non-Chinese characters is replaced by the character itself, as shown below:

[0050] Py h =Sym h

[0051] Step A43: Extract the confused government short sentence dataset {S′1, S′2, …, S′ from the confused-correct text pair dataset DS2 k}, and perform character confusion processing on each confused government sentence to obtain a confused general sentence dataset {S″1, S″2, …, S″ k}; Among them, the confusing general phrase is expressed as follows:

[0052] S″ i =confuse(S′ i)

[0053] Among them, S″ i To confuse common phrases, S′ i To confuse government sentences, confuse(·) is a typo confusion function, which is used to simulate the common typos generated in real conditions, including similar characters, homophones, and random input. The specific implementation method is as follows:

[0054] S′=[w1,w2,…w s ]

[0055] S″=[w′1,w′2,…w′ s ]=confuse(S′)={wordConfuse(w i )|i=1,2,…,s}

[0056] Wherein, S′ is the original character sequence consisting of s characters, that is, the obfuscated government short sentence, S″ is the character sequence obtained after obfuscating each character in the original character sequence, that is, the obfuscated general short sentence, wordConfuse(·) is the character obfuscation function, which is specifically implemented as follows:

[0057] r = random()

[0058]

[0059] Among them, random() is a random number function, which returns a floating point number with a value range of (0, 1); symConfuse(·) is a similar character confusion function, which returns a similar character of the Chinese character when the input character is a Chinese character, and randomly returns a non-Chinese character from the character vocabulary when the input character is a non-Chinese character; pyConffuse(·) is a homophone confusion function, which returns a homophone of the Chinese character when the input character is a Chinese character, and returns the original character when the input character is a non-Chinese character; randConfuse(·) is a random character confusion function, which randomly returns a character from the character vocabulary;

[0060] Step A44: Use the character pronunciation dictionary S2P to convert the confused government short sentence data set {S′1, S′2, …, S′ k} is mapped to the confusing government phonetic data set {P′1, P′2, …, P′ k}, will confuse the general short sentence dataset {S″1,S″2,…,S″ k} is mapped to a confusing general pronunciation dataset {P″1, P″2, …, P″ k};

[0061] Step A45: Compare the character sequence S′=[w1, W2, …w s] and S″=[w′1, w′2,…w′ s ], calculate the corresponding character matching degree sequence M sym =[m sym,1 , m sym,2 ,…m sym，s ]; where m sym，s Represents the character w s With the character w′ s The glyph matching degree is calculated as follows:

[0062] When i and w′ i When all characters are Chinese characters:

[0063]

[0064] Among them, getStrokes(·) is a stroke acquisition function that returns the stroke sequence of a Chinese character, match(x, y) is a matching element calculation function that returns the number of identical elements in sequence x and sequence y, and len(·) is a sequence length calculation function that returns the number of elements in the input sequence;

[0065] When i and w′ i When there is only one Chinese character in m sym，i =0;

[0066] When i and w′ i When both are not Chinese characters, m sym,i =0.8;

[0067] Step A46: Compare the pronunciation sequence P′=[p1, p2, ... p s ] and P″=[p′1,p′2,…p′ s ], calculate the corresponding pronunciation matching sequence M py =[m py,1 , m py,2 ,…m py,s ]; where p i Represents the character w i The corresponding pronunciation is p′ i Represents the character w′ i The corresponding pronunciation is m py，s Indicates the pronunciation p s With the sound p′ s The pronunciation matching degree of is calculated as follows:

[0068] When i and w′ i When all characters are Chinese characters:

[0069]

[0070] When i and w′ i When not all characters are Chinese characters, m py，i =0;

[0071] Step A47: The character sequence S′ and the pronunciation sequence P′ before the misprint confusion process, the character sequence S″ and the pronunciation sequence P″ after the misprint confusion process, and the character shape matching degree sequence M sym and the pronunciation matching sequence M py Combined into a grapheme and pronunciation data sample Q, expressed as follows:

[0072] Q = [(S′, S″, M sym ), (P′, P″, M py )]

[0073] Integrate all data samples to form a character shape and pronunciation data set DS3 = {Q1, Q2, ..., Q k}.

[0074] Furthermore, in step A5, the confusion-correct text pair dataset DS2 and the character shape and pronunciation dataset DS3 are integrated to form an overall error correction dataset DS4 = {C′1, C′2, …, C′ k}, the sample form in the overall error correction data set DS4 is consistent with the sample form in DS2, which is a confusion-correct text pair form C′ i =(S″′ i , S i ), where S i is the real government short sentence in the confusion-correct text pair dataset DS2, S″′ i is a mixed confusing government sentence, and the mixed confusing government sentence has a 50% probability of being the corresponding confused government sentence S′ in the confusion-correct text pair data set DS2 i , 50% probability is the corresponding confused general phrase S″ in the character shape and pronunciation dataset DS3 i ; The overall error correction data set DS4 covers the data of normal sentences, confused government short sentences, and confused general short sentences in the government text error correction scenario.

[0075] Furthermore, the step A6 specifically includes:

[0076] Step A61: Build an initial language feature embedding model The model includes a glyph feature embedding layer, a phonetic feature embedding layer, a language feature extraction layer, a glyph matching calculation layer, a phonetic matching calculation layer, and a language feature restoration layer; the computable parameters of each layer of the model are as follows:

[0077] The glyph feature embedding layer contains parameters l sym is the character vocabulary V sym The length, d m is the number of embedding feature dimensions;

[0078] The phonetic feature embedding layer contains parameters

[0079] The language feature extraction layer contains parameters

[0080] The glyph matching calculation layer contains parameters

[0081] The pronunciation matching calculation layer contains parameters

[0082] The language feature restoration layer contains parameters

[0083] Step A62: Take the sample Q in the data set DS3 = [(S′, S″, M sym ), (P′, P″, M py )], input S′ and S″ into the glyph feature embedding layer to obtain the real glyph feature Feature′ sym and the input glyph feature Feature″ sym , which is expressed as follows:

[0084] OH S′ =onehot(Index(V sym , S′), l sym )

[0085] OH S″ =onehot(Index(V sym , S″), l sym )

[0086]

[0087] Where i is the number of iterations of the current model training. Assume that the number of elements in S′ and S″ is s. The function Index(x, y) returns an integer sequence of the same length as the sequence y, where each element is the subscript of the element at the same position in y in the sequence x. The function onehot(x, l) returns the one-hot matrix obtained by one-hot encoding the integer sequence x.

[0088] Step A63: Input P′ and P″ into the phonetic feature embedding layer to obtain the real phonetic feature Feature′ py and the input phonetic feature Feature″ py , which is expressed as follows:

[0089] OH P′ =onehot(Index(V py , P′), l py )

[0090] OH P″ =onehot(Index(V py , P″), l py )

[0091]

[0092] Assume that the number of elements in P′ and P″ is s, then

[0093] Step A64: Set Feature sym With Feature py The concatenated input embedding feature Feature′ is input into the language feature extraction layer to obtain the abstract language feature Feature″, which is expressed as follows:

[0094] Feature′=concat(Feature″ sym ,Feature″ py )

[0095]

[0096] Att=softmax(score)

[0097] Feature″=V×Att

[0098] in, The function softmax(s) is a normalization function that maps the input matrix to the exponential domain and then performs column normalization to obtain the matrix s′, which is expressed as follows:

[0099]

[0100] Step A65: Input Feature″ into the font matching degree calculation layer to obtain the model font matching degree sequence M′ sym , which is expressed as follows:

[0101]

[0102] M′ sym =sigmoid(MF sym )

[0103] in, The function sigmoid(M) is a normalization function that maps each element in the input sequence to the value range (0, 1) to obtain the sequence M′, which is expressed as follows:

[0104]

[0105] Step A66: Input the abstract language feature Feature″ into the character-pronunciation matching calculation layer to obtain the model character-pronunciation matching sequence M′ py , which is expressed as follows:

[0106]

[0107] M′ py =sigmoid(MF py )

[0108] in,

[0109] Step A67: Input glyph feature Feature″ sym , model glyph matching degree sequence M′ sym , input phonetic feature Feature″ py The matching degree sequence M′ with the model word py Input the language feature restoration layer to obtain the restored language feature Feature″′, which is expressed as follows:

[0110] Feature″′ sym =scale(Feature″ sym , M′ sym )

[0111] Feature″′ py =scale(Feature″ py , M′ py )

[0112] IF = concat(Feature"' sym , Feature ″′ py )

[0113]

[0114] Att′=softmax(score′)

[0115] Feature″′=V′×Att′

[0116] in, The function scale(x, y) is a scaling function that converts the element x in x j Scale to yj Times to get x′, the calculation formula is as follows:

[0117] x′ j =x j *y j

[0118] Step A68: According to the model font matching degree sequence M′ sym , model word-phonetic matching degree sequence M′p y、 Restore language feature Feature″′ and glyph matching degree sequence M sym , word-phonetic matching degree sequence M py and the real glyph feature Feature′ sym , Real phonetic feature Feature′ py Calculate the model loss L and iterate the model parameters using the gradient descent method, as shown below:

[0119] Feature = concat(Feature′ sym , Feature′ py )

[0120] L m_s =MSE(M sym , M′ sym )

[0121] L m_p =MSE(M py , M′ py )

[0122] L f =MSE(Feature,Feature″′)

[0123] L=L m_s +L m_p +L f

[0124] W (i+1) = backwardByLoss(L, W (i) )

[0125] Among them, L m_s Calculate the loss for font matching, L m_p Calculate the loss for the word-phonetic matching, L f is the embedding feature restoration loss, W (i) For all updateable parameters of the model after the i-th iteration, the function MSE(X, Y) returns the root mean square error of the passed object, and the function backwardByLoss(L, W) is the backward update function, which calculates the parameters of the new iteration according to the gradient of the loss function L and chain propagation;

[0126] Step A69: Repeat steps A62-A68 until the model converges or reaches the set number of iterations. The model training is completed. The trained model is the language feature embedding model Model emb .

[0127] Furthermore, the step A7 specifically includes:

[0128] Step A71: Build an initial government text error correction model The model includes a text feature embedding layer, an encoder and a decoder; wherein the text feature embedding layer is the language feature embedding model trained in step A6. emb , used to convert text sequence data into a computer-calculated feature vector sequence. The encoder is composed of multiple layers of encoding layers with the same structure, which are used to extract high-level features of the input text. The decoder is composed of multiple layers of decoding layers with the same structure, which are connected to a classification network and are used to integrate the high-level features given by the encoder with the information of the output sequence to output the sequence bit by bit in an autoregressive manner.

[0129] Step A72: Take sample C′=(S″′, S) from the overall error correction data set DS4, where S″′ is the input text and S is the correct text corresponding to S″′; input the input text S″′ into the text feature embedding layer to obtain the input text embedding feature Feature input , which is expressed as follows:

[0130] P″′=transByDict(S2P,S″′)

[0131] Feature input =Model emb (S″′,P″′)

[0132] Where S2P is the constructed character pronunciation dictionary, the function transByDict(S2P,S″′) is the dictionary conversion function, which maps the key values ​​of the elements in the input text S″′ one by one according to the key-value relationship in the dictionary S2P, and P″′ is the converted input pronunciation sequence. s is the length of the input text S″′, d m is the number of dimensions of the embedding vector;

[0133] Step A73: Embed the input text into the feature input Input encoder to get high-level text features en ;

[0134] Step A74: Take the beginning of the input text S″′ of different lengths as the output sequence, and input the output sequence into the text feature embedding layer according to step A72 to obtain the output text embedding feature Feature output, and then with Feature en Input the decoder together to obtain the distribution prediction sequence result of the model for the output sequence, which is expressed as follows:

[0135] S″′ ，i =S″′[:i]

[0136] P″′ ，i =transByDict(S2P,S″′ ，i )

[0137] Feature outpu t=Model emb (P″′ ，i , S″′ ，i )

[0138] Feature de =Decoder(Feature en , Feature output )

[0139] Result i+1 =Classifier(Feature de,i )

[0140] Among them, S″′[:i] represents the first i elements of the input text S″′, is the i-th element of the distribution prediction sequence result, which represents the probability distribution of the modification result of the i+1-th element of the input text S″′. i+1 The jth element in Indicates that the original character is modified to the font table V sym The probability of the jth font in ;

[0141] Step A75: compare the distribution prediction sequence result with the real output sequence, i.e., the correct text S, element by element, calculate the cross entropy loss of the classification task, and iterate the model parameters using the gradient descent method;

[0142] Step A76: Repeat steps A72-A75 until the model converges or reaches the set number of iterations. The model training is completed. The trained model is the government text error correction model Model fix .

[0143] The present invention also provides a government text correction system, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the above-mentioned method steps can be implemented.

[0144] Compared with the prior art, the present invention has the following beneficial effects: the present invention provides a government text error correction method and system, which combines government text obfuscation rules with conventional text obfuscation rules, and automatically obtains a government text error correction data set by obfuscating public government texts, thereby overcoming the problems of insufficient data set quantity and poor data set quality faced by current text error correction research. At the same time, the present invention proposes a unique language embedding model, explicitly defines text embedding features from the two perspectives of character shape and pronunciation, reduces the difficulty of learning Chinese character semantics by deep learning networks, and designs innovative self-supervised pre-training tasks based on the original data set, which meets the requirements of embedding model training while improving data utilization. Finally, a GPT-like model structure is used to build an encoder-decoder model to complete the error correction task, thereby improving the accuracy of the government text error correction task. BRIEF DESCRIPTION OF THE DRAWINGS

[0145] Figure 1 A flowchart of a method implementation of an embodiment of the present invention;

[0146] Figure 2 is a structural diagram of a language feature embedding model in an embodiment of the present invention;

[0147] Figure 3 It is a structural diagram of the government text error correction model in an embodiment of the present invention. DETAILED DESCRIPTION

[0148] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0149] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0150] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0151] like Figure 1 As shown, this embodiment provides a government text error correction method, including:

[0152] Building and training a government text error correction model includes the following steps:

[0153] Step A1: Obtain government affairs related text as original government affairs chief text.

[0154] Step A2: Preprocess the original government affairs text to obtain the government affairs short sentence dataset DS1.

[0155] Step A3: Perform government text obfuscation processing on the government short sentence dataset DS1 to obtain the obfuscated-correct text pair dataset DS2.

[0156] Step A4: confuse the wrong characters and obtain the pronunciation of the text in the confusion-correct text pair dataset DS2, and calculate the glyph and pronunciation matching characteristics of the samples before and after the confusion to obtain the glyph and pronunciation dataset DS3.

[0157] Step A5: Integrate the confusion-correct text pair dataset DS2 and the character shape and pronunciation dataset DS3 to obtain the overall error correction dataset DS4.

[0158] Step A6: Use the glyph and pronunciation dataset DS3 to train the language feature embedding model Model emb .

[0159] Step A7: Embed language features into the model emb Used as an embedding model for the government text error correction model, the overall error correction dataset DS4 is used to train the government text error correction model Model fix , the text returned by the government text correction model is the final correction result.

[0160] After training the government text error correction model, the government text to be tested is pre-processed by dividing it into sentences to obtain multiple government short sentences to be tested, and then they are input into the government text error correction model one by one to obtain the error correction results; then, the obtained error correction results are used to replace the erroneous government short sentences to form the corrected government text.

[0161] This embodiment provides a training method for a government text error correction model that automatically generates a data set and uses a special self-supervised task to strengthen the embedding model. The implementation idea is: 1. Collect training samples from public websites; 2. Preprocess the training samples, including phonetic annotation, government rule confusion processing, and typo confusion processing; 3. Use the generated special data set to perform self-supervised training on the language embedding model, and the training objectives include glyph matching recognition, glyph matching recognition, and language feature restoration; 4. Use the trained language embedding model as the embedding layer of the overall model, and use the data set to train the overall GPT-like model. This application combines the automatic data set generation method, the embedding model self-supervised training method, and the large language generation model architecture. It aims to solve the language model's demand for large-scale high-quality training data, and at the same time embeds the language from the glyph features and the pronunciation features, and designs a special self-supervised training method to improve data utilization and learning depth. Finally, a deep learning model architecture with stronger learning ability is used to build the overall model. The relevant content involved in the training process of the government text error correction model is further described below.

[0162] In step A1, government-related texts can be obtained from local and central public websites at all levels, including government texts on various topics.

[0163] The step A2 specifically includes:

[0164] Step A21: Using symbols such as commas, periods, semicolons, and line breaks that indicate sentence interruption as separators, the original government minister text obtained is divided into multiple short sentences to obtain a short sentence set {s1, s2, ..., s n}, n is the number of samples in the short sentence set.

[0165] Step A22: In the short sentence set {s1, s2, ..., s n The beginning and end of each short sentence in} are concatenated with the beginning and end identifiers, respectively, as shown below:

[0166] S i =concat([BOS], s i , [EOS])

[0167] Among them, S i is the concatenated short sentence sample, concat(·) is the concatenation function, s i is the i-th short sentence sample in the short sentence set, [BOS] is the sentence start identifier, marking the beginning of the sentence, and [EOS] is the sentence end identifier, marking the end of the sentence.

[0168] Thus, we can obtain the government sentence dataset DS1 required for training = {S1, S2, ..., S n}.

[0169] The step A3 is an automatic obfuscated text generation method driven by the purpose of correcting government texts; the generated obfuscated texts will be paired with the original correct government texts one by one to form an obfuscated-correct text pair data set DS2 = {C1, C2, ..., C k}, used for subsequent derivation of other data sets and error correction model training; specifically including:

[0170] Step A31: Set the government text obfuscation rule set Rule = {R1, R2, ..., R m}; The obfuscation rules are as follows:

[0171] R j =Fake j →Real j

[0172] Among them, R j is the jth obfuscation rule in the government text obfuscation rule set, Real j This is a real government text fragment, Fake j In actual writing, it is often used with Real j Confused government text fragments, j = 1, 2, ..., m, m is the number of samples in the government text obfuscation rule set.

[0173] Step A32: Obfuscate the real government sentences in the government sentence dataset DS1 according to the obfuscation rules in the government text obfuscation rule set Rule to form an initial obfuscated government sentence dataset {S′1, S′2, …, S′ l}; Among them, the confusing government phrases are expressed as follows:

[0174] S′ i =S i .replace(Real t ,Fake t )

[0175] Among them, S′ i is the generated confusing government sentence, S i is the real government sentence in the government sentence dataset DS1, t and Fake t They are respectively the confusion rules R t The real government text fragments and obfuscated government text fragments are recorded in , x.replace(y,z) is a replacement function, which means replacing the fragment y in the string x with the fragment z; i = 1, 2, ..., l, l is the number of samples in the initial obfuscated government short sentence dataset.

[0176] Step A33: Combine the generated obfuscated government short sentences with the corresponding real government short sentences to form an initial obfuscated-correct text pair data set ds2 = {C1, C2, ..., C l}; the obfuscated-correct text pairs are as follows:

[0177] C i =(S′ i , S i )

[0178] Among them, C i For the confusion-correct text pair, S′ i To confuse government short sentences, S i S′ i The corresponding real government short sentences.

[0179] Step A34: In order to enable the model to learn the ability to distinguish correct government texts during training, correct text pairs are added to the initial confusion-correct text pair data set ds2 to obtain the confusion-correct text pair data set DS2 = {C1, C2, ..., C k}; The added error-free text pairs are represented as follows:

[0180] C q =(S′ q , S q )

[0181] Among them, C q is an error-free text pair, S′ q For the confusing government sentences in the correct text pair, S q Align S′ to the correct text q The corresponding real government sentence, and S′ q =S q , that is, the confused government short sentences in the correct text pair are equivalent to the corresponding real government short sentences, q = l+1, l+2, ..., k, k is the number of samples in the confused-correct text pair data set; the confused-correct text pair data set DS2 = {C1, C2, ..., C k} corresponds to the confusing government sentence dataset {S′1, S′2, …, S′ l , S′ l+1 ,…,S′ k}.

[0182] The step A4 performs mischaracter obfuscation and pinyin annotation on the obfuscated text in the data set DS2 to obtain a general error correction sample; calculates the similarity characteristics of the pronunciation and glyph of the samples before and after the obfuscation; and obtains the glyph and pronunciation data set DS3 = {Q1, Q2, ..., Q k}; specifically include:

[0183] Step A41: Extract all characters in the obfuscated-correct text pair dataset DS2 and remove duplicates to build a character vocabulary l sym is the character vocabulary V sym Length.

[0184] Step A42: Using the cnchar library of the JavaScript language, according to the character vocabulary V sym Build a character phonetic dictionary The key-value pairs are as follows:

[0185] kv h =Sym h →Py h

[0186] Among them, kv h is the hth key-value pair in the character-phonetic dictionary, Sym h is the character vocabulary V sym Characters in Py h For the character Sym h The corresponding pronunciation.

[0187] The method for obtaining the pronunciation of Chinese characters is as follows:

[0188] Py h =cnchar.spell(Sym h )

[0189] Among them, Sym h is a Chinese character. cnchar.spell(·) is a pinyin function provided by the cnchar library. The function returns a pinyin string.

[0190] The pronunciation of non-Chinese characters is replaced by the character itself, as shown below:

[0191] Py h =Sym h

[0192] Step A43: Extract the confused government short sentence dataset {S′1, S′2, …, S′ from the confused-correct text pair dataset DS2 k}, and perform character confusion processing on each confused government sentence to obtain a confused general sentence dataset {S″1, S″2, …, S″ k}; Among them, the confusing general phrase is expressed as follows:

[0193] S″ i =confuse(S′ i )

[0194] Among them, S″ iTo confuse common phrases, S′ i To confuse government sentences, confuse(·) is a typo confusion function, which is used to simulate the common typos generated in real conditions, including similar characters, homophones, and random input. The specific implementation method is as follows:

[0195] S′=[w1,w2,…w s ]

[0196] S″=[w′1,w′2,…w′ s ]=confuse(S′)={wordConfuse(w i )|i=1,2,…,s}

[0197] Wherein, S′ is the original character sequence consisting of s characters, that is, the obfuscated government short sentence, S″ is the character sequence obtained after obfuscating each character in the original character sequence, that is, the obfuscated general short sentence, wordConfuse(·) is the character obfuscation function, which is specifically implemented as follows:

[0198] r = random()

[0199]

[0200] Among them, random() is a random number function, which returns a floating point number with a value range of (0, 1); symConfuse(·) is a similar character confusion function, which returns a similar character of the Chinese character when the input character is a Chinese character, and randomly returns a non-Chinese character from the character vocabulary when the input character is a non-Chinese character; pyConfuse(·) is a homophone confusion function, which returns a homophone of the Chinese character when the input character is a Chinese character, and returns the original character when the input character is a non-Chinese character; randConfuse(·) is a random character confusion function, which randomly returns a character in the character vocabulary.

[0201] Step A44: Use the character pronunciation dictionary S2P to convert the confused government short sentence data set {S′1, S′2, …, S′ k} is mapped to the confusing government phonetic data set {P′1, P′2, …, P′ k}, will confuse the general short sentence dataset {S″1,S″2,…,S″ k} is mapped to a confusing general pronunciation dataset {P″1, P″2, …, P″ k}.

[0202] Step A45: Compare the character sequence S′=[w1, W2, …w s ] and S″=[w′1, w′2,…w′ s], calculate the corresponding character matching degree sequence M sym =[m sym,1 , m sym,2 ,…m sym，s ]; where m sym，s Represents the character w s With the character w′ s The glyph matching degree is calculated as follows:

[0203] When i and w′ i When all characters are Chinese characters:

[0204]

[0205] Among them, getStrokes(·) is a stroke acquisition function that returns the stroke sequence of a Chinese character, match(x, y) is a matching element calculation function that returns the number of identical elements in sequence x and sequence y, and len(·) is a sequence length calculation function that returns the number of elements in the input sequence.

[0206] When i and w′ i When there is only one Chinese character in m sym,i =0.

[0207] When i and w′ i When both are not Chinese characters, m sym,i =0.8.

[0208] Step A46: Compare the pronunciation sequence P′=[p1, p2, ... p s ] and P″=[p′1,p′2,…p′ s ], calculate the corresponding pronunciation matching sequence M py =[m py,1 , m py,2 ,…m py,s ]; where p i Represents the character w i The corresponding pronunciation is p′ i Represents the character w′ i The corresponding pronunciation is m py，s Indicates the pronunciation p s With the sound p′ s The pronunciation matching degree of is calculated as follows:

[0209] When i and w′ i When all characters are Chinese characters:

[0210]

[0211] When i and w′ i When not all characters are Chinese characters, m py，i =0.

[0212] Step A47: The character sequence S′ and the pronunciation sequence P′ before the misprint confusion process, the character sequence S″ and the pronunciation sequence P″ after the misprint confusion process, and the character shape matching degree sequence M sym and the pronunciation matching sequence M py Combined into a grapheme and pronunciation data sample Q, expressed as follows:

[0213] Q = [(S′, S″, M sym ), (P′, P″, M py )]

[0214] Integrate all data samples to form a character shape and pronunciation data set DS3 = {Q1, Q2, ..., Q k}.

[0215] In step A5, the confusion-correct text pair dataset DS2 and the character shape and pronunciation dataset DS3 are integrated to form an overall error correction dataset DS4 = {C′1, C′2, …, C′ k}, the sample form in the overall error correction data set DS4 is consistent with the sample form in DS2, which is a confusion-correct text pair form C′ i =(S″′ i , S i ), where S i is the real government short sentence in the confusion-correct text pair dataset DS2, S″′ i is a mixed confusing government sentence, and the mixed confusing government sentence has a 50% probability of being the corresponding confused government sentence S′ in the confusion-correct text pair data set DS2 i , that is, the corresponding confused government sentence S′ in the confused government sentence dataset i , 50% probability is the corresponding confused general phrase S″ in the character shape and pronunciation dataset DS3 i The overall error correction data set DS4 covers the data of normal sentences, confused government short sentences, and confused general short sentences in the government text error correction scenario, which more realistically simulates the data distribution in actual situations and is conducive to comprehensive learning of the error correction model.

[0216] The step A6 models the features of each text character from the two dimensions of character shape and character pronunciation, and uses the single-word error correction pre-training task to strengthen the model's learning of character features, and trains to obtain a language feature embedding model Model emb ; Specifically include:

[0217] Step A61: Build an initial language feature embedding model The model includes a glyph feature embedding layer, a phonetic feature embedding layer, a language feature extraction layer, a glyph matching calculation layer, a phonetic matching calculation layer, and a language feature restoration layer; among which, the computable parameters of each layer of the initial model are as follows:

[0218] The glyph feature embedding layer contains parameters l sym is the character vocabulary V sym The length, d m is the number of embedding feature dimensions;

[0219] The phonetic feature embedding layer contains parameters

[0220] The language feature extraction layer contains parameters

[0221] The glyph matching calculation layer contains parameters

[0222] The pronunciation matching calculation layer contains parameters

[0223] The language feature restoration layer contains parameters

[0224] Step A62: Take the sample Q in the data set DS3 = [(S′, S″, M sym ), (P′, P″, M py )], input S′ and S″ into the glyph feature embedding layer to obtain the real glyph feature Feature′ sym and the input glyph feature Feature″ sym , which is expressed as follows:

[0225] OH S′ =onehot(Index(V sym , S′), l sym )

[0226] OH S′ =onehot(Index(V sym , S″), l sym )

[0227]

[0228] Where i is the number of iterations of the current model training. Assume that the number of elements in S′ and S″ is s. The function Index(x, y) returns an integer sequence of the same length as the sequence y, where each element is the subscript of the element at the same position in y in the sequence x. The function onehot(x, l) returns the one-hot matrix obtained after one-hot encoding of the integer sequence x.

[0229] Step A63: Input P′ and P″ into the phonetic feature embedding layer to obtain the real phonetic feature Feature′ py and the input phonetic feature Feature″ py , which is expressed as follows:

[0230] OH P′ =onehot(Index(V py , P′), l py )

[0231] OH P″ =onehot(Index(V py , P″), l py )

[0232]

[0233] Assume that the number of elements in P′ and P″ is s, then

[0234] Step A64: Set Feature sym With Feature″ py The concatenated input embedding feature Feature′ is input into the language feature extraction layer to obtain the abstract language feature Feature″, which is expressed as follows:

[0235] Feature′=concat(Feature″ sym ,Feature″ py )

[0236]

[0237] Att=softmax(score)

[0238] Feature″=V×Att

[0239] in, The function softmax(s) is a normalization function that maps the input matrix to the exponential domain and then performs column normalization to obtain the matrix s′, which is expressed as follows:

[0240]

[0241] Step A65: Input Feature″ into the font matching degree calculation layer to obtain the model font matching degree sequence M′ sym , which is expressed as follows:

[0242]

[0243] M′ sym =sigmoid(MF sym )

[0244] in, The function sigmoid(M) is a normalization function that maps each element in the input sequence to the value range (0, 1) to obtain the sequence M′, which is expressed as follows:

[0245]

[0246] Step A66: Input the abstract language feature Feature″ into the character-pronunciation matching calculation layer to obtain the model character-pronunciation matching sequence M′ py , which is expressed as follows:

[0247]

[0248] M′ py =sigmoid(MF py )

[0249] in,

[0250] Step A67: Input glyph feature Feature″ sym , model glyph matching degree sequence M′ sym , input phonetic feature Feature″ py The matching degree sequence M′ with the model word py Input the language feature restoration layer to obtain the restored language feature Feature″′, which is expressed as follows:

[0251] Feature″′ sym =scale(Feature″ sym , M′ sym )

[0252] Feature″′ py =scale(Feature″ py , M′ py )

[0253] IF = concat(Feature"' sym , Feature ″′ py )

[0254]

[0255] Att′=softmax(score′)

[0256] Feature″′=V′×Att′

[0257] in, The function scale(x, y) is a scaling function that converts the element x in x j Scale to y j Times to get x′, the calculation formula is as follows:

[0258] x′ j =x j *y j

[0259] Step A68: According to the model font matching degree sequence M′ sym , model word-phonetic matching degree sequence M′ py , restore language feature Feature″′, character shape matching sequence M sym , word-phonetic matching degree sequence M py and the real glyph feature Feature′ sym , Real phonetic feature Feature′ py Calculate the model loss L and iterate the model parameters using the gradient descent method, as shown below:

[0260] Feature = concat(Feature′ sym , Feature′ py )

[0261] L m_s =MSE(M sym , M′ sym )

[0262] L m_p =MSE(M py , M′ py )

[0263] L f =MSE(Feature,Feature″′)

[0264] L=L m_s +L m_p +L f

[0265] W (i+1) = backwardByLoss(L, W (i) )

[0266] Among them, L m_s Calculate the loss for font matching, L m_p Calculate the loss for the word-phonetic matching, L f is the embedding feature restoration loss, W (i)For all updateable parameters of the model after the i-th iteration, the function MSE(X, Y) returns the root mean square error of the passed object, where W (i) For all updateable parameters of the model after the i-th iteration, the function MSE(X, Y) returns the root mean square error of the passed object, expressed as follows:

[0267]

[0268] The function backwardByLoss(L, W) is a backward update function that calculates the parameters of the new iteration based on the gradient of the loss function L and chain propagation.

[0269] Step A69: Repeat steps A62-A68 until the model converges or reaches the set number of iterations. The model training is completed. The trained model is the language feature embedding model Model emb , whose structure is as follows Figure 2 shown.

[0270] Step A7 uses a GPT-like model structure to build an encoder-decoder model and embeds the pre-trained language features into the model emb As the feature embedding layer in the model, the overall error correction dataset DS4 is used to train the model’s error correction capabilities for various types of government texts, and finally the government text error correction model Model is obtained. fix ; Specifically include:

[0271] Step A71: Build an initial government text error correction model The model includes a text feature embedding layer, an encoder and a decoder; wherein the text feature embedding layer is the language feature embedding model trained in step A6. emb , used to convert text sequence data into a computer-calculated feature vector sequence. The encoder is composed of multiple layers of encoding layers with the same structure stacked together, which is used to extract high-level features of the input text. The decoder is composed of multiple layers of decoding layers with the same structure stacked together and connected to a classification network, which is used to integrate the high-level features given by the encoder with the information of the output sequence and autoregressively output the sequence bit by bit.

[0272] Step A72: Take sample C′=(S″′, S) from the overall error correction data set DS4, where S″′ is the input text and S is the correct text corresponding to S″′; input the input text S″′ into the text feature embedding layer to obtain the input text embedding feature Feature input , which is expressed as follows:

[0273] P″′=transByDict(S2P,S″′)

[0274] Feature input =Model emb(S″′,P″′)

[0275] Wherein, S2P is the constructed character pronunciation dictionary, function transByDict(S2P,S″′) is the dictionary conversion function, which maps the key values ​​of the elements in the input text S″′ one by one according to the key-value relationship in the dictionary S2P, and P′″ is the converted input pronunciation sequence. s is the length of the input text S″′, d m is the number of dimensions of the embedding vector.

[0276] Step A73: Embed the input text into the feature input Input encoder to get high-level text features en .

[0277] Step A74: Take the beginning of the input text S″′ of different lengths as the output sequence, and input the output sequence into the text feature embedding layer according to step A72 to obtain the output text embedding feature Feature output , and then with Feature en Input the decoder together to obtain the distribution prediction sequence result of the model for the output sequence, which is expressed as follows:

[0278] S″′ ，i =S″′[:i]

[0279] P″′ ，i =transByDict(S2P,S″′ ，i )

[0280] Feature output =Model emb (P″′ ，i , S″′ ，i )

[0281] Feature de =Decoder(Feature en , Feature output )

[0282] Result i+1 =Classifier(Feature de,i )

[0283] Among them, S″′[:i] represents the first i elements of the input text S″′, is the i-th element of the distribution prediction sequence result, which represents the probability distribution of the modification result of the i+1-th element of the input text S″′. i+1 The jth element in Indicates that the original character is modified to the font table V sym The probability of the jth font in ;

[0284] Step A75: compare the distribution prediction sequence result with the real output sequence, i.e., the correct text S, element by element, calculate the cross entropy loss of the classification task, and iterate the model parameters using the gradient descent method;

[0285] Step A76: Repeat steps A72-A75 until the model converges or reaches the set number of iterations. The model training is completed. The trained model is the government text error correction model Model fix , whose structure is as follows Figure 3 shown.

[0286] After training the government text error correction model, perform text error correction according to the following steps:

[0287] Step B1: The user uploads the text T to be tested, and performs sentence segmentation preprocessing on T to obtain the sample set {S0, S1, ..., S n}, the processing method is as described in step A2;

[0288] Step B2: Use the character pronunciation dictionary S2P to perform pronunciation mapping on the sample, and obtain a sample pronunciation sequence set {P0, P1, ..., P n}, the specific formula is as follows:

[0289] P i =transByDict(S2P,S i )

[0290] The character pronunciation dictionary S2P is obtained as described in step A42;

[0291] Step B3: Input a text sample S and a corresponding character sequence sample P into the embedding layer Model of the government text error correction model emb , get the text embedding feature Feature input , the formula is as follows:

[0292] Feature input =Model emb (S, P)

[0293] Step B4: Embed text into features imput Input model encoderModel en , get the text high-level feature Feature en , the formula is as follows:

[0294] Feature en =Model en(Feature input )

[0295] Step B5: Construct the initial return sequence O (0) , the sequence contains only one sentence start identifier [BOS], and the expression is as follows:

[0296] O (0) =[′[BOS]′]

[0297] Step B6: Same as steps B2-B3, return sequence O (i) Perform character-sound mapping and feature embedding to obtain the output text embedding feature Feature output ;

[0298] Step B7: High-level text features en and output text embedding feature Feature output Input model decoder Model de , get the model's distribution prediction for the i+1th element in the output sequence, take the character number with the highest generation probability and add it to the output sequence. The specific formula is as follows:

[0299] Result i+1 =Model de (Feature en , Feature output )

[0300]

[0301] O (i+1) =concat(O (i) ,[o i+1 ])

[0302] Step B8: Repeat steps B6-B7 until the next character number o is generated i+1 The corresponding sentence end symbol [EOS] in the vocabulary or the output sequence reaches the upper limit of the model length. In this case, the output sequence O = O (i) That is the error correction result of the model for sample S;

[0303] Step B9: Repeat steps B3-B8 for each sample in the sample set to be tested to obtain an error correction result set {O0, O1, ..., O n}, concatenate the error correction results into a short text to obtain the error correction result T′ for the text to be detected T.

[0304] This embodiment also provides a government text error correction system, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the above method steps can be implemented. Specifically, the government text error correction system includes:

[0305] Governmental text collection module: used to collect the original government text required for model training;

[0306] Data processing module: perform sentence segmentation on the original long text to obtain the government short sentence dataset DS1; perform common government text obfuscation processing on dataset DS1 to obtain the obfuscated-correct text pair dataset DS2; perform general error correction obfuscation processing on the text in dataset DS2 and perform feature annotation to obtain the pronunciation and glyph dataset DS3; integrate datasets DS2 and DS3 to obtain the overall error correction dataset DS4;

[0307] Model training module: Use the dataset DS3 to train the language feature embedding model Model emb ; Use the overall error correction dataset DS4 to train the government text error correction model Model fix ;

[0308] Dataset storage module: stores the data sets saved and used during the training process;

[0309] Model training weight management module: manages and stores different training weights obtained by different training data, training strategies, model hyperparameters and other factors; loads corresponding training weights for model training and model inference.

[0310] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0311] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0312] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0313] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0314] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A method for correcting errors in government documents, characterized in that: include: Building and training a government text error correction model includes the following steps: Step A1: obtaining government affairs related text as original government affairs chief text; Step A2: preprocess the original government affairs text to obtain the government affairs short sentence dataset DS1; Step A3: Perform government text obfuscation processing on the government sentence dataset DS1 to obtain the obfuscated-correct text pair dataset DS2; Step A4: Obfuscate the text in the obfuscated-correct text pair dataset DS2 for incorrect characters and obtain the pronunciation, and calculate the glyph and pronunciation matching characteristics of the samples before and after obfuscation to obtain the glyph and pronunciation dataset DS3; Step A5: Integrate the confusion-correct text pair dataset DS2 and the character shape and pronunciation dataset DS3 to obtain the overall error correction dataset DS4; Step A6: Use the glyph and pronunciation dataset DS3 to train the language feature embedding model Model emb ; Step A7: Model emb Used as an embedding model for the government text error correction model, the overall error correction dataset DS4 is used to train the government text error correction model Model fix ; The government text to be tested is preprocessed by dividing it into sentences to obtain multiple government short sentences to be tested, and then they are input into the government text error correction model one by one to obtain error correction results; then, the error correction results are used to replace the erroneous government short sentences to form a corrected government text.

2. A government text error correction method according to claim 1, characterized in that: The step A2 specifically includes: Step A21: The symbol indicating the interruption of the sentence is used as a separator to divide the original government minister text into multiple short sentences, and obtain a short sentence set {s1, s2, …, s n }, n is the number of samples in the short sentence set; Step A22: In the short sentence set {s1,s2,…,s n The beginning and end of each short sentence in} are concatenated with the beginning and end identifiers, respectively, as shown below: S i =concat([BOS],s i ,[EOS]) Among them, S i is the concatenated short sentence sample, concat(·) is the concatenation function, s i is the i-th short sentence sample in the short sentence set, [BOS] is the sentence start identifier, marking the beginning of the sentence, and [EOS] is the sentence end identifier, marking the end of the sentence; Thus, we can obtain the government sentence dataset DS1={S1,S2,…,S n }.

3. A government text error correction method according to claim 1, characterized in that: The step A3 specifically includes: Step A31: Set the government text obfuscation rule set Rule = {R1, R2, ..., R m }; The obfuscation rules are as follows: R j =Fake j →Real j Among them, R j is the jth obfuscation rule in the government text obfuscation rule set, Real j This is a real government text fragment, Fake j In actual writing, it is often used with Real j Obfuscated government text fragments, j = 1, 2, ..., m, m is the number of samples in the government text obfuscation rule set; Step A32: Obfuscate the real government sentences in the government sentence dataset DS1 according to the obfuscation rules in the government text obfuscation rule set Rule to form an initial obfuscated government sentence dataset {S'1, S'2, ..., S' l }; Among them, the confusing government phrases are expressed as follows: S′ i =S i .replace(Real t ,Fake t ) Among them, S' i is the generated confusing government sentence, S i is the real government sentence in the government sentence dataset DS1, t and Fake t They are respectively the confusion rules R t The real government text fragments and obfuscated government text fragments recorded in , x.replace(y,z) is a replacement function, which means replacing the fragment y in the string x with the fragment z; i = 1, 2, ..., l, l is the number of samples in the initial obfuscated government short sentence data set; Step A33: Combine the generated obfuscated government short sentences with the corresponding real government short sentences to form an initial obfuscated-correct text pair data set ds2 = {C1, C2, ..., C l }; the obfuscated-correct text pairs are as follows: C i =(S′ i ,S i ) Among them, C i For the confusion-correct text pair, S' i To confuse government short sentences, S i For S' i The corresponding real government short sentences; Step A34: In order to enable the model to learn the ability to distinguish correct government texts during training, correct text pairs are added to the initial confusion-correct text pair dataset ds2 to obtain the confusion-correct text pair dataset DS2 = {C1, C2, ..., C k }; The added error-free text pairs are represented as follows: C q =(S′ q ,S q ) Among them, C q For an error-free text pair, S' q For the confusing government sentences in the correct text pair, S q Align S' for error-free text q The corresponding real government sentence, and S' q =S q , that is, the confused government short sentences in the correct text pair are equivalent to the corresponding real government short sentences, q = l+1, l+2, ..., k, k is the number of samples in the confused-correct text pair data set; the confused-correct text pair data set DS2 = {C1, C2, ..., C k } corresponds to the confusing government short sentence data set {S'1,S'2,…,S' l ,S' l+1 ,…,S' k }.

4. A government text error correction method according to claim 1, characterized in that: The step A4 specifically includes: Step A41: Extract all characters in the obfuscated-correct text pair dataset DS2 and remove duplicates to build a character vocabulary l sym is the character vocabulary V sym Length; Step A42: According to the character word list V sym Build a character phonetic dictionary The key-value pairs are as follows: kv h =Sym h →Py h Among them, kv h is the hth key-value pair in the character-phonetic dictionary, Sym h is the character vocabulary V sym Characters in Py h For the character Sym h The corresponding pronunciation; the method for obtaining the pronunciation of Chinese characters is as follows: Py h =cnchar.spell(Sym h ) Among them, Sym h is a Chinese character, cnchar.spell(·) is a pinyin function provided by the cnchar library, and the function returns a pinyin string. The pronunciation of non-Chinese characters is replaced by the character itself, as shown below: Py h =Sym h Step A43: Extract the confused government short sentence dataset {S'1, S'2, ..., S' from the confused-correct text pair dataset DS2 k }, and perform spelling confusion on each confused government sentence to obtain a confused general sentence dataset {S″1,S″2,…,S″ k }; Among them, the confusing general phrase is expressed as follows: S″ i =confuse(S′ i ) Among them, S″ i To confuse common phrases, S' i To confuse government sentences, confuse(·) is a typo confusion function, which is used to simulate the common typos generated in real conditions, including similar characters, homophones, and random input. The specific implementation method is as follows: S′=[w1,w2,…w s ] S″=[w′1,w′2,…w′ s ]=confuse(S′)={wordConfuse(w i )| i =1,2,…,s} Wherein, S' is the original character sequence consisting of s characters, that is, the obfuscated government short sentence, S" is the character sequence obtained after obfuscating each character in the original character sequence, that is, the obfuscated general short sentence, wordConfuse(·) is the character obfuscation function, which is specifically implemented as follows: r = random() Among them, random() is a random number function, which returns a floating point number with a value range of (0,1); symConfuse(·) is a similar character confusion function, which returns a similar character of the Chinese character when the input character is a Chinese character, and randomly returns a non-Chinese character from the character vocabulary when the input character is a non-Chinese character; pyConfuse(·) is a homophone confusion function, which returns a homophone of the Chinese character when the input character is a Chinese character, and returns the original character when the input character is a non-Chinese character; randConfuse(·) is a random character confusion function, which randomly returns a character from the character vocabulary; Step A44: Use the character pronunciation dictionary S2P to convert the confused government short sentence data set {S'1, S'2, ..., S' k } is mapped to the confusing government phonetic data set {P'1,P'2,…,P' k }, will confuse the general short sentence dataset {S″1,S″2,…,S″ k } is mapped to the confusion general pronunciation dataset {P″1,P″2,…,P″ k }; Step A45: Compare the character sequence S'=[w1,w2,…w s ] and S' = [w'1,w'2,…w' s ], calculate the corresponding character matching degree sequence M sym =[m sym,1 ,m sym,2 ,…m sym,s ]; where m sym,s Represents the character w s With the character w' s The glyph matching degree is calculated as follows: When i With w' i When all characters are Chinese characters: Among them, getStrokes(·) is a stroke acquisition function that returns the stroke sequence of a Chinese character, match(x,y) is a matching element calculation function that returns the number of identical elements in sequence x and sequence y, and len(·) is a sequence length calculation function that returns the number of elements in the input sequence; When i With w' i When there is only one Chinese character in m sym,i =0; When i With w' i When both are not Chinese characters, m sym,i =0.8; Step A46: Compare the pronunciation sequence P'=[p1, p2, ... p s ] and P' = [p'1,p'2,…p' s ], calculate the corresponding pronunciation matching sequence M py =[m py,1 ,m py,2 ,…m py,s ]; where p i Represents the character w i The corresponding pronunciation is p' i Represents the character w' i The corresponding pronunciation is m py,s Indicates the pronunciation p s With the pronunciation p' s The pronunciation matching degree of is calculated as follows: When i With w' i When all characters are Chinese characters: When i With w' i When not all characters are Chinese characters, m py,i =0; Step A47: The character sequence S' and the pronunciation sequence P' before the misprint confusion process, the character sequence S", the pronunciation sequence P' after the misprint confusion process, and the character shape matching degree sequence M sym and the pronunciation matching sequence M py Combined into a grapheme and pronunciation data sample Q, expressed as follows: Q=[(S′,S″,M sym ),(P′,P″,M py )] Integrate all data samples to form a character shape and pronunciation data set DS3 = {Q1, Q2, ..., Q k }.

5. A government text error correction method according to claim 4, characterized in that: In step A5, the confusion-correct text pair dataset DS2 and the character shape and pronunciation dataset DS3 are integrated to form an overall error correction dataset DS4 = {C'1, C'2, ..., C' k }, the sample form in the overall error correction data set DS4 is consistent with the sample form in DS2, which is a confusion-correct text pair form C' i =(S″′ i ,S i ), where S i is the real government short sentence in the confusion-correct text pair dataset DS2, S″′ i is a mixed confusing government sentence, and the mixed confusing government sentence has a 50% probability of being the corresponding confused government sentence S' in the confusion-correct text pair data set DS2 i , 50% probability is the corresponding confused general phrase S″ in the character shape and pronunciation dataset DS3 i ; The overall error correction data set DS4 covers the data of normal sentences, confused government short sentences, and confused general short sentences in government text error correction scenarios.

6. A government text error correction method according to claim 1, characterized in that: The step A6 specifically includes: Step A61: Build an initial language feature embedding model The model includes a glyph feature embedding layer, a phonetic feature embedding layer, a language feature extraction layer, a glyph matching calculation layer, a phonetic matching calculation layer, and a language feature restoration layer; the computable parameters of each layer of the model are as follows: The glyph feature embedding layer contains parameters lsym is the length of the character vocabulary Vsym, dm is the number of embedding feature dimensions; The phonetic feature embedding layer contains parameters The language feature extraction layer contains parameters The glyph matching calculation layer contains parameters The pronunciation matching calculation layer contains parameters The language feature restoration layer contains parameters Step A62: Take the sample Q in the data set DS3 = [(S', S", M sym ),(P',P",M py )], input S' and S" into the glyph feature embedding layer to obtain the real glyph feature Feature' sym and the input glyph feature Feature″ sym , which is expressed as follows: OH S′' =onehot(Index(V sym ,S′),l sym ) OH S″ =onehot(Index(V sym ,S″),l sym ) Where i is the number of iterations of the current model training. Assume that the number of elements in S' and S' is s, then The function Index(x,y) returns an integer sequence of the same length as the sequence y, where each element is the subscript of the element at the same position in y in the sequence x. The function onehot(x,l) returns the one-hot matrix obtained by one-hot encoding the integer sequence x. Step A63: Input P' and P" into the phonetic feature embedding layer to obtain the real phonetic feature Feature' py and the input phonetic feature Feature″ py , which is expressed as follows: OH P' =onehot(Index(V py ,P′),l py ) OH P″ =onehot(index(V py ,P″),l py ) Assume that the number of elements in P' and P" is s, then Step A64: Set Feature sym With Feature py The concatenated input embedding feature Feature' is input into the language feature extraction layer to obtain the abstract language feature Feature", which is expressed as follows: Feature′=concat(Feature″ sym ,Feature″ py ) Att=softmax(score) Feature″=V×Att in, The function softmax(s) is a normalization function that maps the input matrix to the exponential domain and then performs column normalization to obtain the matrix s', which is expressed as follows: Step A65: Input Feature" into the glyph matching degree calculation layer to obtain the model glyph matching degree sequence M' sym , which is expressed as follows: M′ sym =sigmoid(MF sym ) in, The function sigmoid(M) is a normalization function that maps each element in the input sequence to the range (0,1) to obtain the sequence M', which is expressed as follows: Step A66: Input the abstract language feature Feature" into the character-pronunciation matching calculation layer to obtain the model character-pronunciation matching sequence M' py , which is expressed as follows: M′ py =sigmoid(MF py ) in, Step A67: Input glyph feature Feature″ sym , model glyph matching degree sequence M' sym , input phonetic feature Feature″ py Sequence M' of matching degree with model word pronunciation py Input the language feature restoration layer and obtain the restored language feature Feature', which is expressed as follows: Feature″′ sym =scale(Feature″ sym ,M′ sym ) Feature″′ py =scale(Feature″ py ,M′ py ) IF=concat(Feature″′ sym ,Feature″′ py ) Att′=softmax(score′) Feature″′=V′×Att′ in, The function scale(x,y) is a scaling function that converts the element x in x j Scale to y j Times to get x', the calculation formula is as follows: x′ j =x j *y j Step A68: According to the model font matching sequence M' sym , model word pronunciation matching degree sequence M' py , restore language feature Feature”', character shape matching sequence M sym , word-phonetic matching degree sequence M py With the real glyph feature Feature' sym , Real phonetic features Feature' py Calculate the model loss L and iterate the model parameters using the gradient descent method, as shown below: Feature=concat(Feature′ sym ,Feature′ py ) L m_s =MSE(M sym ,M′ sym ) L m_p =MSE(M py ,M′ py ) L f =MSE(Feature,Feature″′) L=L m_s +L m_p +L f IN (i+1) =backwardByLoss(L,W (i) ) Among them, L m_s Calculate the loss for font matching, L m_p Calculate the loss for the word-phonetic matching, L f is the embedding feature restoration loss, W (i) For all updateable parameters after the i-th iteration of the model, the function MSE(X,Y) returns the root mean square error of the passed object, and the function backwardByLoss(L,W) is the backward update function, which calculates the parameters of the new iteration according to the gradient of the loss function L and chain propagation; Step A69: Repeat steps A62-A68 until the model converges or reaches the set number of iterations. The model training is completed. The trained model is the language feature embedding model Model emb .

7. A government text error correction method according to claim 1, characterized in that: The step A7 specifically includes: Step A71: Build an initial government text error correction model The model includes a text feature embedding layer, an encoder and a decoder; wherein the text feature embedding layer is the language feature embedding model trained in step A6. emb , used to convert text sequence data into a computer-calculated feature vector sequence. The encoder is composed of multiple layers of encoding layers with the same structure, which are used to extract high-level features of the input text. The decoder is composed of multiple layers of decoding layers with the same structure, which are connected to a classification network and are used to integrate the high-level features given by the encoder with the information of the output sequence to output the sequence bit by bit in an autoregressive manner. Step A72: Take sample C'=(S"', S) from the overall error correction data set DS4, where S"' is the input text and S is the correct text corresponding to S"'; input the input text S"' into the text feature embedding layer to obtain the input text embedding feature Feature input , which is expressed as follows: P″′=transByDict(S2P,S″′) Feature input =Model emb (S″′,P″′) Wherein, S2P is the constructed character pronunciation dictionary, function transByDict(S2P,S'') is the dictionary conversion function, which maps the key values ​​of the elements in the input text S'' one by one according to the key-value relationship in the dictionary S2P, and P'' is the converted input pronunciation sequence. s is the length of the input text S'', d m is the number of dimensions of the embedding vector; Step A73: Embed the input text into the feature input Input encoder to get high-level text features en ; Step A74: Take the beginning of the input text S'' of different lengths as the output sequence, input the output sequence into the text feature embedding layer according to step A72, and obtain the output text embedding feature Feature output , and then with Feature en Input the decoder together to obtain the distribution prediction sequence result of the model for the output sequence, which is expressed as follows: S ”',i =S”'[:i] P ”',i =transByDict(S2P,S ”',i ) Feature output =Model emb (P ”',i ,S ”',i ) Feature de =Decoder(Feature en ,Feature output ) result i+1 =Classifier(Feature de,i ) Among them, S"'[:i] represents the first i elements of the input text S"'. The i-th element of the distribution prediction sequence result represents the probability distribution of the modification result of the i+1-th element of the input text S”’. i+1 The jth element in Indicates that the original character is modified to the font table V sym The probability of the jth font in ; Step A75: compare the distribution prediction sequence result with the real output sequence, i.e., the correct text S, element by element, calculate the cross entropy loss of the classification task, and iterate the model parameters using the gradient descent method; Step A76: Repeat steps A72-A75 until the model converges or reaches the set number of iterations. The model training is completed. The trained model is the government text error correction model Model fix .

8. A government text error correction system, characterized in that: The method comprises a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps as described in any one of claims 1 to 7 can be implemented.

Images