A Chinese text error detection method and system based on word order and semantic joint analysis

Abstract

The present invention discloses a Chinese text error detection method and system based on word order and semantic joint analysis. In view of the fact that the existing Chinese text error detection methods cannot deeply understand the semantics of Chinese text and automatically assign weights, a Chinese text prediction error model is designed that regards text as a one-dimensional picture and uses a bidirectional recurrent neural network to fit the text and a self-attention mechanism to assign weights. The present invention adopts a semantic understanding module (FR) composed of a fully convolutional neural network (FCN) and a residual network (ResNet), which has the following two advantages: first, the fully convolutional neural network (FCN) is used to regard one-dimensional text data as a one-dimensional picture to understand the text semantics, solving the problem of lack of semantic processing means in the prior art; second, the residual network (ResNet) is used to deepen the number of network layers, increase the number of features, and deepen the understanding of text semantics.

Description

Technical Field

[0001] The invention belongs to the field of Chinese text processing, text cleaning and text error detection, and relates to a Chinese text error detection method and system based on joint analysis of word order and semantics. Background Art

[0002] With the development of science and technology, the popularization of 4G and 5G, the information level of the whole society is increasing day by day. Online office and remote office are no longer a fantasy. The paperless era has arrived. With the advent of paperless, more and more information is stored in storage devices in the form of electronic information. Because of the particularity of text, only slight differences may bring completely different meanings. It may be that the meaning of the whole sentence becomes different if one word is added. These problems have brought great troubles and losses to people. Official documents, academic papers, legal documents, and case documents are extremely valuable information. If they are misunderstood, they will often bring unpredictable consequences.

[0003] Chinese is the most complex and beautiful language in the world. This complexity and beauty brings about the variability of the language. Often, the semantics of the same sentence will have different meanings in different contexts. With the occurrence of errors in Chinese text, the meaning of the entire text will be greatly different. There are many directions for errors in Chinese text. For example, many times people often misunderstand different characters because of the similarity of pronunciation and pronunciation. Sometimes people mistakenly write characters with different meanings because of the similarity of glyphs. China is a vast country with rich resources and multi-ethnic integration. People in different regions use different dialects. Different dialects have different ways of reading the same word and often have different descriptions of the same thing. These problems are also waiting to be solved. At present, the correction of Chinese texts often lacks common sense knowledge. Therefore, checking Chinese texts for errors in real scenarios has become a hot topic of current research.

[0004] The successful solution of such problems can help people free themselves from the arduous and mechanized manual error detection and comparison. If different errors are compared manually, the cost will increase first, and secondly, for many professional errors, people with professional knowledge are needed to identify the errors, which often results in a waste of human resources. It is imperative to propose methods to solve these problems.

[0005] Looking at the text error detection technology, the current mainstream methods such as convolutional neural networks and recurrent neural networks have achieved good results. However, the effect of applying them to the field of Chinese text is not ideal. The main reason is that the semantics of Chinese text is complex, and the model needs to understand the semantics and perform error detection based on the understanding of the semantics. For example, the original sentence is "Shawshank has a strong desire to survive" and the wrong sentence is "Shawshank has a strong desire to win". There is no problem with the structure of the words in these two sentences, but according to the context, "desire to win" is correct. However, it is difficult for the current mainstream technology to explore the semantic problems of words, so it is impossible to perform error detection well. In addition, the relationship between different words is different, and different weights need to be assigned to represent their correlation. The existing methods are not ideal for the allocation of weights. Summary of the invention

[0006] One purpose of the present invention is to address the above problems and propose a Chinese text error detection method based on word order and semantic joint analysis. The method can take into account both semantic understanding and word-to-word weight allocation while fitting the text.

[0007] The technical solution adopted by the present invention is as follows:

[0008] Step 1: Data preprocessing;

[0009] 1-1 Obtain the original text data, divide all the texts in the original text data into word levels, and construct a Chinese character set D(w); insert an identifier into the Chinese character set D(w), and then use the index to mark the Chinese character set D(w), each word corresponds to a dictionary index, and form a dictionary Dic(w,k);

[0010] 1-2 Tokenize the text sentences in the original text data, add identifiers, and fix the sentence length;

[0011] Preferably, the adding of identifiers in step 1-2 is to add a "START" start symbol at the beginning of the sentence, a "CLS" separator in the middle of the sentence, and an "END" terminator at the end of the sentence;

[0012] Preferably, the fixed sentence length is to truncate the excessive length of the long sentence and fill the short sentence with the "PAD" character to the fixed sentence length;

[0013] 1-3 Serialize the text sentence tokenized in step 1-2 according to the dictionary index in step 1-1;

[0014] 1-4 maps the index serialized data in step 1-3 into a 768-dimensional vector through word embedding technology;

[0015] Step 2: Implement Chinese text error detection through the Chinese text error detection model RFRA based on word order and semantic joint analysis;

[0016] The Chinese text error detection model based on word order and semantic joint analysis includes an information extraction module, a self-attention module (Self-Attention), and an output layer;

[0017] The information extraction module includes a bidirectional gated recurrent neural network (BiGRU) and a semantic understanding module (FR);

[0018] The input of the bidirectional gated recurrent neural network (BiGRU) is the 768-dimensional vector preprocessed in step 1 and the hidden state of the previous moment produced by itself, which is used to extract text time sequence information; specifically:

[0019] The bidirectional gated recurrent unit model includes two gated recurrent units (GRU);

[0020] GRU has a reset gate R and an update gate Z. The reset gate R at time t t and the update gate Z at time t t The calculation is as follows:

[0021]

[0022]

[0023] in is the 768-dimensional vector mapped from step 1 at time t, H t-1 is the hidden state at time t-1, W xr is the reset gate input weight parameter, W xz is the update gate input weight parameter, W hr is the reset gate hidden state weight parameter, W hz is the update gate hidden state weight parameter, b rr and b rz are the bias parameters of the reset gate and update gate respectively; σ is the Sigmoid function, which controls the size range of the reset gate and update gate to be between 0 and 1;

[0024] The reset gate is used to generate candidate hidden states The calculation is expressed as follows:

[0025]

[0026] Where W xh is the candidate hidden state input weight parameter, W hh is the weight parameter of the candidate hidden state with respect to the hidden state, b his the candidate hidden state bias parameter, tahn is the activation function;

[0027] The update gate is used to generate the hidden state H at the current moment t , the calculation is expressed as follows:

[0028]

[0029] in It represents the Hadamard product, which is the multiplication of elements;

[0030] Two GRUs, one for the forward input and one for the reverse input, have a forward hidden state and the reverse hidden state The calculation is expressed as follows:

[0031]

[0032]

[0033] in Indicates the order in which GRU is used to generate hidden states. Indicates the reverse use of GRU to generate hidden states, represents the positive hidden state at time t, represents the reverse hidden state at time t;

[0034] The generation of the hidden state H is not a simple addition but a concatenation, as shown below:

[0035]

[0036] in It represents the Hadamard product, which is the multiplication of elements;

[0037] The input of the semantic understanding module (FR) is the 768-dimensional vector preprocessed in step 1, which is used to extract text semantic information; it includes multiple semantic understanding units, each of which includes a fully convolutional neural network (FCN); each semantic understanding unit is connected using a residual network (ResNet) and an improved Sigmoid function; the input of each semantic understanding unit is the output of the first two layers of units;

[0038] The calculation formula of the residual network ResNet and the improved Sigmoid activation function is as follows:

[0039]

[0040]

[0041] in Represents the output of ResNet at time t, represents the output of the semantic understanding unit at time t-1, Represents the output of the semantic understanding unit at time t-2;

[0042] The input of the self-attention module is the superposition output of the bidirectional gated recurrent neural network (BiGRU) and the semantic understanding module (FR), which is used to assign word weights; the input is differentiated into a key matrix (Key), a query matrix (Query), and a value matrix (Value); then the similarity matrix (Similarity) is calculated based on the key matrix and the query matrix, and then the similarity matrix is ​​normalized, and finally the similarity matrix and the value matrix are weighted to obtain the attention matrix (Attention); specifically:

[0043] (a) The outputs of the bidirectional gated recurrent neural network (BiGRU) and the semantic understanding module (FR) are superimposed and differentiated into a key matrix (Key), a question mark matrix (Query), and a value matrix (Value); specifically

[0044]

[0045]

[0046]

[0047]

[0048] Where W q is the question mark matrix weight parameter, W k is the key matrix weight parameter, W v is the value matrix weight parameter; Represents the output of the bidirectional recurrent neural network BiGRU and the FR semantic understanding module in the information extraction module at time t;

[0049] (b) Calculate the similarity matrix based on the key matrix and the question mark matrix:

[0050] Similarity(Query,Key)=Query×Key (2.14)

[0051] (c) Normalize each row of the similarity matrix

[0052]

[0053] where a ij Represents the value of the normalized similarity matrix in the i-th row and j-th column, and n represents the number of elements in each row of the similarity matrix; similarityij Represents the value of the similarity matrix in the i-th row and j-th column, Indicates similarity based on e ij is the exponential power operation;

[0054] (d) Weight the normalized similarity matrix and value matrix to obtain the attention matrix (Attention)

[0055]

[0056] Among them, attention ij Represents the value of the attention matrix Attention in the i-th row and j-th column, value ij represents the value of the value matrix in the i-th row and j-th column, and l represents the number of elements in each column of the normalized similarity matrix;

[0057] The output layer includes a fully connected layer (Fully Connected Layer) and an activation function Sigmoid, which are used to determine whether the output word is wrong.

[0058] Another object of the present invention is to provide a Chinese text error detection system based on word order and semantic joint analysis, comprising:

[0059] Data preprocessing module, used to convert text data into 768-dimensional vectors;

[0060] The Chinese text error detection module uses a Chinese text error detection model based on joint analysis of word order and semantics to realize Chinese text error detection.

[0061] The technical solution provided by the present invention will produce the following beneficial effects:

[0062] (1) The present invention adopts a semantic understanding module (FR) composed of a fully convolutional neural network (FCN) and a residual network (ResNet), which has the following two advantages: first, the fully convolutional neural network (FCN) is used to treat one-dimensional text data as a one-dimensional picture to understand the text semantics, solving the problem of the lack of semantic processing means in the prior art; second, the residual network (ResNet) is used to increase the number of network layers, increase the number of features, and deepen the understanding of text semantics.

[0063] (2) The present invention uses a bidirectional gated recurrent neural network (BiGRU) to fit text data, which has the following two advantages: first, the gated recurrent neural network (GRU) can avoid the shortcoming that ordinary recurrent neural networks (RNNs) cannot fit long sentences; second, it uses text information from the past and the future at the same time to fit the current text with more feature information.

[0064] (3) The present invention superimposes the output of the semantic understanding module (FR) and the output of the bidirectional gated recurrent network (BiGRU), thereby avoiding the problem of loss of time series information when passing through the pooling layer and the padding layer of the fully convolutional neural network.

[0065] (4) The present invention adopts the self-attention mechanism, which has the following two advantages: first, the attention mechanism has the ability to automatically assign weights, assigning larger weights to words with closer relationships, indicating that their relevance is higher; second, the self-attention mechanism has anti-interference ability, which can effectively avoid the problem of semantics caused by wrong words. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A flow chart of the present invention;

[0067] Figure 2 This is the structure diagram of the semantic understanding module (the dotted lines in the figure are residual network connections);

[0068] Figure 3 Bidirectional gated recurrent network structure diagram

[0069] Figure 4 The residual network structure diagram

[0070] Figure 5 It is the model structure diagram; DETAILED DESCRIPTION

[0071] The specific implementation scheme of the present invention is further described in detail below in conjunction with the accompanying drawings. Figure 1 As shown, where:

[0072] Step 1: Preprocess the input data obtained by the model.

[0073] The preprocessing process is divided into the following four steps:

[0074] 1-1 Create a dictionary. Separate all text sentences into words and build a set of candidate Chinese characters According to the frequency of each word in the set, the words with a frequency lower than 3 are filtered, and the filtered set is deduplicated to form a Chinese character set D(w). Some special symbols such as "START" start symbol, "END" terminator, "CLS" separator, "UNKNOW" unknown symbol, "PAD" filler, etc. are inserted into the Chinese character set D(w). These symbols help the computer to better fit the text. Then use the index to mark each word in the Chinese character set D(w), and each word has a unique mapping to form a dictionary Dic(w,k).

[0075] 1-2 Data tokenization. The data exists in the form of sentences. The "START" start symbol must be added at the beginning of each sentence, the "CLS" separator must be added in the sentence, and the "END" terminator must be added at the end of the sentence. If a word that does not appear in the dictionary is encountered, the "UNKNOW" unknown symbol is used instead. In the judgment theater, the sentence is not of fixed length, and the length of the sentence needs to be processed. Long sentences are truncated for excessive parts, and short sentences need to use the "PAD" filler to fill the remaining part.

[0076] 1-3 Data serialization: Use the dictionary Dic(w,k) obtained in step 1-1 to convert each word in the tokenized text into a dictionary index.

[0077] 1-4 word embedding mapping. There are too many words in the dictionary. Using one-hot encoding will result in a sparse matrix, which will waste storage space and slow down the running speed. The word embedding technology is used to map the index of each word after serialization into a 768-dimensional vector.

[0078] Step 2: Implement Chinese text error detection through the Chinese text error detection model RFRA based on word order and semantic joint analysis;

[0079] The Chinese text error detection model based on word order and semantic joint analysis includes an information extraction module, a self-attention module (Self-Attention), and an output layer;

[0080] The information extraction module includes a bidirectional gated recurrent neural network (BiGRU) and a semantic understanding module (FR);

[0081] The semantic understanding module has the following two advantages: First, the fully convolutional neural network (FCN) is used to treat one-dimensional text data as a one-dimensional picture to understand the text semantics, solving the problem of the lack of semantic processing methods in existing technologies; second, the residual network (ResNet) is used to increase the number of network layers, increase the number of features, and deepen the understanding of text semantics; the bidirectional gated recurrent network has two advantages: first, the gated recurrent neural network (GRU) can avoid the disadvantage that ordinary recurrent neural networks (RNN) cannot fit long sentences; second, text information from the past and the future is used simultaneously to fit the current text with more feature information;

[0082] The output of the semantic understanding module is superimposed with the output of the bidirectional gated recurrent network to avoid the loss of temporal information when it passes through the pooling layer and padding layer of the fully convolutional neural network.

[0083] The input of the bidirectional gated recurrent neural network (BiGRU) is the 768-dimensional vector preprocessed in step 1 and the hidden state of the previous moment, which is used to extract text time sequence information; specifically:

[0084] The bidirectional gated recurrent unit model includes two gated recurrent units (GRU);

[0085] GRU has a reset gate R and an update gate Z. The reset gate R at time t t and the update gate Z at time t t The calculation is as follows:

[0086]

[0087]

[0088] in is the 768-dimensional vector mapped from step 1 at time t, H t-1 is the hidden state at time t-1, W xr is the reset gate input weight parameter, W xz is the update gate input weight parameter, W hr is the reset gate hidden state weight parameter, W hz is the update gate hidden state weight parameter, b rr and b rz are the bias parameters of the reset gate and update gate respectively. σ is the Sigmoid function, which controls the size range of the reset gate and update gate between 0 and 1.

[0089] The reset gate can be used to generate candidate hidden states The calculation is expressed as follows:

[0090]

[0091] Where W xh is the candidate hidden state input weight parameter, W hh is the weight parameter of the candidate hidden state with respect to the hidden state, b h is the candidate hidden state bias parameter and tahn is the activation function.

[0092] The update gate can generate the hidden state H at the current moment t The calculation is expressed as follows:

[0093]

[0094] in It is the Hadamard product, which is the multiplication of elements.

[0095] Two GRUs, one for the forward input and one for the reverse input, have a forward hidden state and the reverse hidden state The calculation is expressed as follows:

[0096]

[0097]

[0098] in Indicates the order in which GRU is used to generate the forward hidden state. Indicates that the hidden state is generated using GRU in reverse order. The generation of the hidden state H is not a simple addition but a concatenation, as shown below:

[0099]

[0100] in It is a dimension join operation.

[0101] The input of the semantic understanding module (FR) is the 768-dimensional vector preprocessed in step 1, which is used to extract text semantic information; it includes 3 units, the input of the first unit is the 768-dimensional vector preprocessed in step 1, the input of the second unit is the 768-dimensional vector preprocessed in step 1 and the output of the first unit, and the input of the second unit is the output of the first and second units;

[0102] Each unit includes a fully convolutional neural network (FCN), which includes a convolution layer, a Relu activation function, an average pooling layer, a deconvolution layer, and an improved Sigmoid activation function; each unit is connected using a residual network (ResNet);

[0103] The residual network ResNet calculation formula and the improved Sigmoid activation function are expressed as follows:

[0104]

[0105]

[0106] in Represents the output of ResNet at time t, Represents the output of ResNet at time t-1, Represents the output of ResNet at time t-2.

[0107] The input of the self-attention module is the superposition output of the bidirectional gated recurrent neural network (BiGRU) and the semantic understanding module (FR), which is used to assign word weights; the input is differentiated into a key matrix (Key), a query matrix (Query), and a value matrix (Value); then the similarity matrix (Similarity) is calculated based on the key matrix and the query matrix, and then the similarity matrix is ​​normalized, and finally the similarity matrix and the value matrix are weighted to obtain the attention matrix (Attention); specifically:

[0108] (a) The outputs of the bidirectional gated recurrent neural network (BiGRU) and the semantic understanding module (FR) are superimposed and differentiated into a key matrix (Key), a question mark matrix (Query), and a value matrix (Value); specifically

[0109]

[0110]

[0111]

[0112]

[0113] Where W q is the question mark matrix weight parameter, W k is the key matrix weight parameter, W v is the value matrix weight parameter.

[0114] (b) Calculate the similarity matrix based on the key matrix and the question mark matrix:

[0115] Similarity(Query,Key)=Query×Key (2.14)

[0116] (c) Normalize each row of the similarity matrix

[0117]

[0118] where a ij It represents the value of the normalized similarity matrix in the i-th row and j-th column, and n represents the number of elements in a row.

[0119] (d) Weight the similarity matrix and the value matrix to obtain the attention matrix (Attention)

[0120]

[0121] Among them, attention ij Represents the value of the attention matrix (Attention) in the i-th row and j-th column, value ij represents the value of the value matrix in the i-th row and j-th column, and l represents the number of rows of elements.

[0122] The output layer includes a fully connected layer and an activation function Sigmoid. The input of the output layer comes from the attention matrix, and after the fully connected layer and the activation function, the probability of the word error is output. If the probability of error is greater than 0.5, it is determined to be a typo.

[0123] The use of the self-attention mechanism has the following two advantages: First, the attention mechanism has the ability to automatically assign weights, assigning larger weights to words with closer relationships, indicating that they are more relevant; second, the self-attention mechanism has anti-interference capabilities, which can effectively avoid semantic problems caused by typos.

[0124] The training of the present invention uses the Merge dataset collected by itself for training, and the performance evaluation uses the SIGHAN15 public Chinese spelling dataset for evaluation. The model performs an experiment on predicting typos on this dataset, and statistics various indicators for comparison. The following table shows the data volume of the Merge and SIGHAN15 datasets.

[0125] Merge SIGHAN15 Number of paragraphs 2390 1100 Number of errors 3740 1602

[0126] The performance evaluation indicators used in this invention are Precesion, Recall, F1, F 0.5 .

[0127] True value 1 True value -1 Prediction value 1 TP (True Positive) FP (False Negative) Prediction value -1 FN (False Negative) TN (True Negative)

[0128] Precision: For the prediction results, the probability of the sample actually being positive among all the samples predicted to be positive.

[0129]

[0130] Recall: For the original sample, the probability of being predicted as a positive sample in the actual positive sample

[0131]

[0132] F1 and F 0.5 It is to find a balance between the two, referring to the precision and recall rate, and comprehensively reflecting the quality of the model.

[0133]

[0134]

[0135] The following table shows the experimental results of gender prediction on the SIGHAN15 dataset of the present invention:

[0136] Precision(%) Recall(%) <![CDATA[F1(%)]]> <![CDATA[F 0.5 (%)]]> LSTM 56.16 47.03 51.19 54.06 GRU 70..17 46.18 55.70 63.57 BiGRU-CNN 81.94 89.38 85.50 83.33 BiGRU-Attention 64.45 99.06 78.09 69.29 RFRA 84.60 98.01 90.81 87.00

[0137] In the above Chinese text error detection experimental results table, LSTM and GRU are traditional recurrent neural network detectors, BiGRU-CNN is a combination of recurrent neural network and convolutional neural network, and BiGRU-Attention is a combination of recurrent neural network and attention mechanism. RFRA is the Chinese text error detection model based on word order and semantic joint analysis in the present invention.

Claims

1. A Chinese text error detection method based on word order and semantic joint analysis, characterized in that The following steps are involved: Step 1: Data preprocessing; 1-1 Obtain the original text data, divide all the texts in the original text data into word levels, and construct a Chinese character set D(w); insert an identifier into the Chinese character set D(w), and then use the index to mark the Chinese character set D(w), each word corresponds to a dictionary index, and form a dictionary Dic(w,k); 1-2 Tokenize the text sentences in the original text data, add identifiers, and fix the sentence length; 1-3 Serialize the text sentence tokenized in step 1-2 according to the dictionary index in step 1-1; 1-4 maps the index serialized data in step 1-3 into a 768-dimensional vector through word embedding technology; Step 2: Implement Chinese text error detection through the Chinese text error detection model RFRA based on word order and semantic joint analysis; The Chinese text error detection model based on word order and semantic joint analysis includes an information extraction module, a self-attention module, and an output layer; The information extraction module includes a bidirectional gated recurrent neural network BiGRU and a semantic understanding module FR; The input of the semantic understanding module FR is the 768-dimensional vector preprocessed in step 1, which is used to extract text semantic information; it includes multiple semantic understanding units, each of which includes a fully convolutional neural network FCN; each semantic understanding unit is connected using a residual network ResNet and uses an improved Sigmoid function; the input of each semantic understanding unit is the output of the first two layers of units; The input of the self-attention module Self-Attention is the superposition output of the bidirectional gated recurrent neural network BiGRU and the semantic understanding module FR, which is used to assign word weights; the input is differentiated into a key matrix Key, a question mark matrix Query, and a value matrix Value, and then the similarity matrix Similarity is calculated according to the key matrix and the question mark matrix, and then the similarity matrix is ​​normalized, and finally the similarity matrix and the value matrix are weighted to obtain the attention matrix Attention; The output layer is used to determine whether the output word is wrong.

2. The method according to claim 1, characterized in that The adding of identifiers in step 1-2 is to add a "START" start symbol at the beginning of the sentence, a "CLS" separator in the middle of the sentence, and an "END" terminator at the end of the sentence.

3. The method according to claim 1, characterized in that The fixed sentence length described in step 1-2 is to truncate the excessive part of the long sentence, and use the "PAD" character to fill the short sentence to the fixed sentence length.

4. The method according to claim 1, characterized in that The input of the bidirectional gated recurrent neural network BiGRU is the 768-dimensional vector preprocessed in step 1 and the hidden state of the previous moment produced by itself, which is used to extract text time sequence information; specifically: The bidirectional gated recurrent unit model includes two gated recurrent units GRU; GRU has a reset gate R and an update gate Z. The reset gate R at time t t and the update gate Z at time t t The calculation is as follows: in is the 768-dimensional vector mapped from step 1 at time t, H t-1 is the hidden state at time t-1, W xr is the reset gate input weight parameter, W xz is the update gate input weight parameter, W hr is the reset gate hidden state weight parameter, W hz is the update gate hidden state weight parameter, b rr and b rz are the bias parameters of the reset gate and update gate respectively; σ is the Sigmoid function, which controls the size range of the reset gate and update gate to be between 0 and 1; The reset gate is used to generate candidate hidden states The calculation is expressed as follows: Where W xh is the candidate hidden state input weight parameter, W hh is the weight parameter of the candidate hidden state with respect to the hidden state, b h is the candidate hidden state bias parameter, tahn is the activation function; The update gate is used to generate the hidden state H at the current moment t , the calculation is expressed as follows: in It represents the Hadamard product, which is the multiplication of elements; Two gated recurrent units GRU, one is the forward input and the other is the reverse input, and its forward hidden state and the reverse hidden state The calculation is expressed as follows: in Indicates the order in which GRU is used to generate hidden states. Indicates the reverse use of GRU to generate hidden states, represents the positive hidden state at time t, represents the reverse hidden state at time t; The generation of the hidden state H is as follows: in Represents a dimension join operation.

5. The method according to claim 1, characterized in that The residual network ResNet calculation formula is as follows: The improved Sigmoid function calculation formula is as follows: in Represents the output of ResNet at time t, represents the output of the semantic understanding unit at time t-1, Represents the output of the semantic understanding unit at time t-2.

6. The method according to claim 1, characterized in that The self-attention module Self-Attention is specifically: (a) The outputs of the bidirectional gated recurrent neural network (BiGRU) and the semantic understanding module (FR) are superimposed and differentiated into a key matrix (Key), a question mark matrix (Query), and a value matrix (Value); specifically Where W q is the question mark matrix weight parameter, W k is the key matrix weight parameter, W v is the value matrix weight parameter; Represents the output of the bidirectional recurrent neural network BiGRU and the FR semantic understanding module in the information extraction module at time t; (b) Calculate the similarity matrix based on the key matrix and the question mark matrix: Similarity(Query,Key)=Query×Key(2.14) (c) Normalize each row of the similarity matrix where a ij Represents the value of the normalized similarity matrix in the i-th row and j-th column, and n represents the number of elements in each row of the similarity matrix; similarity ij Represents the value of the similarity matrix in the i-th row and j-th column, Indicates similarity based on e ij is the exponential power operation; (d) Weight the normalized similarity matrix and value matrix to obtain the attention matrix (Attention) Among them, attention ij Represents the value of the attention matrix Attention in the i-th row and j-th column, value ij It represents the value of the value matrix in the i-th row and j-th column, and l represents the number of elements in each column of the normalized similarity matrix.

7. The method according to claim 1, characterized in that The output layer includes two fully connected layers and two activation functions Gelu.

8. A Chinese text error detection system based on word order and semantic joint analysis for executing the method according to any one of claims 1 to 7, characterized in that include: Data preprocessing module, used to convert text data into 768-dimensional vectors; The Chinese text error detection module uses a Chinese text error detection model based on joint analysis of word order and semantics to realize Chinese text error detection.

Images