A Chinese spelling correction model

Abstract

The present invention discloses a Chinese spelling correction model. The present invention is an encoding and decoding model with consistent input step size and output step size; the sentence X to be corrected = {x1, x2, x3... x n}, is input into the homophone graph convolutional neural network and the dependency syntactic relationship graph convolutional neural network, and a context vector with homophone and relationship information is output; and the context vector is input into the encoding and decoding model to obtain the corrected sentence Y = {y1, y2, y3... y n}. The present invention fully considers the context relationship between words, the dependency syntactic relationship and the relationship between homophones, and proposes a novel Chinese spelling correction model.

Description

Technical Field

[0001] The present invention relates to the field of computer natural language processing, and more specifically to an end-to-end Chinese spelling error correction model. Background Art

[0002] Chinese spelling correction is a kind of text correction, which is applied to intelligent fields such as Chinese input method, search engine, chatbot, voice assistant, etc., and has a good application prospect. It helps Chinese learners better understand the meaning of sentences, helps search engines to match the subject content more accurately, and helps chatbots to obtain user needs more accurately and reply with the most relevant information. Existing models do not fully consider the context of characters or word vectors, dependency syntactic relations, and the relationship between confusing characters. In this regard, this paper proposes a novel end-to-end Chinese spelling correction model that fully considers context information, dependency syntax, and the relationship between confusing characters. Summary of the invention

[0003] The purpose of the present invention is to provide a Chinese spelling error correction model.

[0004] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0005] The model of the present invention is a coding model with the same input step length and output step length. n}, input into the Confusion Word Graph Convolutional Neural Network (CoGcn) and the Dependency Syntax Relationship Graph Convolutional Neural Network (ReGcn), and output the context vector with the confusion word and dependency syntax relationship information. And input the context vector into the encoding and decoding model to obtain the corrected sentence Y = {y1, y2, y3…y n}.

[0006] Furthermore, the described confusing word graph convolutional neural network is specifically implemented as follows:

[0007] Treat each word in the existing confusion set as a node, and the relationship between words as edges, and construct the adjacency matrix A∈R N*N , N refers to the size of the confusion set. If A[i][i] and A[i][j] are confusing characters with similar shapes or similar pronunciations, then A[i][j] = 1, otherwise A[i][j] = 0 (0≤i, j≤n-1). Then the adjacency matrix A is regularized. The formula is as follows:

[0008]

[0009] Where I is the identity matrix, yes The degree matrix of

[0010]

[0011] Through the obfuscated word graph convolutional neural network, similar information between obfuscated words is captured and the obfuscated words are mapped to the same vector space. The graph convolution formula for each layer is as follows:

[0012]

[0013] Among them, H∈R N*D , D is the dimension of the word vector, which is the hidden layer vector. H is initialized with the context word vector pre-trained by Bert 0 .W l ∈R D*D , is a trainable parameter. In order to maintain the original semantics, the superposition operation is performed, and the formula is as follows:

[0014]

[0015] Furthermore, the dependency syntactic relationship graph convolutional neural network:

[0016] Use tools to extract sentence relationships and generate relationship vectors for each word in the input sentence. Take each word as a node and the relationship between words as an edge. Use the dependency syntactic relationship graph convolutional neural network to extract the dependency syntactic relationship between any two words.

[0017] The dependency syntactic relationship vector extracted after graph convolution of each node in each convolutional layer is as follows:

[0018]

[0019] Where l refers to the lth convolutional layer, i refers to the current node, and R refers to all the relationships between words. refers to all nodes that have relationship r with the current node i, c is the scaling factor, w0 l is a trainable parameter, b is a bias parameter, h∈R 1*D , w∈R D*D , b∈R 1*D , D is the dimension of the relation vector. Each node is convolved with its relation nodes to capture the dependency syntactic relation information.

[0020] Since not every word has a confusion word, if the word in the sentence is not in the confusion set, it is represented by the context word vector pre-trained by Bert; otherwise, it is represented by the confusion word vector extracted by the confusion word graph neural network. The output of the confusion word graph neural network is an R N*D The matrix of The vector with context information and confused word information of the i-th word in the confusion set is called h c, the Bert pre-trained context vector becomes h b ,but:

[0021]

[0022] The output of the dependency syntactic relationship graph convolutional neural network is the dependency syntactic relationship vector of each word. The dependency syntactic relationship vector extracted by the dependency syntactic relationship graph convolutional neural network is called h r . The context vector h b Or the confusion word vector h c With the dependency syntactic relation vector h r Blend together, The fused vector is fed to the encoder.

[0023]

[0024] The encoder is a bidirectional long short-term memory network (LSTM), which is a type of recurrent neural network. The hidden state of the encoder at the final time step is used as the initial hidden state of the decoder. The decoder is a unidirectional LSTM recurrent neural network:

[0025]

[0026] in, In the training phase, it is the mixed vector of the target word; in the test phase, it is the target vector inferred from the previous time step. The global context vector is obtained through the attention mechanism where α ij is the weight corresponding to the hidden state at each time step. Concatenate the global context vector corresponding to each time step and the hidden layer vector of the decoding layer and normalize them:

[0027]

[0028] Where W t ∈R M*D , W is a learnable parameter, and M is the size of the Bert vocabulary. The loss function is the cross entropy loss function, Finally, the corrected sentence is output.

[0029] Beneficial effects of the present invention:

[0030] The present invention fully considers the contextual relationship between characters, the dependent syntactic relationship and the relationship between confusing characters, and proposes a novel Chinese spelling error correction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1This is the structural diagram of the model of the present invention

[0032] Figure 2 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0034] like Figure 1 and Figure 2 As shown, the model of the present invention is a coding model with the same input step length and output step length. n}, input into the Confusion Word Graph Convolutional Neural Network (CoGcn) and the Dependency Syntax Relation Graph Convolutional Neural Network (ReGcn), and output the context vector with the confusion word and relationship information. And input the context vector into the encoding and decoding model to get the corrected sentence Y = {y1, y2, y3…y n}.

[0035] Furthermore, the described confusing word graph convolutional neural network is specifically implemented as follows:

[0036] Treat each word in the existing confusion set as a node, and the relationship between words as edges, and construct the adjacency matrix A∈R N*N , N refers to the size of the confusion set. If A[i][i] and A[i][j] are confusing characters with similar shapes or similar pronunciations, then A[i][j] = 1, otherwise A[i][j] = 0 (0≤i, j≤n-1). Then the adjacency matrix A is regularized. The formula is as follows:

[0037]

[0038] Where I is the identity matrix, yes The degree matrix of

[0039]

[0040] Through the obfuscated word graph convolutional neural network, similar information between obfuscated words is captured and the obfuscated words are mapped to the same vector space. The graph convolution formula for each layer is as follows:

[0041]

[0042] Among them, H∈R N*D , D is the dimension of the word vector, which is the hidden layer vector. H is initialized with the context word vector pre-trained by Bert 0 .W l ∈R D*D , is a trainable parameter. In order to maintain the original semantics, the superposition operation is performed, and the formula is as follows:

[0043]

[0044] Furthermore, the dependency syntactic relationship graph convolutional neural network:

[0045] Use tools to extract sentence relationships and generate relationship vectors for each word in the input sentence. Take each word as a node and the relationship between words as an edge. Use the dependency syntactic relationship graph convolutional neural network to extract the dependency syntactic relationship between any two words.

[0046] The syntactic relationship vector obtained after graph convolution of each node in each convolutional layer is as follows:

[0047]

[0048] Where l refers to the lth convolutional layer, i refers to the current node, and R refers to all the relationships between words. refers to all nodes that have relationship r with the current node i, c is the scaling factor, w0 l is a trainable parameter, b is a bias parameter, h∈R 1*D , w∈R D*D , b∈R 1*D , D is the dimension of the relation vector. Each node is convolved with its relation nodes to capture syntactic relation information.

[0049] Since not every word has a confusion word, if the word in the sentence is not in the confusion set, it is represented by the context word vector pre-trained by Bert; otherwise, it is represented by the confusion word vector extracted by the confusion word graph neural network. The output of the confusion word graph neural network is an R N*D The matrix of The vector with context information and confused word information of the i-th word in the confusion set is represented by h. The confused word vector obtained by the confused word graph convolutional neural network is called h c , the context vector of Bert pre-training becomes h b ,but:

[0050]

[0051] The output of the dependency syntactic relationship graph convolutional neural network is the dependency syntactic relationship vector of each word. The dependency syntactic relationship vector extracted by the dependency syntactic relationship graph convolutional neural network is called h r . The context vector h b Or the confusion word vector h c With the dependency syntactic relation vector h r Blend together, The fused vector is fed to the encoder.

[0052]

[0053] The encoder is a bidirectional long short-term memory network (LSTM), which is a type of recurrent neural network. The hidden state of the encoder at the final time step is used as the initial hidden state of the decoder. The decoder is a unidirectional LSTM recurrent neural network:

[0054]

[0055] in, In the training phase, it is the mixed vector of the target word; in the test phase, it is the target vector inferred from the previous time step. The global context vector is obtained through the attention mechanism where α ij is the weight corresponding to the hidden state at each time step. Concatenate the global context vector corresponding to each time step and the hidden layer vector of the decoding layer and normalize them:

[0056]

[0057] Where W t ∈R M*D , W is a learnable parameter, and M is the size of the Bert vocabulary. The loss function is the cross entropy loss function, Finally, the corrected sentence is output.

[0058] Embodiment 1:

[0059] Sentence to be corrected: When we encounter adversity, we must be able to face it.

[0060] Through the confusion word vector matrix output by the confusion word graph convolutional neural network, we find the confusion word vector corresponding to each word in the sentence to be corrected. If the word to be corrected is not in the confusion set, we use the context vector pre-trained by Bert to represent it.

[0061] The tool is used to extract the relationship of the sentences to be corrected, and the relationship vector corresponding to each word is input into the dependency syntactic relationship graph convolutional neural network to obtain the dependency relationship vector of each word.

[0062] The confusion word vector or context vector of the word to be corrected is combined with the dependency vector, E = {e1, e2, e3…e 15}Input into the encoding and decoding model and output the corrected sentence.

[0063] Corrected sentence: When we encounter adversity, we must face it with courage.

Claims

1. A Chinese spelling error correction method, characterized in that The sentence to be corrected X = {x1, x2, x3…x n }, input into the confusion word graph convolutional neural network and the dependency syntactic relationship graph convolutional neural network, output the context vector with the confusion word and dependency syntactic relationship information; and input the context vector into a codec model with the same input step size and output step size to obtain the corrected sentence Y = {y1, y2, y3…y n }; The described confusing word graph convolutional neural network is specifically implemented as follows: Treat each word in the existing confusion set as a node, and the relationship between words as edges, and construct the adjacency matrix A∈R N*N , N refers to the size of the confusion set; if A[i][i] and A[i][j] are confusing characters with similar shapes or pronunciations, then A[i][j] = 1, otherwise A[i][j] = 0, 0≤i,j≤n-1; then the adjacency matrix A is regularized; the formula is as follows: Where I is the identity matrix, yes The degree matrix of Through the obfuscated word graph convolutional neural network, similar information between obfuscated words is captured and the obfuscated words are mapped to the same vector space. The graph convolution formula of each layer is as follows: Among them, H∈R N*D , D is the dimension of the word vector, which is the hidden layer vector; H is initialized with the context word vector pre-trained by Bert 0 ; W l ∈R D*D , is a trainable parameter; in order to maintain the original semantics, the superposition operation is performed, and the formula is as follows:

2. A Chinese spelling error correction method according to claim 1, characterized in that The described dependency syntactic graph convolutional neural network: Use tools to extract sentence relationships, generate relationship vectors for each word in the input sentence, take each word as a node, and the relationship between words as an edge. Use the dependency syntactic relationship graph convolutional neural network to extract the dependency syntactic relationship between any two words. The dependency syntactic relationship vector extracted after graph convolution of each node in each convolutional layer is as follows: Where l refers to the lth convolutional layer, i refers to the current node, and R refers to all the relationships between words. refers to all nodes that have relationship r with the current node i, c is the scaling factor, is a trainable parameter, b is a bias parameter, h∈R 1*D ,w∈R D*D ,b∈R 1*D ,D is the dimension of the relationship vector; each node performs a convolution operation with its relationship node to capture the dependency syntactic relationship information.

3. A Chinese spelling error correction method according to claim 2, characterized in that If the word in the sentence is not in the confusion set, it is represented by the context word vector pre-trained by Bert; otherwise, it is represented by the confusion word vector extracted by the confusion word graph neural network; the output of the confusion word graph neural network is an R N*D The matrix of Represents the vector with context information and confused word information of the i-th word in the confusion set; the confused word vector extracted by the confused word graph convolutional neural network is called h c , the Bert pre-trained context vector becomes h b ,but: The output of the dependency syntactic relationship graph convolutional neural network is the dependency syntactic relationship vector of each word. The dependency syntactic relationship vector extracted by the dependency syntactic relationship graph convolutional neural network is called h r ; The context vector h b Or the confusion word vector h c With the dependency syntactic relation vector h r Blend together, Feed the fused vector to the encoder; The encoder is a bidirectional long short-term memory network; the hidden state of the encoder at the final time step is used as the initial hidden state of the decoder; The decoder is a unidirectional LSTM recurrent neural network: in, In the training phase, it is the mixed vector of the target word; in the testing phase, it is the target vector inferred from the previous time step; the global context vector is obtained through the attention mechanism where α ij is the weight corresponding to the hidden state at each time step; Concatenate the global context vector corresponding to each time step and the hidden layer vector of the decoding layer and normalize them: Where W t ∈R M*D , W is a learnable parameter, M is the size of the Bert vocabulary; The loss function is the cross entropy loss function, Finally, the corrected sentence is output.

Images