A method for out-of-order text correction based on a pointer network

Abstract

The application belongs to the technical field of natural language processing, and particularly relates to a disorderly text correction method based on a pointer network; the method comprises the following steps: obtaining correction text data and constructing labels; extracting a word-level feature vector of the correction text data; extracting a local feature vector according to the word-level feature vector of the correction text; splicing the word-level feature vector and the local feature vector to construct a fusion feature vector; constructing a span pointer score table and a next pointer score table by using a query mechanism according to the fusion feature vector; narrowing the next pointer score table according to the span pointer score table; reordering the disorderly text according to the span score table and the next pointer score table to obtain a final corrected text; calculating a total loss of a model and adjusting model parameters according to the total loss to obtain a trained disorderly text correction model; and using the trained model to correct the text; the application improves the accuracy of an automatic proofreading system and the model reasoning speed is faster.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of natural language processing, and particularly relates to a method for correcting disordered text based on a pointer network. BACKGROUND

[0002] With the rapid development of the Internet, grammar correction is widely used in automatic proofreading systems. The existing grammar correction system has high accuracy for common errors such as pronunciation and character shape. However, the disordered correction module in the grammar correction system often has problems such as long correction time, inability to timely feedback to the user, and low correction accuracy.

[0003] For the disordered text correction task, most methods are based on deep learning, mainly including traditional seq2edit method and encode-decode method. The traditional seq2edit method needs to be labeled at each position, which consumes a large amount of human resources and has a high possibility of labeling errors. The correctness of the labeled position cannot be detected by a machine. In addition, the seq2edit model has an error accumulation problem, which not only causes correction failure, but also changes the original text semantics, which is particularly evident in disordered text. The encode-decode based method needs to be decoded by self-recurrence after encoding. However, for disordered text, the self-recurrence decoding process is uncontrollable and may produce text beyond the original semantics. In addition, the disordered text only occupies a small part of the overall text, and decoding each position by self-recurrence will consume a lot of time, which is not user-friendly for automatic correction systems to provide timely feedback to users.

[0004] Neither the seq2edit model nor the encode-decode model can well solve the disordered text correction problem. The former has an error accumulation problem and can only decide how to correct word by word. The latter increases the complexity of the model by using self-recurrence, reduces the reasoning output time of the model, and cannot be used in real life. Therefore, it is very important to develop an efficient method to quickly correct the disordered text for improving the accuracy of the automatic correction system and improving the user's satisfaction with the correction system. The present application aims to provide a method for correcting disordered text based on a pointer network to solve this problem. SUMMARY

[0005] In view of the deficiencies in the prior art, the present application provides a method for correcting disordered text based on a pointer network, which comprises: obtaining disordered text to be corrected and inputting it into a trained disordered text correction model for processing to obtain corrected text.

[0006] The training process of the disordered text correction model comprises:

[0007] S1: obtaining correction text data and constructing labels;

[0008] S2: extracting a word-level feature vector of the correction text data;

[0009] S3: extracting a local feature vector according to the word-level feature vector of the correction text; concatenating the word-level feature vector and the word-level feature vector to construct a fusion feature vector;

[0010] S4: according to the fusion feature vector, using a query mechanism to construct a span pointer score table and a next pointer score table;

[0011] S5: narrowing the next pointer score table according to the span pointer score table;

[0012] S6: reordering the disordered text according to the span score table and the next pointer score table to obtain the final corrected text;

[0013] S7: calculating the total loss of the model and adjusting the model parameters according to the total loss of the model to obtain the trained disordered text correction model.

[0014] Preferably, the process of extracting the local feature vector comprises: inputting the word-level feature vector of the correction text into one-dimensional convolutional neural networks with convolution kernel sizes of 2, 3 and 4 respectively for processing, and setting padding rules to keep the word-level feature vector unchanged in the length dimension, to obtain the local feature vector.

[0015] Preferably, the process of constructing the span pointer score table and the next pointer score table comprises:

[0016] S41: inputting the fusion feature vector into a fully connected neural network to convert it into a high-dimensional feature space suitable for extracting span pointers and cutting it in the last dimension to obtain matrices Q1 and K1;

[0017] S42: inputting the fusion feature vector into another fully connected neural network to convert it into a high-dimensional feature space suitable for extracting next pointers and cutting it in the last dimension to obtain matrices Q2 and K2;

[0018] S43: according to the query mechanism, respectively using matrices Q1 and K1 and matrices Q2 and K2 to calculate the span pointer score table and the next pointer score table.

[0019] Further, the formula for calculating the span pointer score table is:

[0020]

[0021] wherein, span denotes the span pointer score table, mask denotes the mask matrix, inner_dim1 denotes the size of Q1, K1 matrix in the last dimension, Trailmask span denotes the upper triangular decaying matrix; i, j respectively denote the row, column position of the two-dimensional matrix; seq_len denotes the size of the matrix.

[0022] Further, the formula for calculating the next pointer score table is:

[0023]

[0024] wherein, next denotes the next pointer score table, mask denotes the mask matrix, inner_dim2 denotes the size of Q2, K2 matrix in the last dimension, Trailmask next denotes the coefficient matrix, span_matrix(i, j) denotes the element value of the i-th row and the j-th column in the span_matrix matrix.

[0025] Preferably, the process of reducing the next pointer score table comprises:

[0026] S51: Perform summation operation on the row and column vectors of the span_matrix matrix respectively and set the positions greater than 0 to 1 to obtain the row_matrix and column_matrix matrices;

[0027] S52: Perform or operation on the matrices row_matrix and column_matrix and utilize the broadcast mechanism to perform and operation on the result and the mask matrix to obtain the global_mask matrix;

[0028] S54: Calculate the reduced next pointer score table according to the global_mask matrix.

[0029] Further, the formula for calculating the reduced next pointer score table is:

[0030] next' = next + (1-global_mask)*10 12

[0031] wherein, next' denotes the reduced next pointer score table, next denotes the original next pointer score table.

[0032] Preferably, the process of reordering the out-of-order text comprises:

[0033] S61: Obtain the positions with values greater than 0 in the span_matrix matrix to construct the span_list list;

[0034] S62: Obtain the position where the value in the reduced next pointer score table is greater than 0, and construct a next_list list;

[0035] S63: According to the span_list list and the next_list list, find the double-ended queue of the longest continuous sequence through enumeration, and reorder the disordered text according to the final result of the double-ended queue.

[0036] Preferably, the formula for calculating the total loss of the model is:

[0037]

[0038] Wherein, loss represents the total loss of the model, P a represents the position set of the value greater than 0 in the span_matrix matrix, Q a represents the position set of the value less than or equal to 0 in the span_matrix matrix, s a (i,j) represents the specific score value at position (i,j) in the span pointer score table, and len() represents a function for calculating the number of elements in a set.

[0039] The beneficial effects of the present application are: the present application adopts a scoring mechanism based on a pointer network, solves the problem of error accumulation in the seq2edit model, and simultaneously combines the span mechanism with the convolutional neural network to make the model more effectively focus on the boundary of each span to achieve the effect of only focusing on the part of the text with disordered errors, and solves the problem of slow self-recurrent error correction speed of the encode-decode model without losing accuracy. The disordered text correction method of the present application improves the accuracy of the automatic proofreading system, and the model reasoning speed is faster, and the user experience is improved. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 The disordered text correction model training flowchart in the present application;

[0041] Figure 2 The flowchart of the disordered text correction method based on the pointer network in the present application. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0043] The application provides a disordered text correction method based on a pointer network. Figure 1 、 Figure 2 The method comprises the following steps.

[0044] The disordered text to be corrected is obtained and input into a trained disordered text correction model for processing to obtain the corrected text.

[0045] The training process of the disordered text correction model comprises the following steps.

[0046] S1: Obtain the correction text data and construct labels.

[0047] The correction text data is obtained, the position of the text segment needing to be adjusted in sequence in the text is recorded as the label of the span pointer score table, and the end position of the current text segment needing to be adjusted and the start position of the next adjacent text segment after correction are recorded as the label of the next pointer score table.

[0048] S2: Extract the word-level feature vector of the correction text data.

[0049] The text data to be corrected is input into a pre-trained bert model to extract the word-level feature vector of the correction text.

[0050] S3: Extract the local feature vector according to the word-level feature vector of the correction text; splice the word-level feature vector and the local feature vector to construct a fusion feature vector.

[0051] The word-level feature vector of the correction text is input into a one-dimensional convolutional neural network with convolution kernel sizes of 2, 3 and 4 for processing, and a padding rule is set to keep the word-level feature vector unchanged in the length dimension, thereby obtaining the local feature vector.

[0052] The word-level feature vector and the local feature vector are spliced in the last dimension, so that each word vector has richer semantic information and pays more attention to the surrounding words of each word, thereby constructing the fusion feature vector of the text data.

[0053] S4: According to the fusion feature vector, a query mechanism is used to construct a span pointer score table and a next pointer score table.

[0054] S41: The fusion feature vector is input into a fully connected neural network to convert it into a high-dimensional feature space suitable for extracting span pointers and is cut in the last dimension to obtain two matrices Q1 and K1 with the same size of (batch_size, seq_len, inner_dim_1).

[0055] S42: input the fusion feature vector into another fully connected neural network to convert it into a high-dimensional feature space suitable for extracting the next pointer, and cut it on the last dimension to obtain two matrices Q2 and K2 with the size of (batch_size, seq_len, inner_dim_2).

[0056] S43: according to the query mechanism, the span pointer score table and the next pointer score table are calculated respectively by using the matrices Q1 and K1 and the matrices Q2 and K2.

[0057] The specific formula of the query mechanism is:

[0058]

[0059] Wherein, feature_table represents the score table to be solved, Q i and K i are specific query matrices, inner_dim i is the size of the Q i and K i matrices on the last dimension; mask represents a mask matrix, that is, if the text length does not reach the maximum text length of the batch of training data, fill the "PAD" character in the position less than the maximum text length of the batch of data, and set the position corresponding to the filled "PAD" text of the mask to 0, and set other positions to 1.

[0060] Trail_mask is a dimension size of (seq_len, seq_len) upper triangular decay matrix when constructing the span pointer score table, which is expressed as:

[0061]

[0062] Wherein, i and j represent the row and column position information of the two-dimensional matrix respectively, and seq_len represents the size of the matrix.

[0063] Set the positions greater than 0 in the span pointer score table to 1 and the positions less than or equal to 0 to 0 to obtain the span_matrix matrix; Trail_mask is a dimension size of (seq_len, seq_len) coefficient matrix depending on the result of the span pointer score table when constructing the next pointer score table, which is expressed as:

[0064]

[0065] Wherein, span_matrix(i,j) represents the element value of the i-th row and j-th column in the span_matrix matrix.

[0066] According to the specific formula of the query mechanism, substitute the specific Q, K, Trail_mask matrix, and obtain the span pointer score table span and the next pointer score table next.

[0067] S5: Narrow the next pointer score table according to the span pointer score table.

[0068] S51: Perform summation operation on the row and column vectors of the span_matrix matrix respectively and set the positions greater than 0 to 1 to obtain the row_matrix and column_matrix matrices.

[0069] S52: Perform or operation on the matrices row_matrix and column_matrix and utilize the broadcast mechanism to perform and operation on the result and the mask matrix to obtain the global_mask matrix.

[0070] Use the global_mask matrix for the next pointer score table obtained in S4 to calculate the narrowed next pointer score table next':

[0071] next' = next + (1-global_mask)*10 12

[0072] S6: Reorder the disordered text according to the span score table and the next pointer score table to obtain the final corrected text.

[0073] S61: Obtain the positions with values greater than 0 in the span_matrix matrix to construct the span_list list.

[0074] S62: Obtain the positions with values greater than 0 in the narrowed next pointer score table to construct the next_list list.

[0075] S63: According to the span_list list and the next_list list, find the double-ended queue of the longest continuous sequence through enumeration; reorder the disordered text according to the final result of the double-ended queue.

[0076] For example, according to steps S61 and S62, span_list={(4, 5), (6, 6), (7, 8)} and next_list={ [8, 6], [6, 4]} are obtained. Then: first, put any element n in next_list into the double-end queue, and then find whether there is an element s in span_list that satisfies that the left value of s is equal to the right value of n or the right value of s is equal to the left value of n. If s satisfies that the left value of s is equal to the right value of n, s is put into the queue from the right side; if s satisfies that the right value of s is equal to the left value of n, s is put into the queue from the left side.

[0077] Similarly, find the element queue in next_list that satisfies the same rule as above, and add the left and right sides of the queue to ensure that the elements in the double-end queue are alternately arranged in span_list and next_list. Finally, the elements in the double-end queue are arranged as {(7, 8), [8, 6], (6, 6), [6, 4], (4, 5)}, where [] represents the elements in next_list and () represents the elements in span_list. Then, the elements in () are extracted in turn, and the corresponding positions of the corrected text order are obtained.

[0078] S7: Calculate the total loss of the model and adjust the model parameters according to the total loss of the model to obtain the trained disordered text correction model.

[0079] The formula for calculating the total loss of the model is:

[0080]

[0081] where loss represents the loss function of the entire disordered text correction model, P a represents the position set of span_matrix matrix with a value greater than 0, Q a represents the position set of span_matrix matrix with a value less than or equal to 0, s a (i, j) represents the specific score value at position (i, j) in the span pointer score table, and len() represents a function that calculates the number of elements in a set.

[0082] Adjust the model parameters according to the total loss of the model. When the loss function converges or reaches the preset number of iterations, stop training, save the model parameters, and obtain the trained disordered text correction model. Obtain the disordered text to be corrected and input it into the trained disordered text correction model for processing to obtain the corrected correct text.

[0083] The above examples further illustrate the objects, technical solutions and advantages of the present application. It should be understood that the above examples are only preferred embodiments of the present application and are not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made to the present application within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for out-of-order text correction based on pointer network, characterized in that, The application relates to a disordered text correction method and device. The training process of the disordered text correction model comprises the following steps: S1: obtaining correction text data and constructing labels; the label construction process is as follows: recording the position of the text segment needing to be adjusted in sequence in the text data as the label of a span pointer score table, and simultaneously recording the end position of the current text segment needing to be adjusted and the start position of the next adjacent text segment after correction as the label of a next pointer score table; S2: extracting a word-level feature vector of the correction text data; S3: extracting a local feature vector according to the word-level feature vector of the correction text; and splicing the word-level feature vector and the local feature vector to construct a fusion feature vector; S4: constructing a span pointer score table and a next pointer score table by using a query mechanism according to the fusion feature vector; the process of constructing the span pointer score table and the next pointer score table comprises the following steps: S41: inputting the fusion feature vector into a fully connected neural network to convert the fusion feature vector into a high-dimensional feature space suitable for extracting a span pointer, and performing cutting on the last dimension to obtain a matrix Q1 and a matrix K1; S42: inputting the fusion feature vector into another fully connected neural network to convert the fusion feature vector into a high-dimensional feature space suitable for extracting a next pointer, and performing cutting on the last dimension to obtain a matrix Q2 and a matrix K2; S43: according to the query mechanism, the span pointer score table and the next pointer score table are calculated by using the matrix Q1 and the matrix K1 and the matrix Q2 and the matrix K2 respectively; the formula for calculating the span pointer score table is as follows: the formula for calculating the next pointer score table is as follows: wherein span denotes a span pointer score table, mask denotes a mask matrix, inner_dim1 denotes a size of Q1, K1 matrix in the last dimension, Trailmask span denotes an upper triangular decaying matrix; i, j respectively denote row, column positions of a two-dimensional matrix; seq_len denotes a size of the matrix; S5: narrowing the next pointer score table according to the span pointer score table; wherein next denotes the next pointer score table, inner_dim2 denotes the size of Q2, K2 matrix in the last dimension, Trailmask next denotes the coefficient matrix, span_matrix(i,j) denotes the element value of the i-th row and j-th column of the span_matrix matrix; the span_matrix matrix is obtained by setting the positions greater than 0 in the span pointer score table to 1 and setting the positions less than or equal to 0 to 0; S6: reordering the disordered text according to the span pointer score table and the next pointer score table to obtain the final corrected text; S7: calculating the total loss of the model and adjusting the model parameters according to the total loss of the model to obtain the trained disordered text correction model. The process of extracting the local feature vector comprises the following steps: inputting the word-level feature vector of the correction text into one-dimensional convolutional neural networks with convolution kernel sizes of 2, 3 and 4 respectively for processing, and setting a padding rule to keep the word-level feature vector unchanged in the length dimension to obtain the local feature vector. 2.The method of claim 1, wherein, The process of narrowing the next pointer score table comprises the following steps: 3.The method of claim 1, wherein, S51: performing summation operation on the row and column vectors of the span_matrix matrix respectively, and setting the positions greater than 0 to 1 to obtain row_matrix and column_matrix matrices; S52: performing or operation on the matrices row_matrix and column_matrix, and performing and operation on the result and a mask matrix by using a broadcast mechanism to obtain a global_mask matrix; S54: calculating the narrowed next pointer score table according to the global_mask matrix. the formula for calculating the narrowed next pointer score table is as follows: 4.The method of claim 3, wherein, ​ next' = next + (1 - global_mask) * 10 12 Wherein, next' represents the reduced next pointer score table, and next represents the original next pointer score table.

5. The method of claim 1, wherein, The process of reordering the disordered text comprises: S61: Obtain the positions of the values greater than 0 in the span_matrix matrix, and construct a span_list list. S62: Obtain the positions of the values greater than 0 in the reduced next pointer score table, and construct a next_list list. S63: According to the span_list list and the next_list list, find the double-ended queue of the longest continuous sequence through the enumeration method; and reorder the disordered text according to the final result of the double-ended queue.

6. The method of claim 1, wherein, The formula for calculating the total loss of the model is: where loss denotes the total loss of the model, P a denotes the set of positions in the span_matrix matrix where the value is greater than 0, Q a denotes the set of positions in the span_matrix matrix where the value is less than or equal to 0, s a denotes the specific score value at position (i,j) in the span pointer score table, and len() denotes a function that counts the number of elements in a set.

Images