A multi-threaded sorting error correction method with replication mechanism and dynamic weighting

Abstract

The present invention relates to a multi-threaded sorting error correction method with a copying mechanism and dynamic weighting, which belongs to the field of data processing technology and solves the technical problems of miscorrection, overcorrection and inefficiency in existing error correction methods. The method includes pre-processing the text to be corrected and assembling it into a sentence array; cyclically allocating each sentence in the sentence array to a multi-threaded channel that is dynamically created, destroyed and horizontally scalable; each thread reads the sentence input from the corresponding data input channel and performs error correction using a trained Copying-ERNIE error correction model with a copying mechanism and dynamic weighting, obtaining an output probability distribution for each character in the sentence, selecting the character corresponding to the maximum output probability of each character, obtaining the corrected sentence and outputting it to the corresponding result output channel; reading the corrected sentence in the index order of the sentence array for fault-tolerant convergence to obtain the corrected text. Efficient and correct error correction is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a multi-threaded sorting with replication mechanism and dynamic weighted error correction method. Background Art

[0002] Identifying typos in Chinese is a meaningful yet challenging task, with numerous practical applications, including assisting content editors with pre-publication verification and correcting text in chat conversations. Furthermore, typo correction is a crucial preprocessing step for many natural language processing (NLP) tasks, such as word segmentation, part-of-speech tagging, and syntactic analysis, impacting the performance of many downstream NLP tasks.

[0003] like Figure 2 The existing ERNIE error correction model is shown in Figure 2. The existing ERNIE error correction model suffers from overcorrection: it tends to mistakenly correct low-frequency expressions as high-frequency ones. The basic idea of ​​the ERNIE error correction model is to use ERNIE to extract semantic encoding vectors for text, use softmax to perform multi-class prediction based on the vocabulary size, and select the label with the highest probability as the model output.

[0004] Based on the existing ERNIE error correction model, there are three main technical shortcomings:

[0005] The existing ERNIE error correction model is a MASK language model. Its pre-training method leads to a tendency to predict high-frequency expressions. The existing ERNIE error correction model often over-corrects low-frequency expressions. For example, it corrects "this is not to say" to "this is not to say";

[0006] The generated vocabulary space is large, and the existing ERNIE error correction model has a low prediction probability for the original character, which easily leads to miscorrection. The ERNIE error correction model can be viewed as a restricted generative model. For each character in the input sentence, the model generates a target character from the entire vocabulary space. Due to the large vocabulary space, it is easy to generate characters that are different from the input. Since most characters input in the CSC (Chinese Spelling Check) task are correct, this can lead to overcorrection.

[0007] Single-threaded job execution is inefficient. The existing ERNIE error correction model corrects errors sequentially for each statement, and is inefficient when executing the error correction for a paragraph, article, or long text in series. Summary of the Invention

[0008] In view of the above analysis, the embodiments of the present invention aim to provide a multi-threaded sorting with replication mechanism and dynamic weighted error correction method to solve the technical problems of over-correction, miscorrection and low single-thread operation error correction efficiency in existing error correction methods.

[0009] The purpose of the present invention is mainly achieved through the following technical solutions:

[0010] The present invention provides a multi-threaded sorting with replication mechanism and dynamic weighted error correction method, comprising:

[0011] Preprocess the text to be corrected and assemble it into a sentence array;

[0012] Each sentence in the sentence array is cyclically assigned to a multi-threaded channel that is dynamically created, destroyed, and horizontally scalable; each thread reads the sentence input from the corresponding data input channel and performs error correction using a trained Copying-ERNIE error correction model with a copying mechanism and dynamic weighting, obtaining an output probability distribution for each character in the sentence, selecting the character corresponding to the maximum output probability for each character, and outputting the corrected sentence to the corresponding result output channel;

[0013] The corrected sentences are read in the index order of the sentence array for fault-tolerant aggregation to obtain the corrected text.

[0014] Furthermore, the output probability distribution of each character in the sentence is obtained, including:

[0015] Perform ERNIE encoding conversion on the sentence read from the data input channel to obtain the semantic feature vector of each character position in the sentence;

[0016] Obtaining a generation probability distribution for each character position based on the semantic feature vector of each character position, obtaining a maximum candidate probability and a generation probability of the original character based on the generation probability distribution, and calculating a copy probability based on the maximum candidate probability and the generation probability of the original character;

[0017] The generation probability distribution and the copy probability are dynamically weighted to obtain the output probability distribution of the characters.

[0018] Furthermore, the ERNIE model is used to perform ERNIE encoding conversion to obtain the semantic feature vector of each character position in the sentence;

[0019] The ERNIE model includes an input layer, an embedding layer and an encoding layer;

[0020] The input layer concatenates the sentences read from the corresponding data input channel into character sequences;

[0021] The embedding layer maps each character in the character sequence to a character vector sequence of a preset dimension;

[0022] The encoding layer calculates the contextual semantic features of each character vector sequence through the self-attention mechanism, and generates the semantic feature vector of each character position , , is the number of characters in the sentence.

[0023] Furthermore, generating the generation probability and copy probability of the semantic feature vector of each character position includes:

[0024] Inputting the semantic feature vector of each character position into a Softmax classifier, performing a linear transformation on the semantic feature vector of each character position to obtain a linearly transformed vector, converting the linearly transformed vector into a probability distribution to obtain a generation probability distribution of the word in the vocabulary corresponding to each character;

[0025] Based on the generation probability distribution, the generation probability corresponding to the original character is obtained, and the maximum generation probability is selected as the maximum candidate probability;

[0026] The copy probability of each character position is obtained by dynamically adjusting the difference between the maximum candidate probability and the generation probability of the original character.

[0027] Furthermore, the replication probability is calculated as follows:

[0028] ;

[0029] in, 、 are the weight matrices of the linear transformation layer and the Relu activation layer of the Softmax classifier, is the maximum candidate probability for each generated position, is the generation probability of the original character at each generation position, is the smoothing coefficient.

[0030] Furthermore, the generation probability distribution and the copy probability are dynamically weighted to obtain the output probability distribution of each character position, as follows: ;

[0031] in, is the output probability distribution, is the replication probability distribution, To generate the probability distribution, is the replication probability.

[0032] Furthermore, each sentence in the sentence array is dynamically and cyclically assigned to a multi-thread channel, including:

[0033] Initialize the thread pool;

[0034] Dynamic creation includes A thread pool with 10 threads, each thread is bound to an independent data input channel and result output channel;

[0035] When the number of sentences is less than the product of the initial number of threads and the number of thread channel queues, destroy the redundant threads or scale the number of thread channel queues horizontally;

[0036] When the number of calculated sentence data input channel indexes is greater than the thread channel queue capacity, dynamically add threads or horizontally expand the number of channel queues;

[0037] Calculate the data input channel index to be input for each sentence in the sentence array based on the number of threads and the number of sentences;

[0038] Allocate each sentence poll to the data input channel corresponding to the data input channel index.

[0039] Furthermore, the fault-tolerant convergence includes:

[0040] Each thread sets the timeout T and the number of retries;

[0041] After completing the error correction process, the thread caches the results to the corresponding result output channel;

[0042] For abnormal threads that have timed out and have been executed more than or equal to the number of retries, determine whether the error correction task in the abnormal thread has been completed; if not, directly output the original sentence in the data input channel corresponding to the abnormal thread to the corresponding result output channel, and continue to poll and read the corrected sentence in the next result output channel;

[0043] After all sentences are corrected, the corrected sentences in all result output channels are read, and the corrected sentences are reassembled according to the sentence array index to obtain the corrected text.

[0044] Furthermore, the method of preprocessing the text to be corrected and assembling it into a sentence array includes:

[0045] The text to be corrected is divided into A sentence, ;

[0046] Remove spaces and line breaks, and add start and end markers for each sentence;

[0047] According to the order of the sentences in the text to be corrected, Sentences are assembled into sentence arrays , ; Each array element is a sentence;

[0048] The demarcation elements include commas, periods, exclamation marks, and question marks.

[0049] Furthermore, the Copying-ERNIE error correction model is obtained by the following training method:

[0050] Construct a sample training set with sentences containing incorrect characters as samples and correct sentences as sample labels;

[0051] Construct a Copying-ERNIE error correction model with a replication mechanism and dynamic weighting;

[0052] Weight matrix for the linear transformation layer and Relu activation layer of the Softmax classifier 、 Random initialization; smoothing coefficient Initialized as hyperparameters;

[0053] Input the sample training set into the model, forward propagate to generate the generation probability distribution and replication probability of the character position in each sample, and dynamically weightedly calculate the output probability distribution of the character;

[0054] Calculate the cross entropy loss between the model output and the true label;

[0055] Based on the loss value, backpropagation is used to calculate the gradient; the Adam optimization algorithm is used to update the model weights and parameters;

[0056] Iterate forward propagation, loss calculation, backpropagation, and gradient update until the loss function is minimized;

[0057] Use accuracy, recall, and F1 score to evaluate model performance;

[0058] After the training is completed, save the Copying-ERNIE error correction model weights and hyperparameters to obtain the trained Copying-ERNIE error correction model.

[0059] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0060] 1. Existing text error correction methods detect and correct typos through pre-trained semantic understanding and generation capabilities, but lack the copying mechanism and copying probability of the present invention. The present invention introduces a copying mechanism algorithm to obtain the maximum candidate probability and the generation probability of the original character based on the generation probability distribution, and then dynamically generates a copying probability distribution, effectively reducing the output probability of overcorrecting low-frequency character expressions into high-frequency character expressions. The predicted probability of the original low-frequency character expression is increased, and the original character is copied to the output generated word with a certain probability, increasing the generation probability of the original low-frequency character, and reducing the overcorrection problem in the existing model.

[0061] 2. In the Copying-ERNIE error correction model of the present invention, if the maximum candidate probability is close to the generation probability of the original input character, that is, the confidence of the candidate character with the maximum generation probability is low, it is often impossible to determine whether to output the generated character or copy the original character. Adding a dynamic weighting algorithm can increase the generation probability of the original input character. In this way, the Copying-ERNIE error correction model of the present invention tends to output the original input character. Determining the copy probability based on the maximum candidate probability and the generation probability of the original character reduces the problem of miscorrection, especially when the vocabulary space is large.

[0062] 3. This invention introduces sentence segmentation, distribution, and aggregation, adopts a multi-threaded concurrent processing mechanism, supports dynamic creation and destruction of thread pools, can dynamically adjust the number of threads (including creation and destruction of threads) according to task load and system resources, and horizontally scale the number of queues in the thread channel, significantly improving error correction efficiency; it can simultaneously execute multi-threaded concurrent execution of entire paragraphs or long texts for error correction while ensuring the order of sentences, thereby improving error correction execution efficiency;

[0063] 4. Through the fault-tolerant convergence mechanism, when some threads time out or cannot work (such as threads hang up), the fault-tolerant convergence mechanism in the present invention quickly handles the exception to avoid long-term thread congestion, continuously monitors the thread status and result data channel, and determines whether the error correction task is completed. If not, the data corresponding to the abnormal thread is input into the original sentence in the channel and re-input into the newly created multi-threaded channel for error correction again, ensuring the integrity, sequence, timeliness and reliability of the output text after error correction;

[0064] 5. The dynamic weighted algorithm combines the generation probability distribution and the replication probability to optimize the output probability distribution, thereby improving the accuracy and adaptability of error correction, especially when processing complex texts and low-frequency expressions.

[0065] In the present invention, the above-mentioned technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of the present invention will be described in the following description, and some advantages will become apparent from the description or be learned through practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. Throughout the drawings, the same reference symbols denote the same components.

[0067] Figure 1 This is a flow chart of a multi-threaded sorting with replication mechanism and dynamic weighted error correction method according to an embodiment of the present invention;

[0068] Figure 2It is a schematic diagram of the existing ERNIE model;

[0069] Figure 3 This is a schematic diagram of multi-threaded concurrent processing in an embodiment of the present invention;

[0070] Figure 4 2. This is a diagram of the Copying-ERNIE error correction architecture in an embodiment of the present invention;

[0071] Figure 5 Schematic diagram of the Softmax classifier structure in an embodiment of the present invention. DETAILED DESCRIPTION

[0072] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0073] To address the technical issues of overcorrection, miscorrection, and low error correction efficiency in existing methods, this paper proposes a multi-threaded sorting error correction method with a copying mechanism and dynamic weighting. The Copying-ERNIE error correction model in this invention introduces a variable copying mechanism based on the existing ERNIE model. When generating each position, if the maximum candidate probability generated is close to the generation probability of the original input word, the original input is weighted during generation, increasing the model's prediction probability for the original word, thereby reducing overcorrection and miscorrection. Furthermore, a multi-threaded channel is dynamically created, destroyed, and horizontally scalable, addressing the low error correction efficiency of existing methods.

[0074] A specific embodiment of the present invention discloses a multi-threaded sorting with replication mechanism and dynamic weighted error correction method, such as Figure 1 As shown, the following steps are included:

[0075] Step S1: pre-process the text to be corrected and assemble it into a sentence array;

[0076] Step S2: cyclically assigning each sentence in the sentence array to a multi-threaded channel that is dynamically created, destroyed, and horizontally scalable; each thread reads the sentence input from the corresponding data input channel and performs error correction using the trained Copying-ERNIE error correction model with a copying mechanism and dynamic weighting, obtaining an output probability distribution for each character in the sentence, selecting the character corresponding to the maximum output probability for each character, and outputting the corrected sentence to the corresponding result output channel;

[0077] Step S3: Read the corrected sentences according to the index order of the sentence array and perform fault-tolerant aggregation to obtain the corrected text.

[0078] Step S1, specifically.

[0079] The text to be corrected can be a sentence, a paragraph or a long text.

[0080] First, preprocess the text to be corrected, including:

[0081] The text to be corrected is segmented using the dividing elements as delimiters to obtain at least one sentence;

[0082] For each sentence obtained, remove spaces and line breaks, and add a start tag [CLS] and an end tag [SEP] to each sentence;

[0083] Assemble the preprocessed sentences into a sentence array in sequence.

[0084] The method of preprocessing the text to be corrected and assembling it into a sentence array includes:

[0085] The text to be corrected is divided into A sentence, ;

[0086] Remove spaces and line breaks, and add start and end markers for each sentence;

[0087] According to the order of the sentences in the text to be corrected, Sentences are assembled into sentence arrays , ; Each array element is a sentence;

[0088] The demarcation elements include commas, periods, exclamation marks, and question marks.

[0089] Each array element in the sentence array is a sentence, for example: is the first sentence of the text to be corrected, The last sentence of the text to be corrected. The sentence array retains the original order of the sentences in the text to be corrected.

[0090] The function of step S1 is to pre-process the text to be corrected, including sentence segmentation, removal of spaces and line breaks, adding start and end marks, and assembling it into a sentence array according to the original sentence order in the text to be corrected for subsequent error correction processing.

[0091] Step S2 is divided into steps S21-S22.

[0092] Step S21: cyclically assign each sentence in the sentence array to a multi-threaded channel that is dynamically created, destroyed, and horizontally scalable.

[0093] Dynamically and cyclically assigning each sentence in the sentence array to a multi-thread channel includes:

[0094] Initialize the thread pool;

[0095] Dynamic creation includes A thread pool with 10 threads, each thread is bound to an independent data input channel and result output channel;

[0096] When the number of sentences is less than the product of the initial number of threads and the number of thread channel queues, destroy the redundant threads or scale the number of thread channel queues horizontally;

[0097] When the number of calculated sentence data input channel indexes is greater than the thread channel queue capacity, dynamically add threads or horizontally expand the number of channel queues;

[0098] Calculate the data input channel index to be input for each sentence in the sentence array;

[0099] Allocate each sentence poll to the data input channel corresponding to the data input channel index.

[0100] For example, use SpringBoot's ThreadPoolTaskExecutor configuration The Springboot thread pool is the thread pool that comes with Springboot, which is a secondary encapsulation of the Java thread pool ThreadPoolExecutor. It is the default thread pool in Spring.

[0101] Initialize the thread pool based on the performance and resource utilization of the thread host, including multiple threads. Define the maximum number of error correction queues in each thread as needed. The goal is to start error correction as soon as the correction sentence is entered, thereby improving efficiency.

[0102] Each dynamically created thread is bound to an independent data input channel and result output channel, such as Figure 3 shown.

[0103] When the number of sentences is less than the product of the initial number of threads and the number of thread channel queues, the explanation for destroying excess threads or horizontally scaling (including increasing and decreasing) the number of thread channel queues is as follows:

[0104] For example, if the number of sentences to be corrected is 30, the initial number of threads is 4, and the number of correction queues for each thread is 10;

[0105] First, the solution to reduce resources is to dynamically destroy the fourth thread;

[0106] Secondly, the solution to improve efficiency is to horizontally reduce the number of channel queues to 8 without destroying threads. Alternatively, the number of queues can be horizontally reduced to 5, while creating two more threads and corresponding data input channels and result output channels.

[0107] Finally, you can also increase the number of thread channel queues. For example, if you increase the number of thread channel queues to 15, it is enough to use two threads for 30 sentences to be corrected, and destroy the third and fourth threads at the same time.

[0108] You can also choose one of the methods to dynamically destroy redundant threads or horizontally scale the number of thread channel queues.

[0109] Array Each array element in is distributed sequentially and passed to the corresponding data input channel in a loop.

[0110] Calculate the index of the data input channel to be assigned to each sentence in the sentence array as follows:

[0111] ;

[0112] in, is the number of thread channels, .

[0113] Take 4 data input channels as an example:

[0114] Distributed to data input channel A1, Distributed to data channel A2, Distributed to data channel A3, Distributed to data channel A4, Sent to data channel A1, Distribute to data input channel A2, and recursively.

[0115] The function of step S21 is to dynamically create, destroy and horizontally scale the thread pool, and cyclically assign each sentence in the sentence array to the multi-thread channel for subsequent efficient error correction processing.

[0116] Step S22: Each thread reads the sentence input from the corresponding data input channel and performs error correction using the trained Copying-ERNIE error correction model with a copying mechanism and dynamic weighting to obtain the output probability distribution of each character in the sentence. The character corresponding to the maximum output probability of each character is selected, and the corrected sentence is output to the corresponding result output channel.

[0117] Specifically.

[0118] Each thread reads the sentence input from the corresponding data input channel and uses the trained multi-threaded sorting with replication mechanism and the dynamically weighted Copying-ERNIE error correction model to perform error correction, obtaining the output probability distribution of each character in the sentence, including:

[0119] Perform ERNIE encoding conversion on the sentence read from the data input channel to obtain the semantic feature vector of each character position in the sentence;

[0120] Obtaining a generation probability distribution for each character position based on the semantic feature vector of each character position, obtaining a maximum candidate probability and a generation probability of the original character based on the generation probability distribution, and calculating a copy probability based on the maximum candidate probability and the generation probability of the original character;

[0121] The generation probability distribution and the copy probability are dynamically weighted to obtain the output probability distribution of the characters.

[0122] First, perform ERNIE encoding conversion:

[0123] The semantic feature vector of each character position in the sentence is obtained by encoding conversion using the ERNIE model;

[0124] The ERNIE model includes an input layer, an embedding layer and an encoding layer;

[0125] The input layer concatenates the sentences read from the corresponding data input channel into character sequences;

[0126] The embedding layer maps each character in the character sequence to a character vector sequence of a preset dimension;

[0127] The encoding layer calculates the contextual semantic features of each character vector sequence through the self-attention mechanism, and generates the semantic feature vector of each character position , , is the number of characters in the sentence.

[0128] To enter the data input channel A1 corresponding to thread 1 For example, generate Semantic feature vector, , the semantic feature vector represents the semantic information of each character in the sentence to be corrected.

[0129] Second, obtain the generation probability and copy probability of the semantic feature vector of each character position.

[0130] Generating a generation probability and a copy probability of the semantic feature vector for each character position includes:

[0131] Inputting the semantic feature vector of each character position into a Softmax classifier, performing a linear transformation on the semantic feature vector of each character position to obtain a linearly transformed vector, converting the linearly transformed vector into a probability distribution to obtain a generation probability distribution of the word in the vocabulary corresponding to each character;

[0132] Based on the generation probability distribution, the generation probability corresponding to the original character is obtained, and the maximum generation probability is selected as the maximum candidate probability;

[0133] The copy probability of each character position is obtained by dynamically adjusting the difference between the maximum candidate probability and the generation probability of the original character.

[0134] The semantic feature vector of each character is input into the softmax classifier for multi-classification. When generating at each position, the corresponding generation probability is obtained , and at the same time copy the input word with a certain probability. The copy probability is determined by the difference between the maximum candidate probability and the generation probability of the input word.

[0135] like Figure 5 As shown in Figure 1, the Softmax classifier includes an input layer, a linear transformation layer, a Relu activation layer, a Sigmoid layer, and an output layer.

[0136] (1) Input layer, which is used to receive the semantic feature vector of each character position of the sentence output by ERNIE encoding conversion ;

[0137] (2) Linear transformation layer: Perform a linear transformation on the semantic feature vector of each character position and map it to the dimension of the vocabulary size.

[0138] Assuming the vocabulary size is V and the feature vector dimension is D, the linear transformation is expressed as follows:

[0139] ;

[0140] in, is the vector after linear transformation, with dimension V;

[0141] is the weight matrix of the linear transformation layer, and the shape of the weight matrix is ​​(V, D);

[0142] is the semantic feature vector of the character position, with dimension D.

[0143] (3) ReLU activation layer: introduces nonlinearity, enabling the classifier to learn more complex features.

[0144] ;

[0145] in, is the vector after Relu activation.

[0146] (4) Sigmoid layer: Converted to probability value for calculating replication probability.

[0147] ;

[0148] Formula (4)

[0149] in, is the Relu activation layer weight matrix.

[0150] Convert the linearly changed vector into a generated probability distribution as follows:

[0151] ;

[0152] in, is the generation probability of character i, For the Sigmoid layer, The i-th probability value after transformation.

[0153] The vocabulary contains about 28,000 Chinese characters. The Softmax classifier converts the semantic feature vector output by the ERNIE encoding layer into the generation probability distribution of characters and performs multi-classification prediction of the vocabulary size.

[0154] The duplication probability is determined based on the difference between the maximum candidate probability and the generation probability of the input word.

[0155] The replication probability is calculated as follows:

[0156] ;

[0157] in, 、 are the weight matrices of the linear transformation layer and the Relu activation layer of the Softmax classifier, is the maximum candidate probability for each generated position, is the generation probability of the original character at each generation position, is the smoothing coefficient.

[0158] and The smaller the difference between the two, the greater the probability of replication. is a smoothing coefficient determined by experiment, which determines the smoothness of the generated probability distribution. For example, The value is 0.1.

[0159] Perform a weighted sum of the generation probability distribution of generating characters from the vocabulary and the copy probability distribution from the original characters to obtain the output probability distribution for each character position.

[0160] Dynamically weight the generation probability distribution and the copy probability to obtain the output probability distribution for each character position as follows:

[0161] ;

[0162] where is the output probability distribution, is the copy probability distribution, is the generation probability distribution, is the copy probability.

[0163] is the copy probability distribution generated by the copy operation and is a one-hot vector. For example, for the character "悔" in "患者错过后悔哭", when represented in the vector, the probability at the position of the character "悔" is 1, and the probabilities at the positions of other characters in the vocabulary are 0. <00=00420>

[0164] is the generation probability distribution obtained by the Softmax classifier.

[0165] Select the character corresponding to the maximum output probability for each character position, and output the corrected sentence to the corresponding result output channel. <000=0425>

[0166] Exemplarily, the original character is "痛";

[0167] The generation probability distribution :

[0168]

[0169] The copy probability :

[0170]

[0171] The copy probability distribution :

[0172]

[0173] Calculation result:

[0174] For "痛": 0.7 * 1.0 + 0.3 * 0.8 = 0.7 + 0.24 = 0.94

[0175] For "通": 0.7 * 0.0 + 0.3 * 0.1 = 0.0 + 0.03 = 0.03

[0176] For "copper": 0.7 * 0.0 + 0.3 * 0.05 = 0.0 + 0.015 = 0.015

[0177] The final output probability distribution is as follows:

[0178] p = "pain": 0.94, "through": 0.03, "copper": 0.015,...

[0179] Select the character "pain" corresponding to the maximum output probability of 0.94.

[0180] The dynamic weighting algorithm makes a trade - off between generation and copying, thus reducing the over - correction problem of existing error - correction models.

[0181] The function of the copying mechanism and the dynamic weighting algorithm is to prevent the model from over - correcting. For example, for the text to be error - corrected "The patient will cry after missing it", if the existing ERNIE model is used, the error - correction result is "The patient will regret crying after missing it". The existing ERNIE model over - corrects and changes "will" to "regret". However, due to the use of the copying mechanism and the dynamic weighting algorithm in the Copying - ERNIE error - correction model of the present invention, the character "will" is successfully retained after being copied, avoiding the situation of over - correction.

[0182] Among them, the Copying - ERNIE error - correction model is obtained through the following training method:

[0183] Construct a sample training set with sentences containing incorrect characters as samples and correct sentences as sample labels;

[0184] Construct a Copying - ERNIE error - correction model with a copying mechanism and dynamic weighting;

[0185] For the weight matrices of the linear transformation layer and the Relu activation layer of the Softmax classifier 、 Randomly initialize; initialize the smoothing coefficient as a hyperparameter;

[0186] Input the sample training set into the model, and through forward propagation, generate the generation probability distribution and copying probability of the character positions in each sample, and calculate the output probability distribution of the characters through dynamic weighting;

[0187] Calculate the cross - entropy loss value between the model output and the true label;

[0188] According to the loss value, calculate the gradient through backpropagation; use the Adam optimization algorithm to update the model weights and parameters;

[0189] Iterate forward propagation, loss calculation, backpropagation, and gradient update until the loss function is minimized;

[0190] Use accuracy, recall, and F1 score to evaluate model performance;

[0191] After the training is completed, save the Copying-ERNIE error correction model weights and hyperparameters to obtain the trained Copying-ERNIE error correction model.

[0192] The cross entropy loss function is as follows:

[0193] ;

[0194] in, is the sentence in the text to be corrected, N is the total number of sentences in the text to be corrected, and the output of the Copying-ERNIE error correction model is , The actual result is .

[0195] The Copying-ERNIE error correction model is measured using accuracy, precision, recall, and F1-score, as shown in formulas (9)-(12):

[0196] ;

[0197] ;

[0198] ;

[0199] ;

[0200] in, 、 、 and represent accuracy, precision, recall, and Fraction; Indicates the number of samples that are actually positive examples and are correctly corrected as positive examples by the Copying-ERNIE error correction model; Indicates the number of negative samples that are actually predicted as negative by the Copying-ERNIE error correction model; Indicates the number of negative samples that are incorrectly predicted as positive by the Copying-ERNIE error correction model; Indicates the number of samples that are actually positive but are mistakenly predicted as negative by the Copying-ERNIE error correction model.

[0201] The following are two examples of overcorrection in existing error correction methods:

[0202] Example 1: We went to the vegetable market to buy vegetables after get off work.

[0203] The existing error correction method is to correct "buy vegetables" to "sell vegetables".

[0204] Example 2: Young people like playing video games.

[0205] With the existing error correction method, “electric” is overcorrected to “telephone”.

[0206] Since the fixed collocation "sell vegetables" is more frequent than "buy vegetables" and "telephone" is more frequent than "electric", the model corrects the correct word input as an incorrect word. This requires the model to mitigate the overcorrection problem.

[0207] In this embodiment, the over-correction problem in the existing ERNIE error correction model is solved mainly through the following technical details:

[0208] (1) Introducing a copying mechanism, the Copying-ERNIE error correction model in this invention directly copies the original character with a certain probability during the error correction generation process, rather than always generating a new character. This increases the probability of generating the original character, especially when the generated word space is large, and reduces the possibility of overcorrection;

[0209] (2) Through a dynamic weighting algorithm, the output of the Copying-ERNIE error correction model is optimized by combining the generation probability and the copy probability. The copy probability is dynamically adjusted based on the difference between the maximum candidate probability and the original character probability. The smaller the difference between the two, the greater the copy probability. Dynamically adjusting the copy probability means that the Copying-ERNIE error correction model is more inclined to retain the original character, thereby reducing overcorrection.

[0210] (3) Softmax classifier is combined with Sigmoid function. Sigmoid function is introduced into Softmax classifier to calculate the replication probability. Converted to a probability value, indicating the probability of copying the original character at the current character position. This probability value determines whether the model directly copies the original character instead of generating a new character.

[0211] These technical details work together to enable the Copying-ERNIE error correction model to more accurately retain correct characters and reduce unnecessary corrections when processing low-frequency expressions and long texts.

[0212] The function of step S22 is to correct each sentence using the trained Copying-ERNIE error correction model through multi-threaded processing and dynamic weighting algorithm, generate the output probability distribution of each character position, and select the character with the maximum probability to reduce the over-correction problem.

[0213] The purpose of step S2 is to process the sentences to be corrected in parallel through a dynamically scalable thread pool, and to use the Copying-ERNIE model with a replication mechanism and dynamic weighting for error correction, thereby improving efficiency while reducing over-correction and miscorrection problems.

[0214] Step S3, specifically.

[0215] According to the original index order of the sentences in the sentence array, the corrected sentences are read and fault-tolerantly aggregated to obtain the corrected text.

[0216] Processing thread 1 reads data from data channel A1 After completing the error correction process using the Copying-ERNIE error correction model, the result data Send to the result output channel B1;

[0217] Recursively, processing thread 2 reads data from data channel A2 After completing the error correction process, the result data Send to data channel B2;

[0218] Processing thread 3 reads data from data channel A3 After completing the error correction process, the result data Send to data channel B3; repeat in sequence.

[0219] The fault-tolerant convergence includes:

[0220] Each thread sets the timeout T and the number of retries;

[0221] After completing the error correction process, the thread caches the results to the corresponding result output channel;

[0222] For abnormal threads that have timed out and have been executed more than or equal to the number of retries, determine whether the error correction task in the abnormal thread has been completed; if not, directly output the original sentence in the data input channel corresponding to the abnormal thread to the corresponding result output channel, and continue to poll and read the error-corrected sentence in the next result output channel;

[0223] After all sentences are corrected, the corrected sentences in all result output channels are read, and the corrected sentences are reassembled according to the sentence array index to obtain the corrected text.

[0224] For threads that time out and have a greater number of reads than the number of retries, there are two exceptions:

[0225] (1) Timeout exception;

[0226] (2) The thread cannot work, for example, the thread hangs up.

[0227] Record the exception log and continue to read the corrected sentence in the next result output channel.

[0228] Check and determine whether the error correction tasks in the channels for these two exceptions have been completed;

[0229] If it has been completed, record the exception log to the log file;

[0230] If it has not been completed, record the exception log to the log file and directly output the original sentence in the data input channel corresponding to the exception thread to the corresponding result output channel.

[0231] Fault-tolerant aggregation avoids the situation of long-term blocking of the aggregation thread.

[0232] Exemplarily, the aggregation sorting algorithm sequentially reads data b[0], b[1], b[2], b[3], b[4] from data channels B1, B2, B3, B4, B1 in a loop, splices each array element, and outputs externally.

[0233] Implement multi-threaded concurrent processing of data streams, improving the processing speed while ensuring the order of the data streams before and after.

[0234] Exemplarily, a typical error correction case is shown in Table 1:

[0235] Table 1: Typical Error Correction Cases

[0236]

[0237] Here, "悔" is more appropriate, with the meaning of "regretting because of not having done it". When there is no copy mechanism introduced, based on the existing ERNIE model, "悔" is corrected to "会". Compared with other positions, the difference between the maximum generation candidate probability of "悔" and the generation probability of the input character is smaller; in the Copying-ERNIE error correction model of the present invention, the copy probability of the input character is increased during generation, making the input character ranked first in the generation candidates, thus predicting correctly;

[0238] Gold usually represents the ideal and accurate error correction result in the field of text error correction. In Table 1, the sentence "患者错过后悔哭" shown by Gold is the correct result after manual proofreading and verification, representing the ideal error correction effect.

[0239] The function of step S3 is to ensure the integrity and order consistency of the corrected text by aggregating the multi-threaded error correction results in the original order of the text to be error-corrected and implementing a fault-tolerant retry aggregation mechanism.

[0240] In summary, a multi-threaded sorting error correction method with a copy mechanism and dynamic weighting in the embodiments of the present invention has the following beneficial effects:

[0241] 1. Existing text error correction methods detect and correct typos through pre-trained semantic understanding and generation capabilities, but lack the copying mechanism and copying probability of the present invention. The present invention introduces a copying mechanism algorithm to obtain the maximum candidate probability and the generation probability of the original character based on the generation probability distribution, and then dynamically generates a copying probability distribution, effectively reducing the output probability of overcorrecting low-frequency character expressions into high-frequency character expressions. The predicted probability of the original low-frequency character expression is increased, and the original character is copied to the output generated word with a certain probability, increasing the generation probability of the original low-frequency character, and reducing the overcorrection problem in the existing model.

[0242] 2. In the Copying-ERNIE error correction model of the present invention, if the maximum candidate probability is close to the generation probability of the original input character, that is, the confidence of the candidate character with the maximum generation probability is low, it is often impossible to determine whether to output the generated character or copy the original character. Adding a dynamic weighting algorithm can increase the generation probability of the original input character. In this way, the Copying-ERNIE error correction model of the present invention tends to output the original input character. Determining the copy probability based on the maximum candidate probability and the generation probability of the original character reduces the problem of miscorrection, especially when the vocabulary space is large.

[0243] 3. This invention introduces sentence segmentation, distribution, and aggregation, adopts a multi-threaded concurrent processing mechanism, supports dynamic creation and destruction of thread pools, can dynamically adjust the number of threads (including creation and destruction of threads) according to task load and system resources, and horizontally scale the number of queues in the thread channel, significantly improving error correction efficiency; it can simultaneously execute multi-threaded concurrent execution of entire paragraphs or long texts for error correction while ensuring the order of sentences, thereby improving error correction execution efficiency;

[0244] 4. Through the fault-tolerant convergence mechanism, when some threads time out or cannot work (such as threads hang up), the fault-tolerant convergence mechanism in the present invention quickly handles the exception to avoid long-term thread congestion, continuously monitors the thread status and result data channel, and determines whether the error correction task is completed. If not, the data corresponding to the abnormal thread is input into the original sentence in the channel and re-input into the newly created multi-threaded channel for error correction again, ensuring the integrity, sequence, timeliness and reliability of the output text after error correction;

[0245] 5. The dynamic weighted algorithm combines the generation probability distribution and the replication probability to optimize the output probability distribution, thereby improving the accuracy and adaptability of error correction, especially when processing complex texts and low-frequency expressions.

[0246] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0247] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A multi-threaded sorting error correction method with replication mechanism and dynamic weighting, characterized in that: include: Preprocess the text to be corrected and assemble it into a sentence array; Each sentence in the sentence array is cyclically assigned to a multi-threaded channel that is dynamically created, destroyed, and horizontally scalable; each thread reads the sentence input from the corresponding data input channel and performs error correction on the trained Copying-ERNIE error correction model with a copying mechanism and dynamic weighting, obtaining an output probability distribution for each character in the sentence, selecting the character corresponding to the maximum output probability of each character, and outputting the corrected sentence to the corresponding result output channel; Read the corrected sentences according to the index order of the sentence array and perform fault-tolerant aggregation to obtain the corrected text; Among them, the output probability distribution of each character in the sentence is obtained, including: Perform ERNIE encoding conversion on the sentence read from the data input channel to obtain the semantic feature vector of each character position in the sentence; Obtaining a generation probability distribution for each character position based on the semantic feature vector of each character position, obtaining a generation probability of the original character based on the generation probability distribution, and selecting the largest generation probability as the maximum candidate probability; dynamically adjusting and calculating a copy probability based on a difference between the maximum candidate probability and the generation probability of the original character; the smaller the difference, the greater the copy probability; The generation probability distribution and the copy probability are dynamically weighted to obtain the output probability distribution of the characters.

2. The method according to claim 1, characterized in that Use the ERNIE model to perform ERNIE encoding conversion to obtain the semantic feature vector of each character position in the sentence; The ERNIE model includes an input layer, an embedding layer and an encoding layer; The input layer concatenates the sentences read from the corresponding data input channel into character sequences; The embedding layer maps each character in the character sequence to a character vector sequence of a preset dimension; The encoding layer calculates the contextual semantic features of each character vector sequence through the self-attention mechanism, and generates the semantic feature vector of each character position , , is the number of characters in the sentence.

3. The method according to claim 2, characterized in that Obtaining the generation probability distribution of each character position includes: The semantic feature vector of each character position is input into the Softmax classifier, the semantic feature vector of each character position is linearly transformed to obtain a linearly transformed vector, and the linearly transformed vector is converted into a probability distribution to obtain the generation probability distribution of the word in the vocabulary corresponding to each character.

4. The method according to claim 3, characterized in that The replication probability is calculated as follows: ； in, 、 are the weight matrices of the linear transformation layer and the Relu activation layer of the Softmax classifier, is the maximum candidate probability for each generated position, is the generation probability of the original character at each generation position, is the smoothing coefficient.

5. The method according to claim 3, characterized in that: The generation probability distribution and the copy probability are dynamically weighted to obtain the output probability distribution of each character position, as follows: ； in, is the output probability distribution, is the replication probability distribution, To generate the probability distribution, is the replication probability.

6. The method according to claim 1, characterized in that Dynamically and cyclically assigning each sentence in the sentence array to a multi-thread channel includes: Initialize the thread pool; Dynamic creation includes A thread pool with 10 threads, each thread is bound to an independent data input channel and result output channel; When the number of sentences is less than the product of the initial number of threads and the number of thread channel queues, destroy the redundant threads or scale the number of thread channel queues horizontally; When the number of calculated sentence data input channel indexes is greater than the thread channel queue capacity, dynamically add threads or horizontally expand the number of channel queues; Calculate the data input channel index to be input for each sentence in the sentence array based on the number of threads and the number of sentences; Allocate each sentence poll to the data input channel corresponding to the data input channel index.

7. The method according to claim 1, characterized in that: The fault-tolerant convergence includes: Each thread sets the timeout T and the number of retries; After completing the error correction process, the thread caches the results to the corresponding result output channel; For abnormal threads that have timed out and have been executed more than or equal to the number of retries, determine whether the error correction task in the abnormal thread has been completed; if not, directly output the original sentence in the data input channel corresponding to the abnormal thread to the corresponding result output channel, and continue to poll and read the corrected sentence in the next result output channel; After all sentences are corrected, the corrected sentences in all result output channels are read, and the corrected sentences are reassembled according to the sentence array index to obtain the corrected text.

8. The method according to claim 1, characterized in that: The method of preprocessing the text to be corrected and assembling it into a sentence array includes: The text to be corrected is divided into A sentence, ; Remove spaces and line breaks, and add start and end markers for each sentence; According to the order of the sentences in the text to be corrected, Sentences are assembled into sentence arrays , ; Each array element is a sentence; The demarcation elements include commas, periods, exclamation marks, and question marks.

9. The method according to any one of claims 1 to 8, characterized in that The Copying-ERNIE error correction model is obtained by the following training method: Construct a sample training set with sentences containing incorrect characters as samples and correct sentences as sample labels; Construct a Copying-ERNIE error correction model with a replication mechanism and dynamic weighting; Weight matrix for the linear transformation layer and Relu activation layer of the Softmax classifier 、 Random initialization; smoothing coefficient Initialized as hyperparameters; Input the sample training set into the model, forward propagate to generate the generation probability distribution and replication probability of the character position in each sample, and dynamically weightedly calculate the output probability distribution of the character; Calculate the cross entropy loss between the model output and the true label; Based on the loss value, backpropagation is used to calculate the gradient; the Adam optimization algorithm is used to update the model weights and parameters; Iterate forward propagation, loss calculation, backpropagation, and gradient update until the loss function is minimized; Use accuracy, recall, and F1 score to evaluate model performance; After the training is completed, save the Copying-ERNIE error correction model weights and hyperparameters to obtain the trained Copying-ERNIE error correction model.

Images