A grammar error correction large model training method and device based on reinforcement learning, equipment and medium

Abstract

The application provides a grammar error correction large model training method and device based on reinforcement learning, equipment and medium, relating to the technical field of data processing. The method comprises the following steps: obtaining an initial grammar correction corpus containing error sentences and corresponding correct sentence pairs, processing the initial grammar correction corpus to generate a reasoning correction training set, adjusting a first preset language model according to the reasoning correction training set to obtain an initial strategy model, performing reinforcement learning training on the initial strategy model based on a preset composite function by using a reinforcement learning algorithm, and finally obtaining a target grammar correction model. The target grammar correction model can fully utilize the reasoning ability, effectively improve the performance of the model in terms of precision and recall, better meet the actual needs of grammar correction, and generate correction results containing reasoning processes, which provides more transparent and interpretable basis for the correction process.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, device and medium for training a large model for grammatical error correction based on reinforcement learning. Background Technology

[0002] Currently, with the rapid growth of digital content, grammatical error correction is a crucial task in the field of natural language processing, playing a key role in improving content quality in scenarios such as news publishing and document review. However, existing training methods for various grammatical error correction models generally lack training mechanisms that can guide the model to achieve a balance between maintaining correct sentences and correcting erroneous sentences, fully utilize the model's reasoning ability, and adapt to the error correction task objectives. This results in models with insufficient precision, recall, and generalization ability to unknown domain data, making it difficult to meet the actual needs of high-precision grammatical error correction. Summary of the Invention

[0003] This application provides a method, apparatus, device, and medium for training a large-scale model for grammatical error correction based on reinforcement learning, in order to solve the technical problems existing in the background art.

[0004] Firstly, this application provides a method for training a large-scale grammatical error correction model based on reinforcement learning, including:

[0005] Obtain an initial grammar correction corpus, which includes sentence pairs of erroneous sentences and their corresponding correct sentences;

[0006] The initial syntax error correction corpus is processed to generate an inference error correction training set;

[0007] Based on the inference error correction training set, the first preset language model is adjusted to obtain the initial policy model;

[0008] Based on a preset composite function, a reinforcement learning algorithm is used to train the initial policy model to obtain a target syntax correction model. The preset composite function includes a first sub-function for evaluating the compliance of the output format and a second sub-function for evaluating the semantic correctness of the correction. The target syntax correction model is used to generate a correction result containing the reasoning process based on the received sentence to be corrected.

[0009] In one possible design, processing the initial syntax error correction corpus to generate an inference error correction training set includes:

[0010] The second preset language model is used to process the erroneous sentences in the initial grammar correction corpus to generate triple data containing the erroneous sentences, the reasoning process and the corresponding correct sentences;

[0011] The triplet data is filtered to form the inference error correction training set.

[0012] In one possible design, adjusting the first preset language model based on the inference error correction training set to obtain an initial policy model includes:

[0013] The first preset language model is initially adjusted using general grammar correction data; the first preset language model after the initial adjustment has basic error correction capabilities.

[0014] Based on the inference and error correction training set, the first preset language model, which has been initially adjusted, is further adjusted to obtain the initial strategy model; the initial strategy model has inference capabilities.

[0015] In one possible design, the reinforcement learning algorithm is a group relative policy optimization algorithm;

[0016] The step of training the initial policy model using a reinforcement learning algorithm based on a preset composite function includes:

[0017] For each input sentence in the inference error correction training set, a preset number of candidate output responses are generated by sampling from the initial policy model;

[0018] The total evaluation value of each candidate output response is calculated using the preset composite function.

[0019] The relative strength within each candidate output response is calculated based on the total evaluation value.

[0020] Based on the relative advantages within the group, the model parameters of the initial policy model are updated using the policy gradient update formula to obtain the target syntax correction model.

[0021] In one possible design, calculating the total evaluation value of each of the candidate output responses using the preset composite function includes:

[0022] The first sub-function is used to evaluate the format compliance of the candidate output response to obtain a format evaluation value; and the second sub-function is used to evaluate the error correction semantic correctness of the candidate output response to obtain a semantic evaluation value.

[0023] The format evaluation value and the semantic evaluation value are weighted and summed based on preset weights to obtain the total evaluation value of the candidate output response.

[0024] In one possible design, the evaluation of the format compliance of the candidate output response using the first sub-function to obtain a format evaluation value includes:

[0025] The first sub-function is used to check whether the inference process label and the correction result label exist and are unique in the candidate output response, and to check whether there is redundant content after the correction result label;

[0026] If both the inference process label and the correction result label exist and are unique, and there is no redundant content after the correction result label, output a positive format evaluation value.

[0027] If neither the inference process label nor the correction result label exists or is unique, and if there is redundant content following the correction result label, output a negative format evaluation value.

[0028] In one possible design, the evaluation of the error-correction semantic correctness of the candidate output response using the second sub-function to obtain a semantic evaluation value includes:

[0029] The error state of the original sentence corresponding to the candidate output response is determined by the second sub-function, wherein the error state includes no syntax error and the presence of syntax error;

[0030] The correction results in the candidate output response are compared with the semantic matching degree of the corresponding correct sentences in the initial syntax error correction corpus;

[0031] Based on the comparison results of the error status and the semantic matching degree, the corresponding semantic evaluation value is determined.

[0032] Secondly, this application provides a large-scale training device for grammatical error correction models based on reinforcement learning, comprising:

[0033] The acquisition module is used to acquire an initial grammar correction corpus, which includes sentence pairs of erroneous sentences and corresponding correct sentences;

[0034] The generation module is used to process the initial syntax error correction corpus to generate an inference error correction training set;

[0035] The adjustment module is used to adjust the first preset language model based on the inference error correction training set to obtain an initial strategy model;

[0036] The training module is used to train the initial policy model using a reinforcement learning algorithm based on a preset composite function to obtain a target grammar correction model. The preset composite function includes a first sub-function for evaluating the compliance of the output format and a second sub-function for evaluating the correctness of the error correction semantics. The target grammar correction model is used to generate a correction result containing the reasoning process based on the received sentence to be corrected.

[0037] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0038] The memory stores computer-executed instructions;

[0039] The processor executes computer execution instructions stored in the memory to implement the method as described in any of the first aspects.

[0040] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any of the first aspects.

[0041] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the first aspects.

[0042] This application provides a method, apparatus, device, and medium for training a large-scale grammar error correction model based on reinforcement learning. It involves acquiring an initial grammar error correction corpus containing pairs of erroneous sentences and their corresponding correct sentences, processing the initial corpus to generate an inference error correction training set, adjusting a first preset language model based on the inference error correction training set to obtain an initial policy model, and then training the initial policy model using a reinforcement learning algorithm based on a preset composite function containing a first sub-function for evaluating output format compliance and a second sub-function for evaluating the semantic correctness of error correction. Finally, a target grammar error correction model is obtained. This target grammar error correction model can fully utilize inference capabilities, effectively improving the model's performance in terms of precision and recall, thus better meeting the actual needs of grammar error correction. The generation of correction results containing the inference process also provides a more transparent and interpretable basis for the error correction process. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0044] Figure 1 An application scenario diagram corresponding to a large-scale model training method for grammatical error correction based on reinforcement learning provided in an embodiment of this application;

[0045] Figure 2 A flowchart illustrating a large-scale model training method for grammatical error correction based on reinforcement learning, provided as an embodiment of this application;

[0046] Figure 3A flowchart illustrating a large-scale model training method for grammatical error correction based on reinforcement learning, provided as another embodiment of this application;

[0047] Figure 4 A flowchart illustrating a large-scale model training method for grammatical error correction based on reinforcement learning, provided as another embodiment of this application;

[0048] Figure 5 A schematic diagram of a large-scale training device for grammatical error correction based on reinforcement learning, provided in an embodiment of this application;

[0049] Figure 6 This is a structural example diagram of an electronic device provided in an embodiment of this application.

[0050] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0051] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0052] To clearly understand the technical solution of this application, the solutions of the prior art will be described in detail first.

[0053] Against the backdrop of rapid growth in digital content, grammatical error correction, as an important task in the field of natural language processing, plays a crucial role in improving content quality in scenarios such as news publishing and document review. However, existing training methods for various grammatical error correction models generally lack training mechanisms that can guide the model to achieve a balance between maintaining correct sentences and correcting erroneous sentences, fully utilize the model's reasoning ability, and adapt to the error correction task objectives. This results in the trained models having deficiencies in precision, recall, and generalization ability to unknown domain data, making it difficult to meet the practical needs of high-precision grammatical error correction.

[0054] Figure 1 An application scenario diagram corresponding to a large-scale model training method for grammatical error correction based on reinforcement learning provided in one embodiment of this application is shown below. Figure 1As shown, the application scenario provided in this embodiment includes a model training server 10 and a corpus management system 11. The corpus management system 11 stores an initial grammar correction corpus and various auxiliary corpora. The model training server 10 and the corpus management system 11 establish a communication connection through a local area network or a distributed computing network, supporting corpus transmission, model parameter iteration, and real-time monitoring of the training process.

[0055] First, the model training server 10 retrieves the initial grammar correction corpus from the corpus management system 11, preprocesses the initial grammar correction corpus, extracts corpus features from different scenarios, and adds inference guidance information to each erroneous sentence based on the corpus features, finally generating an inference correction training set. This inference correction training set not only retains the error correction target of sentence pairs but also guides the model to understand the error correction logic through the inference process, giving full play to the model's inference ability.

[0056] Secondly, the model training server 10 calls a preset language model, divides the inference error correction training set by domain, and inputs it into the preset language model in batches for fine-tuning training. During the training process, the preset language model learns the relationship between "error identification, correction logic, and semantic preservation" based on the inference process, gradually forming a basic error correction strategy. Through multiple rounds of iterative adjustment of model parameters, an initial strategy model that can initially balance the conservatism and aggressiveness of error correction is finally obtained. This initial strategy model already has a certain error identification and correction capability.

[0057] Next, the model training server 10 initiates the reinforcement learning training process, using the initial policy model as the agent, the inference and error correction training set as the environment, and a preset composite function as the reward function. After the initial policy model corrects the input sentence, the first sub-function evaluates the format compliance of the output, and the second sub-function evaluates the semantic correctness after correction. Based on the evaluation results of the two sub-functions, a comprehensive reward value is calculated according to preset weights. The reinforcement learning algorithm backpropagates based on the reward value, iteratively optimizing the parameters of the initial policy model and continuously adjusting the model's error correction strategy to achieve a balance between "not changing correct sentences" and "accurately correcting incorrect sentences," thus improving its adaptability to unknown data in different domains. After multiple rounds of reinforcement learning training, the initial policy model is iteratively upgraded to a target grammar correction model. This grammar correction model can receive sentences to be corrected in different scenarios, not only generating accurate correction results but also outputting the corresponding inference process. This not only meets the actual needs of grammar correction but also provides users with a reference for error correction logic, improving the credibility and interpretability of the correction results.

[0058] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0059] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0060] Figure 2 A flowchart illustrating a large-scale model training method for grammatical error correction based on reinforcement learning, as provided in one embodiment of this application, is shown below. Figure 2 As shown, the execution entity of this embodiment is a large-scale grammar error correction model training device based on reinforcement learning. This device can be implemented through a computer program, or through a medium storing the relevant computer program, such as a USB flash drive and / or optical disc; alternatively, it can be implemented through a physical device integrating or installing the relevant computer program, such as a chip or electronic device. The electronic device can be a computer or a server, etc. The large-scale grammar error correction model training method based on reinforcement learning provided in this embodiment includes the following steps:

[0061] S201. Obtain the initial grammar correction corpus, which includes sentence pairs of erroneous sentences and their corresponding correct sentences.

[0062] The initial grammatical error correction corpus is the fundamental data source for the model to learn the mapping relationship between error identification and error correction. The sentence pair between the erroneous sentence and its corresponding correct sentence means that each corpus contains two parts: the original sentence with grammatical errors (also called the erroneous sentence), and the corrected sentence that conforms to grammatical rules and is semantically complete (also called the correct sentence).

[0063] Optionally, the methods for obtaining the initial grammar correction corpus include, but are not limited to, selecting from publicly available grammar correction datasets, manually constructing the corpus, or automatically generating the corpus. Selecting from publicly available grammar correction datasets can utilize datasets such as MuCGEC, NLPCC18, and CTC. These datasets have already labeled a large number of sentence pairs with different types of grammatical errors, such as word choice errors, word order errors, and missing / redundant grammatical components, and can be directly used as the basis for the initial corpus. Manually constructing the corpus can be tailored to the characteristics of texts in specific domains (such as news, academic papers, and corporate documents), with professionals labeling erroneous sentences and their corresponding correct sentences to ensure the corpus's suitability for the target application scenario. Automatically generating the corpus can involve injecting errors into correct text based on rules or pre-trained models to generate erroneous sentences, which are then paired with the original correct sentences to form sentence pairs, allowing for rapid expansion of the corpus size.

[0064] Optionally, the sentence pairs in the initial grammar correction corpus include various types of grammatical errors and texts from different domains to ensure that the subsequently trained model has a strong error recognition range and generalization ability.

[0065] S202. Process the initial grammar correction corpus to generate an inference correction training set.

[0066] Optionally, by processing sentence pairs in the initial corpus, the error correction reasoning process can be integrated into the training data to generate an error correction training set. This allows the model to not only learn the mapping from incorrect sentences to correct sentences during training, but also to learn the reasoning logic of "why it is corrected in this way".

[0067] Optionally, processing the initial syntax correction corpus may include, but is not limited to, inference information annotation, training sample construction, and data cleaning and enhancement.

[0068] Optionally, during the annotation of inference information, for each pair of erroneous and correct sentences in the initial corpus, the corresponding error correction inference process is annotated. The error correction inference process refers to the logical chain from identifying errors to obtaining correction results, which includes at least error location (indicating the specific character / word position in the erroneous sentence), error type judgment (such as word usage error, word order error, subject-verb disagreement, preposition misuse, etc.), and explanation of the basis for correction (such as based on grammatical rules, semantic coherence requirements, fixed collocation habits, etc.).

[0069] Optionally, when constructing training samples, the labeled “error sentence-reasoning process-correct sentence” are combined to form the basic sample unit of the reasoning error correction training set.

[0070] Optionally, during data cleaning and augmentation, the constructed basic sample units are cleaned to remove invalid samples with annotation errors in the reasoning process, mismatches between erroneous and correct sentences, or incomplete text content. Optionally, data augmentation techniques (such as synonym replacement, sentence word order fine-tuning, and error type transfer) are used to expand the size of the reasoning error correction training set, improve the diversity of training data, and further enhance the model's generalization ability. For example, synonym replacement is performed on the erroneous sentence of a sample to generate a new erroneous sentence, which is then combined with the original reasoning process and the correct sentence to form a new training sample.

[0071] S203. Based on the inference error correction training set, adjust the first preset language model to obtain the initial policy model.

[0072] The first pre-selected language model refers to a pre-trained language model with strong natural language understanding and generation capabilities, selected based on factors including semantic understanding ability, text generation fluency, suitable language type (e.g., Chinese, English), and training efficiency. Optional first pre-selected language models include, but are not limited to, large language models such as Qwen and DeepSeek.

[0073] Optionally, the adjustment process for the first preset language model may include steps such as model initialization, loss function setting, and model training and parameter optimization.

[0074] Optionally, during the model initialization phase, the parameters of the first preset language model are loaded into the training framework as initial parameters. Based on the characteristics of the samples in the inference and error correction training set (such as text length and the format of the inference process), the input and output layer structure of the model is adjusted to ensure that the model can receive erroneous sentences as input and output text that conforms to the format requirements. For example, if the first preset language model is GPT-3, a format constraint layer can be added to the model output layer to guide the model to output text containing the inference process.

[0075] Optionally, during the loss function setting stage, the cross-entropy loss function can be used as the loss function for supervised training, with the similarity between the model's output reasoning process, the correct sentence, and the labeled output in the training samples as the optimization objective. Optionally, the predicted probability of each character in the output text can be cross-entropy calculated with the corresponding character in the labeled text to obtain the total loss value.

[0076] Optionally, during the model training and parameter optimization phase, the inference error correction training set is divided into a training set and a validation set, and the model is trained using a mini-batch gradient descent algorithm. During training, the loss value is calculated using the training set, the model parameters are updated using the backpropagation algorithm, and the model performance is monitored using the validation set, such as the accuracy of the output text compared to the labeled text and the completeness of the inference process. When the performance of the validation set tends to stabilize, supervised training is stopped, and the model obtained at this point is the initial policy model.

[0077] S204. Based on a preset composite function, a reinforcement learning algorithm is used to train the initial policy model to obtain a target syntax correction model. The preset composite function includes a first sub-function for evaluating the compliance of the output format and a second sub-function for evaluating the correctness of the error correction semantics. The target syntax correction model is used to generate a correction result containing the reasoning process based on the received sentence to be corrected.

[0078] Optionally, the reinforcement learning framework includes an agent, environment, state, action, and reward. Optionally, in this embodiment, the agent is an initial policy model to be optimized, whose function is to output a correction result (action) containing the reasoning process based on the input sentence to be corrected (state).

[0079] Optionally, in this embodiment, the environment is a reinforcement learning training environment, consisting of an inference error correction training set and a preset evaluation rule (i.e., a preset composite function). The role of the environment is to receive the correction results (actions) output by the agent, calculate the reward value through the preset composite function, and feed the reward value back to the agent to guide the agent in optimizing parameters.

[0080] Optionally, in this embodiment, the state is the input of the agent, which is an erroneous sentence (sentence to be corrected) in the reasoning error correction training set.

[0081] Optionally, in this embodiment, the action is the output of the agent, which is a correction result including the reasoning process.

[0082] Optionally, in this embodiment, the reward is the environment's evaluation of the agent's actions, calculated by a preset composite function, and serves as a signal guiding the optimization of the agent's parameters. The magnitude of the reward value directly reflects the quality of the correction result; the larger the reward value, the better the correction result.

[0083] Among them, the pre-defined composite function is the evaluation criterion for reinforcement learning. The design goal of the pre-defined composite function is to take into account both the practical requirements of "output format compliance" and "error correction semantic correctness".

[0084] The first sub-function is used to evaluate the format compliance of the correction result. The evaluation dimensions of format compliance can include: whether it contains a complete reasoning process, such as whether it contains the three core elements of error location, error type, and correction basis; whether the reasoning process and the correct sentence are separated by a preset separator; whether there are garbled characters or invalid characters in the text; and whether the sentence length is within a preset reasonable range.

[0085] The second sub-function is used to evaluate the semantic correctness of the correct sentences in the correction results. The evaluation dimensions of semantic correctness can include: whether the corrected sentence conforms to the grammatical rules, whether the semantics of the corrected sentence is consistent with the semantics of the original incorrect sentence, and whether all grammatical errors in the incorrect sentence have been completely corrected.

[0086] Optionally, in this embodiment, the reinforcement learning algorithm may be PPO, GRPO, etc.

[0087] Optionally, the process of training the initial policy model may include initializing reinforcement learning parameters and iterative training. Optionally, the parameters that need to be initialized may include the learning rate, batch size, number of iterations, discount factor (used to balance immediate rewards and future rewards), pruning coefficient of the reinforcement algorithm, etc.

[0088] Optionally, during iterative training, firstly, a batch of erroneous sentences are randomly sampled from the inference error correction training set and input into the current initial policy model, which outputs a batch of corrected results. Then, the sampled corrected results and corresponding erroneous sentences are input into a preset composite function to calculate the composite reward value for each corrected result. Next, based on the objective function of the reinforcement learning algorithm (such as the clip objective function of PPO), the calculated composite reward values ​​are used to update the parameters of the initial policy model through backpropagation. Finally, it is determined whether the performance of the current model has reached a preset termination condition. The termination condition may include: the number of iterations reaching a preset maximum value, the model's reward value on the validation set remaining stable above a preset threshold for multiple consecutive iterations, and the model's output corrected results reaching preset standards in precision and recall (e.g., precision ≥ 95%, recall ≥ 90%). If the termination condition is met, reinforcement learning training stops, and the resulting model is the target syntax error correction model; if the termination condition is not met, iteration continues.

[0089] The target syntax error correction model is designed to receive sentences to be corrected and output corrected results containing the complete reasoning process (error location, error type, and correction basis) and the corrected sentence, while ensuring that the corrected results meet the requirements of format compliance and semantic correctness.

[0090] This application provides a reinforcement learning-based method for training a large-scale grammar error correction model. It involves acquiring an initial grammar error correction corpus containing pairs of erroneous sentences and their corresponding correct sentences, processing this corpus to generate an inference error correction training set, adjusting a first preset language model based on the training set to obtain an initial policy model, and then training the initial policy model using a reinforcement learning algorithm based on a preset composite function containing a first sub-function for evaluating output format compliance and a second sub-function for evaluating the semantic correctness of error correction. This ultimately yields a target grammar error correction model. This target grammar error correction model fully utilizes inference capabilities, effectively improving the model's performance in precision and recall, thus better meeting the actual needs of grammar error correction. The generation of correction results containing the inference process also provides a more transparent and interpretable basis for the error correction process.

[0091] As an optional implementation, based on any of the above embodiments, the initial syntax error correction corpus is processed to generate an inference error correction training set, specifically including the following steps:

[0092] First, the erroneous sentences in the initial grammar correction corpus are processed using the second preset language model to generate triplet data containing the erroneous sentences, the reasoning process, and the corresponding correct sentences.

[0093] The second preset language model is a pre-selected pre-trained language model with contextual understanding and logical reasoning capabilities. The second preset language model must meet the following requirements: first, it can accurately capture the differences between incorrect and correct sentences; second, it can generate logical reasoning chains based on the differences; and third, the fluency and standardization of the generated text meet the requirements of the grammar correction task.

[0094] Optionally, the second preset language model can be a large language model such as GPT-5, Qwen-max, or DeepSeek.

[0095] Optionally, the sentence pairs of erroneous sentences and their corresponding correct sentences from the initial grammar correction corpus are used as input, along with preset prompt words, and then input into a second preset language model. The prompt words guide the model to clarify the generation target and the format requirements of the reasoning process, ensuring that the generated reasoning process includes "error location, error type judgment, and explanation of the basis for correction."

[0096] For example, the prompt could be: "Please analyze the differences between the following incorrect sentences and their corresponding correct sentences, and generate a standardized error-correction reasoning process (which must include: error location, error type, and correction basis). The final output format should be: [Incorrect sentence: XXX; Reasoning process: XXX; Correct sentence: XXX]. Incorrect sentence: {incorrect sentence text}; Correct sentence: {correct sentence text}."

[0097] Optionally, the second preset language model, based on the input sentence pairs and prompt words, automatically identifies the error location and determines the error type in the erroneous sentence through contextual semantic understanding, and combines the reasoning and correction criteria such as grammatical rules and semantic coherence requirements to finally generate triple data containing the erroneous sentence, the reasoning process, and the correct sentence.

[0098] For example, given the input of the incorrect sentence "He insists on exercising every day, so he is very healthy" and the correct sentence "He insists on exercising every day, so he is very healthy", the model generates the following triplet data: [Incorrect sentence: He insists on exercising every day, so he is very healthy; Reasoning process: Error position: "good" after "healthy"; Error type: Semantic redundancy; Correction basis: "healthy" already indicates a good physical condition, which is semantically redundant with "good". Deleting "good" makes the sentence grammatically correct and semantically concise; Correct sentence: He insists on exercising every day, so he is very healthy].

[0099] Optionally, each sentence pair from the initial grammar correction corpus is input into the second preset language model in batches as described above to generate batch triplet data. Furthermore, the generated triplet data is preliminarily formatted by extracting three fields—error sentence, reasoning process, and correct sentence—using regular expressions, and removing invalid data with missing fields or disordered formats to obtain a preliminary triplet dataset.

[0100] Secondly, the triplet data is filtered to form a training set for inference error correction.

[0101] Optionally, the quality of the triplet data automatically generated by the model can be controlled, and low-quality data can be removed to ensure that the final inference error correction training set can effectively support subsequent model training and avoid the negative impact of low-quality inference process on model learning.

[0102] Optionally, data filtering of triplet data may include filtering for the validity of the reasoning process, filtering for the consistency of sentence pair matching, and filtering for semantic rationality.

[0103] Optionally, the reasoning process validity filtering refers to selecting triplet data with rigorous reasoning logic and complete elements by constructing reasoning process validity evaluation rules. Optionally, the evaluation rules may include the following three aspects: First, the reasoning process must completely contain the three elements of "error location, error type, and correction basis," and the absence of any element will result in invalidity; Second, the error location mentioned in the reasoning process must accurately correspond to the character / word in the erroneous sentence, and the error type must conform to grammatical norms, such as avoiding vague expressions like "grammatical error" or "inappropriate word choice," and must be accurately determined as specific types such as "semantic redundancy" or "subject-verb disagreement"; Third, the correction basis must be logical and able to reasonably support the correction process from the erroneous sentence to the correct sentence. Through the above evaluation rules, triplet data with incomplete reasoning processes, inaccurate error location, and logical contradictions are eliminated.

[0104] Optionally, sentence pair matching consistency filtering refers to verifying the matching consistency between incorrect and correct sentences in the triplet data, avoiding sentence pair mismatch problems caused by semantic bias during model generation. Optionally, this can be achieved by calculating the semantic similarity between incorrect and correct sentences (using the cosine similarity algorithm), setting a similarity threshold (e.g., ≥0.8), and removing triplet data with similarity below the similarity threshold.

[0105] Optionally, semantic rationality filtering refers to selecting triples that are semantically fluent and unambiguous. Filtering rules include: the reasoning process and correct sentences must be free of grammatical errors, typos, and garbled characters; the language used in the reasoning process must be concise and clear, without redundant expressions or logical jumps; and correct sentences must conform to natural language expression habits, be semantically complete, and unambiguous. Optionally, this filtering can be done automatically using natural language processing tools (such as grammar checking tools and semantic fluency assessment models).

[0106] This application provides a reinforcement learning-based method for training a large-scale grammar error correction model. First, a second preset language model is used to generate triples containing reasoning processes for erroneous sentences in an initial corpus. Then, these triples are filtered to form a reasoning-based error correction training set. This automatically constructs training data with reasoning information, enriching the training sample content and providing more comprehensive information for model training. This helps improve the model's error correction and reasoning abilities, and the data filtering ensures the quality of the training set, thus improving training effectiveness.

[0107] As an optional implementation, based on any of the above embodiments, the first preset language model is adjusted based on the inference error correction training set to obtain an initial policy model, specifically including the following steps:

[0108] First, the first preset language model is initially adjusted using general grammar correction data; the first preset language model after the initial adjustment has basic error correction capabilities.

[0109] Among them, general grammar correction data refers to sentence pairs that cover a general domain and include various basic grammatical error types (word usage errors, word order errors, subject-verb disagreement, preposition misuse, punctuation errors, etc.) with the correct sentences, but does not include inference process annotations.

[0110] Optionally, general grammar correction data can be obtained by filtering publicly available general grammar correction datasets to extract pure sentence pairs without reasoning processes. Optionally, the sentence pair data can be cleaned to remove data with mismatched sentences, invalid error types, and incomplete text, ultimately forming a general grammar correction dataset that ensures comprehensive coverage of basic grammar error types.

[0111] Optionally, based on the text length characteristics of general grammar correction data, the maximum sequence length of the input layer of the first preset language model is adjusted (e.g., set to 256 characters), and a linear mapping layer adapted to generate correct sentences is added to the output layer to ensure that the first preset language model can receive erroneous sentence inputs and output corresponding correct sentences.

[0112] Optionally, a cross-entropy loss function can be used, with the character-level similarity between the correct sentence output by the model and the labeled correct sentence in the general grammar correction data as the optimization objective, guiding the model to learn basic error correction mappings. To ensure that the model fully learns basic error correction capabilities and avoids overfitting, the learning rate can be set to 3e-5, the batch size to 64, the number of training epochs to 8, the AdamW optimizer to be used, and a dropout mechanism (dropout probability set to 0.1) can be introduced to suppress overfitting. Optionally, when the error correction accuracy of the first preset language model on the validation set (divided in an 8:2 ratio) of the general grammar correction data is consistently above 85% for 3 consecutive epochs, and the error recognition recall is consistently above 80%, the initial adjustment is stopped. At this point, the first preset language model has a solid basic grammar correction capability and can accurately correct common grammatical errors in the general domain.

[0113] Secondly, based on the reasoning and error correction training set, the first preset language model after preliminary adjustment is adjusted to obtain the initial policy model; the initial policy model has reasoning ability.

[0114] Optionally, based on the initially adjusted first preset language model, the output layer structure is adapted to guide the model to distinguish between the generation logic of the reasoning process and the correct sentence, ensuring that the output text conforms to the format requirements of "reasoning process, preset separator, correct sentence". Optionally, a format constraint sublayer is added to the original output layer, allowing the model to learn the generation boundaries and format specifications of the two types of text fragments through preset format encoding, such as assigning different format identifiers to the reasoning process and the correct sentence.

[0115] Optionally, an inference correction training set is used, divided into a training set and a validation set in an 8:2 ratio. A weighted cross-entropy loss function is used to assign weights to the inference process generation part and the correct sentence generation part (e.g., the weight of the inference process is set to 0.4, and the weight of the correct sentence is set to 0.6), ensuring both the integrity and logic of the inference process without weakening the model's existing basic error correction capabilities. Optionally, to avoid drastic adjustments that could degrade the basic error correction capabilities, a low learning rate (1e-5), a batch size of 32, and 6 training epochs are set, with the AdamW optimizer still used. When the model's inference process integrity score on the validation set is ≥90% (i.e., the ratio of error location, error type, and correction basis included in the inference process), and the overall accuracy of the inference process and the correct sentence is ≥88%, adjustments are stopped, and the resulting model is the initial strategy model. This initial strategy model possesses basic error correction capabilities and can stably output logically rigorous and formatted error-correcting inference processes.

[0116] This application provides a method for training a large-scale grammar error correction model based on reinforcement learning. First, a first preset language model is trained to acquire basic error correction capabilities using general grammar error correction data, laying the foundation for subsequent improvements. Then, the model is further adjusted using an inference error correction training set, enabling it to possess inference capabilities while having basic error correction capabilities. This step-by-step adjustment allows for the full utilization of different data, effectively improving model performance and enhancing the ability to handle complex grammar error correction tasks.

[0117] As an optional implementation, based on any of the above embodiments, the reinforcement learning algorithm is a group relative policy optimization algorithm.

[0118] Among them, the group relative policy optimization algorithm is a variant of the algorithm in the field of reinforcement learning that is used for multi-agent or group policy optimization. Its core idea is to improve the overall performance of the group policy by comparing relative performance.

[0119] Figure 3 A flowchart illustrating a large-scale model training method for grammatical error correction based on reinforcement learning, as provided in another embodiment of this application, is shown below. Figure 3 As shown, specifically, based on a preset composite function, a reinforcement learning algorithm is used to train the initial policy model, including:

[0120] S301. For each input sentence in the inference error correction training set, sample from the initial policy model to generate a preset number of candidate output responses.

[0121] It should be noted that by constructing a candidate response group for each input sentence (incorrect sentence), the comparison of multiple candidate responses provides a basis for calculating the relative advantage within the group. Compared to the traditional single-sample sampling mode of reinforcement learning, multi-candidate sampling can more comprehensively cover the potential output space of the model for the same input, making the subsequent relative evaluation more representative.

[0122] Optionally, input sentences (erroneous sentences) can be selected from the training set of the inference error correction training set in batches. The batch size can be set according to hardware computing resources, such as 32 sentences / batch or 64 sentences / batch, to ensure a balance between training efficiency and computational load. The selected input sentences should cover different error types to ensure comprehensive training.

[0123] Optionally, for each input sentence, a preset number of candidate output responses are generated based on the output probability distribution of the initial policy model using random sampling. The format of each candidate output response is a combination of "reasoning process, preset separator, and correct sentence". The preset number is chosen to balance evaluation reliability and computational cost, and can range from 5 to 10.

[0124] Optionally, the sampled candidate output responses are initially filtered to remove invalid responses that contain garbled characters, lack elements of the reasoning process (such as unmarked error locations), or have sentence lengths exceeding a preset range (such as being less than 50% or more than 3 times the length of the input sentence). Optionally, if the number of valid responses is less than the preset number, sampling is performed again to ensure that the number of candidate response groups corresponding to each input sentence meets the preset requirement.

[0125] S302. Calculate the total evaluation value of each candidate output response using a preset composite function.

[0126] In this embodiment, each candidate output response in the candidate response group is quantitatively evaluated using an existing preset composite function to obtain a total evaluation value that characterizes the quality of each response, providing a quantitative basis for subsequent calculation of the relative advantage within the group.

[0127] Optionally, each candidate output response (action A) and its corresponding input sentence (state S) are used as input to a pre-defined composite function. The input sentence is the original erroneous sentence of the candidate response, and the candidate output response is the corrected result generated by the model. The pre-defined composite function calculates the total evaluation value for each candidate output response. The total evaluation value can range from [0,1], with a larger value indicating better format compliance, higher semantic correctness, and superior overall performance. Finally, an evaluation record table is established for each input sentence and candidate response group to record the total evaluation value of each candidate output response within the group, which will be used later for calculating the relative advantage within the group.

[0128] For example, if a candidate response group for an input sentence contains 5 responses with total evaluation values ​​of 0.92, 0.85, 0.78, 0.65, and 0.58 respectively, then the evaluation results are recorded in that order.

[0129] S303. Calculate the relative strength within the group for each candidate output response based on the total evaluation value.

[0130] It should be noted that for the i-th candidate output response in the candidate response group, its relative advantage within the group is defined as the difference between the total evaluation value of the response and the mean of the total evaluation values ​​of all candidate responses in the group. If the difference is positive, it means that the response is better than the average level in the group and has a positive relative advantage; if the difference is negative, it means that the response is worse than the average level in the group and has a negative relative advantage.

[0131] Optionally, to avoid the impact of differences in evaluation scales between different candidate response groups on training results, the calculated relative advantages within the group can be normalized and mapped to the [-1,1] interval.

[0132] S304. Based on the relative advantage within the group, update the model parameters of the initial policy model using the policy gradient update formula to obtain the target syntax correction model.

[0133] It should be noted that the relative advantage within the group is used as the guiding signal for policy updates. The parameters of the initial policy model are optimized by the policy gradient descent algorithm, making the model more inclined to generate candidate responses with positive relative advantages, and finally obtaining a target syntax error correction model with excellent overall performance.

[0134] Optionally, based on the relative advantage within the group, the traditional strategy gradient formula is adapted and adjusted. For candidate responses with a positive relative advantage, the weight of their generation probability is increased to guide the model to generate such high-quality responses more easily in the future; for candidate responses with a negative relative advantage, the weight of their generation probability is reduced to suppress the model from generating such low-quality responses.

[0135] Optionally, for all "input sentences and candidate response groups" samples in the current batch, the policy gradient corresponding to each sample is calculated, and then the gradients of all samples within the batch are averaged to obtain the batch gradient. Further, to avoid gradient explosion leading to training instability, the calculated batch gradient is pruned by setting a gradient norm threshold (e.g., 1.0). When the gradient norm exceeds the threshold, the gradient is scaled proportionally to within the gradient norm threshold range. A gradient descent optimizer (e.g., AdamW) is used to backpropagate the pruned batch gradient to the initial policy model, updating all trainable parameters of the model.

[0136] This application provides a reinforcement learning-based method for training a large-scale grammar error correction model. It employs a group relative policy optimization algorithm for reinforcement learning training. By sampling each input sentence to generate multiple candidate output responses, and combining this with a pre-defined composite function to evaluate both format compliance and semantic correctness, it utilizes the relative advantage within the group to calculate accurate policy gradient updates. This approach ensures both the stability and convergence efficiency of the training process, effectively guiding the model to achieve a balance between maintaining correct sentences and correcting erroneous sentences. Simultaneously, it fully leverages the model's reasoning capabilities. The resulting target grammar error correction model exhibits significant improvements in precision, recall, and generalization ability in unknown domains, and can generate correction results that include the reasoning process, thus meeting the practical needs of grammar error correction.

[0137] Figure 4 A flowchart illustrating a large-scale model training method for grammatical error correction based on reinforcement learning, as provided in another embodiment of this application, is shown below. Figure 4 As shown, as an optional implementation, based on any of the above embodiments, the total evaluation value of each candidate output response is calculated using a preset composite function, including the following steps:

[0138] S401. The first sub-function is used to evaluate the format compliance of the candidate output response and obtain the format evaluation value; and the second sub-function is used to evaluate the error correction semantic correctness of the candidate output response and obtain the semantic evaluation value.

[0139] In this embodiment, the first sub-function is specifically designed to evaluate the format compliance of candidate output responses. By constructing a multi-dimensional and quantifiable evaluation index system based on the output format requirements of actual error correction tasks, it is ensured that the evaluation results accurately reflect whether the output format conforms to the specifications of subsequent application processes.

[0140] Optionally, the evaluation dimensions of the first sub-function include, but are not limited to: completeness of the reasoning process, correctness of the use of delimiters, and text standardization. The completeness of the reasoning process refers to whether the reasoning process in the candidate output response is complete and includes the three elements of "error location, error type judgment, and explanation of correction basis." Missing elements directly affect format compliance. The correctness of the use of delimiters refers to whether the candidate output response uses a preset delimiter (such as "[End of Reasoning]") to distinguish the reasoning process from the correct sentence, and whether the delimiter is positioned accurately, such as being immediately adjacent to the end of the reasoning process or the beginning of the correct sentence, without any extra characters separating them. Text standardization refers to whether the candidate output response has basic formatting issues such as garbled characters, typos, and incorrect punctuation.

[0141] Optionally, to adapt to the priority requirements of format dimensions in different scenarios, each evaluation dimension can be configured with an adjustable weight coefficient. For example, the weight of the integrity of the reasoning process can be set to 0.5, the weight of the correctness of the use of delimiters can be set to 0.3, and the weight of the text standardization can be set to 0.2.

[0142] Optionally, for each evaluation dimension, a scoring rule can be adopted: score for compliance, graded score for partial compliance, and zero score for non-compliance. The scores for each dimension can be calculated, and then the format evaluation value can be obtained by weighted summation.

[0143] The second sub-function is specifically designed to evaluate the semantic correctness of the correct sentences in the candidate output response, while balancing the preservation of the original semantics with the thorough correction of errors. Specifically, it evaluates the corrected sentences from two dimensions: error correction effectiveness and semantic consistency, ensuring that the corrected sentences conform to grammatical norms without deviating from the semantics of the original erroneous sentences.

[0144] Optionally, the evaluation dimensions of the second sub-function may include grammatical conformity score and semantic consistency score. The grammatical conformity score evaluates the grammatical compliance of the correct sentences in the candidate output response, i.e., whether the corrected sentence completely eliminates the original error without adding new grammatical errors; the semantic consistency score evaluates whether the corrected sentence is semantically consistent with the original erroneous sentence, avoiding semantic distortion caused by over-correction.

[0145] Optionally, the priority of the two dimensions can be adjusted by weighting coefficients. When the weighting coefficient is larger, more aggressive error correction is given, that is, ensuring grammatical correctness and thoroughly correcting errors is prioritized. When the weighting coefficient is smaller, more conservative error correction is given, that is, ensuring semantic consistency is prioritized and avoiding over-correction is avoided.

[0146] The scores for the two dimensions are calculated separately, and then the semantic evaluation value is obtained by weighted summation.

[0147] S402. The format evaluation value and semantic evaluation value are weighted and summed based on preset weights to obtain the total evaluation value of the candidate output response.

[0148] It should be noted that by fusing the format evaluation value and the semantic evaluation value, a total evaluation value that can comprehensively reflect the quality of the candidate output response is obtained, so that the total evaluation value can accurately match the comprehensive requirements of the actual error correction task.

[0149] Optionally, the preset weight values ​​need to be determined according to the needs of the actual application scenario. The higher the requirement for format standardization in the scenario, the greater the weight value corresponding to the format evaluation value; the higher the requirement for semantic correctness in the scenario, the greater the weight value corresponding to the semantic evaluation value.

[0150] Optionally, the obtained format evaluation value and semantic evaluation value are substituted into a preset composite function, and the total evaluation value is calculated by weighted summation. The total evaluation value ranges from [0,1], and the larger the value, the better the overall performance of the candidate output response.

[0151] Optionally, to avoid evaluation bias caused by a single weighted calculation, a verification step can be added: if the total evaluation value is ≥0.9, it is directly judged as a high-quality candidate response; if 0.6 ≤ total evaluation value <0.9, it is judged as a qualified candidate response; if the total evaluation value <0.6, it is judged as a low-quality candidate response. Low-quality candidate responses can be further analyzed for their format or semantic shortcomings, providing more accurate optimization directions for subsequent model training.

[0152] This application provides a reinforcement learning-based method for training a large-scale grammar error correction model. First, a first sub-function independently evaluates the format compliance of candidate output responses, ensuring the output meets the format requirements of the grammar error correction task. Second, a second sub-function accurately evaluates the semantic correctness of the correction, ensuring the corrected sentence is semantically accurate. Finally, the evaluation values ​​of the two dimensions are weighted and summed based on preset weights to form a total evaluation value. This approach ensures both format and semantic constraints while flexibly guiding the balance between maintaining correct sentences and correcting errors through weight configuration. This allows the reinforcement learning training process to more accurately adapt to the error correction task objective, ultimately improving the overall performance of the target grammar error correction model in terms of precision, recall, and generalization ability.

[0153] As an optional implementation, based on any of the above embodiments, the format compliance of the candidate output response is evaluated using the first sub-function to obtain a format evaluation value, including the following steps:

[0154] First, the first sub-function is used to check whether the inference process label and the correction result label exist and are unique in the candidate output response, and to check whether there is redundant content after the correction result label.

[0155] It should be noted that in this embodiment, by defining the inspection object of the first sub-function, evaluation bias caused by ambiguous label definitions is avoided. Specifically, the inference process label refers to a specific character combination used to identify the starting position of the inference process (e.g., "[Inference Start]"); the correction result label refers to a specific character combination used to identify the ending position of the inference process and the starting position of the correction result. Both labels are preset, dedicated format identifiers that are not easily confused with text content. The specific character combinations of the inference process label and the correction result label can be set according to the actual application scenario, but they must be fixed before model training to ensure consistency of evaluation standards.

[0156] For example, the reasoning process is labeled "[Reasoning Begins]", the correction result is labeled "[Reasoning Ends]", and the standard format of the candidate output response is "[Reasoning Begins] + Reasoning Process + [Reasoning Ends] + Correct Sentence". The two labels constitute the format boundary between the reasoning process and the correction result.

[0157] Optionally, a text retrieval algorithm (such as regular expression matching, string exact matching, etc.) can be used to retrieve the existence status of the reasoning process label and the correction result label in the candidate output response, and output a binary judgment result of "exists" or "does not exist".

[0158] Optionally, provided that the inference process label and the correction result label exist, the number of times the inference process label and the correction result label appear in the candidate output response is counted. If a label appears once, it is determined to be "unique", and if it appears twice or more, it is determined to be "not unique".

[0159] Optionally, assuming the correction result label exists, all text content following the correction result label is extracted, and redundancy is determined using preset rules. Redundant content refers to invalid content after the correction result label, excluding the standard correct sentence, including but not limited to garbled characters, duplicate characters, comments unrelated to the correct sentence, and extra tags. Optionally, the preset judgment rule can be that the text following the correction result label must be a continuous, natural language-compliant correct sentence without the aforementioned invalid content; otherwise, it is considered redundant content, and vice versa.

[0160] Specifically, if both the inference process label and the correction result label exist and are unique, and there is no redundant content following the correction result label, a positively formatted evaluation value is output. If neither the inference process label nor the correction result label exists or is unique, and there is redundant content following the correction result label, a negatively formatted evaluation value is output.

[0161] Optionally, the inference process label exists and is unique, the correction result label exists and is unique, and there is no redundant content after the correction result label. In this case, it indicates that the format of the candidate output response fully conforms to the preset specifications, the format boundaries are clear, there is no invalid content, and it can be directly used for subsequent automated parsing. In this case, a positive format evaluation value is output. The positive format evaluation value is a preset high score; for example, the positive format evaluation value is 1.0 or 0.95.

[0162] Optionally, the inference process label and the correction result label are either nonexistent or non-unique, and there is redundant content following the correction result label. In this case, the format representing the candidate output response is severely violated, the format boundary is invalid, and it cannot be directly used for subsequent applications. The negative format evaluation value is taken as -1 or -0.95 to significantly distinguish it from the positive evaluation value and guide the model to avoid such outputs.

[0163] This application provides a reinforcement learning-based method for training a large-scale grammar error correction model. By checking the existence and uniqueness of the inference process labels and the correction result labels, and ensuring that there is no redundant content following the correction result labels, a positive / negative vectorized expression of the format evaluation value is achieved. This ensures that the output strictly conforms to the format specifications of the grammar error correction task, while avoiding interference from invalid information through redundant content detection, enabling the format evaluation value to accurately reflect the compliance level of the output format.

[0164] As an optional implementation, based on any of the above embodiments, the semantic correctness of the error correction of the candidate output response is evaluated using a second sub-function to obtain a semantic evaluation value, including the following steps:

[0165] First, the error state of the original sentence corresponding to the candidate output response is determined by the second sub-function. The error state includes no syntax error and the presence of a syntax error.

[0166] The candidate output response is the response text generated by the initial policy model for the original input sentence, which includes the reasoning process and the correction result. The correction result refers to the corrected text of the original sentence in the candidate output response, which is the object of semantic correctness evaluation.

[0167] Among them, the original sentence is the input error sentence in the inference error correction training set, which is the original object of the model error correction, and its error state directly determines the direction of semantic evaluation.

[0168] Among them, "no grammatical errors" means that the original sentence fully conforms to the grammatical rules of the target language, without any grammatical problems such as incorrect word choice, word order, semantic contradiction, missing / redundant grammatical components, unclear pronouns, or incorrect fixed collocations, and the semantic expression is complete and logically coherent. "Existence of grammatical errors" means that the original sentence has at least one of the above-mentioned grammatical problems, regardless of whether the error type is explicit (such as incorrect word choice) or implicit (such as subtle semantic contradictions).

[0169] Optionally, grammatical analysis tools or pre-trained models fine-tuned with large-scale grammatical error corpora can be used to determine the error status, ensuring the accuracy, consistency and repeatability of the judgment results and avoiding the subjective bias of manual judgment.

[0170] Secondly, the correction results in the candidate output response are compared with the semantic matching degree of the corresponding correct sentences in the initial syntax error correction corpus.

[0171] Among them, the corresponding correct sentences in the initial grammar correction corpus are standard correct sentences that are paired one-to-one with the original sentences in the initial corpus and have been confirmed by annotation. They serve as the benchmark for evaluating the effectiveness of candidate corrected sentences and ensure that the evaluation has a clear reference basis.

[0172] The semantic matching degree refers to the semantic similarity between the candidate corrected sentence and the standard correct sentence. It is used to quantify the degree of semantic consistency between the two. The value range is strictly limited to [0,1]. The closer the value is to 1, the more consistent the two are semantically; the closer the value is to 0, the greater the semantic deviation between the two.

[0173] It should be noted that by quantifying the semantic consistency between the candidate corrected sentence and the standard correct sentence, invalid outputs that are grammatically correct but semantically deviate from the original meaning after error correction are avoided, and a quantitative basis is provided for subsequent evaluation of the effectiveness of error correction. The higher the semantic matching degree between the candidate corrected sentence and the standard correct sentence, the more accurate the error correction is and the more it conforms to the expressive intent of the original sentence.

[0174] Optionally, a semantic matching degree calculation algorithm with strong contextual semantic understanding capabilities is selected to ensure that deep semantic relationships at the sentence level are captured, rather than relying solely on word overlap. Optionally, the semantic matching degree calculation algorithm can be a semantic similarity algorithm based on a pre-trained language model or a cosine similarity algorithm based on word vectors.

[0175] Optionally, both the candidate corrected sentence and the standard correct sentence undergo unified preprocessing to eliminate interference from non-semantic factors. Preprocessing operations may include: removing irrelevant punctuation, standardizing character case, standardizing full-width and half-width characters, and removing meaningless filler words. Then, using a selected algorithm, the preprocessed candidate corrected sentence and standard correct sentence are transformed into fixed-dimensional semantic vectors (such as the 768-dimensional vector output by the BERT model). Next, the similarity between the two semantic vectors is calculated using the cosine similarity formula to obtain the semantic matching degree, with the semantic matching degree ranging from [0,1].

[0176] Finally, based on the comparison results of the error status and semantic matching degree, the corresponding semantic evaluation value is determined.

[0177] The semantic evaluation value is a quantitative indicator that comprehensively reflects the "correct semantic correctness" of the candidate output response. The value range is [0,1]. The larger the value, the better the semantic correctness of the candidate output response, which completely eliminates the grammatical errors of the original sentence, does not deviate from the semantics of the original sentence, and has no problems of over-correction or incomplete correction.

[0178] Optionally, when the original sentence error status is "no grammatical error", the semantic matching result is analyzed as follows: Optionally, if the semantic matching degree is ≥ the first preset matching degree value (e.g., 0.9), it indicates that the candidate corrected sentence has not excessively modified the original sentence, and the semantics are basically completely consistent with the standard correct sentence (i.e., the original sentence itself), with no semantic distortion problem. A high semantic evaluation value is output, with an example value of 0.9-1.0, such as 1.0. Optionally, if the second preset matching degree value (e.g., 0.6) ≤ the semantic matching degree < the first preset matching degree value, it indicates that the candidate corrected sentence has slightly modified the original sentence, but the core semantics have not deviated, which is an acceptable fine-tuning. A medium semantic evaluation value is output, with an example value of 0.5-0.8, such as 0.7. Optionally, if the semantic matching degree < the second preset matching degree value, it indicates that the candidate corrected sentence has excessively modified the original sentence, and the semantics have seriously deviated from the original meaning, which is an invalid error correction. A low semantic evaluation value is output, with an example value of 0-0.4, such as 0.1.

[0179] Optionally, when the original sentence error status is "grammatical error exists," the semantic matching result is analyzed as follows. Specifically, optionally, if the semantic matching degree is ≥ the first preset matching degree value, it means that the candidate corrected sentence has completely eliminated all grammatical errors in the original sentence and is semantically basically completely consistent with the standard correct sentence, with no semantic deviation, and a high semantic evaluation value, such as 1.0, is output. Optionally, if the second preset matching degree value is ≤ the semantic matching degree value < the first preset matching degree value, it means that the candidate corrected sentence has eliminated some or all grammatical errors in the original sentence, and the semantics have not deviated from the original sentence, which is considered a qualified error correction, and a medium semantic evaluation value, such as 0.7, is output.

[0180] Optionally, if the semantic matching degree is less than the second preset matching degree value, it means that the candidate correction sentence has not eliminated the core grammatical error of the original sentence, or although the error has been eliminated, the semantics have seriously deviated from the original meaning, which is an invalid error correction, and a low semantic evaluation value, such as 0, is output.

[0181] Optionally, to further improve the accuracy of the assessment, the number of errors in the candidate corrected sentence can be compared with that in the original sentence using a grammatical analysis tool. If the number of grammatical errors in the candidate corrected sentence is reduced by ≥80% compared with that in the original sentence, it is considered that the errors have been basically eliminated; if the reduction is <50%, it is considered that the errors have not been effectively eliminated.

[0182] This application provides a reinforcement learning-based method for training a large-scale grammar error correction model. First, it determines whether the original sentence contains grammatical errors. Then, it compares the corrected result with correct sentences in a corpus to quantify the accuracy of the correction. Finally, it combines the error state and the semantic matching result to determine the semantic evaluation value. This avoids unnecessary penalties for the model when the original sentence is error-free, while accurately measuring the semantic fit between the corrected result and the standard answer, ensuring that the semantic evaluation value truly reflects the model's semantic processing capability in the error correction task.

[0183] Figure 5 A schematic diagram of a large-scale training device for grammatical error correction based on reinforcement learning is provided in one embodiment of this application, as shown below. Figure 5 As shown, the large-scale grammar error correction model training device based on reinforcement learning provided in this embodiment is located in an electronic device. The large-scale grammar error correction model training device 50 based on reinforcement learning provided in this embodiment includes: an acquisition module 51, a generation module 52, an adjustment module 53, and a training module 54.

[0184] Specifically, the acquisition module 51 is used to acquire an initial grammar correction corpus, which includes sentence pairs of erroneous sentences and their corresponding correct sentences; the generation module 52 is used to process the initial grammar correction corpus to generate an inference correction training set; the adjustment module 53 is used to adjust the first preset language model based on the inference correction training set to obtain an initial policy model; the training module 54 is used to perform reinforcement learning training on the initial policy model based on a preset composite function using a reinforcement learning algorithm to obtain a target grammar correction model; wherein, the preset composite function includes a first sub-function for evaluating the compliance of the output format and a second sub-function for evaluating the correctness of the correction semantics; the target grammar correction model is used to generate a correction result containing the inference process based on the received sentence to be corrected.

[0185] Optionally, the generation module 52, when processing the initial grammar correction corpus to generate the inference correction training set, specifically performs the following: using the second preset language model to process the erroneous sentences in the initial grammar correction corpus, generating triple data containing erroneous sentences, inference processes, and corresponding correct sentences; filtering the triple data to form the inference correction training set.

[0186] Optionally, when adjusting the first preset language model based on the inference error correction training set to obtain the initial strategy model, the adjustment module 53 is specifically used to: perform preliminary adjustment on the first preset language model using general syntax error correction data; the first preset language model after preliminary adjustment has basic error correction capabilities; adjust the first preset language model after preliminary adjustment based on the inference error correction training set to obtain the initial strategy model; the initial strategy model has inference capabilities.

[0187] Optionally, the reinforcement learning algorithm is a group relative policy optimization algorithm. Optionally, the training module 54, when performing reinforcement learning training on the initial policy model based on a preset composite function, specifically performs the following: for each input sentence in the inference correction training set, it samples from the initial policy model to generate a preset number of candidate output responses; it calculates the total evaluation value of each candidate output response using the preset composite function; it calculates the intra-group relative advantage of each candidate output response based on the total evaluation value; and it updates the model parameters of the initial policy model according to the intra-group relative advantage using the policy gradient update formula to obtain the target syntax correction model.

[0188] Optionally, the training module 54, when calculating the total evaluation value of each candidate output response using a preset composite function, specifically performs the following: using the first sub-function to evaluate the format compliance of the candidate output response and obtain a format evaluation value; and using the second sub-function to evaluate the error correction semantic correctness of the candidate output response and obtain a semantic evaluation value; and performing a weighted summation of the format evaluation value and the semantic evaluation value based on preset weights to obtain the total evaluation value of the candidate output response.

[0189] Optionally, when training module 54 evaluates the format compliance of candidate output responses using the first sub-function to obtain a format evaluation value, it specifically performs the following: checks whether the inference process label and the correction result label exist and are unique in the candidate output response, and checks whether there is redundant content after the correction result label; if both the inference process label and the correction result label exist and are unique, and there is no redundant content after the correction result label, it outputs a positive format evaluation value; if neither the inference process label nor the correction result label exists or is unique, and there is redundant content after the correction result label, it outputs a negative format evaluation value.

[0190] Optionally, when training module 54 evaluates the semantic correctness of the candidate output response using the second sub-function to obtain a semantic evaluation value, it specifically performs the following: determining the error state of the original sentence corresponding to the candidate output response through the second sub-function, wherein the error state includes no grammatical error and the presence of grammatical error; comparing the correction result in the candidate output response with the semantic matching degree of the corresponding correct sentence in the initial grammatical error correction corpus; and determining the corresponding semantic evaluation value based on the comparison result of the error state and the semantic matching degree.

[0191] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, as shown below. Figure 6 As shown, the electronic device 60 provided in this embodiment includes a processor 61 and a memory 62 communicatively connected to the processor 61.

[0192] The memory 62 stores computer execution instructions; the processor 61 executes the computer execution instructions stored in the memory 62 to implement the large-scale model training method for syntax error correction based on reinforcement learning provided in any of the above embodiments.

[0193] The program may include program code, which includes computer-executable instructions. Memory 62 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device.

[0194] In this embodiment, the memory 62 and the processor 61 are connected via a bus. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The bus is represented by a single straight line, but this does not mean that there is only one bus or one type of bus.

[0195] This application also provides a computer-readable storage medium, which stores computer-executable instructions. When executed by a processor, the computer-executable instructions are used to implement a large-scale model training method for syntax error correction based on reinforcement learning provided in any of the above embodiments.

[0196] This application also provides a computer program product, including a computer program that, when executed by a processor, implements a large-scale model training method for syntax error correction based on reinforcement learning provided in any of the above embodiments.

[0197] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0198] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0199] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0200] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0201] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0202] The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0203] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic control unit or main control device.

[0204] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0205] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for training a large-scale grammatical error correction model based on reinforcement learning, characterized in that, include: Obtain an initial grammar correction corpus, which includes sentence pairs of erroneous sentences and their corresponding correct sentences; The initial syntax error correction corpus is processed to generate an inference error correction training set; Based on the inference error correction training set, the first preset language model is adjusted to obtain the initial policy model; Based on a preset composite function, a reinforcement learning algorithm is used to train the initial policy model to obtain a target syntax correction model. The preset composite function includes a first sub-function for evaluating the compliance of the output format and a second sub-function for evaluating the semantic correctness of the correction. The target syntax correction model is used to generate a correction result containing the reasoning process based on the received sentence to be corrected.

2. The method according to claim 1, characterized in that, The process of processing the initial syntax error correction corpus to generate an inference error correction training set includes: The second preset language model is used to process the erroneous sentences in the initial grammar correction corpus to generate triple data containing the erroneous sentences, the reasoning process and the corresponding correct sentences; The triplet data is filtered to form the inference error correction training set.

3. The method according to claim 1, characterized in that, The step of adjusting the first preset language model based on the inference error correction training set to obtain an initial policy model includes: The first preset language model is initially adjusted using general grammar correction data; the first preset language model after the initial adjustment has basic error correction capabilities. Based on the inference and error correction training set, the first preset language model, which has been initially adjusted, is further adjusted to obtain the initial strategy model; the initial strategy model has inference capabilities.

4. The method according to claim 1, characterized in that, The reinforcement learning algorithm is a group relative policy optimization algorithm; The step of training the initial policy model using a reinforcement learning algorithm based on a preset composite function includes: For each input sentence in the inference error correction training set, a preset number of candidate output responses are generated by sampling from the initial policy model; The total evaluation value of each candidate output response is calculated using the preset composite function. The relative strength within each candidate output response is calculated based on the total evaluation value. Based on the relative advantages within the group, the model parameters of the initial policy model are updated using the policy gradient update formula to obtain the target syntax correction model.

5. The method according to claim 4, characterized in that, The calculation of the total evaluation value of each candidate output response using the preset composite function includes: The first sub-function is used to evaluate the format compliance of the candidate output response to obtain a format evaluation value; and the second sub-function is used to evaluate the error correction semantic correctness of the candidate output response to obtain a semantic evaluation value. The format evaluation value and the semantic evaluation value are weighted and summed based on preset weights to obtain the total evaluation value of the candidate output response.

6. The method according to claim 5, characterized in that, The step of evaluating the format compliance of the candidate output response using the first sub-function to obtain a format evaluation value includes: The first sub-function is used to check whether the inference process label and the correction result label exist and are unique in the candidate output response, and to check whether there is redundant content after the correction result label; If both the inference process label and the correction result label exist and are unique, and there is no redundant content after the correction result label, output a positive format evaluation value. If neither the inference process label nor the correction result label exists or is unique, and if there is redundant content following the correction result label, output a negative format evaluation value.

7. The method according to claim 5, characterized in that, The step of evaluating the semantic correctness of the candidate output response using the second sub-function to obtain a semantic evaluation value includes: The error state of the original sentence corresponding to the candidate output response is determined by the second sub-function, wherein the error state includes no syntax error and the presence of syntax error; The correction results in the candidate output response are compared with the semantic matching degree of the corresponding correct sentences in the initial syntax error correction corpus; Based on the comparison results of the error status and the semantic matching degree, the corresponding semantic evaluation value is determined.

8. A training device for a large-scale grammatical error correction model based on reinforcement learning, characterized in that, include: The acquisition module is used to acquire an initial grammar correction corpus, which includes sentence pairs of erroneous sentences and corresponding correct sentences; The generation module is used to process the initial syntax error correction corpus to generate an inference error correction training set; The adjustment module is used to adjust the first preset language model based on the inference error correction training set to obtain an initial strategy model; The training module is used to train the initial policy model using a reinforcement learning algorithm based on a preset composite function to obtain a target syntax correction model. The preset composite function includes a first sub-function for evaluating the compliance of the output format and a second sub-function for evaluating the correctness of the error correction semantics. The target syntax correction model is used to generate a correction result containing the reasoning process based on the received sentence to be corrected.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

Images