Answer correction method, correction model training method and related device

Abstract

The invention discloses an answer correction method, a correction model training method and a related device, and relates to the technical field of data processing, and the method comprises the steps: obtaining answer context data, including first data of an image type and second data of a text type, generating a first visual token set according to the first data through a correction model after reinforcement learning training, the first visual token set is subjected to token screening, a correction result is generated according to the first visual tokens reserved after screening and the second data, rewards used in the reinforcement learning process of the correction model comprise an accuracy reward and a token screening reward, and the accuracy reward and the token screening reward are used for correcting the correction result. The accuracy reward is used for encouraging the model to retain visual tokens with high contribution degree to correction accuracy in token screening, and the token screening reward is used for encouraging the model to retain few visual tokens in token screening. According to the method, the correction model is trained by reinforcement learning through double rewards, so that the correction efficiency is improved by screening out redundant visual tokens under the condition of ensuring high correction accuracy.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method for grading answers, a method for training grading models, and related apparatus. Background Technology

[0002] In automatic grading scenarios, when it is necessary to grade the answer context data related to the respondent's answer, the grading process is as follows: using optical character recognition (OCR) technology, the image-type data in the answer context data is recognized as text data, and then grading is performed based on the recognized text data and the text-type data in the answer context data.

[0003] However, OCR technology is prone to misidentification. For example, it may identify incorrect text as correct text in image-type data, or vice versa, reducing the accuracy of subsequent grading. Furthermore, when image-type data includes graphics, OCR technology can only identify the type of graphics but cannot retain key visual information such as the spatial structure, size proportions, and element positions of the graphics. This affects the accurate judgment of the respondent's answer logic and problem-solving steps during the subsequent grading process, further reducing the accuracy of grading. Summary of the Invention

[0004] In view of the above problems, this application provides a method for grading answers, a method for training a grading model, and related apparatus to achieve high accuracy in grading answer context data and improve grading efficiency. The specific solution is as follows:

[0005] The first aspect of this application provides a method for grading responses, including:

[0006] Obtain answer context data related to the respondent's answer, wherein the answer context data includes first data of image type and second data of text type;

[0007] The grading model trained by reinforcement learning generates a first set of visual tokens based on the first data, performs token filtering on the first set of visual tokens, and generates a grading result based on the first visual tokens retained after filtering and the second data.

[0008] The reinforcement learning process of the grading model uses rewards including accuracy rewards and token removal rewards. The accuracy rewards are used to encourage the model to retain visual tokens that contribute highly to the grading accuracy during token removal, while the token removal rewards are used to encourage the model to retain fewer visual tokens during token removal.

[0009] In one possible implementation, the training process of the reinforcement learning-trained grading model includes:

[0010] Obtain answer context data samples and grading result labels, wherein the answer context data samples include a first data sample of image type and a second data sample of text type;

[0011] The grading model generates a second set of visual tokens based on the first data sample, performs token filtering on the second set of visual tokens, and generates a predicted grading result based on the second set of visual tokens retained after filtering and the second data sample.

[0012] The accuracy reward is generated based on the predicted correction result and the correction result label, and the token elimination reward is generated based on the second visual token and the second visual token set.

[0013] The grading model is trained using reinforcement learning based on the accuracy reward and the token elimination reward to obtain the grading model after reinforcement learning training.

[0014] In one possible implementation, generating the token filtering reward based on the second visual token and the second set of visual tokens includes:

[0015] Determine the number of the second visual tokens and the total number of tokens in the second visual token set;

[0016] The target reward is obtained based on the number of the second visual tokens and the total number of tokens, and the target reward is inversely proportional to the number of the second visual tokens;

[0017] If the predicted correction result has the same label as the correction result, then the token elimination reward is the target reward;

[0018] If the predicted correction result is different from the correction result label, then the token screening reward is a preset minimum value that tends to or equals 0.

[0019] In one possible implementation, training the grading model using reinforcement learning based on the accuracy reward and the token removal reward includes:

[0020] Based on the accuracy reward and the token elimination reward, the expected reward of reinforcement learning is obtained, and the expected reward is inverted as the target loss.

[0021] Determine the rejection ratio loss, which is used to measure the positive deviation between the visual token retention ratio and the expected retention ratio. The visual token retention ratio is the ratio of the number of the second visual tokens to the total number of tokens. The positive deviation means that the difference between the visual token retention ratio and the expected retention ratio is positive.

[0022] The batch correction model is trained using reinforcement learning based on the target loss and the screening ratio loss.

[0023] In one possible implementation, obtaining the expected reward for reinforcement learning based on the accuracy reward and the token elimination reward includes:

[0024] A format reward is obtained based on the format of the output data generated by the correction model, and / or a length reward is obtained based on the length of the output data, wherein the output data includes at least the predicted correction result. The format reward is used to encourage the model to generate output data that conforms to the target format, and the length reward is used to encourage the model to generate output data whose length does not exceed the ideal length.

[0025] The expected return is obtained by combining the accuracy reward and the token elimination reward with the format reward and / or the length reward.

[0026] In one possible implementation, the target format is to use a preset thought process marker to stitch together the correction results after wrapping the thought chain content or empty content.

[0027] The method for grading answers also includes:

[0028] During the process of generating thought chain content by the grading model after reinforcement learning training, the confidence level of the new token and the current thought chain length are determined for each new token generated.

[0029] Based on the confidence level of the new token and the current thought chain length, the next operation is determined. The next operation is to continue reasoning to generate a new token or to add an end marker from the thought process marker pair after the currently generated thought chain content.

[0030] In one possible implementation, determining the next operation based on the confidence level of the new token and the current thought chain length includes:

[0031] If the current thought chain length is greater than the ideal length, and the confidence level of the new token is greater than the first confidence level threshold, or the confidence level of the new token is less than the second confidence level threshold, then the end marker is added after the currently generated thought chain content, where the first confidence level threshold is greater than the second confidence level threshold.

[0032] If the current thought chain length is less than or equal to the ideal length, or the confidence level of the new token is within the target range, then the reasoning continues to generate a new token. The target range is a closed interval composed of the first confidence level threshold and the second confidence level threshold.

[0033] In one possible implementation, the token filtering of the first visual token set includes:

[0034] Obtain a weight matrix, which is derived from the self-attention weights between visual tokens and / or the cross-attention weights between visual tokens and text tokens of the second data;

[0035] Each visual token in the first visual token set is weighted according to the weight matrix to obtain the target attention weight of each visual token; thus obtaining the target attention weight of each visual token in the first visual token set.

[0036] Token filtering is performed based on the target attention weight of each of the visual tokens.

[0037] The second aspect of this application provides a method for batch model training, including:

[0038] Obtain answer context data samples and grading result labels, wherein the answer context data samples include a first data sample of image type and a second data sample of text type;

[0039] The second visual token set is generated by the grading model based on the first data sample. The second visual token set is then screened out. The predicted grading result is generated based on the second visual tokens retained after screening and the first data sample.

[0040] An accuracy reward is generated based on the predicted correction result and the correction result label, and a token removal reward is generated based on the second visual token and the second visual token set. The accuracy reward is used to encourage the model to retain visual tokens that contribute highly to the correction accuracy during token removal, and the token removal reward is used to encourage the model to retain fewer visual tokens during token removal.

[0041] The grading model is trained using reinforcement learning based on the accuracy reward and the token elimination reward.

[0042] A third aspect of this application provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement the response correction method of the first aspect or any implementation thereof, or the correction model training method of the second aspect.

[0043] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0044] The memory is used to store computer programs;

[0045] The processor is used to execute the computer program so that the electronic device can implement the answer grading method of the first aspect or any implementation thereof, or the grading model training method of the second aspect.

[0046] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the answer correction method of the first aspect or any implementation thereof, or the correction model training method of the second aspect.

[0047] By employing the above technical solution, the answer correction method provided in this application obtains answer context data related to the respondent's answer. This answer context data includes first data of image type and second data of text type. A correction model trained through reinforcement learning generates a first set of visual tokens based on the first data. Token filtering is performed on the first set of visual tokens, and a correction result is generated based on the retained first visual tokens and second data. The reinforcement learning process of the correction model uses rewards including accuracy rewards and token filtering rewards. Accuracy rewards encourage the model to retain visual tokens that contribute significantly to correction accuracy during token filtering, while token filtering rewards encourage the model to retain fewer visual tokens. Therefore, this application can convert first data of image type into visual tokens. Compared to OCR recognition, visual tokens can correctly retain more key visual information of the image, thus enabling more accurate correction based on this key visual information and improving the accuracy of the correction result.

[0048] Meanwhile, this application employs an accuracy reward mechanism to ensure that the grading model accurately identifies and retains visual information crucial to the grading task when filtering visual tokens, guaranteeing high accuracy of the grading results. The token filtering reward mechanism effectively suppresses the retention of redundant visual information that has little or no impact on the grading task. This allows the grading model to filter out more useless visual tokens, reducing the computational complexity of the grading process, improving subsequent grading efficiency, and reducing memory usage. The synergistic optimization of these two reward mechanisms enables the grading model to learn the contribution value of different visual tokens to the final grading result during training. This allows it to adaptively adjust the token selection strategy according to the specific grading scenario, achieving more efficient and accurate grading. Attached Figure Description

[0049] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0050] Figure 1 A schematic diagram of a system architecture provided in this application;

[0051] Figure 2 A flowchart illustrating a method for grading responses provided in this application;

[0052] Figure 3 A schematic diagram of the correction process for a correction model provided in this application;

[0053] Figure 4 A flowchart illustrating a graded model training method provided in this application;

[0054] Figure 5 A schematic diagram of a response correction device provided in this application;

[0055] Figure 6 A schematic diagram of a model training device provided in this application;

[0056] Figure 7 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0057] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0058] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0059] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0060] In order to grade the response context data related to the respondent's answer, the inventors of this case conducted in-depth research. The initial approach was as follows: the visual encoder in the multimodal grading model extracts features from the first data of image type in the response context data to obtain multiple visual tokens. Then, the text encoder in the grading model extracts features from the second data of text type in the response context data to obtain multiple text tokens. Finally, grading is performed based on the multiple visual tokens and multiple text tokens to obtain the grading result.

[0061] By tokenizing data, more key visual and textual information from the context of the response can be retained, resulting in more accurate grading results.

[0062] However, most current visual encoders are based on the transformer architecture. The computational cost of the transformer architecture is quadratically positively correlated with the number of visual tokens, which leads to a significant decrease in the efficiency of high-resolution image processing and a sharp increase in memory usage as the image resolution increases.

[0063] To address the aforementioned issues, the inventors conducted further in-depth research. This research revealed that multiple visual tokens generated from the first data of the image type contain a large amount of repetitive or irrelevant visual information, such as creases on the edges of the exam paper, noise in the image background, handwritten doodles in non-critical areas, and overlapping interference between the question text and the image. Removing these visual tokens does almost no reduction in the accuracy of the grading results. Based on this, this application provides a grading method, a grading model training method, and related apparatus based on adaptive image token selection technology, which will be described in detail below.

[0064] Optionally, the answer grading method and grading model training method provided in this application can be applied to, for example... Figure 1 The system architecture shown includes a terminal 100 and a server 200. The server 200 may include one or more servers (…). Figure 1 (This example uses a server as an illustration).

[0065] Either terminal 100 or server 200 can be used independently to execute the answer grading method and grading model training method provided in the embodiments of this application. Alternatively, terminal 100 and server 200 can also be used collaboratively to execute the answer grading method and grading model training method provided in the embodiments of this application.

[0066] The following description Figure 1 The product form of the mid-terminal 100;

[0067] The terminal 100 in this application embodiment can be a mobile phone, tablet computer, wearable device, vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.

[0068] To enable those skilled in the art to better understand this application, the response and correction methods of the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0069] Reference Figure 2 , Figure 2 This is a flowchart illustrating a method for correcting responses provided in an embodiment of this application, as shown below. Figure 2 As shown, the method for grading answers may include:

[0070] Step S201: Obtain answer context data related to the respondent's answer. The answer context data includes first data of image type and second data of text type.

[0071] In this embodiment, the answer context data includes at least the respondent's answer data. Optionally, the answer context data may also include questions and / or standard answers related to the answer data.

[0072] The response context data can include two types of data: first data of image type and second data of text type.

[0073] Optionally, the first data of the image type can be an image obtained by taking pictures of graphic data of the respondent's answer, graphic data in the question, graphic data in the standard answer, etc.; it can also be an image obtained by taking pictures of text data of the respondent's answer, text data in the question, text data in the standard answer, etc.

[0074] For example, in some scenarios of answering and grading math geometry problems, the questions contain textual data and related geometric figures. Students need to add auxiliary lines to the figures and provide textual solutions based on the added auxiliary lines. In such scenarios, the questions, standard answers, and student answer data all contain textual data and image data.

[0075] In some scenarios, the questions only contain text data, and students need to provide relevant graphics based on the text data. In this scenario, both the standard answer and the student's answer data contain only image-type graphic data, while the questions only contain text-type text data.

[0076] Of course, there are many other scenarios besides these, which will not be introduced one by one here.

[0077] To ensure more accurate subsequent corrections, optionally, after obtaining the first and second data, preprocessing can be performed on the first and second data respectively. The preprocessing of the first data includes, but is not limited to, the following: normalization and size adjustment. The preprocessing of the second data includes, but is not limited to, the following: word segmentation and part-of-speech tagging.

[0078] Step S202: The grading model trained by reinforcement learning generates a first visual token set based on the first data, performs token filtering on the first visual token set, and generates a grading result based on the first visual tokens and the second data retained after filtering.

[0079] The reinforcement learning process of the grading model uses rewards including accuracy rewards and token removal rewards. Accuracy rewards are used to encourage the model to retain visual tokens that contribute highly to the accuracy of grading during token removal, while token removal rewards are used to encourage the model to retain fewer visual tokens during token removal.

[0080] Optionally, the correction model in this embodiment can be a multimodal large model, such as the LLaVA (Large Language and Vision Assistant) model.

[0081] Optionally, the grading model can be a large multimodal model consisting of a visual encoder, a text encoder, a token filtering module, and a grading module. For example, see [link to relevant documentation]. Figure 3 The diagram shown is a schematic representation of the correction process of a correction model provided in this application.

[0082] Here, the visual encoder adopts a transformer architecture, which can encode the first data of the image type, thereby encoding the key information in the image into M visual tokens, that is, the first visual token set.

[0083] Optionally, the visual encoder first divides the first data into blocks to obtain multiple image blocks, and then encodes each image block separately to obtain a total of M visual tokens. For ease of explanation, the first visual token set consisting of M visual tokens is defined below. , D represents the dimension of the visual token. For example, assuming the first data is a 224×224 image, then dividing it into 16×16 dimensions will yield M=576 visual tokens.

[0084] Then, the M visual tokens are filtered by the token filtering module to select N visual tokens, where M is greater than N and both M and N are positive integers. In this embodiment, the set of N visual tokens is used... express.

[0085] The second data, which is text-type, can be fed into a text encoder for encoding to obtain multiple text tokens.

[0086] Finally, multiple text tokens and N visual tokens are input into the grading module to generate the grading results.

[0087] Optionally, the grading module can be a large language model, in which case multiple text tokens can be concatenated with N visual tokens, and the concatenated token sequence is then input into the large language model. After deep reasoning by the large language model, the grading result is obtained.

[0088] It should be noted that, Figure 3 The structure of the grading model shown is only an example. Other model structures are also possible. For example, a mapping layer can be added before or after the token filtering module to map the visual tokens output by the visual encoder to visual tokens that the large language model can understand. Another example is that the second data of text type can be directly concatenated with N visual tokens and input into the large language model without encoding.

[0089] In order to train the grading model to retain as many useful visual tokens as possible and filter out as many useless visual tokens as possible while ensuring grading accuracy, this embodiment can use reinforcement learning to train the grading model. The accuracy reward and the token filtering reward are used as rewards in the reinforcement learning process of the grading model to train the grading model's ability to filter visual tokens.

[0090] Among them, the accuracy reward is used to encourage the model to retain visual tokens that contribute highly to the accuracy of the grading during token elimination, so as to improve the accuracy of the grading results by retaining more of these useful visual tokens;

[0091] The token removal reward is used to encourage the model to retain fewer visual tokens during token removal, that is, to train the grading model to remove more visual tokens, thereby reducing computation and memory usage and improving grading efficiency.

[0092] By combining accuracy rewards and token elimination rewards, the grading model can be trained to retain more useful visual tokens that contribute to grading accuracy and eliminate more redundant visual tokens that do not contribute to grading accuracy. Through multiple rounds of training, the grading model can learn to distinguish the importance of different visual tokens. Thus, even when faced with unfamiliar grading tasks, it can adaptively adjust its token selection strategy based on its learned knowledge, achieving more efficient token selection and thereby improving the accuracy and efficiency of grading results.

[0093] Optionally, this embodiment can employ an attention mechanism to filter tokens from the first set of visual tokens. To obtain the weight matrix in the attention mechanism, self-attention weights can be calculated between the generated visual tokens. Optionally, cross-attention weights can also be calculated between the visual tokens and the text tokens of the second data. Then, the weight matrix can be obtained from the self-attention weights and / or cross-attention weights, which will be referred to below. express.

[0094] Furthermore, each visual token in the first visual token set can be weighted according to the weight matrix to obtain the attention weight of each visual token. Optionally, the weighting formula can be: , Indicates the target attention weight. Indicates the bias term. This represents the sigmoid activation function.

[0095] right Each visual token in the set is weighted according to the formula above, which yields the target attention weights for each visual token in the first visual token set.

[0096] Finally, token filtering can be performed based on the target attention weights of each visual token.

[0097] Optionally, the token filtering process is as follows: if the target attention weight of a visual token is greater than a preset threshold... If the visual token is selected, it is retained; otherwise, if the target attention weight of the visual token is less than or equal to the threshold, it is discarded. If so, the visual token will be filtered out. It is a hyperparameter in the interval [0,1).

[0098] For example, in one possible implementation, it could be based on whether the target attention weight is greater than... The logic sets the filter marker Z; if the target attention weight of a visual token is greater than... If Z=1, it means that the visual token is retained if the target attention weight of a visual token is less than or equal to If Z=0, it means that the visual token is removed.

[0099] Therefore, based on the filtering marker Z, visual tokens with Z=1 can be extracted from the first visual token set as the first visual tokens, i.e., according to the formula... Filtered Then All first visual tokens are assembled in their original order for accurate correction.

[0100] The answer correction method provided in this application obtains answer context data related to the respondent's answer. This answer context data includes first data of image type and second data of text type. A correction model trained through reinforcement learning generates a first set of visual tokens based on the first data. The first set of visual tokens is then filtered out, and a correction result is generated based on the retained first visual tokens and second data. The reinforcement learning process of the correction model uses rewards including accuracy rewards and token filtering rewards. Accuracy rewards encourage the model to retain visual tokens that contribute significantly to correction accuracy during token filtering, while token filtering rewards encourage the model to retain fewer visual tokens. Therefore, this application can convert first data of image type into visual tokens. Compared to OCR recognition, visual tokens can correctly retain more key visual information of the image, thus enabling more accurate correction based on this key visual information and improving the accuracy of the correction result.

[0101] Meanwhile, this application employs an accuracy reward mechanism to ensure that the grading model accurately identifies and retains visual information crucial to the grading task when filtering visual tokens, guaranteeing high accuracy of the grading results. The token filtering reward mechanism effectively suppresses the retention of redundant visual information that has little or no impact on the grading task. This allows the grading model to filter out more useless visual tokens, reducing the computational complexity of the grading process, improving subsequent grading efficiency, and reducing memory usage. The synergistic optimization of these two reward mechanisms enables the grading model to learn the contribution value of different visual tokens to the final grading result during training. This allows it to adaptively adjust the token selection strategy according to the specific grading scenario, achieving more efficient and accurate grading.

[0102] In some embodiments of this application, a training method for a graded model trained using reinforcement learning is described.

[0103] First, this embodiment can obtain training data for training the grading model. The training data includes answer context data samples and grading result labels. The answer context data samples include a first data sample of image type and a second data sample of text type.

[0104] Similar to step S202 above, a second set of visual tokens can be generated based on the first data sample through the grading model, and tokens can be filtered out from the second set of visual tokens to obtain the second set of visual tokens that are retained after filtering. Then, a predicted grading result can be generated based on the second set of visual tokens that are retained after filtering and the second data sample.

[0105] Optionally, this embodiment may employ reinforcement learning strategies such as PPO (Proximal Policy Optimization) or GRPO (Group Relative Policy Optimization). For ease of understanding, GRPO will be used as an example in the following description.

[0106] In GRPO reinforcement learning, for the same set of first and second data samples, the grading model can generate multiple different prediction grading results. Then, the accuracy reward and token elimination reward are calculated for each prediction grading result, and then the combined reward is calculated. The specific process is as follows.

[0107] This embodiment can generate an accuracy reward based on the predicted grading results and the grading result labels. For example, optionally, for tasks with definite answers, such as "what is the degree measure of each angle of an equilateral triangle", the accuracy reward can be expressed as the following formula (1).

[0108] Formula (1);

[0109] in, This refers to the output data generated by the grading model, specifically the predicted grading results in the output data. Indicates the label for the correction result. This indicates an accuracy-based reward.

[0110] A token elimination reward can also be generated based on the second visual token and the set of second visual tokens. Optionally, for tasks with deterministic answers, the generation process of the token elimination reward may include: first determining the number of second visual tokens and the total number of tokens in the set of second visual tokens, and then obtaining the target reward based on the number of second visual tokens and the total number of tokens. Here, the target reward is inversely proportional to the number of second visual tokens. If the predicted correction result is the same as the correction result label, the token elimination reward is the target reward. If the predicted correction result is different from the correction result label, the token elimination reward is a preset minimum value that tends to 0 or is equal to 0.

[0111] Alternatively, a possible token screening reward can be expressed as formula (2) below.

[0112] Formula (2);

[0113] in, Indicates the number of second visual tokens. Indicates the total number of tokens. This indicates that the token will be used to filter out rewards. Indicates the target reward. This refers to the output data generated by the grading model, specifically the predicted grading results in the output data. Labels indicating the correction results.

[0114] It should be noted that the above method of dividing the number of second visual tokens by the total number of tokens to obtain the target reward is only an example. In addition, there are other ways to implement it, such as subtracting the number of second visual tokens from the total number of tokens to obtain the target reward, or multiplying the difference by a preset adjustment coefficient to obtain the target reward, etc.

[0115] Furthermore, the token elimination reward shown above is for grading tasks with definitive answers. In grading tasks without definitive answers, the aforementioned "..." The condition "" can also be replaced with others, for example, and The semantic similarity is less than the preset similarity threshold.

[0116] Furthermore, the grading model can be trained using reinforcement learning based on accuracy rewards and token elimination rewards to obtain a grading model trained by reinforcement learning.

[0117] In one possible implementation, the expected reward for reinforcement learning can be obtained based on the accuracy reward and the token elimination reward. The expected reward is then inverted and used as the target loss. The model is then trained using reinforcement learning based on the target loss.

[0118] One possible way to achieve the expected return is as follows.

[0119] The accuracy reward and the token elimination reward can be weighted and summed to obtain the combined reward. , where R represents the set of all rewards included in the combined reward.

[0120] As mentioned earlier, in GRPO reinforcement learning, multiple prediction results can be obtained, and the combined rewards corresponding to each prediction result can be obtained. Then, the relative advantage can be calculated based on the combined rewards corresponding to each prediction result, and the expected return can be obtained based on the relative advantage.

[0121] In GRPO, for each batch change context data sample q, based on the old parameters of the batch change model... Sample multiple prediction correction results Combination set Then, by maximizing the objective function in formula (3) To optimize the parameters of the batch correction model .

[0122]

[0123] in, Parameters representing the grading model; The distribution of the grading context data sample q represents all possible input combinations in the training data; Indicates the predicted correction result The length of , which is the number of tokens included in the predicted correction result; t represents the time step; Indicate the relationship between the new strategy and the old strategy The probability ratio of the t-th token in the algorithm is used to compare the probabilities of the new strategy and the old strategy in choosing an action given the history. It represents relative advantage, estimated based on the difference in scores between the same set of predicted and graded results; This represents the clipping hyperparameter, which is usually a small value; Denotes the pruning function, restricting the probability ratio to... Range, to avoid over-updating; This represents the KL (Kullback-Leibler divergence) divergence weight hyperparameter, which controls the strength of the KL penalty term and is used to prevent the policy from deviating from the reference policy. Too far; The KL divergence is defined as follows:

[0124] Formula (4).

[0125] Substitute the relative advantage calculated from the combined rewards corresponding to the multiple prediction results mentioned above into the formula (3) above. , obtained The function value is the expected return.

[0126] Considering that there may be format requirements for the output of the model in some scenarios, this embodiment can set a format reward to encourage the model to generate output data that conforms to the target format.

[0127] In other scenarios, there may be length requirements for the output of the model. For this purpose, a length reward can be set to encourage the model to generate output data that does not exceed the preset ideal length.

[0128] Based on this, the process of “obtaining the expected reward of reinforcement learning based on accuracy reward and token elimination reward” can include: obtaining a format reward based on the format of the output data generated by the model, and / or obtaining a length reward based on the length of the output data, wherein the output data includes at least the predicted correction result, and obtaining the expected reward based on the accuracy reward and token elimination reward, combined with the format reward and / or the length reward.

[0129] One possible format for representing the reward can be expressed as follows (5):

[0130] Formula (5);

[0131] in, Indicates a formatted reward. It is a function used to check whether the output data generated by the batch correction model conforms to the target format. This represents the output data generated by the correction model. Labels indicating the correction results.

[0132] One possible representation of the length reward can be given by the following formula (6):

[0133] Formula (6);

[0134] in, Indicates length bonus. Indicates the ideal length. This represents the output data generated by the correction model.

[0135] Taking the rewards used in the reinforcement learning process, including accuracy rewards, token elimination rewards, format rewards, and length rewards, as an example, the optional formula for calculating combined rewards is:

[0136] Formula (7);

[0137] in, The weighting of the accuracy reward indicates the weighting of the reward. Indicates a formatted reward. This indicates the weight of the format reward. Indicates length bonus. This indicates the weight of the length reward. This indicates the weight of the token filtering reward.

[0138] It should be noted that the above method of obtaining the combined reward by weighted summation is only an example. In addition, direct summation or weighted averaging can also be used. This application does not impose any specific limitations.

[0139] After obtaining the combined reward, the target loss can be obtained as described above, and then the correction model can be trained.

[0140] Considering that the token filtering reward only trains the model to filter out more redundant visual tokens, but does not limit the specific number or proportion of visual tokens to be filtered out, the grading model may be able to filter out redundant visual tokens, but the filtering effect may not meet expectations.

[0141] Therefore, in one possible implementation, this embodiment may also set a screening ratio loss, which is used to measure the positive deviation between the visual token retention ratio and the expected retention ratio. Here, the visual token retention ratio is the ratio of the number of second visual tokens to the total number of tokens, and the positive deviation means that the difference between the visual token retention ratio and the expected retention ratio is positive.

[0142] Optionally, the screening ratio loss can be expressed as the following formula (7).

[0143] Formula (7);

[0144] in, Indicates the screening ratio loss. This represents the proportion of visual tokens expected to be removed, and its sum with the expected retention proportion is 1.

[0145] In other words, in this embodiment, when the deviation between the visual token retention ratio and the expected retention ratio is greater than or equal to 0, the deviation is used as the screening ratio loss. If the deviation between the visual token retention ratio and the expected retention ratio is less than 0, the screening ratio loss is set to 0 since it has already dropped below the expected retention ratio.

[0146] Furthermore, in this embodiment, the total loss can be obtained based on the target loss and the screening ratio loss, and the correction model can be trained using reinforcement learning based on the total loss.

[0147] Optionally, the target loss and the screening ratio loss can be summed, and the sum can be used as the total loss. ,in, Indicates the total loss. This represents the target loss, which is the inverse of the expected loss.

[0148] It should be noted that the above method of summing the target loss and the screening ratio loss is only an example. In addition, there are other methods to obtain the total loss, such as weighted summation of the target loss and the screening ratio loss.

[0149] The embodiments of this application can combine multiple rewards and losses to train the grading model, which can filter out as many redundant visual tokens as possible while ensuring grading accuracy, thereby effectively improving computational efficiency and reducing storage resource consumption.

[0150] By employing the GRPO reinforcement learning training method, relative advantage can be calculated through comparison of samples within a group, which helps to balance the variance of rewards and makes the training process of the grading model more stable. Furthermore, in grading tasks with multiple rewards and elimination ratio losses, GRPO can automatically balance different reward objectives and elimination ratio objectives, achieving more efficient token elimination and improving grading efficiency.

[0151] As mentioned earlier, the output data generated by the grading model should at least include the predicted grading results. Therefore, the output data generated by the grading model after reinforcement learning training should at least include the grading results.

[0152] Optionally, the output data may also include thought chain content, that is, the grading model may output the reasoning process when the grading result was obtained at the same time as the grading result.

[0153] Based on this, optionally, the target format in the above-mentioned format reward can be to use preset thinking process markers to mark the content of the wrapped thinking chain or empty content and then splice the correction results.

[0154] That is, when the model outputs thought chain content, the output thought chain content is wrapped with thought process tag pairs, and then the correction result is output, with the thought process tag pairs as " <think>" and< / think> For example, the output format can be " <think>COT< / think> "Correction Results", where COT represents the content of the thought chain.

[0155] For example, the output data of the grading model is: <think>The student gave the answer 6 directly, but did not show the calculation process. The original question is "5-2+3", and the correct steps should be: 5-2=3, 3+3=6. The student's answer is correct but lacks process demonstration.< / think> Score: 3 / 5 (Correct answer but deducted points for process).

[0156] In cases where the model does not output thought chain content—for example, simple tasks that do not require thought chain reasoning—the model does not output thought chain content. In this scenario, the thought process can be marked with empty content before outputting the grading result, i.e., " <think> < / think> Correction results.

[0157] Based on this, a token can be forcibly inserted during the reasoning process to terminate the output of the thought chain when the reasoning content becomes too long. Optionally, the answer correction method provided in this embodiment may further include: during the generation of thought chain content by the correction model after reinforcement learning training, for each new token generated, determining the confidence level of the new token and the current thought chain length, and determining the next operation based on the confidence level of the new token and the current thought chain length. The next operation may be to continue reasoning to generate a new token or to add an end marker from the thought process marker pair after the currently generated thought chain content.

[0158] In other words, this embodiment enables the correction model to output the reasoning process in real time when using chain-like reasoning. During the reasoning process, each newly generated token has its confidence level and the current chain length calculated. Then, based on the confidence level of the new token and the current chain length, it is determined whether to insert "" to end the chain reasoning or to continue reasoning to generate the next token.

[0159] Optionally, the confidence level of the new token can be calculated using the following formula:

[0160] Formula (8);

[0161] in, This represents the new token at time step t. confidence level This represents the new token generated by the grading model at time step t. This represents the probability that the grading model generates a new token at time step t, and the calculation formula is as follows (9). This represents the vocabulary, which is a set of all possible generated tokens. This indicates the length of the currently generated thought chain content.

[0162] Formula (9);

[0163] in, This indicates a prompt for inputting the grading model. This represents the first t-1 tokens generated before time step t. This indicates the correction model.

[0164] Optionally, the process of "determining the next step based on the confidence level of the new token and the current thought chain length" may include: if the current thought chain length is greater than the ideal length, and the confidence level of the new token is greater than the first confidence threshold, or the confidence level of the new token is less than the second confidence threshold, then an end marker is added to the content of the currently generated thought chain, where the first confidence threshold is greater than the second confidence threshold; if the current thought chain length is less than or equal to the ideal length, or the confidence level of the new token is within the target range, then the reasoning continues to generate a new token, where the target range is the closed interval formed by the first confidence threshold and the second confidence threshold.

[0165] In other words, the new token can be determined using the following formula. :

[0166] Formula (10);

[0167] in, This represents the first confidence threshold. This represents the second confidence threshold. For the correction model based on the included Tips A function to predict and generate the next token.

[0168] In the above formula (10), if the confidence level of the currently generated thought chain content is lower than the second confidence threshold... If the current thought chain length has exceeded the specified ideal length, it indicates that the correction model believes there may be problems or high uncertainty in the current reasoning process. In this case, the token "" can be forcibly inserted to end the thought chain in advance, thereby avoiding further error accumulation and unnecessary waste of computing resources.

[0169] If the confidence level of the current thought chain is higher than the first confidence threshold This indicates that the model considers the current reasoning process to be reliable. However, since the current thought chain length has exceeded the specified ideal length, the token "" can be forcibly inserted to terminate the thought chain early, so that an excessively long thought chain may still lead to unnecessary waste of computing resources and may introduce additional complexity or redundant information.

[0170] By ending the thought chain reasoning early, the grading model can further improve reasoning efficiency and reduce the waste of computing resources while ensuring the accuracy of reasoning.

[0171] It should be noted that the above conditions for determining the next step are merely examples and are not intended to limit this application. In addition, this embodiment may omit the first confidence threshold or set more confidence thresholds, etc.

[0172] This application also provides a method for batch model training; please refer to [reference needed]. Figure 4 This is a flowchart illustrating a batch model training method provided in an embodiment of this application. Figure 4 The method for correcting model training may include:

[0173] Step S401: Obtain the answer context data sample and the grading result label. The answer context data sample includes a first data sample of image type and a second data sample of text type.

[0174] Step S402: Generate a second visual token set based on the first data sample using the grading model, perform token filtering on the second visual token set, and generate a predicted grading result based on the second visual tokens and the second data sample retained after filtering.

[0175] Step S403: Generate an accuracy reward based on the predicted correction results and correction result labels, and generate a token elimination reward based on the second visual token and the second visual token set.

[0176] Among them, the accuracy reward is used to encourage the model to retain visual tokens that contribute highly to the accuracy of the grading during token elimination, while the token elimination reward is used to encourage the model to retain fewer visual tokens during token elimination.

[0177] Step S404: The grading model is trained using reinforcement learning based on the accuracy reward and the token elimination reward to obtain the grading model after reinforcement learning training.

[0178] The training process in the grading model training method provided in this application embodiment is the same as the training process of the grading model after reinforcement learning training described above. For details, please refer to the previous description, which will not be repeated here.

[0179] The above describes a response correction method provided by the embodiments of this application. The following describes the apparatus for performing the above response correction method.

[0180] Please see Figure 5 , Figure 5 This is a schematic diagram of a response correction device provided in an embodiment of this application. Figure 5 As shown, the answer grading device may include:

[0181] The first data input unit 501 is used to acquire answer context data related to the respondent's answer. The answer context data includes first data of image type and second data of text type.

[0182] The first data processing unit 502 is used to generate a first visual token set based on the first data through a grading model trained by reinforcement learning, perform token sieving on the first visual token set, and generate a grading result based on the first visual tokens and the second data after sieving. The rewards used in the reinforcement learning process of the grading model include accuracy rewards and token sieving rewards. Accuracy rewards are used to encourage the model to retain visual tokens that contribute highly to the accuracy of grading during token sieving, and token sieving rewards are used to encourage the model to retain fewer visual tokens during token sieving.

[0183] The modules in the aforementioned answer grading device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0184] The above also introduced a method for training a modified model provided by the embodiments of this application. The following will introduce the apparatus for performing the above-described method for training a modified model.

[0185] Please see Figure 6 , Figure 6 This is a schematic diagram of a graded model training device provided in an embodiment of this application. Figure 6 As shown, the correction model training device may include:

[0186] The second data input unit 601 is used to acquire answer context data samples and grading result labels. The answer context data samples include a first data sample of image type and a second data sample of text type.

[0187] The second data processing unit 602 is used to generate a second visual token set based on the first data sample through the grading model, perform token screening on the second visual token set, and generate a predicted grading result based on the second visual tokens retained after screening and the first data sample.

[0188] The data calculation unit 603 is used to generate an accuracy reward based on the predicted correction result and the correction result label, and to generate a token elimination reward based on the second visual token and the second visual token set. The accuracy reward is used to encourage the model to retain visual tokens that contribute highly to the correction accuracy in the token elimination, and the token elimination reward is used to encourage the model to retain fewer visual tokens in the token elimination.

[0189] The third data processing unit 604 is used to perform reinforcement learning training on the grading model based on accuracy rewards and token elimination rewards.

[0190] Each module in the aforementioned correction model training device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0191] This application also provides an electronic device, which may include at least one processor and a memory connected to the processor, wherein:

[0192] Memory is used to store computer programs;

[0193] The processor is used to execute computer programs to enable the electronic device to implement any of the response correction methods or correction model training methods provided in the embodiments of this application.

[0194] refer to Figure 7 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 7 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0195] like Figure 7 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. When the electronic device is powered on, the RAM 703 also stores various programs and data required for the operation of the electronic device. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0196] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, memory cards, hard drives, etc.; and communication devices 709. Communication device 709 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0197] This application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are run on an electronic device, the electronic device enables the electronic device to implement any of the answer correction methods or correction model training methods provided in this application.

[0198] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the answer correction methods or any of the correction model training methods provided in this application.

[0199] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0200] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0201] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0202] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method of responding to grading, characterized by, The method comprises: obtaining answer context data related to the answer of the answerer, the answer context data comprising first data of an image type and second data of a text type; generating, by a grading model trained through reinforcement learning, a first visual token set according to the first data, performing token screening on the first visual token set, and generating a grading result according to the first visual token retained after the screening and the second data; wherein the reward used in the reinforcement learning process of the grading model comprises an accuracy reward and a token screening reward, the accuracy reward is used to encourage the model to retain visual tokens with high contribution to grading accuracy in token screening, and the token screening reward is used to encourage the model to retain fewer visual tokens in token screening.

2. The method according to claim 1, wherein, The training process of the grading model trained through reinforcement learning comprises: obtaining answer context data samples and grading result labels, the answer context data samples comprising first data samples of an image type and second data samples of a text type; generating, by the grading model, a second visual token set according to the first data samples, performing token screening on the second visual token set, and generating a predicted grading result according to the second visual token retained after the screening and the second data samples; generating the accuracy reward according to the predicted grading result and the grading result label, and generating the token screening reward according to the second visual token and the second visual token set; training the grading model through reinforcement learning according to the accuracy reward and the token screening reward to obtain the grading model trained through reinforcement learning.

3. The method according to claim 2, wherein, The generation of the token screening reward according to the second visual token and the second visual token set comprises: determining the number of the second visual tokens and the total number of tokens in the second visual token set; obtaining a target reward according to the number of the second visual tokens and the total number of tokens, the target reward being inversely proportional to the number of the second visual tokens; if the predicted grading result is the same as the grading result label, the token screening reward is the target reward; if the predicted grading result is different from the grading result label, the token screening reward is a preset minimum value tending to 0 or equal to 0.

4. The method according to claim 3, wherein, The reinforcement learning training of the grading model according to the accuracy reward and the token screening reward comprises: obtaining an expected return of reinforcement learning according to the accuracy reward and the token screening reward, taking the inverse of the expected return as a target loss; determining a screening proportion loss, the screening proportion loss being used to measure the positive deviation of a visual token retention proportion from an expected retention proportion, the visual token retention proportion being the ratio of the number of the second visual tokens to the total number of tokens, and the positive deviation being that the difference between the visual token retention proportion and the expected retention proportion is positive; training the grading model through reinforcement learning according to the target loss and the screening proportion loss.

5. The method according to claim 4, wherein, The obtaining of the expected return of reinforcement learning according to the accuracy reward and the token screening reward comprises: The output data generated according to the batch correction model is rewarded in terms of format, and / or, length, the format reward is used to encourage the model to generate output data in a target format, and the length reward is used to encourage the model to generate output data with a length not exceeding an ideal length; The expected return is obtained according to the accuracy reward and the token filtering reward, combined with the format reward and / or the length reward.

6. The method according to claim 5, wherein, The target format is to splice the correction result after wrapping the thought chain content or empty content with a preset thinking process marker; The answer correction method further comprises: During the generation of the thought chain content by the reinforcement learning trained correction model, the confidence of a new token and the current thought chain length are determined every time a new token is generated; According to the confidence of the new token and the current thought chain length, the next operation is determined, which is to continue reasoning to generate a new token or to add an end marker in the thinking process marker pair after the currently generated thought chain content.

7. The method according to claim 6, wherein, The determination of the next operation according to the confidence of the new token and the current thought chain length comprises: In the case that the current thought chain length is greater than the ideal length, if the confidence of the new token is greater than a first confidence threshold or the confidence of the new token is less than a second confidence threshold, the end marker is added after the currently generated thought chain content, and the first confidence threshold is greater than the second confidence threshold; If the current thought chain length is less than or equal to the ideal length, or the confidence of the new token is within a target range, continue to reason to generate a new token, and the target range is a closed interval composed of the first confidence threshold and the second confidence threshold.

8. The method of claim 1, wherein, The token filtering on the first visual token set comprises: Obtain a weight matrix, which is obtained from the self-attention weight between visual tokens and / or the cross-attention weight between visual tokens and text tokens of the second data; According to the weight matrix, each visual token in the first visual token set is weighted to obtain the target attention weight of each visual token, so as to obtain the target attention weight of each visual token in the first visual token set; Token filtering is performed according to the target attention weight of each visual token. 9.A method for training a correction model, the method comprising: It comprises: Obtain answer context data samples and correction result labels, the answer context data samples comprise image type first data samples and text type second data samples; Generate a second visual token set from the first data sample through a correction model, perform token filtering on the second visual token set, and generate a predicted correction result according to the retained second visual token after filtering and the first data sample; generate an accuracy reward according to the predicted correction result and the correction result label, and generate a token screening reward according to the second visual token and the second visual token set, the accuracy reward being used to encourage the model to retain visual tokens with high contribution to correction accuracy in token screening, and the token screening reward being used to encourage the model to retain less visual tokens in token screening; perform reinforcement learning training on the correction model according to the accuracy reward and the token screening reward.

10. A computer program product, characterised in that, The computer readable instructions, when executed on an electronic device, cause the electronic device to implement the answer correction method of any one of claims 1-8 or the correction model training method of claim 9.

11. An electronic device, comprising: The memory is configured to store computer programs. The processor is configured to execute the computer programs to enable the electronic device to implement the answer correction method of any one of claims 1-8 or the correction model training method of claim 9. The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the answer correction method of any one of claims 1-8 or the correction model training method of claim 9.

12. A computer storage medium, characterized in that ​

Images