Big language model noise data reinforcement learning training method based on double rewards

Abstract

A big language model noise data reinforcement learning training method based on double rewards comprises the steps that candidate actions containing noise evaluation values are obtained through exploratory evaluation for any state of a decision-making task, and noise originates from insufficient exploration of an exploratory evaluation method under limited resources; sorting and dividing positive and negative example action sets according to noise evaluation values, and calculating preference result rewards; the model is required to carry out multi-situation-dimension qualitative evaluation on the task state, and a situation awareness reward is obtained through comparison with a real label; a comprehensive signal is constructed in combination with double rewards, and model parameters are updated by adopting a reinforcement learning strategy optimization algorithm; and iterative training is carried out until convergence conditions are met. The technical problems that when a large language model depends on noise data training, the strategy gradient variance is high, training is not stable, and hackers are likely to be awarded are solved. Through the synergistic effect of double rewards, the learning signal variance is effectively reduced, the speculative decision is avoided, the training process is stabilized, and the accuracy and robustness of a large language model in a complex decision task are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the training technology of large language model (LLM) of artificial intelligence, and in particular to a large language model noise data reinforcement learning training method based on double rewards. BACKGROUND

[0002] It is a hot research topic to use a large language model as a decision-making agent to solve complex problems. When training such an agent, a large amount of "state-action" data with supervision signals is usually required. A commonly used method to obtain supervision signals is Monte Carlo Tree Search (MCTS), which evaluates the pros and cons of performing different actions in a particular state through simulation. However, due to the extremely large decision space of actual problems, MCTS can only perform insufficient exploration under limited computing power, and the evaluation values (such as Q values) it gives therefore have significant noise.

[0003] Directly using these noisy evaluation values as rewards or regression targets to train LLMs poses a core technical challenge: high variance of policy gradient. Inaccurate reward signals can lead to unstable training processes, and the model may not converge or converge to a suboptimal strategy, or even produce "reward hacking" phenomena, i.e., the model learns to exploit the loopholes of the reward function to get high scores, but does not truly master effective strategies to solve problems. Existing advanced fine-tuning algorithms such as Group Relative Policy Optimization (GRPO) can improve model performance, but their effectiveness is highly dependent on large-scale, high-quality, and low-noise training data, making it difficult to effectively apply them in real-world scenarios where data acquisition costs are high and noise is prevalent.

[0004] In practical applications, the decision-making tasks handled by large language models as decision-making agents rely mainly on text forms (such as text-based decision task descriptions, candidate action instructions, state feedback information, and evaluation value records, etc.), and the acquisition and annotation of such text data are costly. The noisy evaluation values obtained by existing methods based on Monte Carlo Tree Search and other methods further amplify the variance of the policy gradient when guiding model learning, leading to a more unstable model training process and exacerbating the model's misjudgment of the matching relationship between the task state and action selection described in the text. This makes it difficult for existing methods to form a reliable decision-making strategy even after optimizing the model training process when facing such text data-driven complex decision-making tasks, further highlighting the constraints of noisy data processing on the improvement of large language model decision-making capabilities. How to design a training method that can effectively resist training data noise, reduce the variance of the policy gradient, and ensure stable learning of LLMs is a technical challenge that needs to be addressed in the training process of large language models.

[0005] It should be noted that the information disclosed in the above BACKGROUND section is only for understanding the background of the present application, and thus can include information that is not prior art known to those of ordinary skill in the art. SUMMARY

[0006] The main purpose of the present application is to overcome the defects existing in the background art, provide a large language model noise data reinforcement learning training method based on double rewards.

[0007] To achieve the above-mentioned purpose, the present application adopts the following technical solutions: A large language model noise data reinforcement learning training method based on double rewards, comprising the following steps: S1, obtaining noise training data: for any state of a decision task, a set of candidate actions and their respective noise evaluation values are obtained by an exploratory evaluation method; the state, candidate actions and evaluation values of the noise training data are input into a large language model in text form, and the noise is caused by insufficient exploration of the exploratory evaluation method under limited resources; S2, calculating the preference outcome reward (POR): the noise evaluation values of the candidate actions are sorted, the sorted action list is divided into a positive example action set and a negative example action set according to a preset division ratio, and the corresponding rewards are given according to the belonging relationship between the actual action generated by the large language model in this state and the positive and negative example action sets; S3, calculating the context-aware reward (CPR): the large language model is required to qualitatively evaluate a plurality of preset context dimensions of the current decision task state and output evaluation results, and the evaluation results are compared with preset true labels to obtain a context-aware reward value; S4, executing policy optimization: combining the preference outcome reward and the context-aware reward, a comprehensive reward signal is constructed, and a reinforcement learning policy optimization algorithm is used to update the parameters of the large language model according to the comprehensive reward signal; S5, iterative training: steps S1 to S4 are repeatedly executed until the training result of the large language model meets the preset convergence condition.

[0008] Further, in the step S1, the noise evaluation value is obtained by a limited resource evaluation method, and the limited resource evaluation method includes Monte Carlo tree search, which obtains a rough estimate of the value by exploring and evaluating each candidate action within a limited number of iterations.

[0009] Further, in the step S2, when the actual action belongs to the positive example set, the size of the positive reward given is inversely proportional to the ranking of the action within the positive example set; the optimal value of the division ratio is determined by maximizing the separation degree of the expected value of the positive example action set and the negative example action set.

[0010] Further, the target function of the separation degree is defined as the difference between the average cost expectation value of the negative example action set and the average cost expectation value of the positive example action set, and the optimal value of the division ratio is determined by optimizing the target function.

[0011] Further, in the step S3, the preset context dimensions include at least one of a question type, a solution space state, and a cost distribution characteristic of a current solution; and the true label is automatically generated by a rule-based method.

[0012] Further, in the step S3, the context-aware reward is based on the evaluation accuracy of each preset context dimension by the large language model, and a part of positive reward is obtained for each correctly evaluated dimension.

[0013] Further, in the step S4, the strategy optimization algorithm includes a group relative strategy optimization algorithm, and the model parameters are updated by a post-training method of reinforcement learning.

[0014] Further, the strategy optimization algorithm generates multiple candidate replies for each state, the candidate replies include an inference process and a final answer, and the candidate replies are sorted according to the high and low of the comprehensive reward signal to update the large language model parameters.

[0015] Further, in the step S4, the comprehensive reward signal is a weighted combination of the preference result reward, the context-aware reward, and other auxiliary rewards.

[0016] A computer program product comprising a computer program which, when executed by a processor, implements the double-reward-based large language model noise data reinforcement learning training method.

[0017] The present application has the following beneficial effects: The double-reward-based large language model noise data reinforcement learning training method provided by the present application is an innovative scheme for reinforcement learning fine-tuning of a large language model (LLM), and is particularly suitable for solving complex problems (text data including text decision task descriptions, candidate action instructions, state feedback information, and evaluation value records) driven by text data such as combinatorial optimization and sequential decision making. The present application designs a training framework that can effectively resist training data noise, reduce strategy gradient variance, and ensure stable learning of the LLM. Through the innovative double-reward mechanism, when relying on imperfect and noisy text training data, the variance of the learning signal can be significantly reduced, the training process is stabilized, and thus the processing capability of the LLM for text-driven complex decision-making tasks is effectively improved.

[0018] Specifically, the present application stabilizes the result signal by the relative ranking of the noise evaluation value through the preferred outcome reward (POR), helping the model to determine "what (WHAT)" is a relatively better action; at the same time, the model's task state evaluation is supervised through the context perception reward (CPR), strengthening the process signal understanding, guiding the model to understand "why (WHY)" the action needs to be selected in the task state described by the current text. The two work together to fundamentally solve the core pain point of using noise data for LLM reinforcement learning. Through the combination of the two reward modes, not only can the model make correct action selection, but also can ensure that the model makes decisions based on accurate understanding of the problem state constructed by the text data; with the addition of process supervision, the model can avoid speculative answers of "reward hacking" type generated by reasoning, and realize more robust decisions based on reasonable reasoning. Ultimately, the model can further improve the accuracy and stability of the decision result when processing complex decision-making tasks driven by text.

[0019] Other beneficial effects of the embodiments of the present application will be further described below. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 The overall flowchart of the present application based on the dual reward large language model noise data reinforcement learning training method.

[0021] Figure 2 The overall technical roadmap of the training method of the embodiments of the present application.

[0022] Figure 3 The reward plasticity diagram of the embodiments of the present application.

[0023] Figure 4 The guide and correction mechanism diagram of the context perception reward (CPR) of the embodiments of the present application. DETAILED DESCRIPTION

[0024] The embodiments of the present application are described in detail below. It should be emphasized that the following description is merely exemplary and is not intended to limit the scope of the present application and its applications.

[0025] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0026] The present application aims to solve the technical problems of high strategy gradient variance and easy "reward hacking" behavior when a large language model relies on noisy text training data (such as text classification decision task description, candidate action instruction, state feedback information, etc.) to carry out complex decision training, and proposes a dual reward reinforcement learning training method based on the synergistic effect of preference outcome reward (POR) and context perception reward (CPR). The method uses POR to use the relative ranking stability of noise evaluation value to determine high-quality action selection, and uses CPR to supervise the process of model task state evaluation to strengthen decision logic understanding, effectively reducing the variance of learning signal and stabilizing the training process, and ultimately greatly improving the decision accuracy and robustness of the model in text-driven complex tasks such as combinatorial optimization and sequential decision making.

[0027] Referring to Figure 1 The embodiment of the present application provides a dual reward based large language model noise data reinforcement learning training method, comprising the following steps: Step S1, obtaining noise training data: for any state of a decision task, a set of candidate actions and their respective noise evaluation values are obtained by an exploratory evaluation method. The state, candidate action and evaluation value of the noise training data are input into the large language model in text form, and the noise is caused by insufficient exploration of the exploratory evaluation method under limited resources. That is, the noise training data refers to the "state-action-evaluation value" data obtained by insufficient exploration of candidate actions under each state of the decision task through exploratory evaluation methods such as Monte Carlo tree search under limited iteration times or limited computing resources, with noise caused by insufficient exploration.

[0028] In some embodiments, the noise evaluation value in step S1 is obtained by a limited resource evaluation method, which includes Monte Carlo tree search, which obtains a rough estimate of the value by exploring and evaluating each candidate action within a limited number of iterations.

[0029] Step S2, calculating the preference outcome reward (POR): according to the noise evaluation value of the candidate action, the candidate action is sorted, the sorted action list is divided into a positive example action set and a negative example action set according to a preset division ratio, and the actual action generated by the large language model in the state is assigned a corresponding reward according to the belonging relationship of the positive and negative example action sets.

[0030] In some embodiments, in step S2, when the actual action belongs to the positive example set, the size of the positive reward assigned is inversely proportional to the ranking of the action within the positive example set; and the optimal value of the division ratio is determined by maximizing the separation degree of the expected value of the positive example action set and the negative example action set.

[0031] In some embodiments, the target function of the separation degree is defined as the difference between the average cost expectation value of the negative example action set and the average cost expectation value of the positive example action set, and the optimal value of the division ratio is determined by optimizing the target function.

[0032] Step S3, calculating a context-aware reward (CPR): requiring the large language model to qualitatively evaluate a plurality of preset context dimensions of the current decision task state and output evaluation results, comparing the evaluation results with preset true labels to obtain a context-aware reward value.

[0033] In some embodiments, in the step S3, the preset context dimensions include at least one of a question type, a space state of a solution, and a cost distribution feature of a current solution; and the true labels are automatically generated by a rule-based method.

[0034] In some embodiments, in the step S3, the context-aware reward is based on the evaluation accuracy of the large language model for each preset context dimension, and a partial positive reward is obtained for each correctly evaluated dimension.

[0035] Step S4, performing policy optimization: combining the preference result reward and the context-aware reward to construct a comprehensive reward signal, and updating parameters of the large language model according to the comprehensive reward signal by using a reinforcement learning policy optimization algorithm; In some embodiments, in the step S4, the policy optimization algorithm includes a group relative policy optimization algorithm, and the model parameters are updated by a reinforcement learning post-training method.

[0036] In some embodiments, the policy optimization algorithm generates a plurality of candidate replies for each state, the candidate replies include an inference process and a final answer, and the candidate replies are sorted according to the high and low of the comprehensive reward signal to update the parameters of the large language model.

[0037] In some embodiments, in the step S4, the comprehensive reward signal is a weighted combination of the preference result reward, the context-aware reward, and other auxiliary rewards. As Figure 2 shown, the comprehensive reward signal R t By assigning weights to the preference result reward ( R POR ), the context-related reward ( R QRP , corresponding to the “context-aware reward” in the claim), and other auxiliary rewards ( R other ), respectively w POR , w QRP , wother ) post-weighted summation, for quantifying the large language model in the current state s t performing the action a t the overall reward, providing a signal for reinforcement learning policy optimization.

[0038] Step S5, iterative training: steps S1 to S4 are repeatedly performed until the training result of the large language model meets the preset convergence condition.

[0039] The present application aims at the technical challenge of high strategy gradient variance and easy "reward hacker" behavior caused by noisy text training data when a large language model conducts complex decision training relying on text training data such as text type decision task description, candidate action instruction, state feedback information and evaluation value record. A dual reward mechanism of preference result reward (POR) and situation awareness reward (CPR) is innovatively adopted: on the one hand, the preference result reward (POR) uses the relative ordering stable result signal of the noise evaluation value to help the model to clearly define the selection standard of the high-quality action in the text description task; on the other hand, the situation awareness reward (CPR) supervises the process through the task state evaluation of the model, and strengthens the understanding of the decision logic of the model on "why to select the action in the current text defined task state". This mechanism can significantly reduce the variance of the learning signal to stabilize the training process, effectively avoid the model to generate "reward hacker" type speculative answers, and ensure the model to make decisions based on the accurate understanding of the problem state constructed by the text, finally greatly improve the decision accuracy and robustness of the large language model in text-driven complex tasks such as combinatorial optimization and sequential decision making, fully embodying the significant improvement of technical effect in the noisy data training scene.

[0040] The specific embodiments, algorithm examples and experimental verification of the present application are further described below.

[0041] A large language model noisy data reinforcement learning training method based on dual rewards, comprising the following steps: S1, obtaining noisy training data: for any state of a decision task, a set of candidate actions and their respective noise evaluation values are obtained through an exploratory evaluation method.

[0042] S2, calculating the preference result reward (POR): the noise evaluation values of the candidate actions are sorted, and the sorted action list is divided into a positive example action set containing the top k% actions and a negative example action set containing the remaining actions according to a preset division ratio k. The actual action generated by the large language model in this state is compared with the positive and negative example action sets, if the action belongs to the positive example set, a positive reward is obtained, if the action belongs to the negative example set, a negative reward is obtained, thereby forming a preference result reward signal.

[0043] S3, calculating a context-aware reward (CPR): requiring the large language model to qualitatively evaluate a plurality of preset context dimensions of a current decision task state and output evaluation results; comparing the evaluation results with preset true labels to obtain a context-aware reward value for measuring the cognitive accuracy of the model on the current state.

[0044] S4, executing policy optimization: combining the preference result reward and the context-aware reward to construct a comprehensive reward signal; using a policy optimization algorithm in reinforcement learning to update the parameters of the large language model according to the comprehensive reward signal.

[0045] S5, iterative training: repeating steps S1 to S4 until the training result of the large language model meets the preset convergence condition.

[0046] Further, in step S1, the evaluation value with noise is obtained by a limited resource evaluation method such as Monte Carlo tree search (MCTS), which explores and evaluates each candidate action within a limited number of iterations to obtain a rough estimate of the value.

[0047] Further, in step S2, when the actual action belongs to the positive example set, the size of the positive reward obtained is inversely proportional to the ranking of the action within the positive example set; and the optimal value of the division ratio k is determined by maximizing the separation degree of the expected values of the positive and negative example action sets.

[0048] Further, the objective function of the separation degree is J(k) = E[avg_cost(δ⁻) - avg_cost(δ⁺)], where δ⁺ and δ⁻ are the positive and negative example action sets, and avg_cost is the average cost expectation of the actions in the set.

[0049] Further, in step S3, the preset context dimensions include macroscopic judgments such as problem type, solution space state, and cost distribution characteristics of the current solution; and the true labels are automatically generated by a rule-based method.

[0050] Further, in step S4, the policy optimization algorithm used is a group relative policy optimization (GRPO) algorithm or other RL post-training method.

[0051] A large language model training system based on double rewards, comprising: a data acquisition module adapted to acquire a set of candidate actions and noise-containing evaluation values corresponding to each candidate action for a given decision task state; a preference result reward calculation module adapted to sort and divide according to the noise evaluation values of the candidate actions, and calculate a preference result reward according to the ownership of the action selected by the large language model; a context perception reward calculation module adapted to receive the qualitative evaluation result of the current state by the large language model, and compare it with the real label to calculate a context perception reward; and a model training module adapted to update the parameters of the large language model using a strategy optimization algorithm in combination with the preference result reward and the context perception reward.

[0052] The present application proposes a training method named noise-aware double reward optimization (NaDRO), the core principle of which is to decompose a single, noisy reward signal into two mutually complementary and more robust learning signals: preference-based outcome reward (POR) and context perception reward (CPR).

[0053] 1. Preference-based outcome reward (POR): This reward aims to solve the noise problem of the outcome signal. Its design is based on a key insight: although the absolute evaluation value given by MCTS and other methods is unreliable, its relative ranking of actions largely retains valid information. The calculation steps of POR are as follows: All candidate actions evaluated by MCTS are sorted according to the evaluation value.

[0054] Instead of using specific evaluation values, the sorted list is divided into a positive example set δ⁺ consisting of "good actions" and a negative example set δ⁻ consisting of "bad actions" according to a division ratio k.

[0055] When the action selected by the LLM falls into the positive example set, a positive reward is given; otherwise, a penalty is given.

[0056] 2. Context perception reward (CPR): This reward aims to solve the sparsity problem of the process signal. In long-term sequential decision-making, the reward signal of the final result is delayed and sparse. CPR makes up for this by providing a dense reward for the intermediate process. The calculation steps are as follows: In the design of different qualitative evaluation dimensions (for example, the quality of the current solution is "high", "medium" or "low") for different problems, the real labels of these evaluation dimensions can be easily generated by rules, so they are very accurate and consistent with human intuitive judgment.

[0057] Before making the final decision, the LLM is required to make a multi-dimensional qualitative assessment of the current state.

[0058] The LLM is given an instant reward based on the accuracy of its assessment.

[0059] Examples and experimental verification: The following will detail the examples and experimental results in the context of solving the Traveling Salesman Problem (TSP).

[0060] Step 1, Obtain noisy training data: In a certain intermediate state s of TSP solving, there is an action space A = {2-opt, simulated annealing, nearest neighbor insertion,...} composed of multiple heuristic algorithms. For each action a ∈ A, run MCTS for N = 1000 simulations to obtain its noisy Q value, i.e., Q(s, a).

[0061] Step 2, Calculate the optimal partition k: To find the optimal k, define a target function J(k) = E[avg(δ⁻) - avg(δ⁺)], aiming to maximize the cost separation degree of positive and negative example sets. By modeling and analyzing the cost distribution in the training data set, the optimal k value that maximizes J(k) can be theoretically derived.

[0062] Step 3, Calculate POR: According to the calculated partition ratio k, all algorithms in the action space A are sorted by Q value from high to low. If the LLM selects a_t ∈ δ⁺, it gives a positive reward; if a_t ∈ δ⁻, it gives a negative reward.

[0063] Step 4, Calculate CPR: Before the LLM outputs the final selected algorithm name, it is required to first complete a qualitative judgment and output the following format: { "problem_state": "partially_visited", "algorithm_type": "exploration", "current_cost": "low_cost" } The true labels of these judgments are provided by the rule generator. For example, "current_cost" is determined by comparing the average cost of the current path with a pre-computed benchmark cost of a basic greedy algorithm (e.g., below 80% of the benchmark is "low_cost" and above 115% is "high_cost"). The LLM gets a portion of the CPR positive reward for each correct answer.

[0064] Step 5, comprehensive reward plasticity: POR, CPR, and other rewards such as language consistency, format constraints, etc. are weighted and combined into a comprehensive reward signal used in reinforcement learning training.

[0065] Step 6, model optimization and training: The total reward is input into the GRPO training process. GRPO generates multiple candidate complete replies (including reasoning process and final answer) for each state, and sorts these replies according to the high and low of the total reward, thereby updating the LLM parameters. The process is repeated on the entire dataset until the model converges.

[0066] Experimental results show that the method has very high robustness. In a stress test, the inventor artificially randomized the action order of MCTS at a probability of 10% at each step (simulating catastrophic data damage), and the performance of the NaDRO framework of the present invention did not collapse, and still maintained stable and competitive performance, which fully proves the great potential of the present invention for handling various complex noises in the real world.

[0067] The embodiment of the present application also provides a storage medium for storing a computer program, which is executed to perform at least the method described above.

[0068] The embodiment of the present application also provides a control device, which includes a processor and a storage medium for storing a computer program; wherein the processor is used to execute the computer program to perform at least the method described above.

[0069] The embodiment of the present application also provides a processor, which executes a computer program to perform at least the method described above.

[0070] The storage medium can be implemented by any type of nonvolatile storage device, or a combination thereof. The nonvolatile memory can be a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a Ferromagnetic Random Access Memory (FRAM), a Flash memory, a magnetic surface storage, an optical disc or a Compact Disc Read-Only Memory (CD-ROM). The magnetic surface storage can be a disk memory or a tape memory. The storage medium described in the embodiments of the present application is intended to include, but is not limited to, these and any other suitable type of memory.

[0071] In several embodiments provided by the present application, it should be understood that the disclosed system and method can be implemented in other manners. The described device embodiments are merely schematic, and the division of the units is merely a logical function division. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling or direct coupling or communication connection between the components can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0072] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or distributed on a plurality of network units; some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0073] In addition, each functional unit in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in the form of hardware, or in the form of hardware plus software functional units.

[0074] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware, and the foregoing program can be stored in a computer readable storage medium, and the program performs the steps of the above-mentioned method embodiments when executed; and the foregoing storage medium includes a mobile storage device, a read-only memory (ROM), a random access memory (RAM), a magnetic disc or an optical disc and various storage medium capable of storing program codes.

[0075] Alternatively, the integrated unit of the present application can be stored in a computer readable storage medium if it is realized in the form of a software function module and sold or used as an independent product. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, includes several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the methods described in the embodiments of the present application. The foregoing storage medium includes a mobile storage device, a ROM, a RAM, a magnetic disc or an optical disc and various storage medium capable of storing program codes.

[0076] The methods disclosed in the several method embodiments provided by the present application can be combined arbitrarily without conflict to obtain new method embodiments.

[0077] The features disclosed in the several product embodiments provided by the present application can be combined arbitrarily without conflict to obtain new product embodiments.

[0078] The features disclosed in the several method or device embodiments provided by the present application can be combined arbitrarily without conflict to obtain new method embodiments or device embodiments.

[0079] The above is a further detailed description of the present application in combination with specific preferred embodiments, and the specific implementation of the present application cannot be limited to these descriptions. For those skilled in the art of the present application, without departing from the concept of the present application, a number of equivalent substitutions or obvious modifications can be made, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present application.

Claims

1. A method for training a large language model based on double reward noise data reinforcement learning, characterized in that, The method comprises the following steps: S1, obtaining noise training data: for any state of a decision task, a set of candidate actions and their respective noise evaluation values are obtained through an exploratory evaluation method; wherein the state, candidate actions and evaluation values as the noise training data are input into a large language model in text form, and the noise is caused by insufficient exploration of the exploratory evaluation method under limited resources; S2, calculating a preference result reward (POR): the noise evaluation values of the candidate actions are sorted, the sorted action list is divided into a positive example action set and a negative example action set according to a preset division ratio, and rewards are given according to the belonging relationship between the actual action generated by the large language model in the state and the positive and negative example action sets; S3, calculating a context perception reward (CPR): the large language model is required to qualitatively evaluate a plurality of preset context dimensions of the current decision task state and output evaluation results, the evaluation results are compared with preset true labels to obtain a context perception reward value; S4, executing policy optimization: combining the preference result reward and the context perception reward, a comprehensive reward signal is constructed, and a reinforcement learning policy optimization algorithm is used to update the parameters of the large language model according to the comprehensive reward signal; S5, iterative training: steps S1 to S4 are repeatedly executed until the training result of the large language model meets the preset convergence condition.

2. The dual-reward-based large language model noise data reinforcement learning training method of claim 1, wherein, In step S1, the noise evaluation value is obtained by a limited resource evaluation method, and the limited resource evaluation method includes Monte Carlo tree search, which obtains a rough estimate of the value by exploring and evaluating each candidate action within a limited number of iterations.

3. The dual-reward-based large language model noise data reinforcement learning training method according to claim 1 or 2, characterized in that, In step S2, when the actual action belongs to the positive example set, the size of the positive reward given is inversely proportional to the ranking of the action within the positive example set; the optimal value of the division ratio is determined by maximizing the separation degree of the expected value of the positive example action set and the negative example action set.

4. The dual-reward-based large language model noise data reinforcement learning training method of claim 3, wherein, The target function of the separation degree is defined as the difference between the average cost expected value of the negative example action set and the average cost expected value of the positive example action set, and the optimal value of the division ratio is determined by optimizing the target function.

5. The dual-reward-based large language model noise data reinforcement learning training method according to any one of claims 1 to 4, characterized in that, In step S3, the preset context dimensions include at least one of problem type, solution space state and cost distribution characteristics of the current solution; the true label is automatically generated by a rule-based method.

6. The dual-reward-based large language model noise data reinforcement learning training method of claim 5, wherein, In step S3, the context perception reward is based on the evaluation accuracy of the large language model for each preset context dimension, and a part of the positive reward is obtained for each correctly evaluated dimension.

7. The dual-reward-based large language model noise data reinforcement learning training method of any one of claims 1 to 6, wherein, In step S4, the policy optimization algorithm includes a group relative policy optimization algorithm, and the model parameters are updated by a reinforcement learning post-training method.

8. The dual-reward-based large language model noise data reinforcement learning training method of claim 7, wherein, The policy optimization algorithm generates a plurality of candidate replies for each state, the candidate replies include an inference process and a final answer, and the candidate replies are sorted according to the level of the comprehensive reward signal to update the parameters of the large language model.

9. The dual-reward-based large language model noise data reinforcement learning training method of any one of claims 1 to 8, wherein, In step S4, the comprehensive reward signal is a weighted combination of the preference result reward, the context perception reward and other auxiliary rewards.

10. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the double-reward-based large language model noise data reinforcement learning training method according to any one of claims 1 to 9.

Images