A reinforcement learning method and device applied to a self-programming task, an electronic device, and a storage medium

Abstract

The present application relates to the technical field of reinforcement learning, and more particularly to a reinforcement learning method and device applied to self-programming tasks, an electronic device and a storage medium, the method comprising: obtaining an example data set applied to a self-programming task, and fine-tuning an arbitrary open-source large language model according to the example data set; connecting a first value output end and a second value output end to the large language model after fine-tuning is completed; adjusting a temperature parameter to encourage the large language model to generate diversified codes for a selected self-programming task, and collecting positive and negative example data; performing Monte Carlo tree search to generate a series of texts according to the generated diversified codes and record them; and training the large language model using the data collected by the Monte Carlo tree search to optimize its code generation and evaluation capabilities. The present application overcomes the limitations of existing frameworks in the field of self-programming, and realizes more efficient and more accurate code generation and optimization.

Description

Technical Field

[0001] This invention relates to the field of reinforcement learning technology, and in particular to a reinforcement learning method, apparatus, electronic device, and storage medium for self-programming tasks. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially the widespread application of Large Language Models (LLMs) in natural language processing, reinforcement learning based on human feedback (RLHF) has emerged as a key means to improve the quality of model-generated content. RLHF fine-tunes the model by collecting and ranking human preferences for different text generation results, aiming to enable the model to more accurately generate text content that meets human expectations. The implementation of this technique typically involves three stages: supervised fine-tuning, preference sampling and reward function learning, and reinforcement learning optimization. Among these, the preference sampling and reward function learning stage is particularly important, as it directly relates to how human preferences are quantified into signals that the model can understand.

[0003] However, when applying the RLHF framework to the field of self-programming, we face a series of unique challenges and limitations that significantly differ from traditional text generation tasks. First, the core evaluation criterion for self-programming tasks is not simply based on text quality assessment ("how good" or "how bad"), but rather focuses on the actual execution performance of the code—that is, whether it can be successfully compiled and whether it can solve the user-defined specific task. This shift requires the evaluation system to move from subjective preference judgments to objective verification of execution capabilities. When both code snippets meet the compilation and execution requirements, traditional preference ranking loses its practical significance because they are functionally equivalent. Conversely, if the code has problems, precise error messages are needed to guide corrections, rather than general preference scores.

[0004] Secondly, data annotation is a major bottleneck hindering the application of RLHF in the field of self-programming. For large-scale code generation tasks, collecting sufficient human-annotated data is an extremely time-consuming and costly process. In the absence of large-scale support from professional data annotators, relying on manual annotation to obtain training data becomes particularly difficult, which greatly limits the promotion and application of RLHF technology in the field of self-programming. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this application provides a reinforcement learning method for self-programming tasks, aiming to overcome the limitations of existing frameworks in the field of self-programming and achieve more efficient and accurate code generation and optimization.

[0006] The technical means adopted by this invention to solve its technical problem is: a reinforcement learning method applied to self-programming tasks, the improvement of which is that the method includes: obtaining an example dataset applied to self-programming tasks, and fine-tuning an arbitrary open-source large language model based on the example dataset; connecting a first value output terminal and a second value output terminal to the fine-tuned large language model; the first value output terminal is used to predict whether the subsequently generated text can be compiled; the second value output terminal is used to predict whether the subsequently generated text can complete the self-programming task; by adjusting the temperature parameter, the large language model is encouraged to generate diverse code for the selected self-programming task, and positive and negative examples are collected; a series of texts are generated and recorded based on the generated diverse code using Monte Carlo tree search; the large language model is trained using the data collected by Monte Carlo tree search to optimize its code generation and evaluation capabilities.

[0007] The above technical solution, after fine-tuning any open-source large language model based on the example dataset, further includes: using a code compilation discriminator to check the syntax of the fine-tuned large language model generation program; and using a task execution discriminator to determine whether the code generated by the fine-tuned large language model can successfully complete the task.

[0008] The above technical solution describes generating a series of texts by performing Monte Carlo tree search based on the generated diverse code, including: the Monte Carlo tree search includes a selection phase, an expansion and evaluation phase, and a backtracking phase, wherein the selection phase selects the next leaf node to be expanded according to the selection formula; the expansion and evaluation phase performs value evaluation by executing a large model or triggering a termination condition based on the node state; and the backtracking phase updates the statistical information of each node.

[0009] The selection phase described in the above technical solution selects the next leaf node to be expanded according to the following formula:

[0010] (1);

[0011] Where c1, c2, and c3 are any constants greater than or equal to 0, c2 is much smaller than c1, c1 and c3 are on the same order of magnitude, c4+c5 is on the same order of magnitude as c1, and c4>0 and c5>0, U(s, a) is any formula, but it must satisfy the following characteristics:

[0012] U(s, a) is always greater than 0;

[0013] It is calculated from P(s,a) and N, where N is the total number of explorations;

[0014] The value of U(s,a) decreases as N increases, and the larger P(s,a) is, the larger U(s,a) is.

[0015] The values ​​are in the same finite interval as Q1(s,a) and Q2(s,a), and the maximum and minimum values ​​are similar to those of Q1(s,a) and Q2(s,a).

[0016] When N(s, a) = 0, U(s, a) = P(s, a).

[0017] The expansion and evaluation phase described in the above technical solution performs value evaluation based on the node state by executing a large model or triggering termination conditions, including:

[0018] Create a new leaf node at the end of the selected edge;

[0019] If the new node meets the termination condition, the text corresponding to the new node is passed to the compiler and simulator, feedback is collected, the true values ​​v1 and v2 are obtained, and the backtracking phase is entered.

[0020] If the new node does not meet the termination condition, but the Monte Carlo tree search needs to be terminated early due to computational consumption considerations, then the large language model is executed first to obtain the outputs of p and the two value output ends, and then the backtracking phase is entered.

[0021] If the new node does not meet the termination condition and the Monte Carlo tree search continues, the large language model still needs to be executed, but the outputs of the two value output ends will be ignored.

[0022] The backtracking phase described in the above technical solution uses the following formula to iteratively update all variables:

[0023] (2);

[0024] (3);

[0025] (4);

[0026] (5);

[0027] (6);

[0028] Where s' is the result of the model prediction of a certain state s.

[0029] The above technical solution describes using data collected via Monte Carlo tree search to train the large language model, thereby optimizing its code generation and evaluation capabilities, including:

[0030] We prepared an open-source large language model with preliminary fine-tuning as a control group.

[0031] The recorded data is randomly sampled. First, a large control model is used to receive s to generate p0. Then, a large language model is trained using a loss function. The loss function includes any formula that is positively correlated with the error, always greater than 0, and equal to 0, which represents no error.

[0032] The technical means adopted by this invention to solve its technical problem is: a reinforcement learning device applied to self-programming tasks, the device comprising:

[0033] The preparation module is used to obtain example datasets applicable to self-programming tasks and to fine-tune any open-source large language model based on the example datasets;

[0034] The prediction module is used to connect the first value output terminal and the second value output terminal after the large language model is fine-tuned; the first value output terminal is used to predict whether the subsequently generated text can be compiled; the second value output terminal is used to predict whether the subsequently generated text can complete the self-programming task.

[0035] The initial exploration module is used to encourage the large language model to generate diverse code for selected self-programming tasks by adjusting temperature parameters, and to collect positive and negative example data;

[0036] The secondary exploration module is used to perform Monte Carlo tree search based on the generated diverse codes to generate a series of texts and record them.

[0037] The training module is used to train the large language model using data collected through Monte Carlo tree search to optimize its code generation and evaluation capabilities.

[0038] The technical means adopted by this invention to solve its technical problem is: an electronic device, characterized in that it comprises: at least one processor and at least one memory, wherein,

[0039] The memory stores program instructions or code;

[0040] The program instructions or code are loaded and executed by the processor, enabling the electronic device to implement the reinforcement learning method described above for self-programming tasks.

[0041] The technical means adopted by the present invention to solve its technical problem is: a storage medium storing program instructions or code thereon, characterized in that the program instructions or code are loaded and executed by a processor to realize the reinforcement learning method applied to self-programming tasks as described above.

[0042] The beneficial effects of this invention are: it provides a reinforcement learning method for compiler feedback, the self-learning process does not require any manual intervention or much manual data preparation, most of the data is generated by the Monte Carlo tree exploration phase, overcoming the limitations of existing frameworks in the field of self-programming, and achieving more efficient and accurate code generation and optimization. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating a reinforcement learning method applied to self-programming tasks, as shown in an embodiment of the present invention.

[0044] Figure 2 This is a model structure diagram illustrating an embodiment of the present invention;

[0045] Figure 3 This is a schematic diagram of a code compilation discriminator according to an embodiment of the present invention;

[0046] Figure 4 This is a schematic diagram of a task execution discriminator according to an embodiment of the present invention;

[0047] Figure 5 This is a structural block diagram of a reinforcement learning device applied to a self-programming task, as shown in an embodiment of the present invention.

[0048] Figure 6 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0049] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0050] The following will clearly and completely describe the concept, specific structure, and technical effects of the present invention in conjunction with embodiments and accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are all within the scope of protection of the present invention. Furthermore, all connections and relationships involved in the patent do not simply refer to direct connection of components, but rather to the ability to form a better connection structure by adding or reducing connecting accessories according to specific implementation conditions. The various technical features in this invention can be combined interactively without contradicting each other.

[0051] As mentioned earlier, when applying the RLHF framework to the field of self-programming, traditional preference ranking loses its practical significance; at the same time, large-scale code generation tasks require collecting sufficient human preference annotation data, which is an extremely time-consuming and costly task.

[0052] To address these issues, the industry has begun exploring new solutions, such as Direct Preference Optimization (DPO) algorithms, which attempt to bypass the explicit training process of the reward function and directly optimize the model to satisfy specific preferences. However, these methods still rely on some form of "reward function" assumption and fail to fundamentally solve the specificities of evaluation criteria and the challenges of data labeling in self-programming tasks.

[0053] Based on this, this application provides a reinforcement learning method applied to the field of self-programming, such as... Figure 1 As shown, the method includes:

[0054] Step 10: Obtain an example dataset for the self-programming task, and fine-tune any open-source large language model based on the example dataset.

[0055] The example datasets applied to self-programming tasks may include, but are not limited to:

[0056] GitHub code repositories: These contain a large amount of open source project code, which can be used as datasets for tasks such as learning programming, analyzing programming habits, or performing code auto-completion.

[0057] Stack Overflow dataset: This is a programming question and answer website whose dataset contains a large number of programming questions and answers, which is very useful for training models that can understand programming questions;

[0058] Jupyter Notebook datasets: This is a popular code execution and sharing tool whose datasets may contain code examples from various programming languages ​​and domains, suitable for a variety of self-programming tasks.

[0059] The open-source large language model includes, but is not limited to:

[0060] GPT series: Although the latest models such as GPT-4 may not be fully open source, earlier versions such as GPT-2 and GPT-Neo are open source. These models have powerful language generation capabilities and can be used for various NLP (Natural Language Processing) tasks.

[0061] LLaMA: Supports multiple programming languages ​​and tasks;

[0062] ChatGLM is an open-source large language model optimized for Chinese, featuring bilingual question-answering and dialogue capabilities. It is particularly suitable for Chinese-related self-programming tasks.

[0063] In one possible implementation, using the Stack Overflow dataset and ChatGLM model as an example, the specific steps are as follows:

[0064] Data collection: A large number of programming questions and answers were collected from Stack Overflow as a training dataset.

[0065] Data preprocessing: The collected data is cleaned, segmented, encoded, and other preprocessed to enable the model to understand and process it.

[0066] Model Training: The ChatGLM model is trained using the preprocessed dataset. By adjusting the model's parameters and training strategies, the model is enabled to accurately understand programming problems and generate useful answers.

[0067] Model evaluation: The trained model is evaluated using a test dataset to verify its performance and accuracy.

[0068] Deployment: Deploy the trained model to the chatbot so that it can answer users' programming questions in real time.

[0069] Step 20: After the large language model is fine-tuned, connect the first value output terminal and the second value output terminal.

[0070] like Figure 2 As shown, the LLM head and Transformers modules are derived from a finely tuned pedestal model, while the Value head and EST head require complete retraining. These two heads can be any neural network.

[0071] The first value output terminal is used to predict whether the subsequently generated text can be compiled; the second value output terminal is used to predict whether the subsequently generated text can complete the self-programming task.

[0072] Since the correlation between the two "can or cannot" is very low, this application uses a tuple (s, a) to determine the result. The first number s in the tuple represents the reward for the task of "can or cannot compile", and the second number a represents the reward for "can or cannot complete the task".

[0073] The input to the value output terminal is either the whole or a part of the output of the last layer transformer block of the open-source large language model. That is, the value output terminal can receive the whole of all token encodings output by the last layer transformer, or it can receive the encoding of a specified token or tokens.

[0074] The output of the value output terminal is a scalar that falls within a continuous, finite interval, where the minimum and maximum values ​​of the finite interval represent the predictions of "cannot" and "can," respectively. However, it can also be the other way around, with the minimum value representing "can" and the maximum value representing "cannot."

[0075] In one exemplary embodiment, a value output can be a linear layer with a sigmoid activation function, and the input receives the encoding corresponding to the last token in the last layer of the transformer. The output has only one neuron, and because of the sigmoid activation function, the output result will only be between 0 and 1, where 1 represents meeting the standard and 0 represents not meeting it.

[0076] In another exemplary embodiment, a value output can be a neural network such as an RNN or LSTM. RNNs, LSTMs, and similar networks can aggregate serialized signals to obtain a fixed-length hidden state, which can then be used as the input to the first example.

[0077] In another exemplary embodiment, a value output can have a simple aggregation of the transformer block output. The aggregation method yields a semantic encoding for the entire string, which can then be input into the network of the first example to obtain a value estimate.

[0078] In another exemplary embodiment, a value output can be a transformer-like structure such as BERT (Bidirectional Encoder Representations from Transformers). This transformer-based language model represents the semantic information of an entire text segment through the encoding of special tokens. Such a network can also be placed at the front end of the first example to form a value network.

[0079] Step 30: By adjusting the temperature parameters, the large language model is encouraged to generate diverse code for the selected self-programming task, and positive and negative examples are collected.

[0080] By adjusting the temperature parameter (used to control the diversity and randomness of the model's generated results), the large language model is encouraged to generate diverse code for the selected self-programming tasks. For each self-programming task and each evaluation criterion, at least 5 positive examples and 5 negative examples are required.

[0081] Step 40: Perform a Monte Carlo tree search based on the generated diverse codes to generate a series of texts and record them.

[0082] Monte Carlo Tree Search (MCTS) is a decision tree-based search algorithm used to solve the optimal choice problem in the decision-making process. This algorithm approximates the optimal solution by combining stochastic simulation and tree search.

[0083] The generated series of texts includes: state, action, probability distribution π(*|s), P(*|s), and value assessment information v1(s), v2(s), EST(s), and EL(s).

[0084] Where π(*|s) represents the probability distribution of selecting the next action during the actual execution of MCTS. In actual deployment, this can be represented by a tensor of the form (vocabulary size, 1).

[0085] p(*|s) represents the probability distribution of the large language model for selecting the next action based on the current parameters;

[0086] v1(s): A scalar that measures whether subsequent text can be accepted by the compiler;

[0087] v2(s): A scalar measure of whether the current task can be solved;

[0088] EST: The expected number of tokens to be generated in the future;

[0089] EL: The estimated length of the optimal solution after a certain state s.

[0090] In this patent, each edge of the decision tree needs to have the following information:

[0091] Traversal count N(s, a): After the program starts, record the number of times this edge is traversed in all backtracking phases;

[0092] Q1(s, a): Defined as v1(s'), in actual calculation, except for the edges that directly connect the leaf nodes, the value of v1 is obtained by backtracking the process.

[0093] Q2(s, a): Defined as v2(s'), in actual calculation, apart from the edges directly connecting the leaf nodes, the value of v2 is obtained by backtracking the process.

[0094] EST(s,a): equal to EST(s'), output by the neural network of the next node.

[0095] At the same time, each node needs to record: the state itself; the neural network output p; and v. 1-pred (s), that is, the result of v1(s) predicted by the neural network; v 2-pred EST(s), i.e., the result of v2(s) predicted by the neural network; EST-pred(s), i.e., the result of EST(s) predicted by the neural network and the actual v1(s), v2(s), and EST(s) obtained by MCTS.

[0096] The Monte Carlo tree search includes a selection phase, an expansion and evaluation phase, and a backtracking phase. In the selection phase, the next leaf node to be expanded is selected according to the selection formula. In the expansion and evaluation phase, the value is evaluated by executing a large model or triggering a termination condition based on the node state. In the backtracking phase, the statistical information of each node is updated.

[0097] In one possible implementation, during the selection phase, if no initialization is performed, all N and Q are initialized to 0, and EL and EST are initialized to ∞.

[0098] After initialization, select the next leaf node to be expanded according to the following formula:

[0099] (1);

[0100] Where c1, c2, and c3 are any constants greater than or equal to 0, c2 is much smaller than c1, c1 and c3 are on the same order of magnitude, c4+c5 is on the same order of magnitude as c1, and c4>0 and c5>0; EST is the expected number of tokens to be generated in the future; EL is the estimated length of the optimal solution after a certain state s; U(s, a) is any formula, but it must satisfy the following characteristics:

[0101] U(s, a) is always greater than 0;

[0102] It is calculated from P(s,a) and N, where N is the total number of explorations and P(s,a) is the number of leaf nodes;

[0103] The value of U(s,a) decreases as N increases, and the larger P(s,a) is, the larger U(s,a) is.

[0104] The values ​​are in the same finite interval as Q1(s,a) and Q2(s,a), and the maximum and minimum values ​​are similar to those of Q1(s,a) and Q2(s,a).

[0105] When N(s, a) = 0, U(s, a) = P(s, a).

[0106] It is worth mentioning that the last term in the above formula (1), which is used to represent the relationship between EL and EST, only needs to be negatively correlated with EL and EST and always greater than 0. It is not limited to the above formula (1) and does not constitute a specific limitation here.

[0107] In one exemplary embodiment, Then the five constants c1, c2, c3, c4, and c5 mentioned above take values ​​of 1, 0.1, 1, 0.7, and 0.3, respectively.

[0108] In another exemplary embodiment, This refers to the confidence interval upper bound algorithm UCB1.

[0109] After an edge is selected, its corresponding N is incremented by 1 and recorded. , where * represents all possible values ​​of a.

[0110] In one possible implementation, the expansion and evaluation phase performs a value assessment based on the node state, either by executing a large model or by triggering a termination condition, including:

[0111] Create a new leaf node at the end of the selected edge;

[0112] If the new node meets the termination condition, the text corresponding to the new node is passed to the compiler and simulator, feedback is collected, the true values ​​v1 and v2 are obtained, and the backtracking phase is entered.

[0113] If the new node does not meet the termination condition, but the Monte Carlo tree search needs to be terminated early due to computational consumption considerations, then the large language model is executed first to obtain the outputs of p and the two value output ends, and then the backtracking phase is entered.

[0114] If the new node does not meet the termination condition and the Monte Carlo tree search continues, the large language model still needs to be executed, but the outputs of the two value output ends will be ignored.

[0115] The values ​​corresponding to the probability distribution output by the large language model will be stored. Next, new edges will be initialized at the end of the new leaf node by setting N, Q1(s,a), and Q2(s,a) to 0.

[0116] In one possible implementation, the backtracking phase iteratively updates all variables using the following formula.

[0117] (2);

[0118] (3);

[0119] (4);

[0120] (5);

[0121] (6);

[0122] Where s' is the result of the model prediction of a certain state s.

[0123] Step 50: Train the large language model using the data collected by Monte Carlo tree search to optimize its code generation and evaluation capabilities.

[0124] In one possible implementation, step 50 includes:

[0125] We prepared an open-source large language model with preliminary fine-tuning as a control group.

[0126] The recorded data is randomly sampled. First, the large language model in the control group receives s to generate p0. The large language model is then trained using a loss function. The loss function includes any formula that is positively correlated with the error, always greater than 0, and equal to 0, which represents no error.

[0127] In one exemplary embodiment, the loss function is as follows:

[0128] (7);

[0129] in Let p0 and p be the KL divergence. Let be the cross-entropy of two probability distributions.

[0130] It is worth mentioning that in the above formula (7), the relationship between v1 and v 1-pred The relationship between v2 and v 2-pred The relationship can also be expressed in absolute value form, that is:

[0131] ;

[0132] The exploration process of MCTS is trained using the above formula (7) until the termination condition is triggered.

[0133] In one exemplary embodiment, the termination condition may be a pre-set maximum number of iterations.

[0134] After training, you can return to step 10 and re-execute the process. To reuse the previous results, there are two re-execution options:

[0135] The first approach is to retain all the information from the previous Monte Carlo tree iteration and further explore other child nodes, but this approach requires significant computational resources such as memory size and processing speed.

[0136] The second approach is to select a child node from the Monte Carlo tree of the previous iteration (e.g., prioritize the leaf node with the highest preference), and then execute the MCTS algorithm with this node as the root node. In this case, the Monte Carlo tree and its data from the previous iteration have no effect in the new iteration (if the selected node is a leaf node) or are of little use (if it is not a leaf node), and are completely ignored in subsequent iterations.

[0137] The final large model trained through the above process is the model used for self-programming tasks in this application. In this model, the self-learning process requires no human intervention and does not require much data preparation. Most of the data is generated by the Monte Carlo tree exploration phase, overcoming the limitations of existing frameworks in the field of self-programming and achieving more efficient and accurate code generation and optimization.

[0138] In one possible implementation, after step 20, as follows: Figure 3-4 The method further includes:

[0139] The syntax of the fine-tuned large language model generator is checked using a code compiler discriminator.

[0140] The task execution discriminator is used to determine whether the code generated by the fine-tuned large language model can successfully complete the task.

[0141] The code compilation discriminator has the following characteristics:

[0142] ① It can accept strings of any length as input;

[0143] ② The output can only be selected from two values, 0 and 1. 1 means that the compilation was successful and 0 means that the compilation was unsuccessful.

[0144] ③ During the compilation process, intermediate compilation or execution results can be selectively output, but this step is not mandatory. Even if these intermediate compiled results are not output, the compilation process can continue and complete.

[0145] In one exemplary embodiment, the code compilation discriminator can be a Python compiler plus an error detector. If the error checker detects an error in the compilation process, it indicates that the compilation has failed.

[0146] The task execution discriminator has the following characteristics:

[0147] ① It can accept strings of any length as input;

[0148] ② The output can only be selected from two values, 0 and 1. 1 means that the task can be successfully completed, and 0 means that the task cannot be successfully completed.

[0149] In one exemplary embodiment, the task execution discriminator can call open-source simulation tools such as mujoco and ISAAC sim at the underlying level to quickly determine whether the termination condition can be successfully simulated.

[0150] Through the above embodiments, the code compilation discriminator can detect potential errors, inconsistencies, or inefficient code structures early on, thereby prompting developers to improve the code; at the same time, it can identify and correct many problems before the code runs, thereby reducing the possibility of program crashes or unexpected events during runtime; the task execution discriminator can monitor the execution status of tasks in real time and provide immediate feedback when problems occur, and by predicting the risk of task failure or identifying potential problems, measures can be taken to reduce the probability of task failure and improve the success rate of task execution.

[0151] It should be understood that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart above may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0152] The following are embodiments of the apparatus described in this application, which can be used to execute the reinforcement learning method for self-programming tasks involved in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments of the reinforcement learning method for self-programming tasks involved in this application.

[0153] Please see Figure 5 This application provides a reinforcement learning device 60 for self-programming tasks. The device 60 includes a preparation module 610, a prediction module 620, a preliminary exploration module 630, a secondary exploration module 640, and a training module 650.

[0154] The preparation module 610 is used to obtain an example dataset for self-programming tasks and to fine-tune any open-source large language model based on the example dataset.

[0155] The prediction module 620 is used to connect a first value output terminal and a second value output terminal after the large language model has been fine-tuned; the first value output terminal is used to predict whether the subsequently generated text can be compiled; the second value output terminal is used to predict whether the subsequently generated text can complete the self-programming task.

[0156] The initial exploration module 630 is used to encourage the large language model to perform diverse code generation for selected self-programming tasks by adjusting temperature parameters, and to collect positive and negative example data;

[0157] The secondary exploration module 640 is used to perform Monte Carlo tree search based on the generated diverse codes to generate a series of texts and record them.

[0158] Training module 650 is used to train the large language model using data collected by Monte Carlo tree search to optimize its code generation and evaluation capabilities.

[0159] It should be noted that the reinforcement learning device for self-programming tasks provided in the above embodiments is only illustrated by the division of the above functional modules when performing reinforcement learning for self-programming tasks. In practical applications, the above functions can be assigned to different functional modules as needed. That is, the internal structure of the reinforcement learning device for self-programming tasks will be divided into different functional modules to complete all or part of the functions described above. The above modules can be embedded in hardware or independent of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to the above modules.

[0160] Furthermore, the reinforcement learning device for self-programming tasks provided in the above embodiments and the method embodiments for reinforcement learning methods for self-programming tasks belong to the same concept. The specific way in which each module performs operations has been described in detail in the method embodiments, and will not be repeated here.

[0161] Please see Figure 6 This application provides an electronic device 4000.

[0162] exist Figure 6 In this design, data interaction between the processor 4001 and the memory 4003 can be achieved through at least one communication bus 4002. This communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0163] Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0164] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc. The memory 4003 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program instructions or code in the form of instructions or data structures and accessible by the electronic device 400, but not limited thereto.

[0165] The memory 4003 stores program instructions or code, and the processor 4001 can read the program instructions or code stored in the memory 4003 through the communication bus 4002.

[0166] When the program instructions or code are executed by the processor 4001, they implement the reinforcement learning methods applied to self-programming tasks in the above embodiments.

[0167] Furthermore, this application provides a storage medium storing program instructions or code, which is loaded and executed by a processor to implement a reinforcement learning method for self-programming tasks as described above.

[0168] This application provides a computer program product, which includes program instructions or code stored in a storage medium. The processor of an electronic device reads the program instructions or code from the storage medium, loads and executes the program instructions or code, thereby enabling the electronic device to implement the reinforcement learning method applied to self-programming tasks as described above.

[0169] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A reinforcement learning method applied to a self-programming task, characterized in that, The method comprises: obtaining an example data set applied to a self-programming task, and fine-tuning an arbitrary open-source large language model according to the example data set; connecting a first value output end and a second value output end after fine-tuning the large language model; the first value output end is used for predicting whether the subsequently generated text can be compiled; and the second value output end is used for predicting whether the subsequently generated text can complete the self-programming task; by adjusting the temperature parameter, encouraging the large language model to generate diversified code for the selected self-programming task, and collecting positive and negative example data; generating a series of texts by Monte Carlo tree search according to the generated diversified code and recording; training the large language model using the data collected by the Monte Carlo tree search to optimize its code generation and evaluation capabilities; the Monte Carlo tree search comprises a selection phase, an expansion and evaluation phase, and a backtracking phase, wherein the selection phase selects the next leaf node to be expanded according to a selection formula; the expansion and evaluation phase performs a large model or triggers a termination condition according to the node state to evaluate the value; and the backtracking phase updates the statistical information of each node; the expansion and evaluation phase performs a large model or triggers a termination condition according to the node state to evaluate the value, comprising: a new leaf node is created at the end of the selected edge; if the new node meets the termination condition, the text corresponding to the new node is transmitted into the compiler and the simulator, the feedback is collected, the true values v1 and v2 are obtained, and the backtracking phase is entered; if the new node does not meet the termination condition, but needs to terminate the Monte Carlo tree search in consideration of the calculation consumption, the large language model is executed first, and then the backtracking phase is entered; if the new node does not meet the termination condition and the Monte Carlo tree search still needs to be executed, the large language model still needs to be executed, but the outputs of the two value output ends will be ignored. After fine-tuning the arbitrary open-source large language model according to the example data set, the method further comprises: 2.The reinforcement learning method applied to a self-programming task according to claim 1, wherein, checking the syntax of the program generated by the fine-tuned large language model using a code compilation discriminator; judging whether the code generated by the fine-tuned large language model can successfully complete the task using a task execution discriminator. the training of the large language model using the data collected by the Monte Carlo tree search to optimize its code generation and evaluation capabilities, comprising: 3.The reinforcement learning method applied to a self-programming task according to claim 1, wherein, preparing an open-source large language model after preliminary fine-tuning as a control group; randomly sampling the recorded data, and training the large language model using a loss function; the loss function comprises any formula that is positively correlated with error, is always greater than 0, and is equal to 0 when there is no error. The device is used to implement the reinforcement learning method applied to a self-programming task according to any one of claims 1 to 3, and the device comprises:

4. A reinforcement learning device applied to a self-programming task, characterized by, a preparation module for obtaining an example data set applied to a self-programming task, and fine-tuning an arbitrary open-source large language model according to the example data set; ​ The prediction module is configured to connect a first value output end and a second value output end after the large language model is fine-tuned; the first value output end is configured to predict whether the subsequently generated text can be compiled; and the second value output end is configured to predict whether the subsequently generated text can complete the self-programming task. The preliminary exploration module is configured to encourage the large language model to generate diversified codes for the selected self-programming task by adjusting temperature parameters, and collect positive example and negative example data. The secondary exploration module is configured to perform Monte Carlo tree search based on the generated diversified codes to generate a series of texts and record them. The training module is configured to train the large language model based on the data collected by the Monte Carlo tree search to optimize the code generation and evaluation capabilities of the large language model.

5. An electronic device, comprising: The application comprises: at least one processor and at least one memory, program instructions or codes are stored on the memory; The program instructions or codes are loaded and executed by the processor, so that the electronic device implements the reinforcement learning method for self-programming tasks as claimed in any one of claims 1 to 3.

6. A storage medium having stored thereon program instructions or code, characterized in that, The program instructions or codes are loaded and executed by the processor to implement the reinforcement learning method for self-programming tasks as claimed in any one of claims 1 to 3.

Images