Reinforcement learning method and device applied to self-programming task, electronic equipment and storage medium

Abstract

The invention relates to the technical field of reinforcement learning, in particular to a reinforcement learning method and device applied to a self-programming task, electronic equipment and a storage medium, and the method comprises the steps: obtaining an example data set applied to the self-programming task, and carrying out the fine adjustment of any open-source large language model according to the example data set; connecting a first value output end and a second value output end behind the fine-tuned large language model; encouraging the big language model to perform diversified code generation for the selected self-programming task by adjusting temperature parameters, and collecting positive example and counter example data; performing Monte Carlo tree search according to the generated diversified codes to generate a series of texts and recording the texts; and training a large language model by using data collected by Monte Carlo tree search to optimize the code generation and evaluation capability. The limitation of an existing framework in the self-programming field is overcome, and more efficient and more accurate code generation and optimization are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of reinforcement learning, and in particular to a reinforcement learning method, device, electronic device and storage medium applied to a self-programming task. Background Art

[0002] With the rapid development of artificial intelligence technology, especially the widespread application of large language models (LLM) in the field of natural language processing, reinforcement learning based on human feedback (RLHF) technology has emerged and become a key means to improve the quality of model-generated content. RLHF technology collects human preference rankings for different text generation results and fine-tunes the model, aiming to enable the model to more accurately generate text content that meets human expectations. The implementation of this technology usually includes three stages: supervised fine-tuning stage, preference sampling and reward function learning stage, and reinforcement learning optimization stage. Among them, the preference sampling and reward function learning stage is particularly important, which is directly related to how human preferences are quantified into signals that the model can understand.

[0003] However, when applying the RLHF framework to the field of self-programming, we face a series of unique challenges and limitations that are significantly different from traditional text generation tasks. First, the core evaluation criteria of self-programming tasks are not simply based on "good or bad" text quality assessment, but focus on the actual execution effect of the code, that is, "whether it can be successfully compiled and "whether it can" solve specific tasks defined by users. This shift requires the evaluation system to shift from subjective preference judgment to objective execution capability verification. When two pieces of code can meet the compilation and execution requirements, the traditional preference ranking loses its practical significance because the two can be regarded as equivalent in function; on the contrary, if there is a problem with the code, precise error information is needed to guide the correction, rather than a general preference score.

[0004] Secondly, the data labeling problem has become a major bottleneck hindering the application of RLHF in the field of self-programming. For large-scale code generation tasks, collecting enough human-preference labeled data is an extremely time-consuming and costly task. In the absence of large-scale support from professional "data labelers", it becomes particularly difficult to rely on manual labeling to obtain training data, which greatly limits the promotion and application of RLHF technology in the field of self-programming. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present application provides a reinforcement learning method applied to self-programming tasks, aiming to overcome the limitations of existing frameworks in the field of self-programming and achieve more efficient and accurate code generation and optimization.

[0006] The technical means adopted by the present invention to solve its technical problems are: a reinforcement learning method applied to self-programming tasks, and the improvement thereof is that the method comprises: obtaining an example data set applied to the self-programming task, and fine-tuning any open source large language model according to the example data set; connecting a first value output terminal and a second value output terminal after the large language model after fine-tuning; the first value output terminal is used to predict whether the subsequently generated text can be compiled; the second value output terminal is used to predict whether the subsequently generated text can complete the self-programming task; by adjusting the temperature parameter, the large language model is encouraged to perform diversified code generation for the selected self-programming task, and positive and negative example data are collected; a Monte Carlo tree search is performed according to the generated diversified code to generate a series of texts and record them; and the large language model is trained using the data collected by the Monte Carlo tree search to optimize its code generation and evaluation capabilities.

[0007] After fine-tuning any open source large language model according to the example data set as described in the above technical solution, the method also includes: using a code compilation discriminator to check the syntax of the fine-tuned large language model generation program; using a task execution discriminator to determine whether the code generated by the fine-tuned large language model can successfully complete the task.

[0008] The above technical solution generates a series of texts by performing Monte Carlo tree search according to the generated diversified code, including: the Monte Carlo tree search includes a selection phase, an expansion and evaluation phase, and a backtracking phase, wherein the selection phase selects the next leaf node to be expanded according to a specific selection formula; the expansion and evaluation phase executes a large model according to the node status or triggers a termination condition for value evaluation; and the backtracking phase updates the statistical information of each node.

[0009] The selection phase in the above technical solution selects the next leaf node to be expanded according to the following formula:

[0010]

[0011] Among them, c1, c2, c3 are any constants greater than or equal to 0, c2 is much smaller than c1, c1 and c3 are in the same order of magnitude, c4+c5 and c1 are in the same order of magnitude, and c4>0, c5>0, U(s,a) can be any formula, but it needs to meet the following characteristics:

[0012] U(s,a) is always greater than 0;

[0013] Calculated by P and N, where N is the total number of explorations;

[0014] The value of U decreases as N increases. The larger P is, the larger U is.

[0015] The value is in the same finite interval as Q1 and Q2, and the maximum and minimum values ​​are similar to Q1 and Q2;

[0016] When N(s,a)=0, U(s,a)=P(s,a).

[0017] The expansion and evaluation phase described in the above technical solution executes a large model or triggers a termination condition to perform value evaluation according to the node status, including:

[0018] Create a new leaf node at the end of the selected edge;

[0019] If the new node meets the termination condition, the text corresponding to the new node is passed to the compiler and simulator, feedback is collected, the true value v1, v2 is obtained, and the backtracking phase is entered;

[0020] If the new node does not meet the termination condition, but the Monte Carlo tree search needs to be terminated early due to computational consumption, the large language model is first executed to obtain the output of p and the two value output terminals, and then the backtracking phase is entered;

[0021] If the new node does not meet the termination condition and the Monte Carlo tree search is to be executed, the large language model still needs to be executed, but the outputs of the two value output terminals will be ignored.

[0022] In the backtracking phase described in the above technical solution, all variables are iteratively updated using the following formula:

[0023]

[0024] Q(s,a)=v(s')(3);

[0025]

[0026] EST(s,a)=EST(s')(5);

[0027] EL(s)=depth(s)+EST(s) (6);

[0028] Among them, s' is the output result of a certain state s after being predicted by the model.

[0029] The above technical solution uses the data collected by Monte Carlo tree search to train the large language model to optimize its code generation and evaluation capabilities, including:

[0030] Prepare a preliminary fine-tuned open source large language model as a control group;

[0031] The recorded data is randomly sampled, s is first received by the control large model to generate p0, and the large language model is trained with the loss function; the loss function includes any formula that is positively correlated with the error, is always greater than 0, and represents no error when equal to 0.

[0032] The technical means adopted by the present invention to solve the technical problem is: a reinforcement learning device applied to self-programming tasks, the device comprising:

[0033] A preparation module, used to obtain an example dataset applied to the self-programming task and fine-tune any open source large language model based on the example dataset;

[0034] A prediction module, used to connect the first value output terminal and the second value output terminal after the large language model is fine-tuned; the first value output terminal is used to predict whether the subsequently generated text can be compiled; the second value output terminal is used to predict whether the subsequently generated text can complete the self-programming task;

[0035] A preliminary exploration module, for encouraging the large language model to generate diversified codes for the selected self-programming task by adjusting a temperature parameter, and collecting positive and negative example data;

[0036] A secondary exploration module, used for performing a Monte Carlo tree search according to the generated diversified codes to generate a series of texts and record them;

[0037] The training module is used to train the large language model using data collected by Monte Carlo tree search to optimize its code generation and evaluation capabilities.

[0038] The technical means adopted by the present invention to solve the technical problem is: an electronic device, characterized in that it includes: at least one processor and at least one memory, wherein:

[0039] The memory stores program instructions or codes;

[0040] The program instructions or codes are loaded and executed by the processor, so that the electronic device implements the reinforcement learning method applied to self-programming tasks as described above.

[0041] The technical means adopted by the present invention to solve its technical problem is: a storage medium on which program instructions or codes are stored, characterized in that the program instructions or codes are loaded and executed by a processor to implement the reinforcement learning method applied to self-programming tasks as described above.

[0042] The beneficial effects of the present invention are: providing a reinforcement learning method for compiler feedback, the self-learning process does not require any human intervention, nor does it require manual preparation of too much data. Most of the data is generated by the Monte Carlo tree exploration stage, overcoming the limitations of existing frameworks in the field of self-programming and achieving more efficient and accurate code generation and optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flow chart of a reinforcement learning method applied to a self-programming task shown in an embodiment of the present invention;

[0044] Figure 2 A model structure diagram showing an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of a code compilation discriminator shown in an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of a task execution discriminator shown in an embodiment of the present invention;

[0047] Figure 5 A structural block diagram of a reinforcement learning device applied to a self-programming task according to an embodiment of the present invention;

[0048] Figure 6 The present invention is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0050] The following will clearly and completely describe the concept, specific structure and technical effects of the present invention in combination with the embodiments and drawings, so as to fully understand the purpose, characteristics and effects of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, other embodiments obtained by technicians in this field without creative work are all within the scope of protection of the present invention. In addition, all the connections and connection relationships involved in the patent do not refer to the direct connection of components, but refer to the formation of a better connection structure by adding or reducing connection accessories according to the specific implementation situation. The various technical features in the invention can be combined interchangeably without conflicting with each other.

[0051] As mentioned earlier, when the RLHF framework is applied to the field of self-programming, traditional preference ranking loses its practical significance; at the same time, for large-scale code generation tasks, collecting sufficient human preference annotation data is an extremely time-consuming and costly task.

[0052] In response to the above problems, the industry has begun to explore new solutions, such as direct preference optimization (DPO) and other algorithms, which attempt to bypass the explicit training process of the reward function and directly optimize the model to meet specific preferences. However, these methods still rely on some form of "reward function" assumptions and fail to fundamentally solve the particularity of evaluation criteria and data labeling problems in self-programming tasks.

[0053] Based on this, this application provides a reinforcement learning method applied to the field of self-programming, such as Figure 1 As shown, the method includes:

[0054] Step 10: Obtain an example dataset applied to the self-programming task, and fine-tune any open source large language model based on the example dataset.

[0055] The example data set applied to the self-programming task may include but is not limited to:

[0056] GitHub code library: It contains a large amount of open source project codes. These code libraries can be used as datasets for learning programming, analyzing programming habits, or performing code auto-completion tasks.

[0057] Stack Overflow dataset: This is a programming question-and-answer website. Its dataset contains a large number of programming questions and answers, which is very useful for training models that can understand programming problems.

[0058] Jupyter Notebook datasets: This is a popular tool for executing and sharing code. Its datasets may contain code examples in various programming languages ​​and domains, suitable for a variety of self-programming tasks.

[0059] The open source large language model includes but is not limited to:

[0060] GPT series: Although the latest models such as GPT-4 may not be fully open source, early versions such as GPT-2 and GPT-Neo are open source. These models have strong language generation capabilities and can be used for various NLP (natural language processing) tasks;

[0061] LLaMA: can support multiple programming languages ​​and tasks;

[0062] ChatGLM: is an open source large language model optimized for Chinese, with bilingual question-answering and conversation capabilities. It is particularly suitable for Chinese-related self-programming tasks.

[0063] In one possible implementation, taking the Stack Overflow dataset and ChatGLM model as an example, the specific steps are as follows:

[0064] Data collection: A large number of programming questions and answers are collected from Stack Overflow as training datasets.

[0065] Data preprocessing: Perform preprocessing operations such as cleaning, word segmentation, and encoding on the collected data so that the model can understand and process it.

[0066] Model training: Use the preprocessed dataset to train the ChatGLM model. By adjusting the model parameters and training strategy, the model can accurately understand programming questions and generate useful answers.

[0067] Model Evaluation: The trained model is evaluated using the test dataset to verify its performance and accuracy.

[0068] Deploy the application: Deploy the trained model to the chatbot so that it can answer users' programming questions in real time.

[0069] Step 20: After fine-tuning the large language model, connect the first value output terminal and the second value output terminal.

[0070] like Figure 2 As shown in the figure, the LLM head and Transformers modules are obtained from the fine-tuned base model, while the Value head and EST head need to be completely retrained. These two heads can be any neural network.

[0071] Among them, the first value output end is used to predict whether the subsequently generated text can be compiled; the second value output end is used to predict whether the subsequently generated text can complete the self-programming task.

[0072] Since the correlation between the two "can or cannot" is very low, this application uses a tuple (s, a) to decide. The first number s in the tuple represents the reward corresponding to the task "can or cannot be compiled", and the second number a represents the reward corresponding to "can or cannot complete the task".

[0073] Among them, the input of the value output end is the whole or part of the output of the last layer transformer block of the open source large language model, that is, the value output end can receive the whole of all token encodings output by the last layer transformer, and can also receive the encoding of a specified token or tokens.

[0074] The output of the value output terminal is a scalar that falls within a continuous, finite interval, and the minimum and maximum values ​​of the finite interval represent the predictions of "cannot" and "can", respectively. However, it can also be reversed, with the minimum value representing "can" and the maximum value representing "cannot".

[0075] In an exemplary embodiment, a value output terminal can be a linear layer with a sigmoid activation function, and the input terminal receives the code corresponding to the last token in the last layer of the transformer as input. The output terminal has only one neuron, and because of the sigmoid activation function, the output result will only be between 0 and 1, where 1 represents that the standard can be met and 0 represents that it cannot be met.

[0076] In another exemplary embodiment, a value output terminal may be a neural network such as RNN, LSTM, etc. RNN, LSTM, etc. networks may aggregate serialized signals to obtain a fixed-length hidden state, which may be used as the input of the first example.

[0077] In another exemplary embodiment, a value output terminal may have a simple aggregation of the transformer block output. The aggregation method is used to obtain a semantic encoding for the entire string, and the obtained semantic encoding can be input into the network of the first example to obtain a value estimate.

[0078] In another exemplary embodiment, a value output terminal can be a transformer-like structure such as BERT (Bidirectional Encoder Representations from Transformers). This transformer-based language model represents the semantic information of the entire text by encoding special tokens. This type of network can also be placed at the front end of the first example to form a value network.

[0079] Step 30: By adjusting the temperature parameter, the large language model is encouraged to perform diversified code generation for the selected self-programming task, and collect positive and negative example data.

[0080] By adjusting the temperature parameter (used to control the diversity and randomness of the model generation results), the large language model is encouraged to generate diversified code for the selected self-programming tasks. For each self-programming task and each evaluation criterion, at least 5 positive examples and 5 negative examples are required.

[0081] Step 40: Perform a Monte Carlo tree search based on the generated diversified code to generate a series of texts and record them.

[0082] Among them, Monte Carlo Tree Search (MCTS) is a search algorithm based on decision trees, which is used to solve the optimal choice problem in the decision-making process. The algorithm approaches the optimal solution by combining random simulation and tree search.

[0083] The generated series of texts include: state state, action action, probability distribution π(*|s), P(*|s), value evaluation information v1(s), v2(s), EST(s), EL(s).

[0084] Among them, π(*|s) represents the probability distribution of the next action selected when MCTS is actually executed. In actual deployment, it can be represented by a tensor of the form (vocabulary size, 1);

[0085] p(*|s) represents the probability distribution of selecting the next action output by the large language model with the current parameters;

[0086] v1(s): a scalar that measures whether the subsequent text is acceptable to the compiler;

[0087] v2(s): a scalar that measures whether the current task can be solved;

[0088] EST: the expected number of tokens to be generated in the future;

[0089] EL: is the estimated length of the optimal solution passing through a certain state s.

[0090] In this patent, each edge of the decision tree needs to have the following information:

[0091] Traversal times N(s,a): After the program starts, record the number of traversals of this edge in all backtracking stages;

[0092] Q1(s,a): defined as v1(s'). In actual calculation, except for the edges directly connecting leaf nodes, the v1 value is obtained by reverse inference through the backtracking process.

[0093] Q2(s,a): defined as v2(s'). In actual calculation, except for the edges directly connecting leaf nodes, the v2 value is obtained by backtracking the process.

[0094] EST(s,a): equal to EST(s'), output by the neural network of the next node.

[0095] At the same time, each node needs to record: the state itself; the neural network output p; v 1-pred (s), that is, the result of v1(s) predicted by the neural network; v 2-pred (s), that is, the result of v2(s) predicted by the neural network; EST-pred(s), that is, the result of EST(s) predicted by the neural network and the actual v1(s), v2(s), EST(s) obtained by MCTS.

[0096] The Monte Carlo tree search includes a selection phase, an expansion and evaluation phase, and a backtracking phase. The selection phase selects the next leaf node to be expanded according to a specific selection formula; the expansion and evaluation phase executes a large model or triggers a termination condition to perform value evaluation according to the node status; and the backtracking phase updates the statistical information of each node.

[0097] In a possible implementation, in the selection phase, if there is no initialization, all N and Q are initialized to 0, and EL and EST are initialized to ∞.

[0098] After initialization is completed, the next leaf node to be expanded is selected according to the following formula:

[0099]

[0100] Among them, c1, c2, c3 are any constants greater than or equal to 0, c2 is much smaller than c1, c1 and c3 are in the same order of magnitude, c4+c5 and c1 are in the same order of magnitude, and c4>0, c5>0, EST is the expected number of tokens to be generated in the future; EL is the estimated length of the optimal solution passing through a certain state s; U(s,a) can be any formula, but it needs to meet the following characteristics:

[0101] U(s,a) is always greater than 0;

[0102] Calculated by P and N, N is the total number of explorations, and P is the number of leaf nodes;

[0103] The value of U decreases as N increases. The larger P is, the larger U is.

[0104] The value is in the same finite interval as Q1 and Q2, and the maximum and minimum values ​​are similar to Q1 and Q2;

[0105] When N(s,a)=0, U(s,a)=P(s,a).

[0106] It is worth mentioning that the last term in the above formula (1), which is used to express the relationship between EL and EST, only needs to be negatively correlated with EL and EST and always greater than 0, and is not limited to the above formula (1) and does not constitute a specific limitation here.

[0107] In an exemplary embodiment, Then the above five constants c1, c2, c3, c4, and c5 take values ​​of 1, 0.1, 1, 0.7, and 0.3 respectively.

[0108] In another exemplary embodiment, That is, the confidence interval upper bound algorithm UCB1.

[0109] After an edge is selected, its corresponding N is increased by 1, and π(*|s)=softmax(preference(s,*)) is recorded, where * represents all possible values ​​of a.

[0110] In a possible implementation, the expansion and evaluation phase performs value evaluation by executing a large model or triggering a termination condition according to the node status, including:

[0111] Create a new leaf node at the end of the selected edge;

[0112] If the new node meets the termination condition, the text corresponding to the new node is passed to the compiler and simulator, feedback is collected, the true value v1, v2 is obtained, and the backtracking phase is entered;

[0113] If the new node does not meet the termination condition, but the Monte Carlo tree search needs to be terminated early due to computational consumption, the large language model is first executed to obtain the output of p and the two value output terminals, and then the backtracking phase is entered;

[0114] If the new node does not meet the termination condition and the Monte Carlo tree search is to be executed, the large language model still needs to be executed, but the outputs of the two value output terminals will be ignored.

[0115] The value corresponding to the probability distribution output by the large language model will be stored, and then a new edge will be initialized at the end of the new leaf node by setting N, Q1, and Q2 to 0.

[0116] In a possible implementation, the backtracking phase iteratively updates all variables using the following formula:

[0117]

[0118] Q(s,a)=v(s')(3);

[0119]

[0120] EST(s,a)=EST(s')(5);

[0121] EL(s)=depth(s)+EST(s) (6);

[0122] Among them, s' is the output result of a certain state s after being predicted by the model.

[0123] Step 50: Use the data collected by Monte Carlo tree search to train the large language model to optimize its code generation and evaluation capabilities.

[0124] In a possible implementation, step 50 includes:

[0125] Prepare a preliminary fine-tuned open source large language model as a control group;

[0126] The recorded data are randomly sampled, and the large language model in the control group is first used to receive s to generate p0, and the large language model is trained using a loss function; the loss function includes any formula that is positively correlated with the error, is always greater than 0, and represents no error when equal to 0.

[0127] In an exemplary embodiment, the loss function is as follows:

[0128] loss = c1(v1-v 1-pred ) 2 +c2(v2-v 2-pred ) 2 -πlogp+c6KL(p0||p) (7);

[0129] where KL(p0||p) is the KL divergence between p0 and p, πlog p is the cross entropy of two probability distributions.

[0130] It is worth mentioning that in the above formula (7), the relationship between v1 and v 1-pred The relationship between v2 and v 2-pred The relationship can also be expressed in the form of absolute value, that is:

[0131] loss = c1 | v1 - v 1-pred |+c2|v2-v 2-pred |-πlogp+c6KL(p0||p);

[0132] The MCTS exploration process is trained using the above formula (7) until the termination condition is triggered.

[0133] In an exemplary embodiment, the termination condition may be a maximum number of iterations preset in advance.

[0134] After the training is completed, you can return to step 10 and re-execute. In order to reuse the previous results, there are two re-execution schemes:

[0135] The first is to retain all the information of the last iteration of the Monte Carlo tree and further explore other child nodes, but this solution requires a lot of computing resources such as memory size and computing speed;

[0136] The second is to select a child node in the last iteration of the Monte Carlo tree (such as the leaf node with the largest preference), and then use this node as the root node to execute the MCTS algorithm. At this time, the Monte Carlo tree and the data in the last iteration have no effect in the new iteration (if the selected node is a leaf node) or are of little use (if it is not a leaf node), and are completely ignored in the following iterations.

[0137] The final large model trained by the above process is the model used for self-programming tasks in this application. In this model, the self-learning process does not require any human intervention, nor does it require much manual preparation of data. Most of the data is generated by the Monte Carlo tree exploration phase, which overcomes the limitations of existing frameworks in the field of self-programming and achieves more efficient and accurate code generation and optimization.

[0138] In a possible implementation, after step 20, as Figure 3-4 The method further comprises:

[0139] Using a code compilation discriminator to check the syntax of the fine-tuned large language model generation program;

[0140] A task execution discriminator is used to determine whether the code generated by the fine-tuned large language model can successfully complete the task.

[0141] The code compilation discriminator has the following characteristics:

[0142] ①Can accept strings of any length as input;

[0143] ② The output can only be selected from two values, 0 and 1, 1 represents successful compilation, 0 represents unsuccessful compilation;

[0144] ③During the compilation process, you can selectively output the intermediate results of compilation or execution, but this step is not mandatory. Even if these intermediate compiled results are not output, the compilation process can continue and be completed.

[0145] In an exemplary embodiment, the code compilation discriminator may be a python compiler plus an error detector. If the error checker detects that there is an error in the compilation process, it means that the compilation has failed.

[0146] The task execution discriminator has the following characteristics:

[0147] ①Can accept strings of any length as input;

[0148] ② The output can only be selected from two values, 0 and 1. 1 represents that the task can be successfully completed, and 0 represents that the task cannot be successfully completed.

[0149] In an exemplary embodiment, the task execution discriminator may call open source simulation tools such as mujoco and ISAAC sim at the bottom layer to quickly determine whether the termination condition can be successfully simulated.

[0150] Through the above embodiments, the code compilation discriminator can be used to discover potential errors, inconsistent or inefficient code structures early, thereby prompting developers to improve the code; at the same time, many problems can be identified and corrected before the code runs, thereby reducing the possibility of program crashes or accidents during runtime; the task execution discriminator can be used to monitor the execution status of the task in real time and provide instant feedback when problems arise, and by predicting the risk of task failure or identifying potential problems, measures can be taken to reduce the probability of task failure and improve the success rate of task execution.

[0151] It should be understood that, although the various steps in the above-mentioned flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0152] The following is an embodiment of the device of the present application, which can be used to execute the reinforcement learning method for self-programming tasks involved in the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the method embodiment of the reinforcement learning method for self-programming tasks involved in the present application.

[0153] See also Figure 5 In an embodiment of the present application, a reinforcement learning device 60 for self-programming tasks is provided, and the device 60 includes: a preparation module 610, a prediction module 620, a preliminary exploration module 630, a secondary exploration module 640 and a training module 650.

[0154] The preparation module 610 is used to obtain an example data set applied to the self-programming task, and fine-tune any open source large language model according to the example data set;

[0155] Prediction module 620, used to connect the first value output terminal and the second value output terminal after the large language model is fine-tuned; the first value output terminal is used to predict whether the subsequently generated text can be compiled; the second value output terminal is used to predict whether the subsequently generated text can complete the self-programming task;

[0156] A preliminary exploration module 630, for encouraging the large language model to perform diversified code generation for the selected self-programming task by adjusting a temperature parameter, and collecting positive and negative example data;

[0157] A secondary exploration module 640 is used to perform a Monte Carlo tree search based on the generated diversified code to generate a series of texts and record them;

[0158] The training module 650 is used to train the large language model using data collected by Monte Carlo tree search to optimize its code generation and evaluation capabilities.

[0159] It should be noted that the reinforcement learning device for self-programming tasks provided in the above embodiments only uses the division of the above functional modules as an example when performing reinforcement learning for self-programming tasks. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the reinforcement learning device for self-programming tasks will be divided into different functional modules to complete all or part of the functions described above. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0160] In addition, the reinforcement learning device applied to self-programming tasks and the method embodiment of the reinforcement learning method applied to self-programming tasks provided in the above embodiments belong to the same concept, and the specific manner in which each module performs operations has been described in detail in the method embodiment and will not be repeated here.

[0161] See also Figure 6 , an electronic device 4000 is provided in an embodiment of the present application.

[0162] exist Figure 6 In the embodiment, the data interaction between the processor 4001 and the memory 4003 can be realized through at least one communication bus 4002. The communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0163] Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0164] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program instructions or codes in the form of instructions or data structures and can be accessed by the electronic device 400, but is not limited to this.

[0165] The memory 4003 stores program instructions or codes, and the processor 4001 can read the program instructions or codes stored in the memory 4003 through the communication bus 4002 .

[0166] When the program instructions or codes are executed by the processor 4001, the reinforcement learning method applied to the self-programming task in the above-mentioned embodiments is implemented.

[0167] In addition, an embodiment of the present application provides a storage medium on which program instructions or codes are stored, and the program instructions or codes are loaded and executed by a processor to implement the reinforcement learning method applied to self-programming tasks as described above.

[0168] In an embodiment of the present application, a computer program product is provided. The computer program product includes program instructions or codes. The program instructions or codes are stored in a storage medium. A processor of an electronic device reads the program instructions or codes from the storage medium, loads and executes the program instructions or codes, so that the electronic device implements the reinforcement learning method applied to self-programming tasks as described above.

[0169] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A reinforcement learning method for self-programming tasks, characterized in that: The method comprises: Obtain an example dataset for a self-programming task and fine-tune any open source large language model based on the example dataset; After the fine-tuning of the large language model, a first value output terminal and a second value output terminal are connected; the first value output terminal is used to predict whether the subsequently generated text can be compiled; the second value output terminal is used to predict whether the subsequently generated text can complete the self-programming task; By adjusting the temperature parameter, the large language model is encouraged to generate diversified codes for the selected self-programming task, and positive and negative example data are collected; Performing a Monte Carlo tree search based on the generated diversified code generates a series of texts and records them; The large language model is trained using data collected by Monte Carlo tree search to optimize its code generation and evaluation capabilities.

2. The reinforcement learning method for self-programming tasks according to claim 1, characterized in that: After fine-tuning any open source large language model according to the example data set, the method further includes: Using a code compilation discriminator to check the syntax of the fine-tuned large language model generation program; A task execution discriminator is used to determine whether the code generated by the fine-tuned large language model can successfully complete the task.

3. The reinforcement learning method for self-programming tasks according to claim 1, characterized in that: The Monte Carlo tree search is performed according to the generated diversified code to generate a series of texts, including: The Monte Carlo tree search includes a selection phase, an expansion and evaluation phase, and a backtracking phase. The selection phase selects the next leaf node to be expanded according to a specific selection formula; the expansion and evaluation phase executes a large model or triggers a termination condition to perform value evaluation according to the node status; and the backtracking phase updates the statistical information of each node.

4. The reinforcement learning method for self-programming tasks according to claim 3, characterized in that: The selection stage selects the next leaf node to be expanded according to the following formula: Among them, c1, c2, c3 are any constants greater than or equal to 0, c2 is much smaller than c1, c1 and c3 are in the same order of magnitude, c4+c5 and c1 are in the same order of magnitude, and c4>0, c5>0, EST is the expected number of tokens to be generated in the future; EL is the estimated length of the optimal solution passing through a certain state s; U(s,a) can be any formula, but it needs to meet the following characteristics: U(s,a) is always greater than 0; Calculated by P and N, where N is the total number of explorations; The value of U decreases as N increases. The larger P is, the larger U is. The value is in the same finite interval as Q1 and Q2, and the maximum and minimum values ​​are similar to Q1 and Q2; When N(s,a)=0, U(s,a)=P(s,a).

5. The reinforcement learning method for self-programming tasks according to claim 3, characterized in that: The expansion and evaluation phase executes the large model or triggers the termination condition to perform value evaluation according to the node status, including: Create a new leaf node at the end of the selected edge; If the new node meets the termination condition, the text corresponding to the new node is passed to the compiler and simulator, feedback is collected, the true value v1, v2 is obtained, and the backtracking phase is entered; If the new node does not meet the termination condition, but the Monte Carlo tree search needs to be terminated early due to computational consumption, the large language model is first executed to obtain the output of p and the two value output terminals, and then the backtracking phase is entered; If the new node does not meet the termination condition and the Monte Carlo tree search is to be executed, the large language model still needs to be executed, but the outputs of the two value output terminals will be ignored.

6. The reinforcement learning method for self-programming tasks according to claim 3, characterized in that: The backtracking phase iteratively updates all variables using the following formula: Q(s,a)=v(s')(3); EST(s,a)=EST(s')(5); EL(s)=depth(s)+EST(s) (6); Among them, s' is the output result of a certain state s after being predicted by the model.

7. The reinforcement learning method for self-programming tasks according to claim 3, characterized in that: The data collected by Monte Carlo tree search is used to train the large language model to optimize its code generation and evaluation capabilities, including: Prepare a preliminary fine-tuned open source large language model as a control group; The recorded data are randomly sampled, and the large language model of the control group is first used to receive s to generate p0, and the large language model is trained with a loss function; the loss function includes any formula that is positively correlated with the error, is always greater than 0, and represents no error when equal to 0.

8. A reinforcement learning device for self-programming tasks, characterized in that: The device comprises: A preparation module, used to obtain an example dataset applied to the self-programming task and fine-tune any open source large language model based on the example dataset; A prediction module, used to connect the first value output terminal and the second value output terminal after the large language model is fine-tuned; the first value output terminal is used to predict whether the subsequently generated text can be compiled; the second value output terminal is used to predict whether the subsequently generated text can complete the self-programming task; A preliminary exploration module, for encouraging the large language model to generate diversified codes for the selected self-programming task by adjusting a temperature parameter, and collecting positive and negative example data; A secondary exploration module, used for performing a Monte Carlo tree search according to the generated diversified codes to generate a series of texts and record them; The training module is used to train the large language model using data collected by Monte Carlo tree search to optimize its code generation and evaluation capabilities.

9. An electronic device, characterized in that: include: at least one processor, at least one memory, wherein: The memory stores program instructions or codes; The program instructions or codes are loaded and executed by the processor, so that the electronic device implements the reinforcement learning method applied to self-programming tasks as described in any one of claims 1 to 7.

10. A storage medium having program instructions or codes stored thereon, characterized in that: The program instructions or codes are loaded and executed by a processor to implement the reinforcement learning method applied to self-programming tasks as described in any one of claims 1 to 7.

Images