Method and device for generating reinforcement learning reward function based on large language model

Abstract

The invention discloses a reinforcement learning reward function generation method and device based on a large language model. The method comprises the steps of obtaining original task description data, environment code data and cue word calling template data; inputting the original task description data, the environment code data, the initialized cue word template and the code summary cue word template into a preset large language model to obtain a first environment variable output result; in response to the condition that the first environment variable output result meets a first preset condition, inputting the temperature parameter and a reward function generation cue word template into a preset large language model, and generating a first reward function; and inputting the first reward function into a reinforcement learning environment for training to obtain second reward functions, and taking the reward function corresponding to the maximum value of the corresponding task success rate mean values in the second reward functions as a final reward function, the second reward function is a reward function obtained after the reward function which does not meet the second preset condition in the first reward function is deleted.

Description

Technical Field

[0001] The present disclosure generally relates to the field of artificial intelligence algorithm technology, and in particular to the field of natural language processing technology. More specifically, it relates to a method and device for generating a reinforcement learning reward function based on a large language model. Background Art

[0002] In related fields, reinforcement learning algorithms are an important part of machine learning algorithms, which are generally used to provide action strategies for intelligent agents. Reinforcement learning systems are usually composed of reinforcement learning environments and reinforcement learning algorithms. In reinforcement learning systems, the reward function is the core of the reinforcement learning algorithm. The reward function directly reflects the purpose of the reinforcement learning task, thereby guiding the intelligent agent to explore and try and fail in the reinforcement learning environment. Specifically, the size of the reward value is used to encourage or punish the behavior of the intelligent agent, so that the intelligent agent can learn an action strategy that meets the task requirements through interaction with the reinforcement learning environment.

[0003] At present, the generation of reward function code for reinforcement learning algorithms mainly relies on manual design, a process also known as reward engineering. Specifically, algorithm design personnel design corresponding reward function codes according to the different requirements of reinforcement learning tasks, and repeatedly improve the code through manual debugging until a usable reward function code is obtained. However, this manually designed reward function code is not only complex, cumbersome and time-consuming in the design process, but also the manually designed reward function code cannot well reflect the purpose of the task in most cases.

[0004] Therefore, in order to solve the above problems, an improved reinforcement learning algorithm is needed. Summary of the invention

[0005] The embodiments of the present disclosure provide a method and device for generating a reinforcement learning reward function based on a large language model, thereby reducing the difficulty of designing a reinforcement learning algorithm and improving the training efficiency of the reinforcement learning algorithm.

[0006] In one general aspect, a method for generating a reinforcement learning reward function based on a large language model is provided, the reward function generating method comprising: obtaining original task description data, reinforcement learning environment code data and prompt word calling template data, wherein the prompt word calling template is used to call prompt words to be input into a preset large language model for generating a reward function, and the prompt word calling template data comprises an initialization prompt word template, a code summary prompt word template, and a reward function generation prompt word template; inputting the original task description data, the reinforcement learning environment code data, the initialization prompt word template and the code summary prompt word template into the preset large language model, obtaining a first Environmental variable output result; in response to the first environmental variable output result satisfying the first preset condition, the temperature parameter corresponding to the preset large language model and the reward function generation prompt word template are input into the preset large language model to generate a first reward function; the first reward function is input into the reinforcement learning environment corresponding to the reinforcement learning environment code data for training to obtain a second reward function, and the reward function corresponding to the maximum value of the corresponding task success rate mean in the second reward function is used as the final reward function, wherein the second reward function is the reward function obtained after deleting the reward function in the first reward function that does not satisfy the second preset condition.

[0007] Optionally, in response to the first environmental variable output result satisfying the first preset condition, the temperature parameter corresponding to the preset large language model and the reward function generation prompt word template are input into the preset large language model, and the step of generating the first reward function may include: in response to the first environmental variable output result satisfying the first preset condition, in the first iteration cycle, n temperature parameters corresponding to the preset large language model and the reward function generation prompt word template are input into the preset large language model to generate n groups of initial reward functions; in the second to jth iteration cycles, max In each of the jth iteration cycles, the following processing is performed: n temperature parameters, the reward function generation prompt word template, and the n groups of reward functions and corresponding improvement suggestion data generated in the previous iteration cycle are input into the preset large language model to generate n groups of reward functions corresponding to the current iteration cycle, and the reward functions corresponding to the jth ... max The reward function corresponding to the iteration cycle is used as the first reward function, where n is an integer greater than or equal to 1, j max is an integer greater than or equal to 2, and the n temperature parameters have a one-to-one correspondence with the n groups of reward functions generated in the current iteration cycle.

[0008] Optionally, the step of inputting the first reward function into a reinforcement learning environment corresponding to the reinforcement learning environment code data for training to obtain a second reward function may include: in the second to the jth max In each of the iteration cycles, the following processing is performed: in the training process corresponding to the current iteration cycle, the task success rate corresponding to the first reward function and the reward value for each reward item are counted once after a predefined number of training times to obtain a task success rate sequence and a reward value sequence; based on the task success rate sequence corresponding to each first reward function, the mean success rate of each task corresponding to each first reward function is calculated, and the reward function corresponding to the maximum value of the mean success rate of each task is used as the best reward function of the current iteration cycle; all the best reward functions of all iteration cycles are used as the second reward function.

[0009] Optionally, the prompt word calling template data may further include reverse generation of the prompt word template, and the reward function generation method may further include: in the first to jth max In each of the iteration cycles, the following processing is further performed: the n groups of reward functions generated in the current iteration cycle and the reverse generated prompt word template are input into the preset large language model to generate n groups of restored task description data; the similarity between the n groups of restored task description data and the original task description data is determined, and the reward functions corresponding to the several groups of restored task description data with the lowest similarity values ​​are used as the third reward function.

[0010] Optionally, the reward function generation method may further include: in the first to jth max In each of the iteration cycles, the following processing is further performed: based on the reward value sequences corresponding to the n groups of reward functions generated in the current iteration cycle, the mean values ​​of the n groups of reward scores are obtained; in response to the reward function corresponding to the minimum value in the mean values ​​of the n groups of reward scores being the same as the third reward function, the following processing is performed: the reward function corresponding to the minimum value is deleted from the first reward function, the temperature parameter corresponding to the deleted reward function is deleted, and the value of n is updated to a value reduced by the number of reward functions corresponding to the deleted reward functions.

[0011] Optionally, the prompt word calling template data may further include a feedback prompt word template, and the reward function generating method may further include: in the first to jth max In each of the iteration cycles, the following processing is further performed: the feedback prompt word template and the n groups of reward functions and corresponding task success rate sequences generated in the current iteration cycle are input into the preset large language model to generate corresponding improvement suggestion data.

[0012] Optionally, the prompt word calling template data may also include a result sorting prompt word template, and the step of determining the similarity between the n groups of restored task description data and the original task description data may include: inputting the result sorting prompt words, the n groups of restored task description data and the original task description data into the preset large language model to generate n groups of similarity values.

[0013] In another general aspect, a reinforcement learning reward function generation device based on a large language model is provided, the reward function generation device comprising: a data acquisition module, configured to: acquire original task description data, reinforcement learning environment code data and prompt word calling template data, wherein the prompt word calling template is used to call prompt words to be input into a preset large language model for generating a reward function, and the prompt word calling template data includes an initialization prompt word template, a code summary prompt word template, and a reward function generation prompt word template; a variable verification module, configured to: input the original task description data, the reinforcement learning environment code data, the initialization prompt word template and the code summary prompt word template into the preset large language model; model, and obtain a first environment variable output result; the function generation module is configured to: in response to the first environment variable output result satisfying the first preset condition, input the temperature parameter corresponding to the preset large language model and the reward function generation prompt word template into the preset large language model to generate a first reward function; and input the first reward function into the reinforcement learning environment corresponding to the environment code data for training to obtain a second reward function, and use the reward function corresponding to the maximum value of the corresponding task success rate mean in the second reward function as the final reward function, wherein the second reward function is the reward function obtained after deleting the reward function in the first reward function that does not satisfy the second preset condition.

[0014] Optionally, in response to the output result of the first environmental variable satisfying the first preset condition, the function generation module inputs the temperature parameter corresponding to the preset large language model and the reward function generation prompt word template into the preset large language model, and the operation of generating the first reward function may include: in response to the output result of the first environmental variable satisfying the first preset condition, in the first iteration cycle, inputting n temperature parameters corresponding to the preset large language model and the reward function generation prompt word template into the preset large language model to generate n groups of initial reward functions; in the second to jth iteration cycles, max In each of the jth iteration cycles, the following processing is performed: n temperature parameters, the reward function generation prompt word template, and the n groups of reward functions and corresponding improvement suggestion data generated in the previous iteration cycle are input into the preset large language model to generate n groups of reward functions corresponding to the current iteration cycle, and the reward functions corresponding to the jth ... maxThe reward function corresponding to the iteration cycle is used as the first reward function, where n is an integer greater than or equal to 1, j max is an integer greater than or equal to 2, and the n temperature parameters have a one-to-one correspondence with the n groups of reward functions generated in the current iteration cycle.

[0015] Optionally, the function generation module inputs the first reward function into a reinforcement learning environment corresponding to the reinforcement learning environment code data for training, and the operation of obtaining the second reward function may include: in the second to jth max In each of the iteration cycles, the following processing is performed: in the training process corresponding to the current iteration cycle, the task success rate corresponding to the first reward function and the reward value for each reward item are counted once after a predefined number of training times to obtain a task success rate sequence and a reward value sequence; based on the task success rate sequence corresponding to each first reward function, the mean success rate of each task corresponding to each first reward function is calculated, and the reward function corresponding to the maximum value of the mean success rate of each task is used as the best reward function of the current iteration cycle; all the best reward functions of all iteration cycles are used as the second reward function.

[0016] Optionally, the prompt word calling template data may further include reverse generation of prompt word templates, and the function generation module may further be configured to: max In each of the iteration cycles, the following processing is further performed: the n groups of reward functions generated in the current iteration cycle and the reverse generated prompt word template are input into the preset large language model to generate n groups of restored task description data; the similarity between the n groups of restored task description data and the original task description data is determined, and the reward functions corresponding to the several groups of restored task description data with the lowest similarity values ​​are used as the third reward function.

[0017] Optionally, the function generation module may be further configured to: max In each of the iteration cycles, the following processing is further performed: based on the reward value sequences corresponding to the n groups of reward functions generated in the current iteration cycle, the mean values ​​of the n groups of reward scores are obtained; in response to the reward function corresponding to the minimum value in the mean values ​​of the n groups of reward scores being the same as the third reward function, the following processing is performed: the reward function corresponding to the minimum value is deleted from the first reward function, the temperature parameter corresponding to the deleted reward function is deleted, and the value of n is updated to a value reduced by the number of reward functions corresponding to the deleted reward functions.

[0018] Optionally, the prompt word calling template data may further include a feedback prompt word template, and the function generating module may further be configured to: maxIn each of the iteration cycles, the following processing is further performed: the feedback prompt word template and the n groups of reward functions and corresponding task success rate sequences generated in the current iteration cycle are input into the preset large language model to generate corresponding improvement suggestion data.

[0019] Optionally, the prompt word calling template data may also include a result sorting prompt word template, and the operation of the function generation module determining the similarity between the n groups of restored task description data and the original task description data may include: inputting the result sorting prompt words, the n groups of restored task description data and the original task description data into the preset large language model to generate n groups of similarity values.

[0020] In another general aspect, a computer program product is provided, the computer program product comprising a computer program / instruction, which, when executed by a processor, implements the reinforcement learning reward function generation method based on a large language model as described above.

[0021] In another general aspect, a computer-readable storage medium is provided, which, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device / server, enables the electronic device / server to perform the reinforcement learning reward function generation method based on a large language model as described above.

[0022] In another general aspect, a computer device is provided, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, cause the at least one processor to perform the reinforcement learning reward function generation method based on a large language model as described above.

[0023] According to the method and device for generating a reinforcement learning reward function based on a large language model according to the embodiments of the present disclosure, the code generation of the reinforcement learning reward function based on a large language model is realized, which reduces the design difficulty of the reinforcement learning algorithm. In addition, by automating the generation of the reward function, the work efficiency of the reinforcement learning task development is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other objects and features of the embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings showing the embodiments, in which:

[0025] Figure 1 is a flowchart showing a method for generating a reinforcement learning reward function based on a large language model according to an embodiment of the present disclosure;

[0026] Figure 2is a flowchart showing an example of a method for generating a reinforcement learning reward function based on a large language model according to an embodiment of the present disclosure;

[0027] Figure 3 is a structural block diagram showing a device for generating a reinforcement learning reward function based on a large language model according to an embodiment of the present disclosure;

[0028] Figure 4 is a block diagram illustrating a computer device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of features known in the art may be omitted.

[0030] Reference will now be made in detail to the embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like parts throughout. The embodiments will be described below with reference to the drawings in order to explain the present disclosure.

[0031] In the related art, as mentioned above, the generation of reward functions currently still relies on manual design. In order to achieve the purpose of non-artificially designed reward functions, some technical solutions proposed recently can directly use the large language model (hereinafter referred to as LLM or large model) as the reward function. The above method directly uses the large model as a generator of reward values ​​during the training process of the reinforcement learning algorithm. For each action taken by the agent, the large model is used to evaluate whether the result of the action meets the goal of the reinforcement learning task, and outputs the reward value obtained by the action. During the entire process of reinforcement learning algorithm training, the large model must output a reward value after each interaction between the reinforcement learning algorithm and the environment, thereby replacing the reward function code.

[0032] The above existing solutions have the following disadvantages:

[0033] (1) When using reward engineering methods to generate reinforcement learning algorithms, a lot of manual design work is required, which is inefficient and relies heavily on the experience of designers.

[0034] (2) When the reinforcement learning algorithm is generated by directly using the large model as the reward function, each iteration in the training process of the reinforcement learning algorithm needs to wait for the large model to provide rewards, and the processing speed of the large model is very slow, which will cause the training process to take a very long time.

[0035] In order to solve the above problems in the related art, the present disclosure proposes a method and device for generating a reinforcement learning reward function based on a large language model. The present disclosure does not directly use the large language model as a reward function, but uses the large language model to replace the algorithm designer to generate the code of the reward function, and in the training process of reinforcement learning, the large language model does not participate in the evaluation of rewards. Through such a technical solution, not only can the workload of designing the reward function be reduced, but also the problem of slow training caused by the large language model being directly used as the reward value each time can be avoided.

[0036] Furthermore, the present disclosure proposes a method and device for generating a reinforcement learning reward function code based on a large language model, which automatically generates a reinforcement learning reward function code by using a large language model, and applies a "generate-evaluate-feedback" and forward-reverse generation training mechanism to the generated reward function code, thereby effectively reducing the workload and complexity of the generation of the reward function related to the reinforcement learning task. The present disclosure can effectively overcome the defects of traditional manual reward engineering design, automate the design of reward engineering, and has greater advantages in scenarios where a large number of reinforcement learning tasks need to be quickly implemented.

[0037] Refer to the following Figures 1 to 4 The present invention describes in detail a method and device for generating a reinforcement learning reward function based on a large language model according to an embodiment of the present invention.

[0038] Figure 1 is a flowchart illustrating a method 100 for generating a reinforcement learning reward function based on a large language model according to an embodiment of the present disclosure.

[0039] Reference Figure 1 According to an embodiment of the present disclosure, in step S101, original task description data, reinforcement learning environment code data and prompt word calling template data are obtained.

[0040] In the present disclosure, the cue word calling template is used to call the cue word to be input into a preset large language model for generating a reward function.

[0041] For example, the prompt word calling template data may include at least one of an initialization prompt word template, a code summary prompt word template, a reward function generation prompt word template, a reverse generation prompt word template, a feedback prompt word template, and a result sorting prompt word template, but the present disclosure is not limited thereto.

[0042] In the present disclosure, regarding task description data, it can be data that describes various natural language processing tasks, such as data in text form, image or video data converted into text form, etc., and is not specifically limited in the present disclosure.

[0043] According to an embodiment of the present disclosure, in step S102, the original task description data, the reinforcement learning environment code data, the initialization prompt word template and the code summary prompt word template are input into a preset large language model to obtain a first environment variable output result.

[0044] According to an embodiment of the present disclosure, various input parameters are input into the LLM to output the environment parameters, so as to confirm whether there are omissions in the environment parameters in the environment used for reinforcement learning, thereby ensuring the completeness of the environment parameters.

[0045] According to an embodiment of the present disclosure, in step S103, in response to the first environmental variable output result satisfying the first preset condition, the temperature parameter and reward function generation prompt word template corresponding to the preset large language model are input into the preset large language model to generate a first reward function.

[0046] As an example, step S103 may further include: in response to the first environment variable output result satisfying the first preset condition, executing the following S1031 and S1032:

[0047] In S1031, in the first iteration cycle, n temperature parameters corresponding to the preset large language model and a reward function generation prompt word template are input into the preset large language model to generate n groups of initial reward functions.

[0048] In S1032, in the 2nd to jth max In each of the jth iteration cycles, the following processing is performed: n temperature parameters, reward function generation prompt word templates, and n groups of reward functions and corresponding improvement suggestion data generated in the previous iteration cycle are input into the preset large language model to generate n groups of reward functions corresponding to the current iteration cycle, and the reward functions corresponding to the jth ... max The reward function corresponding to the iteration cycle is taken as the first reward function.

[0049] Here, n is an integer greater than or equal to 1, and j max is an integer greater than or equal to 2, which represents the upper limit of the iteration cycle.

[0050] In addition, in the present disclosure, there is a one-to-one correspondence between the n temperature parameters and the n groups of reward functions generated in the current iteration cycle.

[0051] According to the embodiments of the present disclosure, by utilizing a large language model instead of algorithm designers to generate the code for the reward function and the large language model does not participate in evaluating the reward during the reinforcement learning training process, not only the automation of the reward function generation is achieved, but also the workload associated with the reward function design is reduced, and the training speed and efficiency are improved.

[0052] According to an embodiment of the present disclosure, after generating n groups of reward functions corresponding to the current iteration cycle in the current iteration cycle, the following steps S31 and S32 may be further performed (i.e., in the first to the jth max The following processing can be further performed in each iteration cycle:

[0053] In S31, n groups of reward functions and reversely generated prompt word templates generated in the current iteration cycle are input into a preset large language model to generate n groups of restoration task description data.

[0054] In S32, the similarity between the n groups of restored task description data and the original task description data is determined, and the reward functions corresponding to the groups of restored task description data having the lowest similarity values ​​are used as the third reward function.

[0055] According to an embodiment of the present disclosure, by reversely restoring the relevant operations of the task description data, a reference basis is provided for the subsequent execution of the operation of removing the reward function that will be judged as invalid (i.e., the third reward function that minimizes the similarity between the restored task description and the original task description).

[0056] As an example, the step of determining the similarity between n groups of restored task description data and original task description data in S32 may include: inputting the result sorting prompt words, n groups of restored task description data and original task description data into a preset large language model to generate n groups of similarity values.

[0057] According to an embodiment of the present disclosure, parameters related to the restoration task description are input into the LLM, thereby achieving the purpose of generating similarity using the LLM. However, the present disclosure is not limited thereto, and other algorithms or models may also be used to calculate the above similarity value.

[0058] According to an embodiment of the present disclosure, in step S104, the first reward function is input into a reinforcement learning environment corresponding to the reinforcement learning environment code data for training to obtain a second reward function, and the reward function corresponding to the maximum value of the corresponding task success rate mean in the second reward function is used as the final reward function.

[0059] Here, the second reward function is a reward function obtained by deleting the reward function that does not meet the second preset condition in the first reward function. For example, the second preset condition may include a condition corresponding to a situation where the result obtained after training each reward function in the first reward function does not meet the preset requirement.

[0060] As an example, the step of generating the second reward function in step S104 may further include: max In each of the iteration cycles, the following processes S1041 to S1043 are performed:

[0061] In S1041, during the training process corresponding to the current iteration cycle, the task success rate corresponding to the first reward function and the reward value for each reward item are counted once every predefined number of training times to obtain a task success rate sequence and a reward value sequence.

[0062] In S1042, based on the task success rate sequence corresponding to each first reward function, the mean success rate of each task corresponding to each first reward function is calculated, and the reward function corresponding to the maximum value of each task success rate mean is used as the optimal reward function of the current iteration cycle.

[0063] At S1043, all the best reward functions of all iteration cycles are used as the second reward function.

[0064] According to an embodiment of the present disclosure, by adopting the “generate-evaluate-feedback” and forward-reverse generation mechanisms to utilize LLM to obtain the second reward function, the workload and complexity of designing a reward function for a reinforcement learning task are greatly reduced.

[0065] According to an embodiment of the present disclosure, after the n groups of reward functions corresponding to the current iteration cycle are sent to the reinforcement learning environment for training to obtain the corresponding task success rate sequence and reward value sequence in the current iteration cycle, the following steps S41 and S42 may be performed (i.e., in the first to the jth max The following processing can be further performed in each iteration cycle:

[0066] In S41, based on the reward value sequences corresponding to the n groups of reward functions generated in the current iteration cycle, the mean values ​​of the n groups of reward scores are obtained.

[0067] At S42, in response to the reward function corresponding to the minimum value among the mean values ​​of the reward scores of the n groups being the same as the third reward function, the following processing S421 to S423 is performed: at S421, the reward function corresponding to the minimum value is deleted from the first reward function; at S422, the temperature parameter corresponding to the deleted reward function is deleted; at S423, the value of n is updated to a value reduced by the amount corresponding to the deleted reward function.

[0068] According to an embodiment of the present disclosure, by removing the reward function that meets the conditions in the above S42 as an invalid reward function, the workload of calculation related to the reward function in the next iteration cycle is effectively reduced, thereby effectively improving the algorithm execution efficiency.

[0069] In addition, according to the embodiment of the present disclosure, after the n groups of reward functions corresponding to the current iteration cycle are sent to the reinforcement learning environment for training to obtain the corresponding task success rate sequence and reward value sequence in the current iteration cycle, the following steps can also be performed (i.e., in the first to the jth max The following processing can be further performed in each iteration cycle of the iteration cycle): the feedback prompt word template and the n groups of reward functions and corresponding task success rate sequences generated in the current iteration cycle are input into the preset large language model to generate corresponding improvement suggestion data.

[0070] According to an embodiment of the present disclosure, by providing improvement suggestions based on the task success rate corresponding to the reward function and the reward value of each reward item, it can be used to feed back to the forward generation-related operations of the next iteration cycle to improve the reward function generated next time, thereby facilitating that the finally generated reward function meets the predetermined expected requirements.

[0071] According to an embodiment of the present disclosure, by adopting the above-mentioned reinforcement learning reward function generation method 100, driven by relevant prompt words, a large language model is used to iteratively generate, evaluate, feedback and improve the reinforcement learning reward function. In the iterative process, the reward function is automatically generated by forward generating function code and reversely restoring the task description, thereby effectively improving the work efficiency of reinforcement learning task development.

[0072] Below, refer to Figure 2 An example of a method for generating a reinforcement learning reward function based on a large language model according to the present disclosure is described. Figure 2 is a flowchart illustrating an example of a method for generating a reinforcement learning reward function based on a large language model according to an embodiment of the present disclosure.

[0073] In this example, the reward function is generated by using LLM, and then the reinforcement learning model including the generated reward function is trained to complete the learning of the skill strategy of "opening the cabinet". The ultimate goal is to generate an effective reward function for reinforcement learning.

[0074] Specifically, Figure 2As shown, an example of a method for generating a reinforcement learning reward function based on a large language model may include four parts: initialization operation, function generation operation, function evaluation operation, and feedback operation. Here, the three parts of function generation operation, function evaluation operation, and feedback operation may sequentially form a "generation-evaluation-feedback" iteration cycle.

[0075] As an example, the initialization operation may include steps S201 to S204 , the function generation operation may include steps S205 to S207 , the function evaluation operation may include steps S208 and S209 , and the feedback operation may include steps S210 and S211 .

[0076] Refer to the following Figure 2 Each of the above steps is explained in detail.

[0077] In step S201, the prompt word Prompt is initialized. init , the original task description M and the source code of the reinforcement learning environment C env Enter the predetermined LLM to put the LLM into working state.

[0078] Here, the prompt word is initialized. init Contains the specification of the LLM role, the source code of the reinforcement learning environment contains all the environment variables required in the reward function, and the original task description is in natural language form.

[0079] In the present disclosure, all example contents of the following prompt words are expressed in italics, and the contents in the symbols "[]" indicate the parts that can be automatically called according to the data changes in the processing that has been executed.

[0080] As an example of an initialization prompt, the initialization prompt word Prompt init Can be:

[0081] “Now, you are a machine learning algorithm engineer. Your job is to write the reward function code for the reinforcement learning algorithm. The reward function you write should be consistent with the [reinforcement learning environment] and help the agent learn the strategy of [reinforcement learning task]. The reward function you write should use the [environment variables] in the source code of the [reinforcement learning environment], and do not create non-existent environment variables.”

[0082] As an example of an original task description, the original task description M may be:

[0083] “The [reinforcement learning task] you’re trying to solve is: open the door of the cabinet.”

[0084] As an example of environment source code, the environment source code C env Part of it (including environment variables) can be:

[0085] “The source code for the [reinforcement learning environment] is as follows:

[0086]

[0087]

[0088] …(the rest of the code is omitted here)”.

[0089] In step S202, enter the code summary prompt word Prompt check , LLM is required to provide source code C for reinforcement learning environment env Perform code summaries and ask LLM to point out C env All environment variables contained in .

[0090] As an example of a code summary prompt, the code summary prompt word Prompt check Can be:

[0091] “Please summarize the source code for [reinforcement learning environment], describe in words what the code does, and list all the [environment variables] related to the environment in the code.”

[0092] In step S203, it is determined whether all link variables are listed.

[0093] Through the above operations, compared with other methods that do not provide an environment description to LLM for verification or only use natural language to describe the environment, by requiring LLM to summarize the source code of the reinforcement learning environment and confirm that LLM has not missed any environment variables, the method disclosed in the present invention can not only improve LLM's ability to understand the environment, but also be used to ensure that LLM has not missed any environment variable.

[0094] After step S203, step S204 for creating temperature parameters may be performed. In step S204, n random temperature parameters {t i ∣i∈n}.

[0095] Here, it should be noted that the larger the value of n, the higher the probability of generating n reward function codes that meet the requirements, and the longer the calculation time required. The following example assumes that the value of n is 8, for example, 8 different values ​​are randomly selected from [0,1] as the temperature parameter.

[0096] Here, it should also be noted that the order of the operations performed in step S204 is not necessarily after S203, but the operation of S204 can be performed before S203, for example, the operation of S204 can be performed before or after the execution of steps S101 and S102.

[0097] In the following, it is assumed that the current iteration period is: j(1≤j≤j max ), the above function generation operations can be executed in the order of forward generation, reverse generation, and result sorting.

[0098] In step S205, the forward generation step is performed. Specifically, this step includes n operations: each operation sequentially generates i ||i∈n} take out a t i , set the temperature parameter of LLM to t i , generate the prompt word Prompt from the reward function code gen and the reward function code generated at the j-1th iteration cycle and suggestions for improvement Input LLM, and LLM generates n sets of reward function codes {C i |i∈n}. When j is 1, there is no need to enter the reward function code and improvement suggestions.

[0099] For example, if n is assumed to be 8, 8 sets of reward function codes corresponding to 8 temperature parameters may be generated in each iteration cycle.

[0100] As an example of a reward function code generating a prompt word, if the current iteration cycle is 1, the reward function code generates a prompt word Prompt gen Can be:

[0101] "Please generate a reinforcement learning [reward function code] that meets the [reinforcement learning task] in Python based on the source code of the [reinforcement learning environment] above. In the [reward function code], the [reward items] should be reasonably divided, and there may be multiple reward items."

[0102] As another example of a reward function code generating a prompt word, if the current iteration cycle is not 1, the reward function code generates a prompt word Prompt gen Can be:

[0103] “Please modify the [reward function code] according to the [improvement suggestions] and the source code of the [reinforcement learning environment] mentioned above, and generate a reinforcement learning [reward function code] in Python that meets the [reinforcement learning task]. Make necessary adjustments to the [reward item] in the [reward function code].”

[0104] In step S206, the reverse generation step is performed. Specifically, the step includes: generating n groups of reward function codes {C i |i∈n} is sent back to LLM, and the prompt word Prompt is generated in reverse rvs , LLM generates the corresponding n groups of restoration task descriptions

[0105] As an example of reverse generation of prompt words, reverse generation of prompt words Prompt rvs Can be:

[0106] “For the [reward function code] above, summarize the [reinforcement learning task described by the code] in words. The [environment variables] used in the [reward function code] have the same meaning as in the source code of the [reinforcement learning environment]. Do not output content that is not related to this task.”

[0107] In step S207, the result sorting step is performed. Specifically, this step includes: sorting the original task description M with all the restored task descriptions Enter LLM and give the sort prompt word Prompt rank , compared by LLM Each of The similarity with M, and determine the one with the lowest similarity The corresponding C i , marked as C miss .

[0108] As an example of a sort prompt, the sort prompt word Prompt rank Can be:

[0109] “Compare the eight [reinforcement learning tasks described in code] above, find the one that is closest to [reinforcement learning task], and output its serial number.”

[0110] By including the above-mentioned forward generation and reverse generation processes, compared with only setting a forward process of generating reward function code from the original task description, it is beneficial to subsequently improve the code running efficiency by removing invalid reward functions.

[0111] In step S208, the generated result (i.e., n groups of reward function codes) is trained and evaluated. Specifically, this step includes: sequentially training the candidate reward function codes {C i Each C in |i∈n} i The task is sent to the reinforcement learning environment to complete the training. During the training process, the task success rate is recorded every 300 epochs. and the reward value for each reward item In this way each candidate reward function code C i Record the corresponding task success rate and reward value for K times (K is 10 in the following example).

[0112] Thus, the sequence of the corresponding task success rate of each candidate reward function code is obtained and reward item value sequence

[0113] In addition, the candidate reward function code C can be further obtained i The corresponding task success rate is And the task success rate average The maximum value in Corresponding candidate reward function code C i As the optimal reward function code for the jth iteration

[0114] As an example of a task success rate sequence and a reward value sequence of a reward item, the task success rate sequence may be: ['0.00','0.78','1.45','3.18','3.95','4.25','5.66','6.80','7.55','8.57'], and the reward value sequence corresponding to one reward item may be: ['0.05','0.28','0.31','0.39','0.32','0.37','0.37','0.37','0.37','0.36'].

[0115] In step S209, the step of removing invalid codes is performed, specifically including: for the candidate reward function code {C i The candidate reward function code C with the lowest mean reward score among |i∈n} min (can be one or more groups), if C min With C miss If they are the same, then the candidate reward function code C min Confirmed as invalid code, and the candidate reward function code C min From {C i ∣i∈n} is removed and its corresponding t i From {t i |i∈n}, and the number of n is reduced accordingly to the number of its corresponding groups.

[0116] By setting the above-mentioned elimination mechanism for the candidate reward function code which uses the lowest mean reward score and the lowest similarity between the restored task description and the original task description as the basis for elimination, the amount of calculation in each iteration cycle can be effectively reduced compared with the method which includes the elimination mechanism and keeps the number n unchanged in each iteration cycle.

[0117] In step S210, determine whether the maximum iteration period j has been reached. msx If the maximum iteration period j is reached max , then all iterations are completed, and from all candidate reward function codes Select a set of reward functions with the largest mean reward score as the final reward function code C best .

[0118] In step S211, if the maximum iteration period j has not yet been reached max , then the steps of feedback improvement suggestions are executed, including: i ∣i∈n} and the corresponding Enter LLM and give the feedback prompt word Prompt fbk , get the suggested text for improvement described in natural language

[0119] As an example of a feedback prompt word, the feedback prompt word Prompt fbk Can be:

[0120] “You trained a policy for [reward function code] for [reinforcement learning task] and recorded the [sequence of task success rates] obtained during training, as well as the value of each reward item in [reward function code].

[0121] [The sequence of task success rate] is: ['0.00','0.78','1.45','3.18','3.95','4.25','5.66','6.80','7.55','8.57']

[0122] [Reward item value sequence] is: ['0.05','0.28','0.31','0.39','0.32','0.37','0.37','0.37','0.37',0.36']

[0123] Please analyze this [sequence of task success rates] and [sequence of reward item values] and provide [improvement suggestions] in the form of language description to better solve the [reinforcement learning task]. When analyzing the [sequence of task success rates] and [sequence of reward item values], pay attention to the following aspects:

[0124] (1) If the task success rates are all less than 0.1, then it is recommended to re-understand the [reinforcement learning task] and redesign the entire reward function;

[0125] (2) If the value of a certain [reward item] continues to fluctuate, then it should be recommended to delete the reward item, or to rewrite a new reward item;

[0126] (3) If the value of a certain [reward item] is more than 10 times greater than the value of other [reward items], it should be recommended to set or adjust an appropriate coefficient for it so that its value range is consistent with that of other [reward items].

[0127] Please enter [improvement suggestions] below."

[0128] Here, in the example of the feedback prompt word, the feedback prompt word Prompt fbk The values ​​of the "sequence of task success rates" and the "sequence of reward item values" come from the relevant examples of step S208, that is, "[sequence of task success rates]" and "[sequence of reward item values]" can directly call the corresponding values ​​of the two obtained in step S208 without additional input.

[0129] After step S311, return to step S205 to execute the processing of the next iteration cycle.

[0130] Here, it should be noted that the above Figure 2 The description of each step is merely exemplary, and the steps in the method according to the present disclosure are not limited thereto.

[0131] Figure 3 is a structural block diagram showing a device 300 for generating a reinforcement learning reward function based on a large language model according to an embodiment of the present disclosure.

[0132] Reference Figure 3 According to an embodiment of the present disclosure, a reinforcement learning reward function generation device 300 based on a large language model may include a data acquisition module 310, a variable verification module 320 and a function generation module 330.

[0133] According to an embodiment of the present disclosure, the data acquisition module 310 is configured to: acquire original task description data, reinforcement learning environment code data, and prompt word calling template data.

[0134] In the present disclosure, the cue word calling template is used to call the cue word to be input into a preset large language model for generating a reward function.

[0135] For example, the prompt word calling template data may include an initialization prompt word template, a code summary prompt word template, a reward function generation prompt word template, a reverse generation prompt word template, a feedback prompt word template, and a result sorting prompt word template, but the present disclosure is not limited thereto.

[0136] According to an embodiment of the present disclosure, the variable verification module 320 is configured to: input the original task description data, the reinforcement learning environment code data, the initialization prompt word template and the code summary prompt word template into the preset large language model to obtain the first environment variable output result.

[0137] According to an embodiment of the present disclosure, the function generation module 330 is configured to: in response to the first environment variable output result satisfying the first preset condition, input the temperature parameter and reward function generation prompt word template corresponding to the preset large language model into the preset large language model to generate a first reward function; and input the first reward function into the reinforcement learning environment corresponding to the environment code data for training to obtain a second reward function, and use the reward function corresponding to the maximum value of the corresponding task success rate mean in the second reward function as the final reward function.

[0138] Here, the second reward function is a reward function obtained by deleting the reward function that does not meet the second preset condition in the first reward function. For example, the second preset condition may include a condition corresponding to a situation where the result obtained after training each reward function in the first reward function does not meet the preset requirement.

[0139] As an example, the function generation module 330 may also be configured to: max In each of the iteration cycles, the following processing is further performed: the n groups of reward functions and reverse-generated prompt word templates generated in the current iteration cycle are input into the preset large language model to generate n groups of restored task description data; the similarity between the n groups of restored task description data and the original task description data is determined, and the reward functions corresponding to the several groups of restored task description data with the lowest similarity values ​​are used as the third reward function.

[0140] Further, as an example, the operation of the function generation module 330 determining the similarity between n sets of restored task description data and original task description data may include: inputting the result sorting prompt words, n sets of restored task description data and original task description data into a preset large language model to generate n sets of similarity values.

[0141] According to an embodiment of the present disclosure, in response to the first environment variable output result satisfying the first preset condition, the function generation module 330 inputs the temperature parameter corresponding to the preset large language model and the reward function generation prompt word template into the preset large language model, and the operation of generating the first reward function may include operations 3311) and 3312):

[0142] In operation 3311), in response to the first environment variable output result satisfying the first preset condition, in the first iteration cycle, n temperature parameters corresponding to the preset large language model and the reward function generation prompt word template are input into the preset large language model to generate n groups of initial reward functions.

[0143] In operation 3312), in the 2nd to jth maxIn each of the jth iteration cycles, the following processing is performed: n temperature parameters, reward function generation prompt word templates, and n groups of reward functions and corresponding improvement suggestion data generated in the previous iteration cycle are input into the preset large language model to generate n groups of reward functions corresponding to the current iteration cycle, and the reward functions corresponding to the jth ... max The reward function corresponding to the iteration cycle is taken as the first reward function.

[0144] Here, n is an integer greater than or equal to 1, j max is an integer greater than or equal to 2.

[0145] In the present disclosure, there is a one-to-one correspondence between the n temperature parameters and the n groups of reward functions generated in the current iteration cycle.

[0146] According to an embodiment of the present disclosure, the function generation module 330 inputs the first reward function into the reinforcement learning environment corresponding to the reinforcement learning environment code data for training, and the operation of obtaining the second reward function may include: in the second to the jth max In each of the iteration cycles, the following processing 3321) to 3323) is performed:

[0147] At 3321), during the training process corresponding to the current iteration cycle, the task success rate corresponding to the first reward function and the reward value for each reward item are counted once after each predefined number of trainings to obtain a task success rate sequence and a reward value sequence.

[0148] At 3322), based on the task success rate sequence corresponding to each first reward function, the average success rate of each task corresponding to each first reward function is calculated, and the reward function corresponding to the maximum value of each task success rate average is used as the optimal reward function of the current iteration cycle.

[0149] At 3323), all the best reward functions of all iteration cycles are used as the second reward function.

[0150] As an example, the function generation module 330 may also be configured to: max In each of the iteration cycles, the following processing is further performed: based on the reward value sequences corresponding to the n groups of reward functions generated in the current iteration cycle, the mean values ​​of the n groups of reward scores are obtained; in response to the reward function corresponding to the minimum value in the n groups of reward score means being the same as the third reward function, the following processing is performed: the reward function corresponding to the minimum value is deleted from the first reward function, the temperature parameter corresponding to the deleted reward function is deleted, and the value of n is updated to a value reduced by the number of reward functions corresponding to the deleted reward functions.

[0151] In addition, as an example, the function generation module 330 can also be configured as follows:max In each of the iteration cycles, the following processing is further performed: the feedback prompt word template and the n groups of reward functions and corresponding task success rate sequences generated in the current iteration cycle are input into the preset large language model to generate corresponding improvement suggestion data.

[0152] It should be noted that the operations performed on the above-mentioned structural frames can be referred to in Figure 1 The related contents described are similar and will not be repeated here.

[0153] Figure 4 is a block diagram illustrating a computer device 400 according to an embodiment of the present disclosure.

[0154] Reference Figure 4 , the computer device 400 according to an embodiment of the present disclosure may include a processor 410 and a memory 420. The processor 410 may include (but not limited to) a central processing unit (CPU), a digital signal processor (DSP), a microcomputer, a field programmable gate array (FPGA), a system on a chip (SoC), a microprocessor, an application-specific integrated circuit (ASIC), etc. The memory 420 may store computer executable instructions to be executed by the processor 410. The memory 420 includes a high-speed random access memory and / or a non-volatile computer-readable storage medium. When the processor 410 executes the computer executable instructions stored in the memory 420, the reinforcement learning reward function generation method based on the large language model as described above may be implemented.

[0155] The evaluation method according to the embodiment of the present disclosure can be written as a computer program / instruction to form a computer program product and stored on a computer-readable storage medium. When the computer program / instruction is executed by a processor, the reinforcement learning reward function generation method based on a large language model as described above can be implemented. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device / server, the electronic device / server is enabled to execute the reinforcement learning reward function generation method based on a large language model as described above. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer program and any associated data, data files and data structures in a non-transitory manner and provide the computer program and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. In one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0156] According to the method and device for generating a reinforcement learning reward function based on a large language model according to the embodiment of the present disclosure, the code generation of the reinforcement learning reward function based on the large language model is realized, thereby reducing the difficulty of designing the reinforcement learning algorithm. For example, in the present disclosure, the design of the reinforcement learning reward function is completed by using the knowledge of the large language model itself to replace manual design, which can effectively reduce the workload of algorithm designers, reduce the dependence on the experience of algorithm designers, and effectively improve the efficiency of algorithm design.

[0157] On the other hand, according to the method and device for generating a reinforcement learning reward function based on a large language model according to an embodiment of the present disclosure, the generation of the reward function is automated, which effectively improves the work efficiency of the reinforcement learning task development. For example, in the present disclosure, compared with the conventional method of directly using a large language model as a generator of reinforcement learning reward values, the large language model is no longer involved in the training process of the reinforcement learning task after the reward function code is generated, and the running speed of the large language model will not affect the training process of reinforcement learning. Therefore, compared with the conventional method of directly using a large language model as a generator of reinforcement learning reward values, the training speed of the reward function generation method according to the embodiment of the present disclosure is faster and more efficient.

[0158] On the other hand, the reinforcement learning reward function generation method and device based on a large language model according to the embodiments of the present disclosure can effectively overcome the defects of traditional manual reward engineering design, automate the design of reward engineering, and have greater advantages in scenarios where a large number of reinforcement learning tasks need to be designed quickly.

[0159] Although some embodiments of the present disclosure have been disclosed and described, it will be appreciated by those skilled in the art that modifications and variations may be made to the embodiments without departing from the concept and spirit of the present disclosure, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for generating a reinforcement learning reward function based on a large language model, characterized in that: The reinforcement learning reward function generation method comprises: Acquire original task description data, reinforcement learning environment code data, and prompt word calling template data, wherein the prompt word calling template is used to call prompt words to be input into a preset large language model for generating a reward function, and the prompt word calling template data includes an initialization prompt word template, a code summary prompt word template, and a reward function generation prompt word template; Inputting the original task description data, the reinforcement learning environment code data, the initialization prompt word template and the code summary prompt word template into the preset large language model to obtain a first environment variable output result; In response to the first environmental variable output result satisfying a first preset condition, inputting a temperature parameter corresponding to the preset large language model and the reward function generation prompt word template into the preset large language model to generate a first reward function; The first reward function is input into a reinforcement learning environment corresponding to the reinforcement learning environment code data for training to obtain a second reward function, and a reward function corresponding to the maximum value of the corresponding task success rate mean in the second reward function is used as a final reward function, The second reward function is a reward function obtained by deleting the reward function that does not meet the second preset condition in the first reward function.

2. The method for generating a reinforcement learning reward function according to claim 1, characterized in that: In response to the first environmental variable output result satisfying a first preset condition, the temperature parameter corresponding to the preset large language model and the reward function generation prompt word template are input into the preset large language model to generate a first reward function, and the step includes: In response to the first environmental variable output result satisfying the first preset condition, in the first iteration cycle, n temperature parameters corresponding to the preset large language model and the reward function generation prompt word template are input into the preset large language model to generate n groups of initial reward functions; In the 2nd to jth max In each of the iteration cycles, the following processing is performed: Input n temperature parameters, the reward function generation prompt word template, and n groups of reward functions generated in the previous iteration cycle and corresponding improvement suggestion data into the preset large language model to generate n groups of reward functions corresponding to the current iteration cycle, and max The reward function corresponding to the iteration cycle is used as the first reward function. Where n is an integer greater than or equal to 1, j max is an integer greater than or equal to 2, and Among them, the n temperature parameters have a one-to-one correspondence with the n groups of reward functions generated in the current iteration cycle.

3. The method for generating a reinforcement learning reward function according to claim 2, characterized in that: The step of inputting the first reward function into a reinforcement learning environment corresponding to the reinforcement learning environment code data for training to obtain a second reward function comprises: In the 2nd to jth max In each of the iteration cycles, the following processing is performed: In the training process corresponding to the current iteration cycle, the task success rate corresponding to the first reward function and the reward value for each reward item are counted once every predefined number of training times to obtain a task success rate sequence and a reward value sequence; Based on the task success rate sequence corresponding to each first reward function, the mean of each task success rate corresponding to each first reward function is calculated, and the reward function corresponding to the maximum value of the mean of each task success rate is used as the optimal reward function of the current iteration cycle; All the best reward functions of all iteration cycles are used as the second reward function.

4. The method for generating a reinforcement learning reward function according to claim 3, characterized in that: The prompt word calling template data also includes reversely generating a prompt word template, and the reward function generating method also includes: In the 1st to jth max In each of the iteration cycles, the following processing is further performed: Inputting n groups of reward functions generated in the current iteration cycle and the reverse generation prompt word template into the preset large language model to generate n groups of restoration task description data; The similarity between the n groups of restored task description data and the original task description data is determined, and the reward functions corresponding to the groups of restored task description data having the lowest similarity values ​​are used as the third reward function.

5. The method for generating a reinforcement learning reward function according to claim 4, characterized in that: The reward function generation method further includes: In the 1st to jth max In each of the iteration cycles, the following processing is further performed: Based on the reward value sequences corresponding to the n groups of reward functions generated in the current iteration cycle, the mean values ​​of the n groups of reward scores are obtained; In response to the reward function corresponding to the minimum value among the n groups of reward score means being the same as the third reward function, performing the following processing: Deleting the reward function corresponding to the minimum value from the first reward function, The temperature parameter corresponding to the deleted reward function is deleted, and Update the value of n to be reduced by the amount of reward functions that were deleted.

6. The method for generating a reinforcement learning reward function according to claim 3, characterized in that: The prompt word calling template data also includes a feedback prompt word template, and the reward function generating method also includes: In the 1st to jth max In each of the iteration cycles, the following processing is further performed: The feedback prompt word template and the n groups of reward functions and corresponding task success rate sequences generated in the current iteration cycle are input into the preset large language model to generate corresponding improvement suggestion data.

7. The method for generating a reinforcement learning reward function according to claim 4, wherein: The prompt word calling template data also includes a result sorting prompt word template, and the step of determining the similarity between the n groups of restored task description data and the original task description data includes: The result sorting prompt words, the n groups of restored task description data and the original task description data are input into the preset large language model to generate n groups of similarity values.

8. A device for generating a reinforcement learning reward function based on a large language model, characterized in that: The reinforcement learning reward function generating device comprises: A data acquisition module is configured to: acquire original task description data, reinforcement learning environment code data, and prompt word calling template data, wherein the prompt word calling template is used to call prompt words to be input into a preset large language model for generating a reward function, and the prompt word calling template data includes an initialization prompt word template, a code summary prompt word template, and a reward function generation prompt word template; The variable verification module is configured to: input the original task description data, the reinforcement learning environment code data, the initialization prompt word template and the code summary prompt word template into the preset large language model to obtain a first environment variable output result; The function generation module is configured to: in response to the first environment variable output result satisfying the first preset condition, input the temperature parameter corresponding to the preset large language model and the reward function generation prompt word template into the preset large language model to generate a first reward function; and input the first reward function into the reinforcement learning environment corresponding to the environment code data for training to obtain a second reward function, and use the reward function corresponding to the maximum value of the corresponding task success rate mean in the second reward function as the final reward function, The second reward function is a reward function obtained by deleting the reward function that does not meet the second preset condition in the first reward function.

9. A computer program product, characterized in that The computer program product comprises a computer program / instruction, and when the computer program / instruction is executed by a processor, the method for generating a reinforcement learning reward function based on a large language model according to any one of claims 1 to 7 is implemented.

10. A computer device, characterized in that: The computer device includes: at least one processor; and at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, cause the at least one processor to execute the method for generating a reinforcement learning reward function based on a large language model as described in any one of claims 1 to 7.

Images