Large language model enhanced efficient reinforcement learning credit distribution method and device

Abstract

The invention relates to the technical field of reinforcement learning, in particular to a large language model enhanced efficient reinforcement learning credit distribution method and device, and the method comprises the steps: obtaining potential rewards in target environment information through the priori knowledge of a target large language model; based on a return decomposition mode of potential reward enhancement, converting the potential reward into a proxy reward; and adding the proxy reward into reinforcement learning training to obtain an efficient reinforcement learning credit distribution result enhanced by the large language model. Therefore, the problems that in the prior art, difference exists between language knowledge of a large language model and symbolic representation needed by a specific task, meanwhile, randomness and illusion phenomena exist in large language model reasoning, consequently, the credibility of the model is low, and the overall performance of a reinforcement learning system is reduced are solved.

Description

Technical Field

[0001] The present application relates to the technical field of reinforcement learning, and in particular to an efficient reinforcement learning credit allocation method and device enhanced by a large language model. Background Art

[0002] Practical applications of reinforcement learning, such as autonomous driving and healthcare, often encounter situations where only delayed and sparse feedback is received. An extreme case is that only trajectory rewards are received instead of single-step rewards. Credit assignment aims to evaluate the contribution of single-step decisions in trajectories, but previous algorithms are often helpless in the aforementioned extreme cases. In recent years, large language models have shown great potential and have demonstrated impressive capabilities in a variety of downstream tasks. Recently, more and more research has been devoted to leveraging the rich intrinsic knowledge and capabilities of large language models to solve decision-making problems.

[0003] In order to achieve credit allocation in the extreme case of only receiving trajectory rewards, previous work proposed a reward decomposition method, which uses state-action pairs to estimate proxy rewards (reward) to redistribute trajectory rewards (episodic rewards), which has become a widely recognized and promising solution in the field. Subsequent work usually focuses on modifying the model architecture or introducing human-designed regression principles, but often ignores the training difficulties caused by the redundant information in the original state-action pair ([s, a]). Some methods try to filter out features that are not related to rewards through causal analysis to solve this redundancy problem, but still face the challenge of lack of semantic interpretation.

[0004] A common phenomenon in human problem solving is that the evaluation of contribution often covers multiple qualitative and quantitative factors. For example, the performance of a football player is not only evaluated by the number of goals scored, but also includes factors such as injury prevention and teamwork. Similarly, the rewards designed in reinforcement learning are usually a combination of multiple factors. However, previous methods mainly focus on the numerical value of the final reward, ignoring the multidimensional nature of performance evaluation, resulting in poor semantic interpretation and ambiguous reward distribution.

[0005] Therefore, the capabilities demonstrated by pre-trained large language models indicate that integrating their prior knowledge to improve credit assignment is a promising solution. However, there is a difference between the language knowledge of large language models and the symbolic representation required for specific tasks. At the same time, there are randomness and hallucination phenomena in the reasoning of large language models, which leads to low trust in the model and reduces the overall performance of the reinforcement learning system, which needs to be solved urgently. Summary of the invention

[0006] The present application provides a large language model-enhanced efficient reinforcement learning credit allocation method and device to solve the problem in the related art that there is a difference between the language knowledge of the large language model and the symbolic representation required for a specific task, and at the same time, there is randomness and hallucination in the reasoning of the large language model, which leads to low trust in the model and reduces the performance of the reinforcement learning system.

[0007] The first aspect of the present application provides an efficient reinforcement learning credit allocation method enhanced by a large language model, comprising the following steps: using the prior knowledge of the target large language model to obtain potential rewards in the target environment information; based on the reward decomposition method enhanced by the potential reward, converting the potential reward into a proxy reward; adding the proxy reward to the reinforcement learning training to obtain an efficient reinforcement learning credit allocation result enhanced by the large language model.

[0008] Optionally, in one embodiment of the present application, the converting the potential reward into the proxy reward based on the potential reward enhancement return decomposition method includes: generating a potential reward encoding function using the encoding capability of the target large language model; constructing a new neural network model based on the potential reward encoding function, the potential reward enhancement return decomposition method and a pre-set neural network decoder; and converting the potential reward into the proxy reward based on the new neural network model.

[0009] Optionally, in one embodiment of the present application, the target of the new neural network model is expressed as:

[0010]

[0011] Where ψ is the decoder network parameter, τ is the trajectory, R(τ) is the trajectory reward, D is the replay buffer, and φ is the latent reward encoding function.

[0012] Optionally, in one embodiment of the present application, the generating of a potential reward encoding function using the encoding capability of the target large language model includes: obtaining a task description and a state form of a target task; based on the encoding capability of the target large language model, respectively extracting target key information of the task description and the state form, and generating the potential reward encoding function based on the target key information.

[0013] The second aspect of the present application provides an efficient reinforcement learning credit allocation device enhanced by a large language model, including: an acquisition module, used to use the prior knowledge of the target large language model to obtain potential rewards in the target environment information; a conversion module, used to convert the potential rewards into proxy rewards based on the reward decomposition method enhanced by the potential rewards; a control module, used to add the proxy rewards to the reinforcement learning training to obtain the efficient reinforcement learning credit allocation results enhanced by the large language model.

[0014] Optionally, in one embodiment of the present application, the conversion module includes: a generation unit, used to generate a potential reward encoding function using the encoding capability of the target large language model; a construction unit, used to construct a new neural network model based on the potential reward encoding function, the potential reward enhanced return decomposition method and a pre-set neural network decoder; and a conversion unit, used to convert the potential reward into the proxy reward based on the new neural network model.

[0015] Optionally, in one embodiment of the present application, the target of the new neural network model is expressed as:

[0016]

[0017] Where ψ is the decoder network parameter, τ is the trajectory, R(τ) is the trajectory reward, D is the replay buffer, and φ is the latent reward encoding function.

[0018] Optionally, in one embodiment of the present application, the generation unit includes: an acquisition subunit, used to acquire a task description and a state form of a target task; a generation subunit, used to extract target key information of the task description and the state form respectively based on the encoding capability of the target large language model, and generate the potential reward encoding function according to the target key information.

[0019] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the efficient reinforcement learning credit allocation method enhanced by a large language model as described in the above embodiments.

[0020] The fourth aspect embodiment of the present application provides a computer-readable storage medium, which stores a computer program, which, when executed by a processor, implements the above-mentioned efficient reinforcement learning credit allocation method enhanced by a large language model.

[0021] The fifth aspect of the present application provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned efficient reinforcement learning credit allocation method enhanced by a large language model.

[0022] The embodiment of the present application can use the prior knowledge of the target large language model to obtain the potential rewards in the target environment information, and based on the reward decomposition method enhanced by the potential rewards, convert the potential rewards into proxy rewards, and then add the proxy rewards to the reinforcement learning training, so as to obtain the efficient reinforcement learning credit allocation results enhanced by the large language model, and effectively improve the overall performance of the reinforcement learning system. Thus, the difference between the language knowledge of the large language model and the symbolic representation required for a specific task in the related art is solved, and at the same time, the randomness and hallucination phenomena in the reasoning of the large language model lead to low trust in the model, which reduces the overall performance of the reinforcement learning system.

[0023] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0025] Figure 1 A flowchart of an efficient reinforcement learning credit allocation method enhanced by a large language model provided according to an embodiment of the present application;

[0026] Figure 2 A schematic diagram of the structure of a large language model enhanced efficient reinforcement learning credit allocation device provided according to an embodiment of the present application;

[0027] Figure 3 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0029] The following describes the efficient reinforcement learning credit allocation method and device enhanced by a large language model of the embodiment of the present application with reference to the accompanying drawings. In view of the difference between the language knowledge of the large language model in the related technology mentioned in the background technology center and the symbolic representation required for a specific task, and the randomness and hallucination phenomenon in the reasoning of the large language model, which leads to a low trust in the model and reduces the overall performance of the reinforcement learning system, the present application provides a large language model enhanced efficient reinforcement learning credit allocation method, in which the prior knowledge of the target large language model can be used to obtain the potential reward in the target environment information, and the potential reward is converted into a proxy reward based on the reward decomposition method enhanced by the potential reward. Then, the proxy reward is added to the reinforcement learning training, so as to obtain the efficient reinforcement learning credit allocation result enhanced by the large language model, which effectively improves the overall performance of the reinforcement learning system. As a result, the difference between the language knowledge of the large language model in the related technology and the symbolic representation required for a specific task is solved, and the randomness and hallucination phenomenon in the reasoning of the large language model leads to a low trust in the model and reduces the overall performance of the reinforcement learning system.

[0030] Specifically, Figure 1 A flowchart of an efficient reinforcement learning credit allocation method enhanced by a large language model provided in an embodiment of the present application.

[0031] like Figure 1 As shown, the efficient reinforcement learning credit assignment method enhanced by the large language model includes the following steps:

[0032] In step S101, the prior knowledge of the target large language model is used to obtain potential rewards in the target environment information.

[0033] It can be understood that the embodiments of the present application can use the prior knowledge of the target large language model to obtain the latent reward in the target environment information. For example, the embodiments of the present application can design a LaRe framework to obtain the semantically interpretable latent reward (Latent Reward) in the target environment information by combining the prior knowledge of the large language model. The latent reward (Latent Reward) can capture various aspects of task performance through different dimensions, while eliminating redundant information irrelevant to the reward, thereby effectively improving the feasibility of efficient reinforcement learning credit allocation enhanced by the large language model.

[0034] Among them, given that semantically generated code can bridge the linguistic knowledge of large language models and symbolic form targets in symbolic tasks due to its executable nature, LaRe proposes a general paradigm for integrating large language model prior knowledge into symbolic tasks.

[0035] It should be noted that the embodiments of the present application are fully compared with previous methods in multiple commonly used benchmark test environments, including single-agent and multi-agent scenarios. The experimental results show that the LaRe framework outperforms the existing state-of-the-art methods in credit allocation in the time dimension, performs well in contribution allocation among multiple agents, and surpasses strategies trained with real rewards in some tasks.

[0036] In addition, since in human activities, individual contributions are usually evaluated from multiple perspectives to achieve a comprehensive evaluation, current research on trajectory reward distribution often only focuses on the regression of the final reward value, ignoring that the reward is actually derived from the evaluation of multiple implicit factors (such as cost and efficiency). Inspired by the need for multi-dimensional evaluation of task performance, the embodiment of the present application proposes the concept of latent reward. Conceptually, the different dimensions of latent reward can capture various aspects of task performance. Formally, the reward becomes a mapping of latent reward. Therefore, the embodiment of the present application can construct a new trajectory reward probability model ([s, a] → latent reward → reward → episodic reward), which reveals the multi-dimensional characteristics of gradual contribution, thereby better serving the training of reinforcement learning.

[0037] Among them, a simple method is to obtain potential rewards through the information bottleneck method, but since each task requires a separate encoder training, this method is limited in language interpretability and has a high computational cost. In contrast, the pre-training of the large language model has captured richer representations in a more compact token form, promoting better cross-task generalization. Therefore, the embodiments of the present application use the prior knowledge of the large language model to more efficiently extract interpretable multi-dimensional task performance indicators, i.e., potential rewards, from redundant environmental information.

[0038] In step S102, the potential reward is converted into a proxy reward based on a reward decomposition method enhanced by the potential reward.

[0039] It can be understood that the embodiments of the present application can convert potential rewards into proxy rewards based on the reward decomposition method enhanced by potential rewards, so that the proxy rewards alleviate the problems of delay and sparse feedback, effectively improve the performance and robustness of the reinforcement learning system, and improve the transparency and controllability of the learning process.

[0040] Among them, in one embodiment of the present application, based on the potential reward enhancement return decomposition method, the potential reward is converted into a proxy reward, including: using the encoding ability of the target large language model to generate a potential reward encoding function; based on the potential reward encoding function, the potential reward enhancement return decomposition method and a pre-set neural network decoder, a new neural network model is constructed; based on the new neural network model, the potential reward is converted into a proxy reward.

[0041] In the actual execution process, the embodiment of the present application can set the environment prompting (EnvironmentPrompting) of the large language model, that is, standardized prompts that can be easily transferred to different environments. These prompts are composed of templated role instructions and specific task instructions, where the role instructions are consistent between tasks, guiding the large language model to think in a predefined way: understand the task and state → identify factors related to the reward → generate a potential reward encoding function. Only the task description and state form required for a specific task need to be provided, which significantly reduces the burden of tedious prompt engineering between different tasks.

[0042] Next, the embodiment of the present application can set up the self-verification (Latent Reward Self-verification) of the large language model. Since the knowledge of the large language model is encoded in the form of language, and the decision-making task is represented by symbolic states, this inconsistency hinders the direct application of the large language model. In order to effectively integrate the large language model, the embodiment of the present application proposes to use the encoding ability of the large language model to generate a latent reward encoding function. Among them, the semantically generated code can bridge the gap between language knowledge and symbolic latent rewards because its execution is symbolic and customized for specific tasks. Taking into account the inherent randomness and hallucination phenomena in the reasoning of large language models, the embodiment of the present application proposes a latent reward generation process with self-verification, including self-prompts and pre-verification, to enhance stability and reliability.

[0043] In the self-prompt phase, the large language model first generates n candidate responses, each of which contains a code implementation of the potential reward encoding function. These candidate responses are then input into the prompt, and the large language model is asked to summarize an improved response.

[0044] Pre-verification can utilize standardized response templates, and the potential reward encoding function can be extracted from the large language model response. Next, it is verified using pre-collected random states and providing error feedback to the large language model until it is executable.

[0045] Secondly, the embodiment of the present application can set the contribution allocation of the large language model. Based on the potential reward encoding function, the embodiment of the present application can adopt the reward decomposition method enhanced by the potential reward, based on the neural network decoder f ψ , let f ψ As a neural network decoder with parameter ψ, a new neural network model can be constructed, where the target of the new neural network model is expressed as:

[0046]

[0047] Where ψ is the decoder network parameter, τ is the trajectory, R(τ) is the trajectory reward, D is the replay buffer, and φ is the latent reward encoding function.

[0048] Therefore, the embodiments of the present application can convert potential rewards into proxy rewards based on the new neural network model, effectively improving the performance and robustness of the reinforcement learning system.

[0049] Optionally, in one embodiment of the present application, a potential reward encoding function is generated using the encoding capability of a target large language model, including: obtaining a task description and a state form of a target task; based on the encoding capability of the target large language model, extracting target key information in the task description and state form respectively, and generating a potential reward encoding function based on the target key information.

[0050] In some embodiments, the embodiments of the present application can extract the task description and state form required for a specific task from the task document, where the task description includes an environment overview and task objectives, and the state form details the meaning of each dimension in the state space. Then, based on the encoding capability of the large language model, the key information of the task description and the state form can be extracted respectively to generate a potential reward encoding function, thereby improving the performance of the reinforcement learning system.

[0051] In step S103, the proxy reward is added to the reinforcement learning training to obtain an efficient reinforcement learning credit allocation result enhanced by a large language model.

[0052] It can be understood that the embodiments of the present application can add proxy rewards to reinforcement learning training, utilize the multi-dimensional characteristics of potential rewards to enhance the time reward allocation, effectively alleviate the problems of delay and sparse feedback, thereby improving the training performance of reinforcement learning, and further obtain efficient reinforcement learning credit allocation results enhanced by a large language model, effectively improving the overall performance of the reinforcement learning system.

[0053] According to the large language model enhanced efficient reinforcement learning credit allocation method proposed in the embodiment of the present application, the prior knowledge of the target large language model can be used to obtain the potential rewards in the target environment information, and the potential rewards can be converted into proxy rewards based on the reward decomposition method enhanced by the potential rewards. Then, the proxy rewards are added to the reinforcement learning training, thereby obtaining the large language model enhanced efficient reinforcement learning credit allocation results, which effectively improves the overall performance of the reinforcement learning system. Thus, the difference between the language knowledge of the large language model and the symbolic representation required for a specific task in the related technology is solved, and at the same time, the randomness and hallucination phenomena in the large language model reasoning lead to low trust in the model, which reduces the overall performance of the reinforcement learning system.

[0054] Next, the efficient reinforcement learning credit allocation device enhanced by a large language model proposed in an embodiment of the present application is described with reference to the accompanying drawings.

[0055] Figure 2 It is a block diagram of an efficient reinforcement learning credit allocation device enhanced by a large language model according to an embodiment of the present application.

[0056] like Figure 2 As shown, the large language model enhanced efficient reinforcement learning credit allocation device 10 includes: an acquisition module 100, a conversion module 200 and a control module 300.

[0057] Specifically, the acquisition module 100 is used to acquire potential rewards in the target environment information using the prior knowledge of the target large language model.

[0058] The conversion module 200 is used to convert the potential reward into a proxy reward based on the reward decomposition method enhanced by the potential reward.

[0059] The control module 300 is used to add the proxy reward to the reinforcement learning training to obtain an efficient reinforcement learning credit allocation result enhanced by a large language model.

[0060] Optionally, in one embodiment of the present application, the conversion module 200 includes: a generation unit, a construction unit and a conversion unit.

[0061] Among them, the generation unit is used to generate a potential reward encoding function using the encoding ability of the target large language model.

[0062] A construction unit for building a new neural network model based on a potential reward encoding function, a potential reward enhancement return decomposition method, and a pre-set neural network decoder.

[0063] A conversion unit is used to convert potential rewards into proxy rewards based on the new neural network model.

[0064] Optionally, in one embodiment of the present application, the target of the new neural network model is expressed as:

[0065]

[0066] Where ψ is the decoder network parameter, τ is the trajectory, R(τ) is the trajectory reward, D is the replay buffer, and φ is the latent reward encoding function.

[0067] Optionally, in one embodiment of the present application, the generating unit includes: an acquiring subunit and a generating subunit.

[0068] The acquisition subunit is used to acquire the task description and status form of the target task.

[0069] The generation subunit is used to extract the target key information in the form of task description and state respectively based on the encoding capacity of the target large language model, and generate a potential reward encoding function according to the target key information.

[0070] It should be noted that the aforementioned explanation of the embodiment of the efficient reinforcement learning credit allocation method enhanced with a large language model is also applicable to the efficient reinforcement learning credit allocation device enhanced with a large language model of this embodiment, and will not be repeated here.

[0071] According to the large language model enhanced efficient reinforcement learning credit allocation device proposed in the embodiment of the present application, the prior knowledge of the target large language model can be used to obtain the potential rewards in the target environment information, and the potential rewards can be converted into proxy rewards based on the reward decomposition method enhanced by the potential rewards. Then, the proxy rewards are added to the reinforcement learning training, thereby obtaining the large language model enhanced efficient reinforcement learning credit allocation results, which effectively improves the overall performance of the reinforcement learning system. In this way, the difference between the language knowledge of the large language model and the symbolic representation required for a specific task in the related technology is solved, and at the same time, the randomness and hallucination phenomena in the large language model reasoning lead to low trust in the model, which reduces the overall performance of the reinforcement learning system.

[0072] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0073] A memory 301 , a processor 302 , and a computer program stored in the memory 301 and executable on the processor 302 .

[0074] When the processor 302 executes the program, the efficient reinforcement learning credit allocation method enhanced by the large language model provided in the above embodiment is implemented.

[0075] Furthermore, the electronic device further comprises:

[0076] The communication interface 303 is used for communication between the memory 301 and the processor 302 .

[0077] The memory 301 is used to store computer programs that can be run on the processor 302 .

[0078] The memory 301 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0079] If the memory 301, the processor 302 and the communication interface 303 are implemented independently, the communication interface 303, the memory 301 and the processor 302 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0080] Optionally, in a specific implementation, if the memory 301, the processor 302 and the communication interface 303 are integrated on a chip, the memory 301, the processor 302 and the communication interface 303 can communicate with each other through an internal interface.

[0081] The processor 302 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0082] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned efficient reinforcement learning credit allocation method enhanced by a large language model.

[0083] This embodiment also provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned efficient reinforcement learning credit allocation method for large language model enhancement.

[0084] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0085] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0086] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0087] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0088] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0089] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0090] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0091] The storage medium mentioned above may be a read-only memory, a disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. An efficient reinforcement learning credit assignment method enhanced by a large language model, characterized in that: The following steps are involved: Leverage the prior knowledge of the target large language model to obtain potential rewards in the target environment information; Converting the potential reward into a proxy reward based on a reward decomposition method enhanced by the potential reward; The proxy reward is added to the reinforcement learning training to obtain efficient reinforcement learning credit allocation results enhanced by a large language model.

2. The method according to claim 1, characterized in that: The potential reward-enhanced reward decomposition method converts the potential reward into a proxy reward, including: generating a potential reward encoding function using the encoding capability of the target large language model; Constructing a new neural network model based on the potential reward encoding function, the potential reward enhancement return decomposition method and the preset neural network decoder; Based on the new neural network model, the potential reward is converted into the proxy reward.

3. The method according to claim 2, characterized in that The objective of the new neural network model is expressed as: Where ψ is the decoder network parameter, τ is the trajectory, R(τ) is the trajectory reward, D is the replay buffer, and φ is the latent reward encoding function.

4. The method according to claim 2, characterized in that: The generating a potential reward encoding function by utilizing the encoding capability of the target large language model comprises: Get the task description and status form of the target task; Based on the encoding capability of the target large language model, target key information in the form of the task description and the state are extracted respectively, and the potential reward encoding function is generated according to the target key information.

5. An efficient reinforcement learning credit allocation device enhanced by a large language model, characterized in that: include: The acquisition module is used to use the prior knowledge of the target large language model to obtain the potential rewards in the target environment information; A conversion module, configured to convert the potential reward into a proxy reward based on a reward decomposition method enhanced by the potential reward; A control module is used to add the proxy reward to the reinforcement learning training to obtain an efficient reinforcement learning credit allocation result enhanced by a large language model.

6. The device according to claim 5, characterized in that The conversion module comprises: A generating unit, configured to generate a potential reward encoding function using the encoding capability of the target large language model; A construction unit, configured to construct a new neural network model based on the potential reward encoding function, the potential reward enhancement return decomposition method and a preset neural network decoder; A conversion unit is used to convert the potential reward into the proxy reward based on the new neural network model.

7. The device according to claim 6, characterized in that The objective of the new neural network model is expressed as: Where ψ is the decoder network parameter, τ is the trajectory, R(τ) is the trajectory reward, D is the replay buffer, and φ is the latent reward encoding function.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the efficient reinforcement learning credit allocation method enhanced by a large language model as described in any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement an efficient reinforcement learning credit allocation method enhanced by a large language model as described in any one of claims 1-4.

10. A computer program product, comprising a computer program, characterized in that The computer program is executed by a processor to implement the efficient reinforcement learning credit allocation method enhanced by a large language model as described in any one of claims 1-4.

Images