Collaboration method and apparatus for large language model and reinforcement learning model

Abstract

The present disclosure relates to a collaboration method and apparatus for a large language model and a reinforcement learning model. The method comprises: determining basic information corresponding to an agent in an open world, and an environment image where the agent is located; inputting the basic information and a knowledge base into a large language model trained on the basis of a preset hierarchy of needs; selecting, from among a plurality of candidate tools, a plurality of target tools each having an execution sequence, and outputting the plurality of target tools; and on the basis of a reinforcement learning model corresponding to the target tool having an optimal execution sequence, controlling the agent to execute a corresponding event.

Description

Collaborative method and device of large language model and reinforcement learning model

[0001] Cross-reference to related applications

[0002] The present disclosure claims priority to the Chinese patent application No. 202410798640.5, filed on June 20, 2024, entitled “Collaborative method and device of large language model and reinforcement learning model” and assigned to Tsinghua University. TECHNICAL FIELD

[0003] The present disclosure relates to the field of computer technology, and in particular, to a collaborative method and device of a large language model and a reinforcement learning model. BACKGROUND

[0004] Open-world survival games, as a kind of virtual environment, provide an ideal platform for studying the skill application of artificial intelligence agents in complex tasks. Early attempts were mainly based on reinforcement learning, requiring a single model to master multiple skills. However, with the rise of large language models, especially their excellent ability to automatically call various programs, researchers began to integrate large language models into open-world survival game research to further expand the skill range of agents. However, there are two major challenges in practice: first, the understanding of large language models for reinforcement learning models is not comprehensive enough, and it needs to rely on human-injected prior knowledge, but such knowledge is not always available or sufficient, limiting the adaptability and flexibility of agents; second, reinforcement learning models are not clear about when they will be called by large language models, which forces reinforcement learning models to generalize their capabilities for a wide range of situations during training, increasing the complexity of the training process.

[0005] SUMMARY

[0006] In view of this, the present disclosure proposes a collaborative method and device of a large language model and a reinforcement learning model, aiming to improve the flexibility and processing capacity of the collaborative processing process of the large language model and the reinforcement learning model.

[0007] According to a first aspect of the present disclosure, a collaborative method of a large language model and a reinforcement learning model is provided, the method comprising:

[0008] determining basic information corresponding to an agent of an open world and an environment image in which the agent is located;

[0009] inputting the basic information and a knowledge base into a large language model trained based on a preset demand pyramid, selecting and outputting a plurality of target tools with an execution order from a plurality of candidate tools, each of the candidate tools having a corresponding event, the demand pyramid being used to judge the demand level corresponding to different target tools, and the knowledge base including attribute information of the event corresponding to each of the candidate tools;

[0010] controlling the agent to perform the event corresponding to the target tool according to the reinforcement learning model corresponding to the target tool with the optimal execution sequence;

[0011] The method further includes:

[0012] performing the following steps in an iterative manner at least once:

[0013] inputting the environment image into the reinforcement learning model to output an operation instruction for controlling the agent;

[0014] redetermining the environment image in which the agent is located after the agent completes the operation corresponding to the operation instruction.

[0015] In one possible implementation, the inputting the basic information and the knowledge base into the large language model trained based on the preset demand pyramid to select and output a plurality of target tools with execution sequences from a plurality of candidate tools includes:

[0016] inputting the basic information and the knowledge base into the large language model trained based on the preset demand pyramid to select at least one target tool from a plurality of candidate tools by using the large language model;

[0017] sorting the at least one target tool based on the basic information and the knowledge base by using the large language model and outputting the sorted target tool.

[0018] In one possible implementation, the inputting the basic information and the knowledge base into the large language model trained based on the preset demand pyramid to select at least one target tool from a plurality of candidate tools by using the large language model includes:

[0019] inputting the basic information and the knowledge base into the large language model trained based on the preset demand pyramid to determine, by using the large language model, whether each layer of the demand pyramid is satisfied according to the basic information;

[0020] in response to the layer of the demand pyramid being satisfied, determining, by using the large language model, that the layer does not have a corresponding target tool;

[0021] in response to the layer of the demand pyramid not being satisfied, selecting, by using the large language model, at least one target tool corresponding to the layer from a plurality of candidate tools based on the basic information and the knowledge base.

[0022] In a possible implementation, the outputting, by the large language model, the at least one target tool after the sorting based on the basic information and the knowledge base includes:

[0023] determining, by the large language model, the urgency corresponding to each of the target tools according to the attribute information of each of the target tools in the basic information and the knowledge base;

[0024] outputting, by the large language model, the at least one target tool after the sorting according to the urgency corresponding to each of the target tools and the position of the corresponding layer in the demand pyramid.

[0025] In a possible implementation, the method further includes:

[0026] recording the event execution trajectory of each of the reinforcement learning models in the process of the agent performing the event corresponding to the target tool.

[0027] In a possible implementation, the method further includes:

[0028] re-determining the attribute information according to the multiple event execution trajectories in response to the multiple event execution trajectories corresponding to the candidate tool satisfying a preset condition;

[0029] replacing the attribute information in the knowledge base in response to the attribute information being different from the attribute information of the candidate tool in the knowledge base.

[0030] In a possible implementation, the training processes of the large language model and the reinforcement learning model are performed cooperatively.

[0031] According to a second aspect of the present disclosure, a cooperative device of a large language model and a reinforcement learning model is provided, and the device includes:

[0032] an image determination module configured to determine the basic information corresponding to an agent in an open world and an environment image in which the agent is located;

[0033] a tool prediction module configured to input the basic information and a knowledge base into a large language model trained based on a preset demand pyramid, select and output multiple target tools having an execution order from multiple candidate tools, each of the candidate tools having a corresponding event, and the demand pyramid being used to determine the demand level corresponding to different target tools, and the knowledge base including attribute information of the event corresponding to each of the candidate tools;

[0034] an event execution module configured to control the agent to perform the event corresponding to the target tool according to the reinforcement learning model corresponding to the target tool having the optimal execution order;

[0035] The reinforcement learning model controls the agent to perform an event corresponding to the target tool, including:

[0036] The following steps are performed at least once in an iterative manner:

[0037] The environment image is input into the reinforcement learning model, and an operation instruction for controlling the agent is output.

[0038] After the agent completes the operation corresponding to the operation instruction, the environment image in which the agent is located is re-determined.

[0039] In a possible implementation, the tool prediction module is further configured to:

[0040] The basic information and the knowledge base are input into a large language model trained based on a preset demand pyramid, and at least one target tool is selected from a plurality of candidate tools by the large language model.

[0041] The at least one target tool is sorted and output by the large language model based on the basic information and the knowledge base.

[0042] In a possible implementation, the tool prediction module is further configured to:

[0043] The basic information and the knowledge base are input into a large language model trained based on a preset demand pyramid, and whether each layer of the demand pyramid is satisfied is determined by the large language model according to the basic information.

[0044] In response to the layer of the demand pyramid being satisfied, it is determined by the large language model that the layer does not have a corresponding target tool.

[0045] In response to the layer of the demand pyramid not being satisfied, at least one target tool corresponding to the layer is selected from a plurality of candidate tools by the large language model based on the basic information and the knowledge base.

[0046] In a possible implementation, the tool prediction module is further configured to:

[0047] The urgency of each target tool is determined by the large language model according to the attribute information of each target tool in the basic information and the knowledge base.

[0048] The at least one target tool is sorted and output by the large language model according to the urgency of each target tool and the position of the corresponding layer in the demand pyramid.

[0049] In a possible implementation, the apparatus further includes:

[0050] a trajectory recording module, configured to record an event execution trajectory of each of the reinforcement learning models in a process in which the agent executes an event corresponding to the target tool.

[0051] In a possible implementation, the apparatus further includes:

[0052] an attribute determining module, configured to, in response to the plurality of event execution trajectories corresponding to the candidate tool satisfying a preset condition, determine attribute information again according to the plurality of event execution trajectories;

[0053] a knowledge base updating module, configured to, in response to the attribute information being different from attribute information of the candidate tool in the knowledge base, replace the attribute information in the knowledge base.

[0054] In a possible implementation, the training processes of the large language model and the reinforcement learning model are performed cooperatively.

[0055] According to a third aspect of the present disclosure, an electronic device is provided, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0056] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, which stores computer program instructions, wherein the computer program instructions are executed by a processor to implement the above method.

[0057] According to a fifth aspect of the present disclosure, a computer program product is provided, including computer-readable code or a non-volatile computer-readable storage medium carrying computer-readable code, when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the above method.

[0058] In the embodiments of the present disclosure, the basic information corresponding to the agent in the open world and the environment image are determined, the basic information and the knowledge base are input into the large language model trained based on the preset demand pyramid, a plurality of target tools with execution sequences are selected and output from a plurality of candidate tools, and the agent is controlled to execute corresponding events according to a reinforcement learning model corresponding to the target tool with the optimal execution sequence. The execution process of the event includes executing the following steps at least once in an iterative manner: inputting the environment image into the reinforcement learning model to obtain an operation instruction, and redetermining the environment image in which the agent is located after the agent completes the operation corresponding to the operation instruction. The present disclosure realizes flexible control of the agent through the large language model and the reinforcement learning model together, and improves the processing efficiency of the agent on the event.

[0059] Other features and aspects of the present disclosure will become apparent from a detailed description of exemplary embodiments with reference to the following drawings. BRIEF DESCRIPTION OF DRAWINGS

[0060] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments, features, and aspects of the present disclosure and serve to explain the principles of the present disclosure.

[0061] FIG. 1 shows a flowchart of a collaborative method of a large language model and a reinforcement learning model according to an embodiment of the present disclosure;

[0062] FIG. 2 shows a schematic diagram of a collaborative training process of a large language model and a reinforcement learning model according to an embodiment of the present disclosure;

[0063] FIG. 3 shows a schematic diagram of a collaborative device of a large language model and a reinforcement learning model according to an embodiment of the present disclosure;

[0064] FIG. 4 shows a schematic diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0065] Various exemplary embodiments, features, and aspects of the present disclosure will be explained hereinafter with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar elements. Although various aspects of the embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0066] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0067] In addition, for the purpose of convenience and brevity, detailed descriptions of well-known functions and structures incorporated herein can be omitted. It will be apparent, however, to those skilled in the art that the present disclosure can be practiced without such specific details. In some instances, well-known methods, apparatuses, elements and circuits have not been described in detail in order to not unnecessarily obscure aspects of the present disclosure.

[0068] The method for cooperation between the large language model and the reinforcement learning model can be executed by an electronic device such as a terminal device or a server. The terminal device can be any fixed or mobile terminal such as a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, and the like. The server can be a single server or a server cluster composed of multiple servers. Any electronic device can realize the method for cooperation between the large language model and the reinforcement learning model by calling computer-readable instructions stored in a memory through a processor.

[0069] FIG. 1 shows a flowchart of a method for cooperation between a large language model and a reinforcement learning model according to an embodiment of the present disclosure. As shown in FIG. 1, the method for cooperation between the large language model and the reinforcement learning model according to an embodiment of the present disclosure can include the following steps S10-S30.

[0070] Step S10, determining basic information of an agent in an open world and an environment image in which the agent is located.

[0071] In a possible implementation, an agent in the open world that needs to operate an event is determined by an electronic device, and further, basic information of the agent and an environment image in which the agent is located are determined. The open world can be a virtual survival game world, and the agent can be any game character such as a virtual person, a virtual animal, or a virtual plant in the survival game. Events that the agent needs to operate in the survival game can include collecting resources, eating, sleeping, drinking, and building, etc. The environment image in which the agent is located is an environment image obtained by image acquisition of an environment near a location of the agent in the open world, which can include an environment within a range covered by a line of sight of the agent. The basic information of the agent is used to represent a target and a state of the agent in the open world, which can include a preset ultimate target or a phased target, and a self-state representing a basic condition of the agent and an inventory state representing resources already owned by the agent.

[0072] Exemplarily, in the basic information of the agent determined by the electronic device, the preset ultimate goal can be "maximum exploration of the open world", and the self-state of the agent can include at least one of a health value, a food value, a beverage value, an energy value and an intelligence value, respectively representing the health state, the hunger state, the thirst state, the physical state and the mental state of the agent. The inventory state can include basic resources collected by the agent and advanced resources built by the agent, wherein the basic resources can include ore, wood, branches and the like, and the advanced resources built by the agent can include wooden pickaxes, wooden swords, wooden rafts made of wood, fire pits, fences made of ore, and tents built of wood, branches and ore. The environment image determined by the electronic device can include basic backgrounds such as trees, mountains, rivers and grasses in the open world, and animals such as cows, bees, butterflies and fish in the basic backgrounds.

[0073] In step S20, the basic information and the knowledge base are input into the large language model trained based on the preset demand pyramid, and a plurality of target tools with execution sequences are selected and output from a plurality of candidate tools.

[0074] In a possible implementation, after determining the basic information of the agent, the electronic device can input the basic information of the agent and a predetermined knowledge base into a large language model trained based on a preset demand pyramid, so as to select and output a plurality of target tools with execution sequences from a plurality of candidate tools through the large language model. Each candidate tool has a corresponding event and a corresponding reinforcement learning model, and the candidate tool can control the agent to execute the corresponding event through the corresponding reinforcement learning model. The event can be preset in the open world, and can include collecting coal, collecting stones, collecting wood, making an axe, making a fire pit, making food, drinking water, eating, fishing, catching a rabbit, etc. The demand pyramid is a multi-level pyramid set based on the Maslow's hierarchy of needs and the background of the open world survival game, and the demand level of each layer from bottom to top is gradually improved, which is used to judge the demand level corresponding to different target tools while screening the target tools. The knowledge base includes attribute information of the event corresponding to each candidate tool, which is used to limit the characteristics of the execution process of each event. For example, for the event "eating roasted fish", the corresponding attribute information can be "the fish needs to be roasted on the fire pit before fishing". Optionally, the knowledge base can be obtained according to the pre-accumulation in the training process, and can be updated in real time during the execution process of each event of the agent in the open world.

[0075] Optionally, in the open-world survival game, the electronic device can consider key factors such as instant tasks, intelligent agent health factors with long-term impact, and safety factors to build a demand pyramid. Illustratively, it can be determined that the demand pyramid has the demands of the intelligent agent from bottom to top in order of physiological needs, safety needs, cognitive needs, and self-actualization needs. After determining the demand pyramid, the demand pyramid can be input into the large language model as a prerequisite, and the large language model is trained for the scene of the open-world survival game.

[0076] Then, the electronic device inputs the basic information and the knowledge base into the large language model based on the training, and can first select at least one target tool from the multiple candidate tools through the large language model. That is, the large language model can fuse the information in the basic information and the knowledge base as prompt information, and screen the target tool corresponding to each layer of the demand pyramid from the multiple candidate tools. Wherein, whether each layer of the demand pyramid has a corresponding target tool needs to be judged according to whether the demand of this layer is met, that is, after inputting the basic information and the knowledge base into the large language model based on the preset demand pyramid, determining whether each layer of the demand pyramid is met through the large language model according to the basic information. In the case that the layer of the demand pyramid is met, it is determined through the large language model that the layer does not have a corresponding target tool. That is, it is judged that the intelligent agent does not need to perform the event corresponding to the layer of the demand. In the case that the layer of the demand pyramid is not met, at least one target tool corresponding to the layer is selected from the multiple candidate tools through the large language model based on the basic information and the knowledge base.

[0077] Secondly, after selecting at least one target tool from the candidate tools, the large language model is used to sort and output the at least one target tool based on the basic information and the knowledge base. Specifically, the large language model can determine the urgency of each target tool according to the attribute information of each target tool in the basic information and the knowledge base. Then, the large language model sorts and outputs the at least one target tool according to the urgency of each target tool and the position of the corresponding layer in the demand pyramid.

[0078] That is, the large language model of the embodiment of the present disclosure includes a proposal stage of screening target tools based on the demand pyramid, and a ranking stage of ranking target tools based on the importance determined based on the demand pyramid and the urgency determined based on the basic information and the knowledge base. In the proposal stage, the large language model considers the basic information of the agent and the demand pyramid for two-step thinking. First, it is determined whether the demand at each level is met. If it is met, the tool proposal at this level is skipped. If it is not met, the corresponding target tool at this level is screened based on the basic information and the knowledge base. In the ranking stage, the large language model integrates the urgency and the importance and also performs two-step thinking to avoid overly conservative ranking results. Specifically, the large language model can first evaluate the urgency of each target tool, which is affected by different subjective degrees from different demands. In order to alleviate potential illusions, the large language model can refer to the supporting facts in the proposal stage during the evaluation process and consider them during the evaluation process. Among them, the supporting fact is the fact on which the proposal is based, for example, in the case of the agent's "food" state being 1 and the large language model selecting the candidate tool "eat beef" as the proposal, the large language model will also output the supporting fact that the "food" state is 1 in the proposal stage. Subsequently, the large language model ranks the target tools according to the urgency of each proposal and the corresponding demand level (i.e., importance), and outputs the ranking result after obtaining the ranking result. The large language model trained based on the demand pyramid screens and ranks the target tools in the embodiment of the present disclosure, which, under the guidance of Maslow's hierarchy of needs, enables the agent in the open world to develop the ability to solve long-term low-level needs and pursue higher-level goals.

[0079] In step S30, the agent is controlled to perform an event corresponding to the target tool with the optimal execution sequence based on a reinforcement learning model corresponding to the target tool.

[0080] In a possible implementation, after the large language model outputs a plurality of target tools with execution sequences, the electronic device can obtain a reinforcement learning model corresponding to a target tool with the highest priority execution sequence, and control the agent to perform an event corresponding to the target tool based on the reinforcement learning model. Optionally, the process of controlling the agent to perform the event based on the reinforcement learning model can be an iterative process, and the environment image can be input to the reinforcement learning model at least once in an iterative manner to output an operation instruction for controlling the agent. After the agent completes the operation corresponding to the operation instruction, the environment image of the agent is re-determined to enter the next iteration process, and the iteration process is stopped until the agent completes the entire event.

[0081] Optionally, the electronic device can record the event execution trajectory of each reinforcement learning model controlling the agent to perform the event corresponding to the target tool after each event is completed. For example, after completing steps S10 and S20, in step S30, the reinforcement learning model controls the operation instruction of the agent in each iteration, and obtains a point in the event execution trajectory after the agent completes the operation corresponding to the operation instruction. After the iteration process is completed, the event execution trajectory is obtained, which corresponds to the optimal reinforcement learning model corresponding to the target tool. If steps S10, S20 and S30 are re-executed, a new event execution trajectory is obtained. Each point in the event execution trajectory records at least one of the state of the agent after performing the operation instruction and the environmental reward, and the state of the agent can include survival or death, etc., and the environmental reward can include the execution result feedback of the event, such as success or failure, etc.

[0082] Then, the electronic device can obtain a plurality of event execution trajectories corresponding to a plurality of candidate tools after performing steps S10, S20 and S30 multiple times, and can determine in real time whether the knowledge base needs to be updated according to the plurality of event execution trajectories currently corresponding to each candidate tool. Specifically, in the case that the plurality of event execution trajectories corresponding to the candidate tool meet the preset condition, the electronic device can re-determine the attribute information according to the advanced knowledge in the plurality of event execution trajectories. And in the case that the attribute information is different from the attribute information of the candidate tool in the knowledge base, the attribute information is replaced in the knowledge base. The advanced knowledge can include some experience learned, such as the execution order that must be met between certain operations, and the occurrence of certain consequences of the agent after performing certain operations, etc. The preset condition can be that the number of event execution trajectories corresponding to the candidate tool reaches a preset number threshold. The condition for the electronic device to determine whether to update the knowledge base can be that after the record of each event execution trajectory is completed, it is determined whether the plurality of event execution trajectories corresponding to the candidate tool corresponding to the event execution trajectory meet the preset condition. With the increase in the number of event execution trajectories corresponding to each candidate tool, the frequency of noisy events tends to decrease, enabling the electronic device to identify truly important events. The knowledge base summarized by the event execution trajectory according to the embodiments of the present disclosure can promote the tool proposal and ranking process of the large language model. Further, the agent gradually learns the best way to use various tools.

[0083] FIG. 2 shows a schematic diagram of a collaborative training process of a large language model and a reinforcement learning model according to an embodiment of the present disclosure. As shown in FIG. 2, in the present embodiment, the training process of the large language model and the reinforcement learning model is performed collaboratively. Specifically, in the training process, the present embodiment introduces a reflection module composed of an execution log recording event execution trajectories and a knowledge base, and a motivation module composed of a demand pyramid. The electronic device can determine the basic information of the current agent and the environment image in which it is located in real time during the training process. Then the basic information and the knowledge base are input as prompt information into the motivation module, and a plurality of target tools with execution sequences are selected and output from a plurality of candidate tools by the large language model based on the preset demand pyramid. The large language model trained based on the demand pyramid enables the agent to learn to consider its state comprehensively. For example, in the game Crafter, the physiological demand prompts the agent to propose the use of tools such as collecting drinks or eating beef to prevent entering a low state, and the cognitive demand prompts the proposal of the use of advanced tools such as making a wooden pickaxe to enhance the exploration of the world.

[0084] A common approach to grounding the large language model involves using immediate feedback of the current tool, which is a common practice in many previous works. However, this approach is susceptible to noisy trajectories in the game, which can lead to incorrect interpretations. For example, if the agent uses the eat-beef tool before using the make-wooden-pickaxe tool, it can incorrectly conclude that eat-beef is a prerequisite for make-wooden-pickaxe. Therefore, the present embodiment designs a reflection module to better ground the large language model in the environment without prior knowledge. During the model training process, both the execution log and the knowledge base within the reflection module are initialized as empty dictionaries. As the agent interacts with the environment multiple times, the execution log accumulates event execution trajectories for each candidate tool, and the corresponding attribute information is stored in the knowledge base by synthesizing advanced knowledge when the corresponding event execution trajectory meets the preset conditions. As the number of event execution trajectories increases, the frequency of noisy events tends to decrease, enabling the reflection module to identify truly important events. The reflection knowledge is fed into the motivation module to facilitate the tool proposal and ranking process. Therefore, the agent gradually learns the optimal way to use various tools, enabling the large language model to participate in the tool learning process.

[0085] After obtaining the reinforcement learning model corresponding to the target tool according to the execution order optimization, the agent is controlled to execute the event corresponding to the target tool. When the reinforcement learning model controls the agent to execute the event, the environment image and the feedback information are input to the reinforcement learning model in each iteration process, and the operation instruction for controlling the agent is output. After the agent completes the operation corresponding to the operation instruction, the environment image and the feedback information of the agent are re-determined. The feedback information can be provided by the software API of the environment, and specifically includes the new self-state and inventory of the agent, and whether the operation corresponding to the current target tool (such as eating beef) is completed. According to the information whether the operation is completed, the feedback information can be determined, for example, the reinforcement learning model is given a positive reward if the operation is completed, otherwise no reward is given, so as to update the model parameters by using the reinforcement learning algorithm.

[0086] In the integration research of large language models and reinforcement learning models, the dominant method in related technologies involves freezing the reinforcement learning model of the candidate tool and focusing on adjusting the large language model. Although this strategy has shown promising results due to the powerful reasoning ability of the large language model, it ignores the potential of synergizing the large language model and the reinforcement learning model of the tool. Specifically, when training independently, each tool's reinforcement learning model needs to start from scratch to complete its function. However, in collaboration with the large language model, the reinforcement learning models of different tools may only need to solve part of the task according to the prerequisites established by other tools. For example, the reinforcement learning model of the tool "collecting iron ore" only needs to learn to explore iron ore if the tool "making iron pickaxe" has already created the necessary equipment. Compared with starting from scratch, this cooperative method simplifies the learning process, making reinforcement learning model training more accessible and efficient.

[0087] To achieve the above-mentioned goals, the disclosed embodiments can train reinforcement learning models in collaboration with the learning of large language models through a synergistic learning framework. At the beginning of each event, the motivation module proposes a target tool to be used according to the basic information of the agent. Then, the agent takes action by calling the reinforcement learning model of the target tool. After completing the event of the target tool of the reinforcement learning model or exceeding a certain number of steps, the event execution trajectory is appended to the execution log of the reflection module, and the action trajectory is appended to the replay buffer of the tool training of the reinforcement learning model. The action trajectory can include the environment image corresponding to each operation step in the event execution process, the action of the agent, and the feedback information after the operation. Then, the motivation module proposes a new target tool according to the updated state of the agent. Throughout the process, the reflection module is running to update the knowledge base according to the event execution trajectory included in the execution log.

[0088] In the training process of the large language model and the reinforcement school model in the embodiments of the present disclosure, the agent completes the necessary preparation before pursuing more complex goals in view of the motivation module giving priority to low-level tools rather than high-level tools. Therefore, the advanced reinforcement learning model is trained with appropriate prerequisites, thereby reducing the overall learning difficulty. At the same time, the collaborative training framework of the embodiments of the present disclosure is very flexible and does not impose any restrictions on the training method of the original reinforcement learning model, with strong flexibility. The learning method can be online or offline, or model-free or model-based. The training process can combine the training of the reinforcement learning model with the learning process of the large language model to coordinate the use of tools. And through the large language model motivation model based on the demand pyramid, the reinforcement learning model is called according to the state of the agent, and the model training is considered in view of the survival needs and long-term task planning of the agent.

[0089] Based on the above technical features, the embodiments of the present disclosure realize flexible control of the agent based on the large language model and the reinforcement learning model, and improve the processing efficiency of the agent for events. At the same time, through the large language model based on the demand pyramid and the state of the agent, the reinforcement learning model is screened, which improves the accuracy of the screening result, and can efficiently guide the agent to complete the target in the basic information on the premise of meeting the needs of the agent. And in the process of use, the instructions of the tool execution are summarized in real time, and better tool use order and more appropriate timing are learned.

[0090] FIG. 3 shows a schematic diagram of a collaborative device of a large language model and a reinforcement learning model according to an embodiment of the present disclosure. As shown in FIG. 3, the collaborative device of the large language model and the reinforcement learning model in the embodiments of the present disclosure can include:

[0091] An image determination module 30 is configured to determine the basic information corresponding to the agent in the open world and the environment image in which the agent is located.

[0092] A tool prediction module 31 is configured to input the basic information and a knowledge base into a large language model trained based on a preset demand pyramid, select and output a plurality of target tools with execution order from a plurality of candidate tools, each of the candidate tools having a corresponding event, and the demand pyramid is used to judge the demand level corresponding to different target tools, and the knowledge base includes attribute information of the event corresponding to each of the candidate tools.

[0093] An event execution module 32 is configured to control the agent to execute the event corresponding to the target tool according to the reinforcement learning model corresponding to the target tool with the optimal execution order.

[0094] The reinforcement learning model controls the agent to execute the event corresponding to the target tool, including:

[0095] performing the following steps at least once in an iterative manner:

[0096] inputting the environment image into the reinforcement learning model to output operation instructions for controlling the agent;

[0097] redetermining the environment image in which the agent is located after the agent completes the operation corresponding to the operation instructions.

[0098] In a possible implementation, the tool prediction module 31 is further configured to:

[0099] inputting the basic information and the knowledge base into a large language model trained based on a preset demand pyramid, and selecting at least one target tool from a plurality of candidate tools through the large language model;

[0100] sorting the at least one target tool based on the basic information and the knowledge base through the large language model, and outputting the sorted target tool.

[0101] In a possible implementation, the tool prediction module 31 is further configured to:

[0102] inputting the basic information and the knowledge base into a large language model trained based on a preset demand pyramid, and determining whether each layer of the demand pyramid is satisfied according to the basic information through the large language model;

[0103] in response to the layer of the demand pyramid being satisfied, determining that the layer does not have a corresponding target tool through the large language model;

[0104] in response to the layer of the demand pyramid not being satisfied, selecting at least one target tool corresponding to the layer from a plurality of candidate tools based on the basic information and the knowledge base through the large language model.

[0105] In a possible implementation, the tool prediction module 31 is further configured to:

[0106] determining the urgency of each target tool according to the attribute information of each target tool in the basic information and the knowledge base through the large language model;

[0107] sorting the at least one target tool according to the urgency of each target tool and the position of the corresponding layer in the demand pyramid through the large language model, and outputting the sorted target tool.

[0108] In a possible implementation, the apparatus further includes:

[0109] A trajectory recording module is configured to record an event execution trajectory of each of the reinforcement learning models in a process of the agent executing events corresponding to the target tool.

[0110] In a possible implementation, the apparatus further includes:

[0111] An attribute determining module is configured to, in response to the plurality of event execution trajectories corresponding to the candidate tool satisfying a preset condition, determine attribute information again according to the plurality of event execution trajectories.

[0112] A knowledge base updating module is configured to, in response to the attribute information being different from attribute information of the candidate tool in the knowledge base, replace the attribute information into the knowledge base.

[0113] In a possible implementation, the training processes of the large language model and the reinforcement learning model are performed cooperatively.

[0114] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, it will not be repeated here.

[0115] The embodiments of the present disclosure also provide a computer-readable storage medium having computer program instructions stored therein, and the computer program instructions are executed by a processor to implement the above method. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0116] The embodiments of the present disclosure also provide an electronic device, including a processor, a memory for storing processor-executable instructions, and wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0117] The embodiments of the present disclosure also provide a computer program product, including computer-readable code or a non-volatile computer-readable storage medium carrying computer-readable code, and when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the above method.

[0118] FIG. 4 shows a schematic diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 can be provided as a server or a terminal device. Referring to FIG. 4, the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application program, executable by the processing component 1922. The application program stored in the memory 1932 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above method.

[0119] The electronic device 1900 can further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as Windows Server TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM or the like.

[0120] In an exemplary embodiment, a non-transitory computer readable storage medium, such as the memory 1932 including computer program instructions, is also provided, which can be executed by the processing component 1922 of the electronic device 1900 to complete the above method.

[0121] The present disclosure can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0122] Computer readable storage media can be tangible storage media which can retain and store instructions for use by an instruction execution device. Computer readable storage media can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer readable storage media include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0123] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0124] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0125] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0126] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other data storage device. When the computer readable program instructions are loaded into the computer and other programmable data processing apparatus, a series of operational steps are implemented that provide processes such that the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0127] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0128] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logic functions. In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0129] Embodiments of the present disclosure have been described above, and the description is intended to be illustrative of the embodiments and not restrictive. Many modifications and variations of the described embodiments are possible and are within the scope of the disclosure. The selection of terms is intended to best describe the principles of the embodiments, practical application, or technical improvements in the art, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method of collaboration of a large language model and a reinforcement learning model, wherein, The method comprises: determining basic information corresponding to an agent of an open world and an environment image in which the agent is located; inputting the basic information and a knowledge base into a large language model trained based on a preset demand pyramid, selecting and outputting a plurality of target tools with execution sequences from a plurality of candidate tools, each of the candidate tools having a corresponding event, the demand pyramid being used to judge different demand levels of different target tools, and the knowledge base including attribute information of events corresponding to each of the candidate tools; controlling the agent to execute events corresponding to the target tools according to a reinforcement learning model corresponding to the target tool with the optimal execution sequence; wherein the reinforcement learning model controls the agent to execute the events corresponding to the target tools, comprising: performing the following steps at least once in an iterative manner: inputting the environment image into the reinforcement learning model to output operation instructions for controlling the agent; redetermining the environment image in which the agent is located after the agent completes the operation corresponding to the operation instructions.

2. The method of claim 1, wherein, The inputting the basic information and the knowledge base into the large language model trained based on the preset demand pyramid, selecting and outputting a plurality of target tools with execution sequences from a plurality of candidate tools, comprises: inputting the basic information and the knowledge base into the large language model trained based on the preset demand pyramid, and selecting at least one target tool from a plurality of candidate tools through the large language model; sorting and outputting the at least one target tool through the large language model based on the basic information and the knowledge base.

3. The method of claim 2, wherein, The inputting the basic information and the knowledge base into the large language model trained based on the preset demand pyramid, selecting and outputting a plurality of target tools with execution sequences from a plurality of candidate tools, comprises: inputting the basic information and the knowledge base into the large language model trained based on the preset demand pyramid, and determining whether each layer of the demand pyramid is satisfied according to the basic information through the large language model; in response to the layer of the demand pyramid being satisfied, determining that the layer does not have a corresponding target tool through the large language model; in response to the layer of the demand pyramid not being satisfied, selecting at least one target tool corresponding to the layer from a plurality of candidate tools based on the basic information and the knowledge base through the large language model.

4. The method of claim 2 or 3, wherein, The sorting and outputting the at least one target tool through the large language model based on the basic information and the knowledge base, comprises: determining the urgency of each target tool according to the attribute information of each target tool in the basic information and the knowledge base through the large language model; sorting and outputting the at least one target tool according to the urgency of each target tool and the position of the corresponding layer in the demand pyramid through the large language model.

5. The method of any of claims 1-4, wherein, The method further comprises: recording the event execution trajectory of the agent in the process of controlling the agent to execute the events corresponding to the target tools by each reinforcement learning model.

6. The method of claim 5, wherein, The method further comprises: In response to the preset condition being met by the trajectory of the multiple events corresponding to the candidate tool, attribute information is re-determined according to the trajectory of the multiple events; In response to the attribute information being different from attribute information of the candidate tool in the knowledge base, the attribute information is replaced into the knowledge base.

7. The method of any one of claims 1-6, wherein, The training processes of the large language model and the reinforcement learning model are cooperatively performed.

8. A collaborative device of a large language model and a reinforcement learning model, wherein, The apparatus comprises: An image determination module is configured to determine basic information corresponding to an agent in an open world and an environment image in which the agent is located; A tool prediction module is configured to input the basic information and a knowledge base into a large language model trained based on a preset demand pyramid, select and output multiple target tools having execution sequences from multiple candidate tools, each of the candidate tools having corresponding events, and the demand pyramid being used to judge demand levels corresponding to different target tools, and the knowledge base including attribute information of events corresponding to each of the candidate tools; An event execution module is configured to control the agent to execute events corresponding to the target tools according to a reinforcement learning model corresponding to the target tool having the optimal execution sequence. The reinforcement learning model controls the agent to execute the events corresponding to the target tools, comprising: The following steps are performed at least once in an iterative manner: The environment image is input into the reinforcement learning model, and operation instructions for controlling the agent are output; After the agent completes the operation corresponding to the operation instructions, the environment image in which the agent is located is re-determined.

9. An electronic device, comprising: Comprise: A processor; A memory for storing processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 7 when executing the instructions stored in the memory.

10. A non-transitory computer readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions are executed by the processor to implement the method of any one of claims 1 to 7.

Images