Planning method based on feedback and value driving, computer equipment and storage medium

Abstract

The invention provides a planning method based on feedback and value driving, computer equipment and a storage medium, and the method comprises the steps: carrying out the prompt word construction for the current information of an intelligent agent; performing value-driven task planning by using a language model according to the cue word; and analyzing and recalling a task planning result, namely analyzing, checking and correcting an output result of the language model completing the task planning, then matching and recalling skills of the existing intelligent agent, performing analog simulation on the task planning result, and performing feedback and providing feedback information according to the analog simulation result. According to the method, the output of the language model follows a certain value, so that a more reasonable task path can be planned, and the problem that an unavoidable output result possibly has illusion in the application of the existing language model is solved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a planning method based on feedback and value drive, a computer device and a storage medium. Background Art

[0002] The task planning capability of an intelligent agent is the ability of an intelligent agent to generate a series of action sequences based on the current environment and goals, or the ability of an intelligent agent to pre-set an action plan before performing a task to ensure that it moves towards the established goal. The task planning capability of an intelligent agent covers many aspects, including long-term and short-term planning strategies, planning output formats, feedback and iteration mechanisms, etc. The task planning of an intelligent agent is one of its core driving forces. As a key research direction in the field of artificial intelligence, the core of the task planning of an intelligent agent lies in how to enable computer systems to plan and execute a series of complex tasks efficiently and accurately like humans. This requires not only that the system can understand the goals and constraints of the task, but also that it must have the ability to make decisions in an uncertain environment. Traditional task planning algorithms such as fast forward search and fast descent algorithms are all search-based strategy algorithms.

[0003] Fast forward search has the following disadvantages: 1. Accuracy issues: Search engines are susceptible to false information. Some content may be ranked higher through search engine optimization (SEO) technology, pretending to be an authoritative source, causing users to face difficulties in obtaining valid information and increasing the risk of misleading on the Internet. 2. Search tampering: In order to obtain higher click-through rates and advertising revenue, some search engines will tamper with search results, ranking results that are more relevant to advertisers higher, which damages the fairness and accuracy of the search engine results page. 3. Personalization bias: In order to provide a better user experience, some search engines use personalized algorithms to customize search results, which may lead to the generation of "filter bubbles". Users can only see content related to their interests or have been exposed to them, and cannot be exposed to diverse and conflicting views. 4. Advertising interference: Although advertising is an important part of the profit model of many search engines, some search engines have problems in advertising. Similarly, keyword queries often have many advertising promotion links, which interfere with the user's normal experience of obtaining information.

[0004] The fast descent algorithm has the following disadvantages: a. Local optimal solution: The fast descent algorithm is prone to fall into the local optimal solution instead of the global optimal solution. This is because once the algorithm finds a local optimal solution during the iteration process, it will stop iterating and no longer search for other possible solutions, which may cause the algorithm to fail to find the global optimal solution. B. Slow convergence: The fast descent algorithm converges slowly, especially when the objective function is more complex or there are multiple local optimal solutions. The algorithm requires multiple iterations to approach the global optimal solution, which results in high computational costs. C. Sensitive to initial values: The fast descent algorithm is very sensitive to the choice of initial values. Different initial values ​​may cause the algorithm to fall into different local optimal solutions, thus affecting the final results. D. May fall into a plateau area: In some cases, the objective function may change very slowly in a certain area, causing the algorithm to perform a large number of invalid iterations in the area, thereby reducing the efficiency of the algorithm. E. Difficult parameter selection: When using the fast descent algorithm, it is a challenge to choose the right step size and learning rate. The choice of step size and learning rate has a significant impact on the performance of the algorithm. Improper selection may cause the algorithm to converge slowly or fail to converge. E. May not converge: In some cases, the fast descent algorithm may not converge to a stable solution, especially when the objective function is very complex or noisy.

[0005] The above shortcomings limit the application effect and stability of fast forward search and fast descent algorithms on some problems, and their generalization ability is limited. In practical applications, it is necessary to select appropriate optimization algorithms according to specific problems and improve them in combination with the following methods: a. Combine with other optimization strategies: For example, introduce momentum or adaptive learning rate in the fast descent algorithm to improve the stability and convergence speed of the algorithm; combine heuristic search strategies in fast forward search to cope with complex search spaces.

[0006] b. Parameter adjustment: Adjust the algorithm parameters according to the characteristics of the specific problem, such as search depth, breadth, step size, etc., to adapt to different application scenarios.

[0007] c. Data enhancement: By increasing the diversity and complexity of training data, the algorithm can generalize better when faced with new data.

[0008] d. Model integration: Combine multiple strategies or models for integrated learning to improve the overall generalization ability and robustness.

[0009] In recent years, with the rapid development of language model technology, through the training of massive data, the technology has the ability to integrate knowledge from multiple fields, as well as the flexibility and adaptability of task processing based on different background information, showing remarkable results in tasks such as natural language understanding and interactive dialogue. The reasoning ability of language models has a wide range of applications in many fields. For example, in the field of education, it can be used as an intelligent tutoring tool to help students solve math problems and understand scientific principles; in the medical field, it can assist doctors in diagnosing diseases and formulating treatment plans; in the legal field, it can help lawyers analyze cases and write legal documents. The generalization ability of language models brings new ideas to task planning: using the reasoning ability of language models, combined with environmental information, to perform reasonable active task decomposition and planning.

[0010] Although language models have demonstrated certain reasoning capabilities through training with massive amounts of data. However, its hallucination output, that is, the model output results may have "hallucinations", and the output may not conform to the facts or have erroneous content, which is still an inevitable problem in the application of language models. The hallucination problem seriously affects the reliability and credibility of language models, especially in application scenarios that require high accuracy, such as medical diagnosis, legal consultation, etc., which may lead to serious consequences. Some scholars have conducted detailed research on the generation and verification of the hallucination output problem of language models. Experimental results show that although researchers have proposed many technical techniques to solve the problem of hallucination output of language models, hallucinations caused by training data, hallucinations caused by training, hallucinations caused by the randomness of reasoning algorithms, etc., are still the difficulties that language models cannot avoid in generating hallucination outputs. Summary of the invention

[0011] In view of the above problems, the present invention provides a planning method, a computer device and a storage medium based on feedback and value drive.

[0012] The feedback-based and value-driven planning method provided by the present invention may include the following steps: S1. Construct prompt words based on the current information of the intelligent agent; S2. Performing value-driven task planning using a language model according to the prompt words; S3, task planning result analysis and recall, that is, analyzing, verifying and correcting the output result of the language model that completes the task planning, and then matching and recalling the existing skills of the intelligent agent; and S4. Simulate the result of the task planning, provide feedback based on the result of the simulation, and provide feedback information.

[0013] Further, the step S1 may include: Role definition: explain the role of the language model in task planning and provide role information; Output definition: define the data format that the language model output needs to meet, that is, define the language model output format; Provide examples: Provide examples of the specific task planning, that is, provide example information; Environment definition: explaining the scene content information used by the language model when performing the current task planning, that is, defining the environment information; Information feedback: after the simulation is performed on the result of the task planning, feedback information is provided according to the result of the simulation; Task definition: Define the task content that needs to be completed for value-driven proactive task planning, that is, define task information.

[0014] Furthermore, in the step S1, part of the structured data of the current scene is converted into a natural language description and enters the prompt word, and part of it is converted into JSON structured data and enters the prompt word.

[0015] Furthermore, in step 2, the task planning is performed by calling a local model or calling a commercial application program interface.

[0016] Furthermore, all the completed results of the task planning are mapped into the skill library.

[0017] Further, the step S3 may include: Verifying and correcting the output format of the language model, that is, correcting and reorganizing the output content of the language model into a data structure for parsing and operation; The parsed results of the task planning are mapped to a database, i.e., the skill library, to recall the actions of the agent.

[0018] Among them, the step of mapping the parsed result of the task planning to the database, i.e., the skill library, and recalling the action of the agent can include: first, using a vector model to embed the result of the natural language autonomously played by the language model and the action of the agent used for the simulation, and converting the result of the task planning described in natural language into a vector representation, and then performing an inner product similarity calculation on the embedded vectors obtained from the two, that is, calculating the similarity by calculating the inner product of the vector obtained by the conversion and the vector of the action supported by the simulation in the database, and finally selecting the action whose similarity is greater than the expected matching threshold as the final planning result to enter the next step of the simulation.

[0019] Further, in step S4, the feedback result may include the problem that occurs when the task planning result is simulated: 1. Non-content errors: The result of the task planning does not meet the format requirements of the simulation and needs to be re-planned; 2. Content errors may include: a. The language model has a hallucination output problem when performing task planning, and cannot perform the simulation, and needs to be re-planned; b. The result of the task planning cannot improve the utility of the agent, and re-planning is required in combination with the current state; and c. The result of the task planning failed during the simulation, indicating unreasonable planning.

[0020] Further, if the simulation does not select an executable result for the planning result of the language model, the currently collected feedback information automatically enters the next task round planning to assist the planning of the agent, and at the same time, the agent maintains the current state and waits for the planning result of the next round; The simulation refers to the agent simulating the task sequence output by the language model. During the simulation, the agent interacts with the current virtual world scene according to the result of the task planning. For each result of the task planning, multiple rounds of simulation will be performed in parallel when performing the simulation. If all the results of the task planning cannot be completed in the simulation stage, the information generated during the simulation is recorded and fed back to the language model in the next round of task planning, and is used to guide the language model to reflect through chain thinking, so as to refer to the task planning of the current scenario.

[0021] The present invention also provides a computer device, which may include a memory, a first processor, and a first computer program stored in the memory and executable on the first processor, wherein the first computer program implements the above-mentioned feedback-based and value-driven planning method when executed by the first processor.

[0022] The present invention also provides a computer-readable storage medium, which is used to store a second computer program. The second computer program can be executed by at least one second processor so that the at least one second processor executes the above-mentioned feedback-based and value-driven planning method.

[0023] The feedback- and value-driven planning method provided by the present invention combines physical simulation with a language model. On the one hand, it improves the language model's understanding of real scenarios, and on the other hand, it enhances the interpretability of the entire language model and the executability of task planning results, so that the output of the language model follows certain values, thereby planning a more reasonable task path, solving the problem that the output results of existing language models may inevitably have "hallucinations" in applications.

[0024] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 A flow chart of a planning method based on feedback and value-driven according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the embodiments of the present invention more clearly understood, the spirit of the contents disclosed by the present invention will be clearly explained with the accompanying drawings and detailed descriptions below. After understanding the embodiments of the contents of the present invention, any technician in the relevant technical field can change and modify the techniques taught by the contents of the present invention without departing from the spirit and scope of the contents of the present invention.

[0028] The exemplary embodiments and descriptions of the present invention are used to explain the present invention, but are not intended to limit the present invention. In addition, elements / components with the same or similar reference numerals used in the drawings and embodiments are used to represent the same or similar parts.

[0029] The terms “first”, “second”, etc. used in this document do not particularly refer to an order or sequence, nor are they used to limit the present invention. They are only used to distinguish elements or operations described with the same technical terms.

[0030] The directional terms used in this document, such as up, down, left, right, front or back, etc., are only used to refer to the directions of the drawings. Therefore, the directional terms used are used to illustrate and not to limit the present invention.

[0031] The words “include,” “including,” “have,” “contain,” etc. used in this document are open-ended terms, meaning including but not limited to.

[0032] As used herein, "and / or" includes any or all combinations of the items described.

[0033] As used herein, “plurality” includes “two” and “more than two”; as used herein, “plurality groups” includes “two groups” and “more than two groups”.

[0034] The terms "substantially" and "approximately" used herein are used to modify any quantity or error that may vary slightly, but these slight changes or errors do not change their essence. Generally speaking, the range of slight changes or errors modified by such terms may be 20% in some embodiments, 10% in some embodiments, 5% in some embodiments, or other values. Those skilled in the art should understand that the aforementioned values ​​can be adjusted according to actual needs and are not limited thereto.

[0035] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0036] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but not be limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or a system having A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but not be limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or a system having A, B, C, etc.). Those skilled in the art should also understand that any transitional conjunctions and / or phrases that substantially represent two or more optional items, whether in the specification, claims, or drawings, should be understood to give the possibility of including one of these items, either of these items, or both of these items. For example, the phrase "A or B" should be understood to include the possibilities of "A" or "B", or "A and B".

[0037] Figure 1This is a flow chart of the feedback-based and value-driven planning method provided by the present invention. The value in the concept of value-driven refers to the utility of a certain dimension of the agent in the current state. Therefore, by modeling the value of different dimensions of the agent, it is possible to construct an agent whose behavior is driven by value. For example, by modeling the value dimension of the agent's hunger, when the agent's hunger increases, it can be actively driven to eat. See Figure 1 , the feedback-based and value-driven planning method may include the following steps.

[0038] S1. Construct prompt words based on the current information of the intelligent agent.

[0039] Different from the traditional method of providing task requirements and passively letting the language model perform task planning, in the present invention, the value system of an intelligent agent will be modeled and evaluated, that is, simulated. In the subsequent task planning, the language model will combine the current scene information, the value evaluation results, that is, the simulation results, and the role settings to perform autonomous task planning. This step may specifically include: S11. Role definition: Explain the role task of the language model during task planning, that is, the background information of the role played by the language model in the environment, and provide role information. If the role is an embodied intelligent entity, the role information may include its name, gender, age, etc.

[0040] S12. Output definition: Combine the chain of thought technique to define the data format that the model output needs to meet, that is, define the language model output format. The purpose of this step is to support the subsequent simulation to effectively analyze the model output results, thereby improving the accuracy of language model task planning by combining chain thinking.

[0041] S13. Provide cases: provide examples of specific task planning, that is, provide example information, which is a case example: in a certain scenario, define the current state of the intelligent agent and the expected task planning results.

[0042] S14. Environment definition: Instruct the language model to include scene content information that may be used when planning the current task, i.e., define environment information. The environment information may include: whether the environment is a home environment or an open natural environment, what objects and people are in the environment, etc.

[0043] S15. Information feedback: After the simulation of the task planning results, feedback information is provided according to the simulation results. The feedback information is a summary of the results of the task planning simulation, which may include: the language model task planning results, whether the current state of the intelligent agent can be improved, the reasons for the failure of the planning result simulation, and other information.

[0044] S16. Task definition: define the task content that needs to be completed by value-driven active task planning, that is, define task information. The active task planning refers to the task planning that is not passively carried out by relying on the user to specify the task, but is carried out based on the evaluation of the current environment state and the state of the intelligent agent.

[0045] Part of the structured data of the current scene will be converted into natural language descriptions and entered into the prompt words, while part of it will be entered into the prompt words as structured data in JSON (JavaScript Object Notation). The natural language description of the scene provides the scene information to the language model in an interpretable form, which can enhance the interpretability of task planning in subsequent task planning. The structured JSON data is used to list relevant information such as the location coordinates of the object, the color of the object, the orientation of the object, etc.

[0046] The prompt words constructed in this step may include the above-mentioned role information, language model output format, example information, environment information, feedback information, task information, etc. The present invention is not limited thereto.

[0047] S2. Based on the prompt words, use the language model to perform value-driven task planning.

[0048] When performing the task planning, two methods can be used: one is to call the local model, that is, the local model calling method, and the other is to call the commercial application program interface (API), that is, the commercial API calling method. In the local model calling method, users can choose to download the language model by themselves. The model is deployed as a web service through FastAPI for subsequent task planning calls. This method supports users to use different models according to their own hardware conditions, and has a high degree of flexibility. The commercial API calling method is carried out by calling the API interface provided by the commercial language model. Users need to adapt accordingly according to the interface of the API provider.

[0049] S3: Task planning result analysis and recall, that is, analyzing, verifying and correcting the output results of the language model that completes the task planning, and then matching and recalling the skills of the existing intelligent agent.

[0050] Among them, all the results of completed task planning are mapped into a simulatable skill library.

[0051] After completing the task planning through the language model, all planning results will be simulated and fed back, which may include: performing the simulation on all planning results, some planning results, that is, the planned paths, may not be able to be executed at all, some planned paths have no positive impact on the value of the current intelligent agent, and some planned paths have a positive impact on the value of the intelligent agent; when the simulation does not have a reasonable output, that is, the planned path does not have a positive impact on the value of the intelligent agent, feedback information of the simulation is generated, and the feedback information will participate in the next round of task planning. At that time, the language model will be used to combine the current information of the next round and the feedback information for task planning.

[0052] Since the language model is essentially a generative model, the task planning results it outputs are continuous text information, and different models have different reasoning capabilities. Therefore, the task planning results obtained by model reasoning may not meet the needs of subsequent simulation, and the output results of the model need to be reasonably parsed, verified and corrected. This step can include two sub-steps: the first step is to verify and correct the model output format, that is, to correct and reorganize the output content of the language model into a data structure that can be parsed and operated; the second step is to map the parsed task planning results to the database, that is, the skill library, to recall the actions of the intelligent agent. In the second step, the language results of the task planning will be mapped into action results that can be simulated, and then enter the simulation module for simulation and scoring.

[0053] As a generative model, although the language model has been trained with a large amount of data and has demonstrated intelligence under certain conditions, it still lacks understanding of the real environment in essence. The language model may inevitably have the problem of hallucination output, resulting in the inability to find the corresponding results of the task planning in the intelligent agent action library, or the results of the task planning may not be executed by the intelligent agent. In order to ensure the stability of task planning and the simulatability of the results, the second step can also include a recall process of similarity matching for the task planning results of the language model's autonomous performance. First, the vector model is used to embed the natural language results of the model's autonomous performance and the actions that the intelligent agent can simulate, and the task planning results described in natural language are converted into vector representations. Subsequently, the similarity of the inner product of the embedded vectors obtained by the two is calculated, that is, the similarity is calculated by calculating the inner product of the vector obtained by the conversion and the vector of the action supported by the simulation in the database, so as to align the task planning results of the language model's hallucination to the space that can be simulated. Finally, the action of the intelligent agent that can be simulated with the highest matching degree, that is, the similarity greater than the expected matching degree threshold, is selected as the final planning result to enter the next step of simulation.

[0054] S4. Simulate the task planning results, provide feedback based on the simulation results, and provide feedback information.

[0055] The simulation of the result of the previous task planning can be used to verify the executability of the result of the task planning on the one hand, and to provide feedback on the result of the task planning on the other hand, so as to improve the accuracy of the next task planning. The present invention simulates the result of the previous task planning according to the world model to obtain the feedback result, which can ensure the accuracy of the agent when the action is executed on the one hand, and speed up the reaction speed of the agent on the other hand, so that it does not rely on manual verification. The simulation refers to the simulation of the task sequence output by the language model by the agent. In the process of simulation, the agent interacts with the current virtual world scene according to the task planning result, such as changing the position of the object, changing its own state, etc. For each task planning result, the simulation link will use a parallel method to perform multiple rounds of simulation, for example, the task of eating can simulate eating apples or bread if the scene supports it. At the same time, in the simulation link, not all tasks can be performed, such as: the language model plans the agent to perform the task of "eating", and there is no food in the scene, then the task of "eating" cannot be performed. Through simulation, the present invention can objectively and effectively evaluate the tasks planned by the language model. If all the task planning results cannot be completed in the simulation stage, the information generated in the simulation link will be recorded and fed back to the large language model in the next round of task planning, and the language model will be guided to reflect through chain thinking to refer to the task planning of the current scene. The present invention is not limited to this.

[0056] Feedback results can include possible problems that may arise when performing simulations on the language model task planning results: 1. Non-content errors: The results of the language model task planning do not meet the format requirements of the simulation and need to be re-planned.

[0057] 2. Content-based errors may include the following steps: a. There is a problem of hallucination output when planning language model tasks, simulation cannot be performed, and re-planning is required.

[0058] b. The language model task planning results cannot improve the effectiveness of the agent and need to be re-planned based on the current state.

[0059] c. The language model task planning results failed during simulation, and there was unreasonable planning.

[0060] If the simulation cannot select a reasonable and executable result for the planning result of the language model, the currently collected feedback information will automatically enter the next round of task planning to assist the agent in planning. At the same time, the agent maintains the current state and waits for the next round of planning results.

[0061] The present invention also provides a computer device, which may include a memory, a first processor, and a first computer program stored in the memory and executable on the first processor, wherein the first computer program implements the above-mentioned feedback-based and value-driven planning method when executed by the first processor.

[0062] The present invention also provides a computer-readable storage medium, which is used to store a second computer program. The second computer program can be executed by at least one second processor so that the at least one second processor executes the above-mentioned feedback-based and value-driven planning method.

[0063] The memory may be a volatile memory, such as a random access memory (RAM), or an external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM); or non-volatile memory, such as read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk drive (HDD) or solid-state drive (SSD); or a combination of the above types of memory.

[0064] The first processor and the second processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to implement the functions of the processor may also be other, and the embodiment of the present invention does not specifically limit it.

[0065] The feedback- and value-driven planning method provided by the present invention combines physical simulation with a language model. On the one hand, it improves the language model's understanding of real scenarios, and on the other hand, it enhances the interpretability of the entire language model and the executability of task planning results, so that the output of the language model follows certain values, thereby planning a more reasonable task path, solving the problem that the output results of existing language models may inevitably have "hallucinations" in applications.

[0066] According to an embodiment of the present invention, the method flow according to an embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. According to an embodiment of the present invention, the electronic device, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0067] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0068] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus or device.

[0069] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0070] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways, even if such combinations and / or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All of these combinations and / or combinations fall within the scope of the present invention.

[0071] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination. The scope of the present invention is defined by the attached claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A feedback-based and value-driven planning approach, characterized by: The planning approach includes: S1. Construct prompt words based on the current information of the intelligent agent; S2. performing value-driven task planning using a language model according to the prompt word; and S3, task planning result analysis and recall, that is, analyzing, verifying and correcting the output result of the language model that completes the task planning, and then matching and recalling the existing skills of the intelligent agent; S4. Simulate the result of the task planning, provide feedback based on the result of the simulation, and provide feedback information.

2. The feedback-based and value-driven planning method according to claim 1, characterized in that: The step S1 comprises: Role definition: explain the role of the language model in task planning and provide role information; Output definition: define the data format that the language model output needs to meet, that is, define the language model output format; Provide examples: Provide examples of the specific task planning, that is, provide example information; Environment definition: explaining the scene content information used by the language model when performing the current task planning, that is, defining the environment information; Information feedback: after the simulation is performed on the result of the task planning, feedback information is provided according to the result of the simulation; and Task definition: Define the task content that needs to be completed for value-driven proactive task planning, that is, define task information.

3. The feedback-based and value-driven planning method according to claim 2, characterized in that: In the step S1, part of the structured data of the current scene is converted into a natural language description and enters the prompt word, and part of it enters the prompt word as JSON structured data.

4. The feedback-based and value-driven planning method according to claim 2, characterized in that: In the step 2, the task planning is performed by calling a local model or calling a commercial application program interface.

5. The feedback-based and value-driven planning method according to any one of claims 1 to 4, characterized in that: All completed results of the task planning are mapped into the skill library.

6. The feedback-based and value-driven planning method according to claim 5, characterized in that: The step S3 comprises: Verifying and correcting the output format of the language model, that is, correcting and reorganizing the output content of the language model into a data structure for parsing and operation; Map the parsed results of the task planning to the database, i.e., the skill library, and recall the actions of the agent. The step of mapping the result of the analyzed task planning to a database, i.e., the skill library, and recalling the action of the agent includes: Firstly, a vector model is used to embed the natural language results autonomously expressed by the language model and the actions used by the agent for the simulation, and the results of the task planning described in natural language are converted into vector representations. Subsequently, the similarity of the inner product of the embedded vectors obtained from the two is calculated, that is, the similarity is calculated by calculating the inner product of the vector obtained by the conversion and the vector of the action supported by the simulation in the database. Finally, the action whose similarity is greater than the expected matching threshold is selected as the final planning result to enter the next step of the simulation.

7. The feedback-based and value-driven planning method according to claim 6, characterized in that: In step S4, the feedback result includes the problem that the result of the task planning occurs when the simulation is performed:

1. Non-content errors: The result of the task planning does not meet the format requirements of the simulation and needs to be re-planned; 2. Content errors, including: a. The language model has a hallucination output problem when performing task planning, and cannot perform the simulation, and needs to be re-planned; b. The result of the task planning cannot improve the utility of the agent, and re-planning is required in combination with the current state; and c. The result of the task planning failed during the simulation, indicating unreasonable planning.

8. The feedback-based and value-driven planning method according to claim 7, characterized in that: If the simulation does not select an executable result for the planning result of the language model, the currently collected feedback information automatically enters the planning of the next task round to assist the planning of the agent, and at the same time, the agent maintains the current state and waits for the planning result of the next round; The simulation refers to the agent simulating the task sequence output by the language model. During the simulation, the agent interacts with the current virtual world scene according to the result of the task planning. For each result of the task planning, multiple rounds of simulation will be performed in parallel when performing the simulation. as well as If all the results of the task planning cannot be completed in the simulation stage, the information generated during the simulation is recorded and fed back to the language model in the next round of task planning, and is used to guide the language model to reflect through chain thinking, so as to refer to the task planning of the current scenario.

9. A computer device, characterized in that: The method comprises a memory, a first processor and a first computer program stored in the memory and executable on the first processor, wherein the first computer program implements the feedback-based and value-driven planning method described in any one of claims 1 to 8 when executed by the first processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a second computer program, and the second computer program can be executed by at least one second processor, so that the at least one second processor executes the feedback-based and value-driven planning method described in any one of claims 1-8.

Images