Verification method, system, electronic device and storage medium based on scene graph navigation

Abstract

The embodiment of the present invention provides a verification method, system, electronic device and storage medium based on scene graph navigation, which belongs to the field of navigation. The method includes: in the navigation planning stage, the navigation task instruction, the constructed scene graph and the first prompt text are input into the large language model, and the action sequence for implementing the task instruction is output; in the scoring verification stage, the action sequence output by the large language model is parsed to generate corresponding action instructions and operation objects; the task instruction, scene graph, action instruction and operation object scene graph navigation are verified. By proposing a navigation action sequence scoring verification mechanism, the executability of the action sequence generated by the large language model is comprehensively evaluated, and the generation of erroneous or misleading instructions is effectively avoided, thereby improving the efficiency and success rate of robot navigation; by means of multiple iterative evaluations, the instability of the output of the large language model is effectively avoided, and the credibility of the large language model is improved.

Description

Technical Field

[0001] The present invention relates to the field of navigation technology, and in particular to a verification method, system, electronic device and storage medium based on scene graph navigation. Background Art

[0002] Robot SLAM technology is a method that enables robots to collect data in unknown environments with the help of their own sensors, analyze these data through specific algorithms to determine their position and posture, and build environmental maps. It is the basis for realizing autonomous navigation of robots. In recent years, thanks to the leapfrog development of information technologies such as artificial intelligence and knowledge graphs, scene graphs, as a new technical path, provide new solutions for the positioning and navigation of indoor robots by converting the robot's environment into a multidimensional relationship graph composed of nodes and edges. Traditional map-based navigation methods are limited by the accuracy and update efficiency of maps, and are difficult to cope with unknown or changing environments. However, scene graphs provide a more detailed and flexible way to express scenes, which not only includes the spatial position and characteristics of objects, but also captures the complex relationships between objects. During the operation of indoor robots, through the onboard visual cameras and IMU and other sensors, the robot can perceive and identify objects in the scene in real time, create corresponding nodes for each object in the scene graph, and give these nodes rich visual features such as color, texture and material; at the same time, by adding edges to the spatial relationship between adjacent objects, a complete scene graph is formed. As the robot's perceived environment changes, the scene map will continue to be updated and improved to support more accurate and efficient indoor navigation tasks.

[0003] In scene graph navigation based on a large language model, the large language model fully understands the scene graph data and formulates a series of action sequences from the robot's current position to the task target point. However, the instability of the large language model always affects the execution effect of the action sequence. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a verification method, system, electronic device and storage medium based on scene graph navigation, which are used to solve the technical problems existing in the above-mentioned prior art.

[0005] In order to achieve the above object, an embodiment of the present invention provides a verification method based on scene graph navigation, comprising:

[0006] In the navigation planning stage, the navigation task instruction, the constructed scene graph and the first prompt text are input into the large language model, and the action sequence to implement the task instruction is output;

[0007] In the scoring and verification phase, the action sequence output by the large language model is parsed to generate corresponding action instructions and operation objects;

[0008] The task instruction, the scene graph, the action instruction, the operation object and the second prompt text are input into the large language model, the empirical probability is output, and the scene graph navigation is verified based on the output empirical probability.

[0009] Optionally, the first prompt text includes but is not limited to roles, instructions, scene maps, positioning points, object execution operations, outputs and examples, wherein the role representation is the role positioning that defines the large language model, the instruction representation is the task of performing navigation, the scene map representation is the current scene information, the positioning point representation is the current position of the robot, the object execution operation representation is the behavior control of the objects in the current scene, the output representation is the generated action sequence and its thought chain, and the example representation is the input and output reference according to the specific instruction;

[0010] The second prompt text is characterized as a verification instruction.

[0011] Optionally, unreasonable action instructions include, but are not limited to, unexecutable action instructions, actions that are inconsistent with the current state, and robot positioning loss.

[0012] Optionally, verification of scene graph navigation based on the output empirical probability includes:

[0013] Based on the output empirical probability, determine whether the action instruction is reasonable. If it is reasonable, execute the action instruction. If it is unreasonable, return to the navigation planning stage to re-plan the task and continue to output the action sequence of the task instruction.

[0014] Optionally, determine the empirical probability according to the following formula:

[0015] ;

[0016] In the formula, Represents the large language model The verification results of the times, The number of times the large language model needs to be repeatedly verified. is the scene graph data, For task instructions, and It is the room or object operation object corresponding to the action instruction.

[0017] On the other hand, the present invention also provides a verification system based on scene graph navigation, comprising:

[0018] A navigation planning module is used to input the navigation task instruction, the constructed scene graph and the first prompt text into the large language model during the navigation planning stage, and output an action sequence to implement the task instruction;

[0019] A generation module, used to parse the action sequence output by the large language model in the scoring and verification stage, and generate corresponding action instructions and operation objects;

[0020] A verification module is used to input the task instruction, the scene graph, the action instruction, the operation object and the second prompt text into the large language model, output the empirical probability, and verify the scene graph navigation based on the output empirical probability.

[0021] Optionally, the verification module is specifically used to determine whether the action instruction is reasonable based on the output empirical probability. If it is reasonable, execute the action instruction; if it is unreasonable, return to the navigation planning stage to re-plan the task and continue to output the action sequence of the task instruction.

[0022] Optionally, determine the empirical probability according to the following formula:

[0023] ;

[0024] In the formula, Represents the large language model The verification results of the times, The number of times the large language model needs to be repeatedly verified. is the scene graph data, For task instructions, and It is the room or object operation object corresponding to the action instruction.

[0025] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned verification method based on scene graph navigation when executing the program.

[0026] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned verification method based on scene graph navigation.

[0027] Through the above technical solution, a navigation action sequence scoring and verification mechanism based on a large language model is proposed to comprehensively evaluate the executableness of the action sequence generated by the large language model, effectively avoid the generation of erroneous or misleading instructions, and improve the efficiency and success rate of robot navigation; through multiple iterative evaluations, the instability of the large language model output is effectively avoided, the credibility of the large language model is improved, and a reliable basis is provided for the evaluation of navigation action instructions.

[0028] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

[0030] Figure 1 This is a flowchart of an implementation of a verification method based on scene graph navigation provided by an embodiment of the present invention.

[0031] Figure 2 It is a schematic diagram of a verification method based on scene graph navigation provided by an embodiment of the present invention;

[0032] Figure 3 It is a structural diagram of a verification system based on scene graph navigation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

[0034] See also Figure 1 As shown, it is a flowchart of an implementation of a verification method based on scene graph navigation provided by an embodiment of the present invention, including the following execution steps:

[0035] Step 100: In the navigation planning stage, the navigation task instruction, the constructed scene graph and the first prompt text are input into the large language model, and the action sequence for implementing the task instruction is output.

[0036] It should be noted that the first prompt text includes but is not limited to roles, instructions, scene maps, positioning points, object execution operations, outputs and examples, wherein the role representation is the role positioning that defines the large language model, the instruction representation is the task of performing navigation, the scene map representation is the current scene information, the positioning point representation is the current position of the robot, the object execution operation representation is the behavior control of objects in the current scene, the output representation is the generated action sequence and its thinking chain, and the example representation is the input and output reference according to the specific instruction; the second prompt text representation is the verification instruction.

[0037] In some embodiments, the scene map is a text data in JSON format that describes the current scene information and consists of objects, rooms, floors and their corresponding attributes; the positioning point is the current position of the robot; object execution operations include grabbing, dropping, opening, closing objects in the scene, action instructions to go to a specified room, etc.; the output includes the generated action sequence and its thought chain; the example lists the input and output references of specific navigation instructions.

[0038] For example, the first prompt text is as follows:

[0039] Agent role: You are an excellent graph planning agent. Given a scenegraph, you can use this graph to generate a step-by-step task plan that the agent can follow to solve a given instruction.

[0040] Instruction: Natural language description of the task.

[0041] 3D Scene Graph: Text-serialized json description of a 3D scene graph.

[0042] Locate_floor_room: The floor and room where the agent is currently located.

[0043] Environment Functions:

[0044] goto(room / floor): Move the agent to any room or floor node.

[0045] access(object): Provide access to the set of affordances associated with an object.

[0046] pickup(object): Pick up an accessible object from the accessed node.

[0047] release(object): Release grasped object at an object node.

[0048] turn_on / off(object): Toggle object at agent's node, if accessible andhas affordance.

[0049] open / close(object): Open / close object at agent's node, affectingobject accessibility.

[0050] done(): Call when the task is completed.

[0051] Output Response Format:

[0052] "chain_of_thought": break your problem down into a series ofintermediate reasoning steps to help you determine your next command.

[0053] "room_node": room node to perform an operation on.

[0054] "floor_node": floor_node to perform an operation on.

[0055] "plan": task plan

[0056] Examples:

[0057] {'role': 'user,

[0058] 'content': {

[0059] 'Instruction' : 'Find a dining room without a bow1.',

[0060] '3D Scene Graph' :{"nodes":[{"floorB": {"room":[{"24":"lobby"},{"17":"dining_room",

[0061] {"objects": [{"21": "bottle"}, {"87": "dining table"}]}}, {"objects":[{"7": "bottle"},

[0062] {"18": "bowl"}],"20": "kitchen"}, {"22": "living_room"}, {"28": "staircase"}]}},

[0063] {"floorA":{"room":[{"11": "bedroom"}, {"18": "living_room"},{"16": "dining_room",

[0064] "objects": [{"51": "dining_table"}, {"68": "chair"}]},{"2": "lobby"}]}}]},

[0065] 'Locate_floor_room' : ['floorB','kitchen']

[0066] }},

[0067] {'role': 'assistant',

[0068] 'content': {

[0069] 'chain_of_thought': 'According to the given graph, there are no bowlsin the dining_room on

[0070] floorA, which meets the task requirements. ->I will start planningmode>go to dining

[0071] room on floor A',

[0072] 'plan':'goto(staircase_B)>goto(floorA)>goto(dining_room)>done',

[0073] }}

[0074] Step 101: In the scoring and verification phase, the action sequence output by the large language model is parsed to generate corresponding action instructions and operation objects.

[0075] Step 102: Input the task instruction, the scene graph, the action instruction, the operation object and the second prompt text into the large language model, output the empirical probability, and verify the scene graph navigation based on the output empirical probability.

[0076] Among them, the verification of scene graph navigation based on the output empirical probability includes: determining whether the action instruction is reasonable based on the output empirical probability, if reasonable, executing the action instruction, if unreasonable, returning to the navigation planning stage to re-plan the task, and continuing to output the action sequence of the task instruction.

[0077] It should be noted that unreasonable action instructions include but are not limited to unexecutable action instructions, actions that are inconsistent with the current state, and robot positioning loss.

[0078] For example, for the task of "taking ice cream out of the refrigerator", the reasonable action should be to open the refrigerator first, then grab the ice cream, and finally close the refrigerator. However, if the large language model outputs the action of grabbing the ice cream directly without opening the refrigerator or grabbing the ice cream without closing the refrigerator, it is obviously not in line with daily behavioral norms. These are all unexecutable actions; the action is inconsistent with the current state. When the robot has already grabbed an object and the gripper is occupied, it is also unreasonable for the large language model to generate instructions to grab other objects. This is inconsistent with the state that the gripper is already occupied. The object in the gripper must be put down before executing the instruction to grab other objects; positioning is lost. When the robot loses its positioning during navigation, the action instruction should be canceled. Incorrect positioning makes all operations futile.

[0079] In some implementations, the action sequence planned by the large language model is parsed and disassembled to generate corresponding action instructions and their operation objects. The action instructions are to perform operations on the objects defined in the prompt text; the operation objects are the rooms or object nodes in the scene graph data. The obtained action instructions, operation objects, task instructions and scene graph data are input into the large language model together. To improve the reliability of the output of the large language model, the large language model is continuously communicated with for multiple times, and the rationality of each action instruction is evaluated for multiple times. The evaluation results of each time are summarized to obtain the final empirical probability. Action Commands The empirical probability Defined as:

[0080] ;

[0081] In the formula, Represents the large language model The verification results of the times, The number of times the large language model needs to be repeatedly verified. is the scene graph data, For task instructions, and It is the room or object operation object corresponding to the action instruction.

[0082] The solution proposed in this application aims to improve the reliability of the navigation path output by the large language model and avoid the occurrence of unexecutable action sequences. Figure 2 As shown in the figure, when performing robot navigation, the large language model first fully understands the input scene map and task instructions, and outputs a reasonable and efficient action sequence based on the pre-built prompt text. Then, the action sequence is disassembled and fully analyzed, and potentially unreasonable action instructions are sent to the large language model and the rationality of the action is repeatedly asked, thereby obtaining a series of evaluation results (empirical probability), integrating all results and summarizing the score as the basis for the executableness of the action (if the empirical probability is reasonable, the navigation task is executed, and if it is unreasonable, the navigation plan is returned). Through the constructed prompt text and scoring verification mechanism, the uncertainty in the response is effectively reduced before and after the large language model plans the path, further enhancing the credibility of the large language model and avoiding providing incorrect or misleading instructions that may cause the navigation task to fail.

[0083] See also Figure 3 As shown, it is a structural schematic diagram of a verification system based on scene graph navigation provided by an embodiment of the present invention, including:

[0084] The navigation planning module 300 is used to input the navigation task instruction, the constructed scene graph and the first prompt text into the large language model during the navigation planning stage, and output an action sequence to implement the task instruction;

[0085] A generation module 301 is used to parse the action sequence output by the large language model and generate corresponding action instructions and operation objects during the scoring and verification phase;

[0086] The verification module 302 is used to input the task instruction, the scene graph, the action instruction, the operation object and the second prompt text into the large language model, output the empirical probability, and verify the scene graph navigation based on the output empirical probability.

[0087] In some embodiments, the verification module 302 is specifically used to determine whether the action instruction is reasonable based on the output empirical probability. If it is reasonable, execute the action instruction; if it is unreasonable, return to the navigation planning stage to re-plan the task and continue to output the action sequence of the task instruction.

[0088] In some embodiments, the empirical probability is determined according to the following formula:

[0089] ;

[0090] In the formula, Represents the large language model The verification results of the times, The number of times the large language model needs to be repeatedly verified. is the scene graph data, For task instructions, and It is the room or object operation object corresponding to the action instruction.

[0091] The system proposes a navigation action sequence scoring and verification mechanism based on a large language model, which comprehensively evaluates the executableness of the action sequence generated by the large language model, effectively avoids the generation of erroneous or misleading instructions, and improves the efficiency and success rate of robot navigation. It uses multiple iterative evaluations to effectively avoid the instability of the large language model output, improve the credibility of the large language model, and provide a reliable basis for the evaluation of navigation action instructions.

[0092] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the scene graph navigation-based verification method described in any one of the above-mentioned embodiments when executing the program.

[0093] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the verification method based on scene graph navigation described in any of the above embodiments are implemented.

[0094] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0095] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0096] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0097] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0098] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0099] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0100] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0101] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0102] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the present application.

Claims

1. A verification method based on scene graph navigation, characterized in that: include: In the navigation planning stage, the navigation task instruction, the constructed scene graph and the first prompt text are input into the large language model, and the action sequence to implement the task instruction is output; In the scoring and verification phase, the action sequence output by the large language model is parsed to generate corresponding action instructions and operation objects; Inputting the task instruction, the scene graph, the action instruction, the operation object and the second prompt text into the large language model, outputting the experience probability, and verifying the scene graph navigation based on the output experience probability; The first prompt text includes but is not limited to roles, instructions, scene graphs, positioning points, object execution operations, outputs and examples, and the second prompt text is characterized as a verification instruction; Among them, the verification of scene graph navigation based on the output empirical probability includes: Determine whether the action instruction is reasonable based on the output experience probability, if reasonable, execute the action instruction, if unreasonable, return to the navigation planning stage to re-plan the task, and continue to output the action sequence of the task instruction; The empirical probability is determined according to the following formula: ； In the formula, Represents the large language model The verification results of the times, The number of times the large language model needs to be repeatedly verified. is the scene graph data, For task instructions, and is the room or object operation object corresponding to the action instruction. Indicates An action instruction.

2. The verification method according to claim 1, characterized in that: The role representation is to define the role positioning of the large language model, the instruction representation is to perform the navigation task, the scene map representation is the current scene information, the positioning point representation is the current position of the robot, the object execution operation representation is the behavior control of the objects in the current scene, and the output representation is the generated action sequence and its thinking chain.

3. The verification method according to claim 1, characterized in that: Unreasonable action instructions include but are not limited to unexecutable action instructions, actions that are inconsistent with the current state, and robot positioning loss.

4. A verification system based on scene graph navigation, characterized in that: include: A navigation planning module is used to input the navigation task instruction, the constructed scene graph and the first prompt text into the large language model during the navigation planning stage, and output an action sequence to implement the task instruction; A generation module, used to parse the action sequence output by the large language model in the scoring and verification stage, and generate corresponding action instructions and operation objects; A verification module, used for inputting the task instruction, the scene graph, the action instruction, the operation object and the second prompt text into the large language model, outputting the empirical probability, and verifying the scene graph navigation based on the output empirical probability; The first prompt text includes but is not limited to roles, instructions, scene graphs, positioning points, object execution operations, outputs and examples, and the second prompt text is characterized as a verification instruction; The verification module is specifically used to: determine whether the action instruction is reasonable based on the output experience probability, if reasonable, execute the action instruction, if unreasonable, return to the navigation planning stage to re-plan the task, and continue to output the action sequence of the task instruction; Among them, the empirical probability is determined according to the following formula: ； In the formula, Represents the large language model The verification results of the times, The number of times the large language model needs to be repeatedly verified. is the scene graph data, For task instructions, and is the room or object operation object corresponding to the action instruction. Indicates An action instruction.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the verification method based on scene graph navigation as described in any one of claims 1-3 are implemented.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the verification method based on scene graph navigation as described in any one of claims 1 to 3 are implemented.

Images