Object search method, system, medium and robot based on dynamic scene graph

Abstract

The present invention relates to the technical field of object search, and discloses an object search method, system, medium and robot based on a dynamic scene graph, including: obtaining a target search object; obtaining the posture information and visual angle information of the robot at the current moment; judging whether the visual angle information contains the target search object, if so, controlling the robot to move to the target search object; otherwise, combining the posture information and the visual angle information with the dynamic scene graph and the staged target object at the previous moment, integrating to obtain a prompt, based on the prompt, using a large language model to obtain the staged target object at the current moment; adding the posture information and the visual angle information at the current moment to the dynamic scene graph, controlling the robot to move to the staged target object, and returning to continue to obtain the posture information and the visual angle information. The robot can complete the object search task in an unknown environment.

Description

Technical Field

[0001] The present invention relates to the technical field of object search, and in particular to an object search method, system, medium and robot based on a dynamic scene graph. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In the process of robot object search, the environment model constructed is the map mapping. Existing map mapping mainly includes: scale map, topological map, hybrid map, etc. However, these maps lack the consideration of time dimension, have poor timeliness, are difficult to represent dynamically changing open scenes, and lack semantic information, making them difficult to be directly applied in robot autonomous object search.

[0004] In addition, robot environmental perception and exploration are the prerequisites for them to adapt to complex unknown environments. However, current environmental perception methods often focus on passive processing and handling of external information. Robots do not have the ability to actively perceive and understand environmental information and cannot complete object search tasks in unknown environments. Summary of the invention

[0005] In order to solve the above problems, the present invention provides an object search method, system, medium and robot based on a dynamic scene graph, which uses a pre-trained large language model to provide the robot with background knowledge and semantic understanding required for object search; uses a dynamic scene graph constructed in real time as an environment mapping for the robot to search for objects; the dynamic scene graph contains the location and object information of the environment explored by the robot, and the large language model has the ability to integrate and utilize this information for task planning; the two are combined and integrated to enable the robot to complete object search tasks in unknown environments.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A first aspect of the present invention provides an object search method based on a dynamic scene graph, comprising:

[0008] Get the target search object;

[0009] Obtain the robot's posture information and information within the viewing angle at the current moment; determine whether the information within the viewing angle contains the target search object, and if so, control the robot to move to the target search object; otherwise, integrate the posture information and the information within the viewing angle with the dynamic scene graph and the stage target object at the previous moment to obtain a prompt, and based on the prompt, use a large language model to obtain the stage target object at the current moment; the prompt includes: the stage target object at the previous moment, a description of the posture information, a description of the information within the viewing angle, and a description of the historical information perceived by the robot up to the current moment, and the historical information perceived by the robot up to the current moment is extracted from the dynamic scene graph;

[0010] After adding the current pose information and viewpoint information to the dynamic scene graph, the robot is controlled to move to the staged target object and return to continue to obtain the pose information and viewpoint information.

[0011] Furthermore, the method for acquiring the information within the viewing angle is: controlling the robot to rotate several times, taking the environmental information acquired during each rotation as a point cloud set, and taking all objects in the point cloud set as the information within the viewing angle.

[0012] Furthermore, if the objects in the point cloud sets obtained by two consecutive rotations have the same semantic class, the distance between the two is calculated, and if the distance is less than a specified value, they are regarded as the same object.

[0013] Furthermore, the large language model adopts ChatGPT.

[0014] A second aspect of the present invention provides an object search system based on a dynamic scene graph, comprising:

[0015] A task acquisition module is configured to: acquire a target search object;

[0016] The object search module is configured to: obtain the robot's posture information and in-view information at the current moment; determine whether the in-view information contains the target search object, and if so, control the robot to move to the target search object; otherwise, integrate the posture information and in-view information with the dynamic scene graph and the stage target object at the previous moment to obtain a prompt, and based on the prompt, use a large language model to obtain the stage target object at the current moment; the prompt includes: the stage target object at the previous moment, a description of the posture information, a description of the in-view information, and a description of the historical information perceived by the robot up to the current moment, and the historical information perceived by the robot up to the current moment is extracted from the dynamic scene graph;

[0017] The dynamic scene graph update module is configured to: add the current posture information and the information within the viewing angle to the dynamic scene graph, control the robot to move to the staged target object, and return to continue to obtain the posture information and the information within the viewing angle.

[0018] Furthermore, the method for acquiring the information within the viewing angle is: controlling the robot to rotate several times, taking the environmental information acquired during each rotation as a point cloud set, and taking all objects in the point cloud set as the information within the viewing angle.

[0019] Furthermore, if the objects in the point cloud sets obtained by two consecutive rotations have the same semantic class, the distance between the two is calculated, and if the distance is less than a specified value, they are regarded as the same object.

[0020] Furthermore, the large language model adopts ChatGPT.

[0021] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor and implements the steps of an object search method based on a dynamic scene graph as described above when the program is executed by the processor.

[0022] A fourth aspect of the present invention provides a robot that adopts an object search method based on a dynamic scene graph as described above.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] The present invention provides an object search method based on a dynamic scene graph, which uses a pre-trained large language model to provide a robot with background knowledge and semantic understanding required for object search; uses a dynamic scene graph constructed in real time as an environment mapping for the robot to search for objects; the dynamic scene graph contains the location and object information of the environment explored by the robot, and the large language model has the ability to integrate and utilize this information for task planning; the combination and integration of the two enables the robot to complete the assigned task efficiently and stably, which is of great significance for studying how to enable the robot to complete the object search task in an unknown environment.

[0025] The object search method based on a dynamic scene graph provided in this embodiment can help a robot performing object search to perceive and integrate object information in an unknown environment, and then, with the help of a large language model, construct a dynamic scene graph that is conducive to object search.

[0026] The present embodiment provides an object search method based on a dynamic scene graph, which can plan the best object search route for the robot in an open environment according to the object search requirements by using the constructed scene graph information and the pre-trained knowledge of the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings, which constitute a part of the specification of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but do not constitute limitations of the present invention.

[0028] Figure 1 This is a flow chart of an object search method based on a dynamic scene graph according to Embodiment 1 of the present invention;

[0029] Figure 2 This is a schematic diagram of the moving effect after constructing the scene graph according to the first embodiment of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0031] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0032] In the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other. The present invention is further described below with reference to the accompanying drawings and embodiments.

[0033] Embodiment 1

[0034] The purpose of the first embodiment is to provide an object search method based on a dynamic scene graph.

[0035] Large language models have demonstrated extraordinary capabilities in many fields. Among them, ChatGPT (Chat Generation Pre-trained Transformer) has attracted much attention for its outstanding reasoning ability demonstrated by its ultra-large-scale training parameters. In the field of robotics, it has become a new trend to integrate large language models represented by ChatGPT with various robots.

[0036] This embodiment combines ChatGPT to propose a method for embodied real-time construction of dynamic scene graphs, and applies the dynamic scene graph to robot object search; a method for combining the real-time constructed dynamic scene graph with ChatGPT, using the knowledge graph information stored in the exploration process and the semantic information under the current perspective, to obtain complete information input to the large language model for navigation intelligent planning; a perception method using ChatGPT to understand environmental information can combine the background knowledge of ChatGPT and the robot's mobility to achieve active perception of the environment.

[0037] This embodiment provides an object search method based on a dynamic scene graph, which is deployed on a robot platform developed based on the TESSE plug-in (Unity plug-in for developing robotics) in the Unity 3D (fully integrated professional game engine) simulation system. The TESSE plug-in has the following core functions: providing a robot agent; providing a depth camera and semantic segmentation; providing physical information such as the robot's position and speed; and being able to exchange data with Unity (game engine) through a network interface.

[0038] The present embodiment provides an object search method based on a dynamic scene graph, comprising the following steps:

[0039] Step 1: Robot platform settings.

[0040] Customize object attributes and control the semantic labels of objects in the scene through the TESSE plug-in. Divide the semantic labels in detail according to the needs of the robot's object search task.

[0041] TESSE provides two robot motion control methods: continuous control and discrete control. Suppose the robot needs to move from coordinate point A to coordinate point B. In continuous control, the robot moves smoothly from coordinate point A to coordinate point B. In discrete control, the robot stops at fixed position intervals when moving from coordinate point A to coordinate point B.

[0042] In order to enable the robot to efficiently collect environmental information and dynamically face the environment that may change, this embodiment is set to use a discrete control method when the robot looks around before taking action, so as to avoid the overlap of the robot's perspective when turning around and improve the efficiency of collecting information. When the robot moves, a continuous control method is used, so that the robot can update the information collected by the robot in real time when it approaches the staged target, effectively facing the possible changes in the environment.

[0043] Step 2: Set the task background for ChatGPT.

[0044] Before the robot moves, it describes the problem and inputs the prompt (Prompt D) to ChatGPT to set the navigation background.

[0045] For example, the prompt could be: You are a robot in an unfamiliar environment. You can obtain information about your own position and the surrounding environment from sensors... I will integrate this information... If I want you to get a glass of water... How should you go... Which object in this information should you get closer to?

[0046] This embodiment has modularized the design of the prompt words. For each task, it is only necessary to change the search area or object to assign the task to the robot.

[0047] Step 3: Concretely construct the scene graph method.

[0048] This embodiment proposes a method for constructing a dynamic scene graph by embodied exploration of the environment, so that the robot can perceive the environment efficiently and store effective information.

[0049] like Figure 2 As shown, this embodiment proposes that before the robot starts to move to search for a target, it performs five rotations of 60° each time in situ to collect all the environmental information around it and obtain a temporarily stored point cloud set.

[0050] (301) Before each turn, that is, before entering the next information collection, these temporarily stored point cloud sets are processed: the point cloud sets are clustered, and the point clouds of the same object are clustered into one category. The centroid coordinates of the cluster corresponding to a certain object are extracted, and combined with the robot's current posture information, the position coordinates of the object before the robot turns around are calculated.

[0051] (302) After turning around, the point cloud set is acquired again. If the objects in the point cloud set obtained by two consecutive rotations have the same semantic class, the Euclidean distance between the two coordinates is calculated. If the Euclidean distance between the two coordinates is less than the value specified in this embodiment, it is considered that they are the same object, and the position coordinates calculated before turning around are updated to the position coordinates of the object after turning around.

[0052] Repeat steps (301) and (302) in this cycle, and use the objects in the point cloud set obtained by five rotations as the information within the current view (hereinafter referred to as the information within the current view), and the robot will obtain the information of all objects in the surrounding environment. ,in, Indicates the first n objects, including the semantic class of the object (obj:name n ) and position coordinates (pos: ).

[0053] The robot will integrate the information collected from all objects in the surrounding environment with the pre-set prompt module and input it into ChatGPT. According to the object coordinate output of ChatGPT, the robot will move to the target coordinate.

[0054] In this embodiment, the semantic class collects the depth image of the object through the depth camera carried by the robot, converts the object depth image into a point cloud set, and then clusters the point cloud set. The point cloud of the same object is clustered into one class. For a certain object, the RGB (red, yellow, and blue) value of the point cloud of the object is compared with the semantic class-RGB list file constructed in advance to obtain the semantic class of the object.

[0055] Step 4: Object search method.

[0056] Step 401: Acquire the target search object.

[0057] Specifically, the task issued can be one or more of the target search object (such as a water cup), action (such as fetching water) and target area (such as a kitchen). If the task issued is an action and a target area, such as going to the kitchen to fetch water, the large language model can directly obtain the target search object (such as a water cup).

[0058] Step 402: After obtaining the current position and posture information of the robot, the robot performs five rotations of 60° each time to collect all the environmental information around it and obtain a temporarily stored point cloud set. Before each turn, the point cloud set is clustered, and the point clouds of the same object are clustered into one category. The centroid coordinates of the cluster corresponding to a certain object are extracted, and the position coordinates of the object before the robot turns around are calculated in combination with the current position and posture information of the robot. If the objects in the point cloud set obtained before and after the turn have the same semantic class, the Euclidean distance between the two is calculated based on the position coordinates. If the Euclidean distance of the two coordinates is less than the specified value, it is considered that this is the same object. In this way, the robot obtains the information of all objects in the surrounding environment (information in the current perspective) O-obj, that is, .

[0059] Step 403, determine whether the information in the current viewing angle contains the target search object. If so, control the robot to move to the target search object and the task ends; otherwise, go to step 404.

[0060] Step 404: Figure 1 As shown in the figure, the robot posture information at the current moment, the information in the current field of view, the stage target object and dynamic scene graph output by ChatGPT to the robot at the previous moment are integrated to obtain the prompt Prompt (R-xyz, O-obj, H-msg, S-obj-1) input to the large language model ChatGPT, where R-xyz represents the robot posture information, O-obj represents the information in the field of view at the current moment, H-msg represents the historical information perceived by the robot up to the current moment (all object information obtained from the dynamic scene graph up to now), and S-obj-1 represents the stage target object output by ChatGPT to the robot at the previous moment.

[0061] Step 405: Use each prompt to interact with ChatGPT to update the phased target object - .

[0062] like Figure 2As shown in the figure, when the robot is at the starting point, it first collects surrounding information and then interacts with ChatGPT (GPT for short). GPT will select the best action plan based on the background settings and training information to update the staged target object. - For object 7.

[0063] like Figure 2 In the scene shown, there are 5 rooms and a corridor. The starting point and object 7 are located in the corridor. The room corresponding to door 1 contains objects 1 and 2. The room corresponding to door 2 contains objects 3, 4 and 5. The room corresponding to door 3 contains object 6. The room corresponding to door 4 contains objects 8 and 9. The room corresponding to door 5 contains objects 10, 11 and the target object (object 12).

[0064] Step 406: Add the current posture information and the in-view information to the dynamic scene graph, and update the dynamic scene graph.

[0065] Step 407 , control the robot to move to the staged target object, and return to step 402 .

[0066] like Figure 2 As shown, the interaction with GPT is carried out directly, and the target object (object 7) is moved to the target object (target object) at the target object 7. GPT will select the best action plan based on the background setting and training information to update the target object at the target object. - For object 10, explore object 10 until the target object appears in the field of view.

[0067] It should be noted that the robot will collect point clouds in real time during the movement, and sense and identify whether there is a door in the point cloud. If there is a door, the position coordinates of the door are calculated in combination with the robot posture. If the robot passes through the position coordinates of the door, the stage target object is not in the same room as the position before the robot moved. At this time, if the position coordinates of the door are not in the dynamic scene graph, a new room is constructed in the dynamic scene graph, and the robot posture information and the information in the perspective obtained next are added to the new room in the dynamic scene graph. If the position coordinates of the door are in the dynamic scene graph, the room corresponding to the position coordinates of the door is retrieved, and the robot posture information and the information in the perspective obtained next are added to the retrieved room. If the Euclidean distance between the coordinates of an object in the newly added perspective information and the coordinates of an existing object is less than the value specified in this embodiment, the coordinates of the object are updated to the coordinates of the object in the newly added perspective information.

[0068] It should be noted that after the robot enters a room, when the number of stage target objects searched exceeds the set value, it returns to the last stage target object reached before entering the room.

[0069] The modular prompt Prompt is recorded as , then the system design of robot input and ChatGPT output is: - = { ( - , -obj, - , - - )}.

[0070] Among them, the robot’s posture information - , which represents the coordinates of the robot, obtained from the information of the depth camera and inertial sensor. The specific prompt description is: "…If you already know your coordinates are …". Among them, If you already know your coordinates are It means if you already know the coordinates are {(x,y,z)}.

[0071] Among them, the information within the robot’s current perspective -obj, obtained from the first layer of the dynamic scene graph, the metric semantic grid. The specific prompt description is expressed as: "…outside of your field of view, there are:{{obj: name1, pos: }, {obj: name2, pos: }…{obj: name n , pos: }}…”, where “outside of your field of view, there are:” means “within your field of view, there are:”. obj: the collected object; name1: the category name of the first object collected; name2: the category name of the second object collected; and so on…; (x1, y1, z1): the coordinates of the first object collected; (x2, y2, z2): the coordinates of the second object collected; and so on….

[0072] Among them, the historical information perceived by the robot up to the current moment - , obtained from the backend that builds the dynamic scene graph. The specific prompt description is: "...{room1}, which also includes{{obj: name1, pos: : }, {obj: name2, pos: }...";"...{room2}, which also includes{{obj: name1, pos: : }, {obj: name2, pos: }…”, where “{room i}, which also includes{}” means “there are {} in the i-th room”.

[0073] in, - - It represents the interim target object that the last large language model output to the robot. This step enables ChatGPT to understand its own historical output, fine-tune the model itself, and make the next output of the task more accurate.

[0074] This embodiment provides an object search method based on a dynamic scene graph, in which a robot uses a large language model and a dynamic scene graph constructed in real time to efficiently complete a regional object search task in a completely unknown environment.

[0075] This embodiment provides an object search method based on a dynamic scene graph. The robot can perceive environmental information in real time, understand the position of itself and surrounding objects through its own radar sensor, continuously update the dynamic scene graph of the environment, and store various valid information.

[0076] This embodiment provides an object search method based on a dynamic scene graph. The robot needs to dynamically interact with a large language model in real time according to information transformation to understand different navigation instruction information.

[0077] This embodiment provides an object search method based on a dynamic scene graph. The robot perceives environmental information, builds an effective dynamic scene graph in real time and integrates its information into a large language model for utilization, and plans phased goals according to the task.

[0078] This embodiment provides an object search method based on a dynamic scene graph, which uses a pre-trained large language model ChatGPT to provide the robot with the background knowledge and semantic understanding required for object search; and uses a dynamic scene graph constructed in real time as an environmental mapping for the robot to search for objects. The dynamic scene graph contains the location and object information of the environment explored by the robot, and the large language model has the ability to integrate and use this information for task planning; combining the two enables the robot to complete the assigned tasks efficiently and stably, which is of great significance for studying how to enable the robot to complete object search tasks in unknown environments.

[0079] The object search method based on a dynamic scene graph provided in this embodiment can help a robot performing object search to perceive and integrate object information in an unknown environment, and then, with the help of ChatGPT, construct a dynamic scene graph that is conducive to object search.

[0080] This embodiment provides an object search method based on a dynamic scene graph, which can plan the best object search route for the robot in an open environment according to the object search requirements by using the constructed scene graph information and the pre-trained knowledge of ChatGPT.

[0081] Embodiment 2

[0082] The purpose of this second embodiment is to provide an object search system based on a dynamic scene graph.

[0083] A task acquisition module is configured to: acquire a target search object;

[0084] The object search module is configured to: obtain the robot's posture information and in-view information at the current moment; determine whether the in-view information contains the target search object, and if so, control the robot to move to the target search object; otherwise, integrate the posture information and in-view information with the dynamic scene graph and the stage target object at the previous moment to obtain a prompt, and based on the prompt, use a large language model to obtain the stage target object at the current moment; the prompt includes: the stage target object at the previous moment, a description of the posture information, a description of the in-view information, and a description of the historical information perceived by the robot up to the current moment, and the historical information perceived by the robot up to the current moment is extracted from the dynamic scene graph;

[0085] The dynamic scene graph update module is configured to: add the current posture information and the information within the viewing angle to the dynamic scene graph, control the robot to move to the staged target object, and return to continue to obtain the posture information and the information within the viewing angle.

[0086] The method for acquiring the information within the viewing angle is as follows: controlling the robot to rotate several times, taking the environmental information acquired during each rotation as a point cloud set, and taking all objects within the point cloud set as the information within the viewing angle.

[0087] Among them, if the objects in the point cloud set obtained by two consecutive rotations have the same semantic class, the distance between the two is calculated. If the distance is less than the specified value, they are regarded as the same object.

[0088] Among them, the large language model adopts ChatGPT.

[0089] It should be noted here that each module in this embodiment corresponds to each step in Example 1 one by one, and the specific implementation process is the same, which will not be repeated here.

[0090] Embodiment 3

[0091] This embodiment provides a computer-readable storage medium on which a computer program is stored. The program is executed by a processor. When the program is executed by the processor, the steps in the object search method based on a dynamic scene graph as described in the first embodiment above are implemented.

[0092] Embodiment 4

[0093] This embodiment provides a robot that uses the object search method based on a dynamic scene graph as described in the first embodiment above.

[0094] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0095] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. An object search method based on a dynamic scene graph, characterized in that: include: Get the target search object; Get the robot's current position information and viewing angle information; Determine whether the information in the field of view contains the target search object. If so, control the robot to move to the target search object; otherwise, combine the posture information and the information in the field of view with the dynamic scene graph and the stage target object at the previous moment to obtain a prompt. Based on the prompt, use the large language model to obtain the stage target object at the current moment. The prompt includes: a description of the stage target object and posture information at the previous moment, a description of the information within the viewing angle, and a description of the historical information perceived by the robot up to the current moment, and the historical information perceived by the robot up to the current moment is extracted from the dynamic scene graph; After adding the current posture information and the information within the field of view to the dynamic scene graph, the robot is controlled to move to the staged target object, and then returns to continue to obtain the posture information and the information within the field of view; The method for obtaining the information within the viewing angle is as follows: controlling the robot to rotate several times, taking the environmental information obtained during each rotation as a point cloud set, and taking all objects in the point cloud set as the information within the viewing angle; If the objects in the point cloud set obtained by two consecutive rotations have the same semantic class, the distance between them is calculated. If the distance is less than the specified value, they are regarded as the same object; The large language model adopts ChatGPT.

2. An object search system based on a dynamic scene graph, characterized in that: include: A task acquisition module is configured to: acquire a target search object; The object search module is configured to: obtain the robot's current position information and visual information; determine whether the visual information contains the target search object, and if so, control the robot to move to the target search object; otherwise, combine the position information and the visual information with the dynamic scene graph and the stage target object at the previous moment to obtain a prompt, and based on the prompt, use a large language model to obtain the stage target object at the current moment; The prompt includes: a description of the stage target object and posture information at the previous moment, a description of the information within the viewing angle, and a description of the historical information perceived by the robot up to the current moment, and the historical information perceived by the robot up to the current moment is extracted from the dynamic scene graph; The dynamic scene graph update module is configured to: add the current pose information and the in-view information to the dynamic scene graph, control the robot to move to the staged target object, and return to continue to obtain the pose information and the in-view information; The method for obtaining the information within the viewing angle is as follows: controlling the robot to rotate several times, taking the environmental information obtained by each rotation as a point cloud set, and taking all objects in the point cloud set as the information within the viewing angle; if the objects in the point cloud sets obtained by two consecutive rotations have the same semantic class, then calculating the distance between the two, and if the distance is less than a specified value, they are regarded as the same object; The large language model adopts ChatGPT.

3. A computer-readable storage medium having a computer program stored thereon, the program being executed by a processor, characterized in that: When the program is executed by a processor, the steps of the object search method based on a dynamic scene graph as claimed in claim 1 are implemented.

4. A robot, characterized in that: An object search method based on a dynamic scene graph as described in claim 1 is adopted.

Images