Robot autonomous planning method based on scene graph

Abstract

The invention provides a robot autonomous planning method based on a scene graph, and the method comprises the steps: S1, reconstructing a three-dimensional point cloud of a scene based on a multi-view image, and carrying out the over-segmentation of the three-dimensional point cloud, and obtaining a plurality of point cloud segments; meanwhile, extracting a multi-view boundary diagram from the multi-view image; s2, calculating the multi-view boundary affinity between any two adjacent point cloud fragments based on the multi-view boundary graph; s3, based on the multi-view boundary affinity, performing region growing clustering on all the point cloud fragments to obtain object instances in the scene; and S4, endowing each object instance with semantic information, and constructing a Voronoi diagram based on the spatial positions of all the object instances. According to the method, description of spatial information and open semantic features of object instances in scene topology is utilized, a large language model is assisted to plan tasks and behaviors in combination with the ability of a robot, and the problem that task planning and the actual environment are disjointed generally existing in a large language model planner is solved.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of robot perception, in particular to a robot autonomous planning method based on a scene graph. BACKGROUND

[0002] In recent years, the research on robot task planning and operation technology mainly focuses on the combination of embodied intelligence technology and robot system, and the research on embodied intelligence technology mainly focuses on the two abilities of movement and operation.

[0003] In terms of movement ability, the research on embodied intelligence technology mainly focuses on how to analyze the movement demand of the target task based on the existing SLAM technology, calculate the corresponding target position space coordinates, and drive the robot to move through SLAM. Early related research mainly uses the occupancy map in SLAM as the basis for retrieval, and guides the robot to carry out embodied navigation by adding some original reference data, for example, SemExp uses MaskRCNN to project the semantic category label extracted from RGB to the occupancy map, realizes the binding of the map and the semantic information of each position, and uses the semantic map navigation to better guide the movement of the embodied agent than the pure occupancy map navigation.

[0004] In terms of operation ability, among such methods, ALOHA proposes an action sequence prediction (ACT) method to enable the robot to learn fine operation actions through 10 minutes of training. Subsequently, RoboAgent introduces MT-ACT for semantic enhancement, enabling more accurate learning of multiple tasks. In addition to action sequence learning, using a generative model is also a model training method, and Diffusion Policy uses a conditional diffusion model to train a success rate that is significantly higher than that of the previous method on 15 tasks.

[0005] Although a large number of previous works have respectively improved the movement and operation abilities of the robot, for a specific robot task, due to the lack of understanding of the current scene by the robot, there is still a lack of methods that can reasonably combine the movement and operation abilities, hindering the autonomous execution of the robot task. SUMMARY

[0006] The problem solved by the present application is how to avoid the disconnection between the planning result of the large language model planner and the actual environment due to the lack of global understanding of the scene.

[0007] To solve the above problems, the present application provides a robot autonomous planning method and system based on a scene graph, an electronic device and a storage medium.

[0008] In a first aspect, the present application provides a robot autonomous planning method based on a scene graph, comprising the following steps: S1: reconstruct a three-dimensional point cloud of a scene based on multi-view images, and perform over-segmentation on the three-dimensional point cloud to obtain a plurality of point cloud segments; and extract a multi-view boundary map from the multi-view images; S2: calculate a multi-view boundary affinity between any two adjacent point cloud segments based on the multi-view boundary map; S3: perform region growing clustering on all point cloud segments based on the multi-view boundary affinity to obtain object instances in the scene; S4: assign semantic information to each of the object instances, and construct a Voronoi graph based on the spatial positions of all object instances, and further generate a scene topology graph taking object instances as nodes and spatial relationships between objects as edges; S5: receive a natural language task, and decompose the natural language task into a plurality of sub-tasks using a large language model; S6: for each sub-task, retrieve relevant object instances from the scene topology graph based on the task semantics thereof to form a task sub-graph; S7: generate an execution plan containing a robot skill sequence associated with the object instances in the task sub-graph based on the task sub-graph.

[0009] Optionally, in the step S2, the multi-view boundary affinity of two point cloud segments is calculated, comprising: For a single view, the average value of the pixel boundary affinity between all pairs of pixel points in the projected pixel set of the two point cloud segments is calculated as the boundary affinity under the view; The pixel boundary affinity is calculated as follows: ; In the formula, and are two pixel points, represents the line segment connecting and , and represents the probability that the pixel point belongs to the boundary. The average value of the boundary affinity under all valid views is calculated to obtain the multi-view boundary affinity.

[0010] Optionally, in the step S3, the process of the region growing clustering is as follows: The multi-view boundary affinity is taken as the growing criterion, and when the multi-view boundary affinity of two adjacent point cloud segments is higher than a preset threshold, they are clustered into the same object instance.

[0011] Optionally, in the step S4, the semantic information is assigned to each of the object instances, comprising: back-projecting the object instance to the original multi-view images; calculating a semantic vector of the corresponding projection region under each view using a vision-language model; averaging the semantic vectors of all views to aggregate the semantic information of the object instance.

[0012] Optionally, in the step S4, the constructing the Voronoi diagram comprises: taking the point cloud barycenter of each object instance as a generating point; calculating a perpendicular bisector plane of each pair of adjacent generating points, and dividing the three-dimensional space by the perpendicular bisector plane to form the Voronoi diagram; wherein, if the distance between two object instances exceeds a preset distance threshold, the potential relationship between them is ignored.

[0013] Optionally, in the step S6, the forming the task subgraph comprises: calculating the similarity between the task semantics of a subtask and the semantic information of each object instance in the scene topology graph; retrieving the object instances related to the subtask based on the similarity; pruning the scene topology graph to retain only the related object instances and the object instances directly connected thereto, thereby forming the task subgraph.

[0014] Optionally, the calculation of the semantic similarity is based on a cosine similarity formula: ; wherein, denotes a task semantic vector, denotes a semantic vector of an object instance.

[0015] Optionally, in the step S7, the generating an execution plan containing a robot skill sequence comprises: converting the task subgraph into a local scene description in natural language form; inputting the subtask and the local scene description into the large language model; outputting, by the large language model in combination with pre-registered robot skills, a skill sequence associated with the object instances in the task subgraph.

[0016] Optionally, after the step S7, the method further comprises: S8: determining, in the skill sequence, whether a moving action needs to be inserted between two skills according to the spatial distance between the object instances associated with the two skills.

[0017] In a second aspect, the present application provides a robot autonomous planning system based on a scene graph, comprising: a multi-view boundary graph extraction module, which reconstructs a three-dimensional point cloud of a scene based on multi-view images, and performs over-segmentation on the three-dimensional point cloud to obtain a plurality of point cloud segments; and extracts a multi-view boundary graph from the multi-view images; a multi-view boundary affinity calculation module, configured to calculate multi-view boundary affinity between any two adjacent point cloud segments based on the multi-view boundary graph; a growth clustering module, configured to perform region growth clustering on all point cloud segments based on the multi-view boundary affinity to obtain object instances in the scene; a Voronoi graph construction module, configured to assign semantic information to each of the object instances, and construct a Voronoi graph based on the spatial positions of all object instances, and further generate a scene topology graph taking the object instances as nodes and the spatial relationships between the objects as edges; a task acceptance decomposition module, configured to receive a natural language task, and decompose the natural language task into a plurality of subtasks using a large language model; a retrieval module, configured to retrieve relevant object instances from the scene topology graph based on the task semantics of each subtask to form a task subgraph; a generation module, configured to generate an execution plan containing a robot skill sequence based on the task subgraph, the skill sequence being associated with the object instances in the task subgraph.

[0018] In a third aspect, the present application provides an electronic device, comprising a processor, a communication interface, a memory and a bus, wherein the processor, the communication interface and the memory can communicate with each other through the bus, and the processor can call logical instructions in the memory to execute steps of the method provided in the first aspect.

[0019] In a fourth aspect, the present application provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement steps of the robot autonomous planning method based on a scene graph according to the first aspect.

[0020] The robot autonomous planning method based on a scene graph according to the present application has the following advantages: The present application constructs a topological scene graph fusing accurate spatial information and open semantics by introducing boundary affinity guided region growing clustering and Voronoi diagram space division, which on the one hand effectively overcomes the limitations of low instance extraction precision and lack of semantic description of traditional point cloud segmentation methods, and realizes compact and semantic-rich abstract expression of complex geometric environment; on the other hand, by combining this scene graph with a large language model for task planning, it changes the traditional "symbolic reasoning" relying on instantaneous visual perception into "embodied reasoning" based on the overall spatial topological relationship of the environment, fundamentally solving the problem that the planning result of the large language model planner is out of touch with the actual environment due to the lack of global understanding of the scene, and finally realizing autonomous, reliable and efficient decomposition and planning of the executable skill sequence of the robot for natural language tasks. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 A flowchart of the robot autonomous planning method based on the scene graph in the embodiment of the present application; Figure 2 A principle diagram of the robot autonomous planning method based on the scene graph in the embodiment of the present application; Figure 3 A structure block diagram of the robot autonomous planning system based on the scene graph in the embodiment of the present application; Figure 4 A structure block diagram of the electronic device in the embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to more clearly understand the purpose, technical scheme and advantages of the present application, the present application is described and explained below in combination with the drawings and embodiments.

[0023] Unless otherwise defined, technical terms or scientific terms used in the present application shall have the same meaning as those commonly understood by a person skilled in the art to which the present application belongs. The terms "one", "a", "an", "the", "these", and similar terms in the present application do not represent the number of quantity limitation, and they can be singular or plural. The terms "include", "contain", "have", and any variants thereof in the present application are intended to cover non-exclusive inclusion; for example, a process, method and system, product or device containing a series of steps or modules (units) are not limited to the listed steps or modules (units), but can include steps or modules (units) not listed, or can include other steps or modules (units) inherent to the process, method, product or device. The terms "connected", "connected", "coupled" and the like in the present application are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The term "multiple" in the present application refers to two or more. The term "and / or" describes the association between the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent the following three cases: A exists alone, A and B exist together, and B exists alone. Generally, the character " / " represents an "or" relationship between the front and rear associated objects. The terms "first", "second", "third" and the like in the present application are only used to distinguish similar objects, and do not represent a specific order of the objects.

[0024] As shown in Figure 1 and Figure 2 , the robot autonomous planning method based on scene graph provided in the embodiment of the present application comprises the following steps: S1: reconstructing a three-dimensional point cloud of a scene based on multi-view images, and performing over-segmentation on the three-dimensional point cloud to obtain a plurality of point cloud segments; simultaneously, extracting multi-view boundary maps from the multi-view images; Specifically, first, the input multi-view color image and depth map are reconstructed into a scene three-dimensional point cloud using motion result inference (SfM) combined with Colmap, and the size of the color image and the depth map used in the implementation is 1280x720.

[0025] Further, the original scene point cloud is then decomposed into a large number of fine point cloud segments using a local convex connection package (LCCP) method, and the threshold value of decomposition is 2mm; then, for the multi-view images, the segmentation mask and object boundary information in the images are extracted using a segmentation all model (SAM) to obtain multi-view boundary maps corresponding to the multi-view images. Here, the boundary map is obtained by using a Sobel operator on the mask map calculated by SAM.

[0026] S2: calculating the multi-view boundary affinity between any two adjacent point cloud segments based on the multi-view boundary maps; In step S2, the multi-view boundary affinity of the two point cloud segments is calculated, including: For a single view, the average value of the pixel boundary affinity between all pixel pairs in the projected pixel sets of the two point cloud segments is calculated as the boundary affinity under the view; The average value of the boundary affinities under all effective views is calculated to obtain the multi-view boundary affinity.

[0027] Specifically, for the boundary map obtained in step S2, the gray value of each pixel in the map is normalized to the interval [0, 1], at this time, the affinity of any two pixels in the map can be approximately expressed as the product of the probability that each pixel on the line segment connecting the two pixels does not belong to any boundary.

[0028] Pixel boundary affinity The calculation formula is: ; In the formula, and are two pixel points, represents the line segment connecting and , and represents the probability that the pixel point belongs to the boundary.

[0029] Then, by calculating the average value of the pixel boundary affinity between all pixel points in the projected pixel sets and corresponding to the point cloud segments and under a single view m, the boundary affinity of the two point cloud segments under the view m can be obtained: ； Further, by calculating the average value of the effective boundary affinity of the point cloud segments and under multiple views , the multi-view boundary affinity of the two point cloud segments can be obtained, wherein represents the effective view that simultaneously contains the two point cloud segments, which is determined by comparing the reverse projection of the point cloud segment and the corresponding region of the depth map: ； S3: Based on the multi-view boundary affinity, region growing clustering is performed on all point cloud segments to obtain the object instances in the scene; In step S3, the process of region growing clustering is as follows: The multi-view boundary affinity is taken as a growth criterion, and when the multi-view boundary affinity of two adjacent point cloud segments is higher than a preset threshold, the two adjacent point cloud segments are clustered into the same object instance.

[0030] Specifically, for all point cloud segments, the multi-view boundary affinity between any two adjacent point cloud segments is calculated Region growing clustering is performed, and optionally, the clustering threshold is set to 0.8, and each instance object in the scene can be obtained.

[0031] S4: semantic information is given to each object instance, and a Voronoi diagram is constructed based on the spatial positions of all object instances, and a scene topology graph is generated with object instances as nodes and spatial relationships between objects as edges; In step S4, semantic information is given to each object instance, including: The object instance is back-projected onto the original multi-view image; A visual-linguistic model is used to calculate a semantic vector of the corresponding projection region under each view; The semantic vectors of all views are averaged to obtain the semantic information of the object instance.

[0032] Further, in step S4, the Voronoi diagram is constructed, including: The point cloud center of gravity of each object instance is taken as a generating point; The perpendicular bisector surface of all adjacent generating point pairs is calculated, and the three-dimensional space is divided by taking the perpendicular bisector surface as a boundary to form a Voronoi diagram; If the distance between two object instances exceeds a preset distance threshold, the potential relationship between them is ignored.

[0033] Specifically, each instance object in the scene is back-projected onto the original multi-view image, and the CLIP is used to calculate the semantics of the corresponding region, and the multi-view semantic information is aggregated to the instance by averaging.

[0034] Then each object instance in the scene is taken as an initial object group, and the position of the object group is located at the center of gravity of the point cloud corresponding to the object instance by default. Then the initial object groups are taken as nodes, and the perpendicular bisector surface of any two adjacent nodes is taken as a boundary, and a three-dimensional Voronoi diagram is calculated.

[0035] Each of the subspaces obtained by the Voronoi diagram contains an approximate object instance, and each of the divided boundary planes represents the potential relationship between two adjacent objects in the scene. By this method, the instances in the three-dimensional scene graph are converted into a topological graph in which the semantic information and spatial position of the object instances are saved as nodes, and the relationships between the objects are saved as edges, serving as a scene graph to support subsequent task planning. The relationships between the object instances that are more than 0.5 m apart are ignored, so as to avoid mistakenly regarding the objects at a long distance as adjacent.

[0036] S5: receiving a natural language task, and decomposing the natural language task into a plurality of subtasks by using a large language model; Specifically, for a natural language task, the robot uses a large language model as a task planner, which first registers all the skills that can be used by the robot, and then decomposes the task into a set T of subtasks in PDDL format by using the large language model, each subtask containing at most one skill of the robot.

[0037] S6: for each subtask, retrieving relevant object instances from the scene topological graph based on the task semantics of the subtask to form a task subgraph; In step S6, the task subgraph is formed, including: calculating the similarity between the task semantics of the subtask and the semantic information of each object instance in the scene topological graph; retrieving the object instances related to the subtask based on the similarity; pruning the scene topological graph to only keep the relevant object instances and the object instances directly connected thereto, thereby forming the task subgraph.

[0038] Specifically, for each subtask task ∈ T the object semantics related to the task are retrieved from the scene graph based on the task semantics the relevance is calculated based on the cosine similarity , and : ; wherein, denotes the task semantic vector, denotes the semantic vector of the object instance.

[0039] Based on the task-related object information, the scene graph is pruned to only keep the task-related objects and the object instances directly connected thereto, thereby forming a task subgraph.

[0040] S7: generating an execution plan containing a sequence of robot skills based on the task subgraph, the sequence of skills being associated with the object instances in the task subgraph.

[0041] In step S7, the execution plan containing the sequence of robot skills is generated, including: transforming the task subgraph into a local scene description in natural language form; inputting the subtask and the local scene description into a large language model; outputting, by the large language model in combination with pre-registered robot skills, a skill sequence associated with object instances in the task subgraph.

[0042] After step S7, further comprising: S8: In the skill sequence, according to the spatial distance between the object instances associated with the two skills executed in succession, it is judged whether a moving action needs to be inserted between the two skills.

[0043] Specifically, for a subtask, first, its task subgraph is transformed into a local scene description according to the instances in the graph and the relationships between the instances, using a language model, and then the subtask and the scene description are input into a planner, which will try to transform the task subgraph into a skill sequence that can be executed step by step by a robot, each skill being associated with relevant object instances in the task subgraph to let the robot perform tasks such as "pulling open-drawer", "picking up-water cup", etc.

[0044] Finally, according to the spatial distance of the relevant objects in the skill sequence, the planner will add moving actions between tasks, thereby guiding the robot to move between objects that are far apart.

[0045] Thus, based on the scene graph information, the robot task planning is guided, the natural language task is converted into a series of robot basic skill-relevant object instance combinations, thereby realizing the autonomous task planning of the robot.

[0046] In combination with the above, it can be seen from the technical solutions provided by the present application that, by using boundary affinity guided region growing clustering, in combination with CLIP semantic calculation and Voronoi diagram spatial division, a scene topology graph containing spatial information and semantic information can be established. Based on this scene topology graph, using a large language model with pre-set robot skills as a planner, autonomous decomposition of robot natural language tasks into executable skill sequences can be realized. Using the scene graph mapping based on boundary affinity guided region growing clustering proposed by the present application, the problems of low instance extraction accuracy and lack of semantic information existing in traditional point cloud segmentation methods can be effectively enhanced; in combination with the task planning method proposed by the present application, which combines skill reasoning and spatial distance reasoning, the problem of task planning being generally disconnected from the actual environment of the large language model planner can be solved.

[0047] The embodiment of the present application also provides a robot autonomous planning system based on a scene graph, which is used to implement the above method embodiment, and details are not repeated. The terms "module", "unit", "sub-unit" and the like used below can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiment is preferably implemented in software, hardware or a combination of software and hardware is also possible and contemplated.

[0048] As Figure 3 shown, Figure 3 is a structural block diagram of a robot autonomous planning system based on a scene graph in the present application, which comprises: a multi-view boundary graph extraction module 101, which reconstructs a three-dimensional point cloud of a scene based on multi-view images, and performs over-segmentation on the three-dimensional point cloud to obtain a plurality of point cloud segments; meanwhile, extracts a multi-view boundary graph from the multi-view images; a multi-view boundary affinity calculation module 102, which is used to calculate multi-view boundary affinity between any two adjacent point cloud segments based on the multi-view boundary graph; a growth clustering module 103, which is used to perform region growth clustering on all point cloud segments based on the multi-view boundary affinity to obtain object instances in the scene; a Voronoi graph construction module 104, which is used to assign semantic information to each of the object instances, and construct a Voronoi graph based on the spatial positions of all object instances, and further generate a scene topology graph taking the object instances as nodes and the spatial relationship between objects as edges; a task acceptance decomposition module 105, which is used to receive a natural language task, and decompose the natural language task into a plurality of sub-tasks by using a large language model; a retrieval module 106, which is used to retrieve relevant object instances from the scene topology graph based on the task semantics of each sub-task to form a task sub-graph; a generation module 107, which is used to generate an execution plan containing a robot skill sequence based on the task sub-graph, the skill sequence being associated with the object instances in the task sub-graph.

[0049] As Figure 4 shown, the electronic device provided by the embodiment of the present application comprises a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 complete mutual communication through the communication bus 640. The processor 610 can invoke a logical instruction in the memory 630 to execute the following method: S1: reconstruct a three-dimensional point cloud of a scene based on multi-view images, and perform over-segmentation on the three-dimensional point cloud to obtain a plurality of point cloud segments; meanwhile, extract a multi-view boundary graph from the multi-view images; S2: calculating a multi-view boundary affinity between any two adjacent point cloud segments based on the multi-view boundary map; S3: performing region growing clustering on all point cloud segments based on the multi-view boundary affinity to obtain object instances in the scene; S4: assigning semantic information to each of the object instances, and constructing a Voronoi graph based on the spatial positions of all object instances, and further generating a scene topology graph taking object instances as nodes and spatial relationships between objects as edges; S5: receiving a natural language task, and decomposing the natural language task into multiple subtasks using a large language model; S6: for each subtask, retrieving relevant object instances from the scene topology graph based on the task semantics thereof to form a task subgraph; S7: generating an execution plan containing a robot skill sequence based on the task subgraph, the skill sequence being associated with the object instances in the task subgraph.

[0050] In addition, the logic instructions in the memory 630 described above can be implemented in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0051] The embodiments of the present application also provide a non-transitory computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method provided by the above-mentioned embodiments.

[0052] Those skilled in the art can clearly understand the implementation of the various embodiments by means of software and necessary general hardware platforms through the description of the above embodiments, and of course, the implementation can also be through hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, and the computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the method of each embodiment or some parts of the embodiment.

[0053] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A robot autonomous planning method based on scene graphs, characterized in that, Includes the following steps: S1: Reconstruct the 3D point cloud of the scene based on the multi-view images, and over-segment the 3D point cloud to obtain multiple point cloud fragments; at the same time, extract the multi-view boundary map from the multi-view images. S2: Based on the multi-view boundary map, calculate the multi-view boundary affinity between any two adjacent point cloud segments; S3: Based on the multi-view boundary affinity, perform region growing clustering on all point cloud fragments to obtain object instances in the scene; S4: Assign semantic information to each object instance, and construct a Voronoi diagram based on the spatial positions of all object instances, thereby generating a scene topology diagram with object instances as nodes and spatial relationships between objects as edges; S5: Receive the natural language task and decompose the natural language task into multiple sub-tasks using a large language model; S6: For each subtask, retrieve relevant object instances from the scene topology graph based on its task semantics to form a task subgraph; S7: Based on the task subgraph, generate an execution plan containing robot skill sequences, the skill sequences being associated with object instances in the task subgraph.

2. The method according to claim 1, characterized in that, In step S2, the multi-view boundary affinity of the two point cloud segments is calculated, including: For a single viewpoint, the average pixel boundary affinity between all pixel pairs in the projected pixel set of two point cloud segments is calculated as the boundary affinity under that viewpoint. The pixel boundary affinity The calculation formula is: ； In the formula, and For two pixels, Indicates connection and line segments, Represents pixels The probability of belonging to the boundary; The multi-view boundary affinity is obtained by averaging the boundary affinity under all effective viewpoints.

3. The method according to claim 2, characterized in that, In step S3, the process of region growth clustering is as follows: Using the multi-view boundary affinity as a growth criterion, when the multi-view boundary affinity of two adjacent point cloud segments is higher than a preset threshold, they are clustered into the same object instance.

4. The method according to claim 1, characterized in that, In step S4, assigning semantic information to each object instance includes: Back-project the object instance onto the original multi-view image; The semantic vector of the corresponding projection region under each viewpoint is calculated using a visual-language model; The semantic vectors from all perspectives are averaged to aggregate the semantic information of the object instance.

5. The method according to claim 1, characterized in that, In step S4, constructing the Voronoi diagram includes: The centroid of the point cloud for each object instance is used as the generation point; Calculate the perpendicular bisectors of all adjacent generated point pairs, and divide the three-dimensional space using the perpendicular bisectors as boundaries to form the Voronoi diagram; If the distance between two object instances exceeds a preset distance threshold, their potential relationship is ignored.

6. The method according to claim 1, characterized in that, In step S6, forming the task subgraph includes: Calculate the semantic similarity between the task semantics of the subtask and the semantic similarity between each object instance in the scene topology graph; Based on the similarity, retrieve object instances related to the subtask; The scene topology graph is pruned, retaining only the relevant object instances and the object instances directly connected to them, thereby forming the task subgraph.

7. The method according to claim 6, characterized in that, The semantic similarity is calculated based on the cosine similarity formula: ； in, Represents the task semantic vector. A semantic vector representing an object instance.

8. The method according to claim 1, characterized in that, In step S7, generating an execution plan containing robot skill sequences includes: The task subgraph is converted into a local scene description in natural language form; The subtask and the local scene description are input together into the large language model; The large language model, combined with pre-registered robot skills, outputs a skill sequence associated with object instances in the task subgraph.

9. The method according to claim 1, characterized in that, After step S7, the method further includes: S8: In the skill sequence, determine whether a movement action needs to be inserted between two skills based on the spatial distance between the object instances associated with the two skills executed consecutively.

10. An electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the scene graph-based robot autonomous planning method as described in any one of claims 1 to 9.

Images