A scene graph-based robot autonomous planning method

Abstract

The application provides a robot autonomous planning method based on a scene graph, and the method comprises the following steps: S1: reconstructing a three-dimensional point cloud of a scene based on multi-view images, and performing over-segmentation on the three-dimensional point cloud to obtain a plurality of point cloud segments; meanwhile, multi-view boundary maps are extracted from the multi-view images; S2: based on the multi-view boundary maps, the multi-view boundary affinity between any two adjacent point cloud segments is calculated; S3: based on the multi-view boundary affinity, region growing clustering is performed on all the point cloud segments to obtain object instances in the scene; S4: semantic information is given to each object instance, and a Voronoi diagram is constructed based on the spatial positions of all the object instances. The application utilizes the description of the spatial information and open semantic features of the object instances in the scene topology to assist a large language model in combining the task and behavior planning of the robot itself, and solves the problem that the task planner of the large language model is generally disconnected with the actual environment.

Description

Technical Field

[0001] This application relates to the field of robot perception technology, specifically a robot autonomous planning method based on scene graphs. Background Technology

[0002] In recent years, research on robot task planning and operation technology has mainly focused on the integration of embodied intelligence technology with robot systems, and the research on embodied intelligence technology mainly revolves around the two types of capabilities: movement and manipulation.

[0003] In terms of mobility, research on embodied intelligence technology mainly focuses on how to analyze the mobility requirements of a target task based on existing SLAM technology, and calculate the corresponding spatial coordinates of the target location, thereby driving robot movement through SLAM. Early related research mainly used the occupancy map in SLAM as the basis for retrieval, and guided the robot to carry out embodied navigation by adding some raw reference data. For example, SemExp uses Mask R-CNN to project the semantic category labels extracted from RGB onto the occupancy map, realizing the binding of the map with the semantic information of each location. Using this semantic map navigation can better guide the embodied agent's movement compared to pure occupancy map navigation.

[0004] In terms of operational capabilities, ALOHA proposed the Action Sequence Prediction (ACT) method, enabling robots to learn fine-grained maneuvers in just 10 minutes of training. RoboAgent subsequently built upon this, introducing MT-ACT for semantic enhancement, achieving more accurate learning across multiple tasks. Besides action sequence learning, using generative models is also a model training method. Diffusion Policy, using a conditional diffusion model, significantly outperformed previous methods in success rates across 15 tasks.

[0005] Despite extensive prior work improving the robot's mobility and maneuverability, for a specific robot task, the lack of understanding of the current environment prevents the robot from effectively combining these abilities, hindering its autonomous execution. Summary of the Invention

[0006] The problem addressed by this invention is how to avoid the planning results of large language model planners from becoming disconnected from the actual environment due to a lack of global understanding of the scenario.

[0007] To address the aforementioned issues, this invention provides a robot autonomous planning method, system, electronic device, and storage medium based on scene graphs.

[0008] In a first aspect, the present invention provides a robot autonomous planning method based on scene graphs, comprising the following steps:

[0009] S1: Reconstruct the 3D point cloud of the scene based on the multi-view images, and over-segment the 3D point cloud to obtain multiple point cloud fragments; at the same time, extract the multi-view boundary map from the multi-view images.

[0010] S2: Based on the multi-view boundary map, calculate the multi-view boundary affinity between any two adjacent point cloud segments;

[0011] S3: Based on the multi-view boundary affinity, perform region growing clustering on all point cloud fragments to obtain object instances in the scene;

[0012] S4: Assign semantic information to each object instance, and construct a Voronoi diagram based on the spatial positions of all object instances, thereby generating a scene topology diagram with object instances as nodes and spatial relationships between objects as edges;

[0013] S5: Receive the natural language task and decompose the natural language task into multiple sub-tasks using a large language model;

[0014] S6: For each subtask, retrieve relevant object instances from the scene topology graph based on its task semantics to form a task subgraph;

[0015] S7: Based on the task subgraph, generate an execution plan containing robot skill sequences, the skill sequences being associated with object instances in the task subgraph.

[0016] Optionally, in step S2, calculating the multi-view boundary affinity between two point cloud segments includes:

[0017] For a single viewpoint, the average pixel boundary affinity between all pixel pairs in the projected pixel set of two point cloud segments is calculated as the boundary affinity under that viewpoint.

[0018] pixel boundary affinity The calculation formula is:

[0019] ;

[0020] In the formula, and For two pixels, Indicates connection and line segments, Represents pixels The probability of belonging to the boundary;

[0021] The multi-view boundary affinity is obtained by averaging the boundary affinity under all effective viewpoints.

[0022] Optionally, in step S3, the process of region growing clustering is as follows:

[0023] Using the multi-view boundary affinity as a growth criterion, when the multi-view boundary affinity of two adjacent point cloud segments is higher than a preset threshold, they are clustered into the same object instance.

[0024] Optionally, in step S4, assigning semantic information to each object instance includes:

[0025] Back-project the object instance onto the original multi-view image;

[0026] The semantic vector of the corresponding projection region under each viewpoint is calculated using a visual-language model;

[0027] The semantic vectors from all perspectives are averaged to aggregate the semantic information of the object instance.

[0028] Optionally, in step S4, constructing the Voronoi diagram includes:

[0029] The centroid of the point cloud for each object instance is used as the generation point;

[0030] Calculate the perpendicular bisectors of all adjacent generated point pairs, and divide the three-dimensional space using the perpendicular bisectors as boundaries to form the Voronoi diagram;

[0031] If the distance between two object instances exceeds a preset distance threshold, their potential relationship is ignored.

[0032] Optionally, in step S6, forming the task subgraph includes:

[0033] Calculate the similarity between the task semantics of the subtask and the semantic information of each object instance in the scene topology graph;

[0034] Based on the similarity, retrieve object instances related to the subtask;

[0035] The scene topology graph is pruned, retaining only the relevant object instances and the object instances directly connected to them, thereby forming the task subgraph.

[0036] Optionally, the semantic similarity is calculated based on the cosine similarity formula:

[0037] ;

[0038] in, Represents the task semantic vector. A semantic vector representing an object instance.

[0039] Optionally, in step S7, generating an execution plan containing a sequence of robot skills includes:

[0040] The task subgraph is converted into a local scene description in natural language form;

[0041] The subtask and the local scene description are input together into the large language model;

[0042] The large language model, combined with pre-registered robot skills, outputs a skill sequence associated with object instances in the task subgraph.

[0043] Optionally, after step S7, the method further includes:

[0044] S8: In the skill sequence, determine whether a movement action needs to be inserted between the two skills based on the spatial distance between the object instances associated with the two consecutively executed skills.

[0045] Secondly, the present invention provides a scene graph-based autonomous robot planning system, comprising:

[0046] The multi-view boundary map extraction module reconstructs a 3D point cloud of the scene based on multi-view images and over-segments the 3D point cloud to obtain multiple point cloud fragments; at the same time, it extracts a multi-view boundary map from the multi-view images.

[0047] The multi-view boundary affinity calculation module is used to calculate the multi-view boundary affinity between any two adjacent point cloud segments based on the multi-view boundary map.

[0048] The growth clustering module is used to perform region growth clustering on all point cloud fragments based on the multi-view boundary affinity to obtain object instances in the scene;

[0049] Voronoi graph construction module: used to assign semantic information to each object instance and construct a Voronoi graph based on the spatial position of all object instances, thereby generating a scene topology graph with object instances as nodes and spatial relationships between objects as edges;

[0050] Task Acceptance and Decomposition Module: This module receives natural language tasks and decomposes them into multiple sub-tasks using a large language model.

[0051] Retrieval module: For each subtask, it retrieves relevant object instances from the scene topology graph based on its task semantics to form a task subgraph;

[0052] Generation module: Used to generate an execution plan containing robot skill sequences based on the task subgraph, wherein the skill sequences are associated with object instances in the task subgraph.

[0053] Thirdly, the present invention provides an electronic device including a processor, a communication interface, a memory, and a bus, wherein the processor, the communication interface, and the memory communicate with each other through the bus, and the processor can call logical instructions in the memory to execute the steps of the method provided in the first aspect.

[0054] Fourthly, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a scene graph-based autonomous robot planning method as described in the first aspect.

[0055] The beneficial effects of the scene graph-based robot autonomous planning method of the present invention are as follows:

[0056] This invention constructs a topological scene graph that integrates precise spatial information and open semantics by introducing boundary affinity-guided region growing clustering and Voronoi graph spatial partitioning. On the one hand, it effectively overcomes the limitations of traditional point cloud segmentation methods, such as low instance extraction accuracy and lack of semantic description, and achieves a compact and semantically rich abstract expression of complex geometric environments. On the other hand, by combining this scene graph with a large language model for task planning, it transforms the traditional "symbolic reasoning" that relies on instantaneous visual perception into "embodied reasoning" based on the overall spatial topological relationship of the environment. This fundamentally solves the problem of the planning results being out of touch with the actual environment due to the lack of global understanding of the scene in the large language model planner. Ultimately, it enables robots to autonomously, reliably, and efficiently decompose and plan executable skill sequences for natural language tasks. Attached Figure Description

[0057] Figure 1 This is a flowchart of a robot autonomous planning method based on a scene graph, as described in an embodiment of the present invention.

[0058] Figure 2 This is a schematic diagram of the robot autonomous planning method based on scene graph in an embodiment of the present invention;

[0059] Figure 3 This is a structural block diagram of a robot autonomous planning system based on a scene graph, as described in an embodiment of the present invention.

[0060] Figure 4 This is a structural block diagram of the electronic device in an embodiment of the present invention. Detailed Implementation

[0061] To better understand the purpose, technical solution, and advantages of this application, the application is described and explained below in conjunction with the accompanying drawings and embodiments.

[0062] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0063] like Figure 1 and Figure 2 As shown in the figure, a robot autonomous planning method based on a scene graph provided in this embodiment of the invention includes the following steps:

[0064] S1: Reconstruct the 3D point cloud of the scene based on multi-view images, and perform over-segmentation on the 3D point cloud to obtain multiple point cloud fragments; at the same time, extract the multi-view boundary map from the multi-view images.

[0065] Specifically, the input multi-view color images and depth maps are first reconstructed into a 3D point cloud of the scene using motion result inference (SfM) combined with Colmap. The size of the color images and depth maps used in the implementation is 1280x720.

[0066] Furthermore, the Local Convex Connection Bag (LCCP) method is then used to decompose the original scene point cloud into a large number of fine point cloud fragments with a decomposition threshold of 2mm. Then, for the multi-view image, the Segmentation Model (SAM) is used to extract the segmentation mask and object boundary information in the image to obtain the multi-view boundary map corresponding to the multi-view image. Here, the boundary map is obtained by using the Sobel operator on the mask map calculated by SAM.

[0067] S2: Based on the multi-view boundary map, calculate the multi-view boundary affinity between any two adjacent point cloud segments;

[0068] In step S2, the multi-view boundary affinity of the two point cloud segments is calculated, including:

[0069] For a single viewpoint, the average pixel boundary affinity between all pixel pairs in the projected pixel set of two point cloud segments is calculated as the boundary affinity under that viewpoint.

[0070] The average boundary affinity under all effective viewpoints is used to obtain the multi-view boundary affinity.

[0071] Specifically, for the boundary map obtained in step S2, the gray value of each pixel in the map is regularized to the range of [0, 1]. At this time, the affinity between any two pixels in the map can be approximately represented as the product of the probabilities that each pixel on the line segment connecting the two pixels does not belong to any boundary.

[0072] Pixel boundary affinity The calculation formula is:

[0073] ;

[0074] In the formula, and For two pixels, Indicates connection and line segments, Represents pixels The probability of belonging to the boundary.

[0075] Then, point cloud fragments under a single viewpoint m were calculated. and Corresponding set of projected pixels and The average pixel boundary affinity between all pixels in the cloud can be used to obtain the boundary affinity between two point cloud segments at viewpoint m:

[0076] ；

[0077] Furthermore, by calculating point cloud fragments from multiple perspectives... and The average value of the corresponding effective boundary affinity This allows us to obtain the multi-view boundary affinity of two point cloud fragments, where... This represents an effective viewpoint that simultaneously contains two point cloud fragments, determined by comparing the back projection of the point cloud fragments with the corresponding areas in the depth map:

[0078] ；

[0079] S3: Based on multi-view boundary affinity, perform region growing and clustering on all point cloud fragments to obtain object instances in the scene;

[0080] In step S3, the region growing clustering process is as follows:

[0081] Using multi-view boundary affinity as the growth criterion, when the multi-view boundary affinity of two adjacent point cloud segments is higher than a preset threshold, they are clustered into the same object instance.

[0082] Specifically, for all point cloud segments, the multi-view boundary affinity between any two adjacent point cloud segments is calculated. By performing regional growth clustering, and optionally setting the clustering threshold to 0.8, we can obtain the instance objects in the scene.

[0083] S4: Assign semantic information to each object instance and construct a Voronoi diagram based on the spatial location of all object instances, thereby generating a scene topology diagram with object instances as nodes and spatial relationships between objects as edges;

[0084] In step S4, semantic information is assigned to each object instance, including:

[0085] Back-project the object instance onto the original multi-view image;

[0086] The semantic vector of the corresponding projection region under each viewpoint is calculated using a visual-language model;

[0087] The semantic vectors from all perspectives are averaged to aggregate the semantic information of the object instance.

[0088] Further, in step S4, the Voronoi diagram is constructed, including:

[0089] The centroid of the point cloud for each object instance is used as the generation point;

[0090] Calculate the perpendicular bisectors of all adjacent generated point pairs, and divide the three-dimensional space using the perpendicular bisectors as boundaries to form a Voronoi diagram;

[0091] If the distance between two object instances exceeds a preset distance threshold, their potential relationship is ignored.

[0092] Specifically, each instance object in the scene is back-projected onto the original multi-view image, and CLIP is used to calculate the semantics of the corresponding region. The multi-view semantic information is then aggregated onto the instance by averaging.

[0093] Then, each object instance in the scene is treated as an initial object group, with the object group's position defaulting to the centroid of the point cloud corresponding to the object instance. Subsequently, using these initial object groups as nodes and the perpendicular bisectors of any two adjacent nodes as boundaries, a set of 3D Voronoi diagrams is calculated.

[0094] Each subspace derived from the Voronoi diagram approximately contains an object instance, and the boundary plane of each subdivision represents the potential relationship between two adjacent objects in the scene. This method transforms instances in the 3D scene graph into a topological graph where object instances serve as nodes storing semantic information and spatial location, and relationships between objects serve as edges, thus supporting subsequent task planning. Relationships where object instances are more than 0.5m apart are ignored to avoid incorrectly treating distant objects as adjacent.

[0095] S5: Receive natural language tasks and decompose the natural language tasks into multiple sub-tasks using a large language model;

[0096] Specifically, for a natural language task, the robot uses a large language model as a task planner. It first registers all the skills that the robot can use, and then uses the large language model to decompose the task into a set of subtasks T in PDDL format. Each subtask contains at most one robot skill.

[0097] S6: For each subtask, retrieve relevant object instances from the scene topology graph based on its task semantics to form a task subgraph;

[0098] In step S6, a task subgraph is formed, including:

[0099] Calculate the similarity between the task semantics of the subtask and the semantic information of each object instance in the scene topology graph;

[0100] Retrieve object instances related to the subtask based on similarity;

[0101] The scene topology graph is pruned, retaining only relevant object instances and object instances directly connected to them, thus forming a task subgraph.

[0102] Specifically, for each subtask task∈T Based on task semantics from the scene graph Retrieve the semantics of objects relevant to the task The correlation calculation is based on cosine similarity. :

[0103] ;

[0104] in, Represents the task semantic vector. A semantic vector representing an object instance.

[0105] Based on the information of task-related objects, the scene graph is pruned, retaining only the task-related objects and their directly connected object instances to form a task subgraph.

[0106] S7: Based on the task subgraph, generate an execution plan containing robot skill sequences, which are associated with object instances in the task subgraph.

[0107] In step S7, an execution plan containing the robot's skill sequence is generated, including:

[0108] Convert the task subgraph into a local scene description in natural language form;

[0109] Input the subtasks and local scene descriptions together into the large language model;

[0110] The large language model, combined with pre-registered robot skills, outputs a skill sequence associated with object instances in the task subgraph.

[0111] After step S7, the following also includes:

[0112] S8: In a skill sequence, determine whether a movement action needs to be inserted between two skills based on the spatial distance between the object instances associated with the two skills executed consecutively.

[0113] Specifically, for a subtask, its task subgraph is first transformed into a local scene description using a language model based on the relationships between instances in the graph. Then, the subtask and scene description are input into the planner. The planner will combine the robot's available skills to try to transform the task subgraph into a skill sequence that the robot can execute step by step. Each skill will be associated with relevant object instances in the task subgraph so that the robot can perform tasks such as "pull open - drawer" or "take - cup".

[0114] Finally, based on the spatial distance between relevant objects in the skill sequence, the planner will add movement actions between tasks, thereby guiding the robot to move between task objects that are far apart.

[0115] This demonstrates how scene graph information guides robot task planning, transforming natural language tasks into a combination of basic robot skills and related object instances, thereby enabling autonomous task planning for the robot.

[0116] In summary, the technical solution provided by this invention demonstrates that by using boundary affinity-guided region growing clustering, combined with CLIP semantic computation and Voronoi diagram spatial partitioning, a scene topology map containing both spatial and semantic information can be established. Based on this scene topology map, using a large language model with pre-defined robot skills as a planner, the robot's natural language task can be autonomously decomposed into executable skill sequences. Using the scene graph construction based on boundary affinity-guided region growing clustering proposed in this invention effectively addresses the problems of low instance extraction accuracy and lack of semantic information in traditional point cloud segmentation methods. Furthermore, combining the task planning method that integrates skill reasoning and spatial distance reasoning proposed in this invention solves the common problem of task planning being disconnected from the actual environment in large language model planners.

[0117] This invention also provides a scene graph-based autonomous robot planning system for implementing the above-described method embodiments; details already described will not be repeated. The terms "module," "unit," and "subunit," etc., used below refer to combinations of software and / or hardware that perform predetermined functions. Although the system described in the following embodiments is preferably implemented in software, hardware implementation or a combination of software and hardware is also possible and contemplated.

[0118] like Figure 3 As shown, Figure 3 This is a structural block diagram of the scene graph-based autonomous robot planning system of the present invention, which includes:

[0119] The multi-view boundary map extraction module 101 reconstructs a 3D point cloud of the scene based on multi-view images and over-segments the 3D point cloud to obtain multiple point cloud fragments; at the same time, it extracts a multi-view boundary map from the multi-view images.

[0120] The multi-view boundary affinity calculation module 102 is used to calculate the multi-view boundary affinity between any two adjacent point cloud segments based on the multi-view boundary map.

[0121] The growth clustering module 103 is used to perform region growth clustering on all point cloud fragments based on the multi-view boundary affinity to obtain object instances in the scene;

[0122] Voronoi graph construction module 104: used to assign semantic information to each object instance and construct a Voronoi graph based on the spatial position of all object instances, thereby generating a scene topology graph with object instances as nodes and spatial relationships between objects as edges;

[0123] Task receiving and decomposition module 105: used to receive natural language tasks and decompose the natural language tasks into multiple sub-tasks using a large language model;

[0124] Retrieval module 106: For each subtask, it retrieves relevant object instances from the scene topology graph based on its task semantics to form a task subgraph;

[0125] Generation module 107: Used to generate an execution plan containing robot skill sequences based on the task subgraph, the skill sequences being associated with object instances in the task subgraph.

[0126] like Figure 4 As shown in the figure, an electronic device provided by an embodiment of the present invention includes: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute the following method:

[0127] S1: Reconstruct the 3D point cloud of the scene based on the multi-view images, and over-segment the 3D point cloud to obtain multiple point cloud fragments; at the same time, extract the multi-view boundary map from the multi-view images.

[0128] S2: Based on the multi-view boundary map, calculate the multi-view boundary affinity between any two adjacent point cloud segments;

[0129] S3: Based on the multi-view boundary affinity, perform region growing clustering on all point cloud fragments to obtain object instances in the scene;

[0130] S4: Assign semantic information to each object instance, and construct a Voronoi diagram based on the spatial positions of all object instances, thereby generating a scene topology diagram with object instances as nodes and spatial relationships between objects as edges;

[0131] S5: Receive the natural language task and decompose the natural language task into multiple sub-tasks using a large language model;

[0132] S6: For each subtask, retrieve relevant object instances from the scene topology graph based on its task semantics to form a task subgraph;

[0133] S7: Based on the task subgraph, generate an execution plan containing robot skill sequences, the skill sequences being associated with object instances in the task subgraph.

[0134] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0135] This invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the methods provided in the above embodiments.

[0136] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot autonomous planning method based on scene graphs, characterized in that, Includes the following steps: S1: Reconstruct the 3D point cloud of the scene based on the multi-view images, and over-segment the 3D point cloud to obtain multiple point cloud fragments; at the same time, extract the multi-view boundary map from the multi-view images. S2: Based on the multi-view boundary map, calculate the multi-view boundary affinity between any two adjacent point cloud segments; S3: Based on the multi-view boundary affinity, perform region growing clustering on all point cloud fragments to obtain object instances in the scene; S4: Assign semantic information to each object instance, and construct a Voronoi diagram based on the spatial positions of all object instances, thereby generating a scene topology diagram with object instances as nodes and spatial relationships between objects as edges; S5: Receive the natural language task and decompose the natural language task into multiple sub-tasks using a large language model; S6: For each subtask, retrieve relevant object instances from the scene topology graph based on its task semantics to form a task subgraph; The formation of the task subgraph includes: Calculate the semantic similarity between the task semantics of the subtask and the semantic similarity between each object instance in the scene topology graph; Based on the similarity, retrieve object instances related to the subtask; The scene topology graph is pruned to retain only the relevant object instances and the object instances directly connected to them, thereby forming the task subgraph; S7: Based on the task subgraph, generate an execution plan containing a sequence of robot skills associated with object instances in the task subgraph.

2. The method according to claim 1, characterized in that, In step S2, the multi-view boundary affinity of the two point cloud segments is calculated, including: For a single viewpoint, the average pixel boundary affinity between all pixel pairs in the projected pixel set of two point cloud segments is calculated as the boundary affinity under that viewpoint. pixel boundary affinity The calculation formula is: ； In the formula, and For two pixels, Indicates connection and line segments, Represents pixels The probability of belonging to the boundary; The multi-view boundary affinity is obtained by averaging the boundary affinity under all effective viewpoints.

3. The method according to claim 2, characterized in that, In step S3, the process of region growth clustering is as follows: Using the multi-view boundary affinity as a growth criterion, when the multi-view boundary affinity of two adjacent point cloud segments is higher than a preset threshold, they are clustered into the same object instance.

4. The method according to claim 1, characterized in that, In step S4, assigning semantic information to each object instance includes: Back-project the object instance onto the original multi-view image; The semantic vector of the corresponding projection region under each viewpoint is calculated using a visual-language model; The semantic vectors from all perspectives are averaged to aggregate the semantic information of the object instance.

5. The method according to claim 1, characterized in that, In step S4, constructing the Voronoi diagram includes: The centroid of the point cloud for each object instance is used as the generation point; Calculate the perpendicular bisectors of all adjacent generated point pairs, and divide the three-dimensional space using the perpendicular bisectors as boundaries to form the Voronoi diagram; If the distance between two object instances exceeds a preset distance threshold, their potential relationship is ignored.

6. The method according to claim 1, characterized in that, The semantic similarity is calculated based on the cosine similarity formula: ； in, Represents the task semantic vector. A semantic vector representing an object instance.

7. The method according to claim 1, characterized in that, In step S7, generating an execution plan containing robot skill sequences includes: The task subgraph is converted into a local scene description in natural language form; The subtask and the local scene description are input together into the large language model; The large language model, combined with pre-registered robot skills, outputs a skill sequence associated with object instances in the task subgraph.

8. The method according to claim 1, characterized in that, After step S7, the method further includes: S8: In the skill sequence, determine whether a movement action needs to be inserted between the two skills based on the spatial distance between the object instances associated with the two consecutively executed skills.

9. An electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the scene graph-based robot autonomous planning method as described in any one of claims 1 to 8.

Images