Multi-source map fusion scenario graph construction method and system for embodied intelligent robot

Abstract

The application provides a multi-source map fusion scene graph construction method and system for embodied intelligent robots, comprising: in a unified world coordinate system, performing spatial geometric matching and semantic feature fusion on object instances observed by multiple frames and multiple views, eliminating duplicate instances, and obtaining globally unified stable object instances; and in the unified world coordinate system, maintaining a three-dimensional hierarchical scene graph, a three-dimensional voxel semantic feature graph and a two-dimensional navigation occupancy graph, constructing a joint optimization target containing a navigation safety distance, a motion cost and a target occlusion cost, and solving an optimal navigation pose from a candidate approaching pose, which simultaneously satisfies a chassis collision avoidance constraint and a mechanical arm target visibility constraint; taking a component node in the three-dimensional hierarchical scene graph as a candidate operation point, and comprehensively scoring in combination with a task intention, component visibility, grasping feasibility and an occlusion condition, to select an optimal component node as a grasping anchor point for mechanical arm operation.

Description

Technical Field

[0001] This invention relates to, specifically, a method and system for constructing scene graphs through multi-source map fusion for embodied intelligent robots. Background Technology

[0002] Existing robot maps mainly include geometric maps, semantic maps, dynamic scene maps, and open vocabulary maps. Among them, geometric maps are usually used to express the occupancy status, free space, and obstacle information in the environment, and can support the localization, obstacle avoidance, and navigation of mobile chassis. However, they are difficult to directly express the semantic identity of target objects, object relationships, graspable parts, placeable areas, and the feasibility of robotic arm operations.

[0003] Existing 3D semantic maps and dynamic scene graphs can represent objects, regions, rooms, and their topological relationships, and can be used for tasks such as object search, spatial understanding, and semantic navigation. However, such solutions usually focus on environmental understanding and lack a unified expression for candidate robotic arm actions, end-effector pose constraints, grasping / placement feasibility, collision risks, and execution feedback, making it difficult to directly support mobile robots in completing the integrated task of "navigation to position - robotic arm operation".

[0004] Existing multi-platform or multi-robot map fusion solutions mainly focus on geometric consistency issues such as point cloud registration, pose graph optimization, and map stitching. However, they are insufficient in fusion of semantic conflicts, duplicate instances, dynamic changes, information source credibility, and object operation attributes, which can easily lead to duplicate mapping of the same object, erroneous merging of different objects, or untimely updates of object status.

[0005] While existing open vocabulary mapping solutions can support natural language queries for target objects, they typically lack integration with 2D navigation safety boundaries, 3D geometric reachability, robotic arm kinematic constraints, and end effector constraints. Therefore, in real-world scenarios, situations may arise where the target object can be retrieved, but the robot cannot safely reach it, the robotic arm cannot grasp it, or the placement area does not meet operational requirements.

[0006] In summary, the existing technology has at least the following problems: 1. The geometric map, semantic map, and operation planning modules are independent of each other, with inconsistent coordinate systems, update frequencies, and semantic granularity, making it difficult to support the integrated task of navigation and robotic arm operation.

[0007] 2. It lacks stable representation and updates of object-level spatial relationships, such as on, in, belong, part-of, support, and include, making it difficult to describe complex scenarios such as containers, stacks, components, and occlusion.

[0008] 3. Insufficient ability to associate instances across perspectives or across robots can easily lead to problems such as incorrect merging, missed merging, and duplicate instances.

[0009] 4. Map updates focus primarily on static reconstruction, lacking an online topology correction mechanism after robots perform operations such as grabbing and placing.

[0010] 5. The lack of linkage between the navigation map and the operational scene understanding makes it difficult for the robotic arm to simultaneously consider the viewpoint, path, accessibility, and collision risk when performing grasping or placement.

[0011] Therefore, there is an urgent need for a map construction and updating method for mobile operation tasks that can unify and integrate geometric information, semantic information, object spatial relationships, open vocabulary query capabilities, and robot operation constraints, thereby supporting mobile robots to complete target retrieval, navigation planning, feasibility judgment of robotic arm operations, and post-execution state updates. Summary of the Invention

[0012] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for constructing scene graphs through multi-source map fusion for embodied intelligent robots.

[0013] A method for constructing a multi-source map fusion scene graph for embodied intelligent robots according to the present invention includes: Step S1: Acquire RGB-D environment perception data and corresponding camera parameters and pose information, and complete the transformation from 3D coordinates to world coordinates; based on the acquired RGB-D environment perception data, perform target detection, instance segmentation and visual semantic feature extraction on the environment, combine depth information to generate a set of object instances in the world coordinate system, and filter out invalid noise instances; Step S2: Under the unified world coordinate system, perform spatial geometric matching and semantic feature fusion on object instances obtained from multi-frame and multi-view observations to eliminate duplicate instances and obtain globally unified stable object instances. Step S3: Based on the globally unified stable object instance, extract the geometric contour information of the object, infer the multi-class spatial topological relationships between objects, and construct a three-dimensional hierarchical scene graph containing object nodes and component nodes; Step S4: Perform voxelization discretization on the three-dimensional space, and fuse the three-dimensional geometric structure information and visual semantic features incrementally in units of voxels to form a three-dimensional voxel semantic feature map that supports target retrieval and spatial positioning. Step S5: Generate a two-dimensional planar grid map based on three-dimensional perception information, mark passable areas and obstacle areas, and after safety dilation and distance transformation processing, select robot candidate approach poses that meet safety distance constraints to construct a two-dimensional navigation occupancy map; Step S6: When the 3D hierarchical scene map, 3D voxel semantic feature map, and 2D navigation occupancy map output inconsistent representation information for the same spatial location, combine the prior weights and local confidence of each map source, and determine a unified environmental representation result through a weighted voting mechanism to correct the single-source perception bias. Step S7: Based on the unified environmental characterization results, construct a joint optimization objective that includes navigation safety distance, motion cost, and target occlusion cost, and solve for the optimal navigation pose that simultaneously satisfies the chassis collision-free constraint and the robot arm target visibility constraint from the candidate approach poses; Step S8: Based on the unified environmental representation results, the component nodes in the 3D hierarchical scene map are used as candidate operation points. The task intent, component visibility, grasping feasibility and occlusion are combined to make a comprehensive score, and the optimal component node is selected as the grasping anchor point for the robotic arm operation.

[0014] Preferably, step S1 includes: Step S1.1: Acquire environmental images and depth maps based on several RGB-D vision sensors; simultaneously, acquire camera intrinsic parameters K and camera pose. ; Step S1.2: For pixels Its depth value is Then the corresponding 3D point in the camera coordinate system for:

[0015] Among them, superscript For transpose; Step S1.3: Based on camera pose Transform 3D points to the world coordinate system:

[0016] in, Represents a 3D point in the camera coordinate system. Represents a three-dimensional point in the world coordinate system; Step S1.4: Perform target detection and instance segmentation based on the RGB image in the RGB-D observation to obtain a two-dimensional detection box and segmentation mask; Step S1.5: Extract visual semantic features based on the image region corresponding to the segmentation mask in the RGB image to obtain visual semantic features; Step S1.6: Based on the depth map, camera intrinsic parameters K, and camera pose The pixels within the mask area are back-projected onto the world coordinate system to generate a 3D point cloud of the current object instance. Step S1.7: For point cloud numbers below a preset threshold Alternatively, instances located outside the preset robot workspace can be filtered.

[0017] Preferably, step S2 includes: Step S2.1: For the newly observed instance i and the historical instance j, calculate their three-dimensional intersection-union ratio. Represented as:

[0018] in, and Let i and j represent the 3D axis-aligned bounding box volumes of instance i and instance j, respectively. This represents the volume of the intersection of the two bounding boxes; Calculate the projection overlap rate of the instance on the XY plane. :

[0019] in, and Let i and j represent the projected polygons of instance i and instance j on the XY plane, respectively. Indicates area; The visual semantic feature similarity between instances is represented by cosine similarity:

[0020] in, and Let i and j represent the visual semantic feature vectors of instance i and instance j, respectively. For the set of real numbers, The dimension of the visual semantic feature vector; when , and If each value exceeds its respective preset threshold, it is determined that instance i and instance j correspond to the same object; fused instance semantic features Represented as:

[0021] in, and These represent the instance detection confidence scores, and These represent the cumulative number of fusions or the number of observations, respectively. This indicates a normalization operation.

[0022] Preferably, the three-dimensional hierarchical scene graph includes: object nodes, component nodes, and spatial relationships between objects; the spatial relationships include: on, in, belong, part-of, support, and occlude relationships; Inferring object relationships based on directed bounding boxes or alpha-shape contours includes: For instance i, its point cloud's lower bound, upper bound, and center height in the vertical direction are respectively represented as: ; Instance i is considered to be close to the support plane or root node when it meets the following conditions in the vertical direction with respect to the root node or support plane:

[0023] in, The vertical distance threshold. The vertical height of the supporting plane or root node; When instance i and instance j overlap on a horizontal plane and the degree of overlap meets a preset requirement, and the bottom of instance i is close to the top of instance j, while the center height of instance i is above the top of instance j, then instance i is determined to be above instance j.

[0024] in, Let be the projection overlap rate of instance i and instance j on the horizontal plane. The horizontal overlap rate threshold; When instance i and instance j overlap on the horizontal plane and the degree of overlap meets the preset requirements, and the center height of instance i is below the upper limit of instance j, it is determined that instance i belongs to or is located within the internal region of instance j.

[0025] in, Indicates the horizontal overlap threshold; After the robot completes the grasping and placement operations, it updates the position of objects, parent and child nodes, and spatial topological relationships in the 3D hierarchical scene map based on the operation results, so that the map state keeps synchronized with the real environment.

[0026] Preferably, step S4 includes: Step S4.1: For a 3D point in the world coordinate system According to the voxel side length Discretize it into voxel indexes:

[0027] in, Indicates the origin of the voxel map. This indicates a round-down operation; Step S4.2: Index the semantic features observed at different times and from different perspectives according to the same voxel. To achieve incremental voxel feature updates, cumulative and incremental fusion is performed, including: For voxels If its historical voxel semantic features are The cumulative confidence score is The current observed features are The current observation confidence score is The updated weights are then expressed as:

[0028] The voxel semantic features are updated as follows:

[0029] in, This is a normalization operation; The cumulative confidence score has been updated to: ; Step S4.3: When performing open vocabulary retrieval, query semantic features. With voxel semantic features Searching using cosine similarity: .

[0030] Preferably, step S5 includes: Step S5.1: For the two-dimensional position in the world coordinate system The corresponding navigation grid index is represented as:

[0031] in, 'c' indicates that the navigation occupies the origin of the map, and 'c' represents the raster resolution. Step S5.2: Mark the grid cells that fall within the obstacle height range as obstacle areas, and mark the remaining observable grid cells as passable areas; Step S5.3: Expand the obstacle area according to the safety radius r, and determine the expansion core size. Represented as:

[0032] in, This indicates the rounding up operation; Step S5.4: After completing obstacle expansion, calculate the distance from each passable grid cell to the nearest obstacle using distance transformation;

[0033] in, Represents a set of obstacle grids; For obstacle grid index; Step S5.5: When the distance from the passable grid to the nearest obstacle is greater than or equal to a preset value, the safe distance constraint is satisfied; select the grid that satisfies the safe distance constraint as the candidate approach pose and map it to the robot pose in the world coordinate system.

[0034] Preferably, step S6 includes: when the three-dimensional hierarchical scene map, the three-dimensional voxel semantic feature map, and the two-dimensional navigation occupancy map give different labels to the same world point p, a weighted voting method is used to determine the final label. :

[0035] in, For the candidate tag set; The label given to map source m at location p; For indicator functions; The weight of map source m at location p; A three-dimensional scene diagram; A three-dimensional voxel semantic feature map; For two-dimensional navigation occupancy map;

[0036] in, For map source-level prior weights, This represents the confidence level of the corresponding map source at location p. It is the sigmoid function; The representation information of the corresponding map is corrected based on the final label, or an instance update or re-observation is triggered.

[0037] Preferably, step S7 includes: To simultaneously satisfy both chassis collision-free and robotic arm visibility constraints, a joint cost is calculated for candidate navigation poses q, and the pose with the minimum cost is selected. :

[0038] ; The distance to the nearest obstacle; To prevent constants with a denominator of zero; This represents the robot's current or default pose. To observe the target node from the candidate pose The cost of occlusion at that time; and These are the weighting coefficients.

[0039]

[0040] in, This represents the set of nodes that may occlude the target from the candidate viewpoint. This represents the area of ​​the intersection of the occluding node and the target node in the view frust projection; Let be the intersection area of ​​the polygons in the view cone projection. The target node.

[0041] Preferably, step S8 includes: For target instance i, its set of component nodes is represented as:

[0042] Candidate component nodes are scored based on instruction intent, component visibility, crawling feasibility, and occlusion status, and crawling anchor points are selected accordingly:

[0043] in, Semantic embeddings that represent user instructions or task intent; This indicates the semantic similarity between component nodes and instruction intents. Indicates the visibility ratio of component nodes from the current viewpoint; Indicates the feasibility of component grabbing; Indicates the area of ​​the component obstructed by the grasping frustum; These are the weighting coefficients.

[0044] A multi-source map fusion scene graph construction system for embodied intelligent robots, provided by the present invention, includes: Module M1 is used to acquire RGB-D environmental perception data and corresponding camera parameters and pose information, and complete the conversion from 3D coordinates to world coordinates. Based on the acquired RGB-D environmental perception data, it performs target detection, instance segmentation and visual semantic feature extraction on the environment, combines depth information to generate a set of object instances in the world coordinate system, and filters out invalid noise instances. Module M2 is used to perform spatial geometric matching and semantic feature fusion on object instances obtained from multi-frame and multi-view observations in a unified world coordinate system, eliminate duplicate instances, and obtain globally unified stable object instances. Module M3 is used to extract the geometric contour information of objects based on the globally unified stable object instances, infer the multi-class spatial topological relationships between objects, and construct a three-dimensional hierarchical scene graph containing object nodes and component nodes. Module M4 is used to perform voxelization discretization processing on three-dimensional space, and to fuse three-dimensional geometric structure information and visual semantic features incrementally in units of voxels to form a three-dimensional voxel semantic feature map that supports target retrieval and spatial positioning. Module M5 is used to generate a two-dimensional planar grid map based on three-dimensional perception information, mark passable areas and obstacle areas, and after safety dilation and distance transformation processing, select robot candidate approach poses that meet safety distance constraints to construct a two-dimensional navigation occupancy map. Module M6 is used to determine a unified environmental representation result by combining the prior weights and local confidence of each map source when the three-dimensional hierarchical scene map, the three-dimensional voxel semantic feature map, and the two-dimensional navigation occupancy map output inconsistent representation information for the same spatial location, thereby correcting the single-source perception bias. Module M7 is used to construct a joint optimization objective based on unified environmental characterization results, including navigation safety distance, motion cost, and target occlusion cost, and solve for the optimal navigation pose that simultaneously satisfies the chassis collision-free constraint and the robot arm target visibility constraint from candidate approach poses; Module M8 is used to select the optimal component node as the gripping anchor point for the robotic arm operation based on the unified environmental representation results, using component nodes in the 3D hierarchical scene map as candidate operation points, and combining task intent, component visibility, grasping feasibility and occlusion situation for comprehensive scoring.

[0045] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention integrates a three-dimensional hierarchical scene map, a three-dimensional voxel semantic feature map, and a two-dimensional navigation occupancy map in a unified world coordinate system, enabling the robot to simultaneously utilize object semantics, three-dimensional geometry, spatial relationships, and navigation constraints to support target retrieval, navigation approach, robotic arm operation, and post-execution map updates. 2. This invention unifies multi-camera RGB-D observations, robot poses, robotic arm TCP poses, and camera extrinsic parameters into the same world coordinate system, reducing coordinate inconsistencies, redundant mapping, and object state conflicts caused by cross-viewpoint and cross-time observations, thereby improving the consistency of multi-source maps. 3. This invention combines three-dimensional bounding box overlap, planar projection overlap rate and visual semantic feature similarity for instance matching, and expresses relationships such as on, in, belong, support, and include through a three-dimensional hierarchical scene graph, thereby reducing the probability of false merging, missed merging and misjudgment of spatial relationships, and improving the accuracy of instance fusion and spatial relationship judgment. 4. This invention associates visual semantic features with three-dimensional spatial location through three-dimensional voxel semantic feature maps. Robots can retrieve targets based on natural language or open category descriptions and locate the corresponding three-dimensional regions, which can improve open vocabulary target retrieval and three-dimensional positioning capabilities. 5. This invention generates a collision-free approach pose by using a two-dimensional navigation occupancy map, and combines a three-dimensional scene map and a voxel feature map to determine viewpoint occlusion, target visibility, and robotic arm reachability, reducing situations where "the target has been found but cannot be grasped" or "a second adjustment is still required after the chassis is in place," thereby improving the coordination between navigation and robotic arm operation. 6. After the robot of this invention completes the grasping, placing or moving operation, it can update the object position, parent-child relationship, on / in / belong relationship and navigation occupation state according to the execution result, so that the map is dynamically updated with the changes of the real environment, forming a closed loop of perception, mapping, planning, execution and feedback update, and can support online map update after execution. 7. This invention can form a unified map representation and closed-loop execution framework for embodied operations, improving the robot's target retrieval, spatial understanding, navigation approach, robotic arm operation, and post-execution self-update capabilities in complex dynamic environments. Attached Figure Description

[0046] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A schematic diagram of a multi-source map fusion scene map construction system for embodied intelligent robots.

[0047] Figure 2 A flowchart for constructing a multi-source map fusion scene graph for embodied intelligent robots. Detailed Implementation

[0048] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0049] Example 1 This invention provides a method and system for constructing a multi-source map fusion scene graph for embodied intelligent robots. Object instances are generated through RGB-D observation, and a three-dimensional hierarchical scene graph, a three-dimensional voxel semantic feature graph, and a two-dimensional navigation occupancy graph are maintained in a unified world coordinate system. These three types of maps are interconnected and continuously updated based on robot observation results and execution feedback, thereby supporting target retrieval, navigation approach, robotic arm grasping / placement, and post-operation topology correction.

[0050] The method for constructing scene graphs through multi-source map fusion for embodied intelligent robots, such as Figure 2 As shown, it includes: Step S1: Acquire multi-camera RGB-D observations; The robot acquires environmental images, depth maps, camera intrinsic parameters K, and camera pose through one or more RGB-D vision sensors. And acquire robot state information such as the end-effector pose and chassis pose; For pixels Its depth value is The corresponding 3D points in the camera coordinate system are: (1) Among them, superscript For transpose; Based on camera pose Transform 3D points to the world coordinate system: (2) in, Represents a 3D point in the camera coordinate system. Represents a three-dimensional point in the world coordinate system.

[0051] Step S2: Generate object instances and perform filtering; Object detection, instance segmentation, visual semantic feature extraction, and point cloud generation are performed on RGB-D observations to obtain a set of object instances. Each object instance includes a 2D bounding box, a segmentation mask, a category label, visual semantic features, a 3D point cloud, and an optional six-DOF pose. More specifically, object detection and instance segmentation are performed based on the RGB images from the RGB-D observations to obtain 2D bounding boxes and segmentation masks; visual semantic features are extracted from the image regions corresponding to the segmentation masks in the RGB images to obtain visual semantic features; and visual semantic features are generated based on the depth map, camera intrinsics K, and camera pose. The pixels within the mask area are back-projected to the world coordinate system according to formulas (1) and (2) to generate the three-dimensional point cloud of the object instance.

[0052] For point cloud counts below a preset threshold Instances located outside the robot's workspace are filtered to reduce the impact of noisy instances on subsequent map updates.

[0053] Step S3: Instance cross-frame matching and fusion; For the newly observed instance i and the historical instance j, a coarse screening is first performed using a 3D axis-aligned bounding box. The 3D intersection-union ratio (IUU) of the two instances can be expressed as: (3) in, and Let i and j represent the 3D axis-aligned bounding box volumes of instance i and instance j, respectively. This represents the intersection volume of the bounding boxes of the two instances. The purpose of calculating the 3D intersection-union ratio is to perform a fast coarse screening of candidate instance pairs: when Greater than the preset threshold If instance i and instance j are significantly overlapping in three-dimensional space and may correspond to the same object, they are retained as candidate matching pairs and entered into subsequent fine matching based on projection overlap rate and semantic similarity; otherwise, they are directly excluded, thereby reducing the computational cost of subsequent fine matching.

[0054] After passing through the 3D coarse sieve, the projection overlap rate of the instance on the XY plane is further calculated: (4) in, and Let i and j represent the projected polygons of instance i and instance j on the XY plane, respectively. The projected polygons can be represented by alpha-shapes, directed bounding boxes, or other two-dimensional contours. Indicates area.

[0055] The visual semantic feature similarity between instances can be represented by cosine similarity: (5) in, and Let i and j represent the visual semantic feature vectors of instance i and instance j, respectively. For the set of real numbers, The dimension of the visual semantic feature vector; When both geometric overlap and semantic similarity meet the preset threshold conditions, instance i and instance j are determined to correspond to the same object, and point cloud alignment, point cloud merging, and feature fusion are performed. The geometric overlap includes the three-dimensional intersection-union ratio of the above formula (3). The overlap rate of planar projections with formula (4) The semantic similarity is the cosine similarity of visual semantic features according to formula (5). ;when , and If each value exceeds its respective preset threshold, instance i and instance j are determined to correspond to the same object.

[0056] The fused instance semantic features can be represented as: (6) in, and These represent the instance detection confidence scores, and These represent the cumulative number of fusions or the number of observations, respectively. This indicates a normalization operation.

[0057] Formula (3) is used for coarse-grained intersection screening, formula (4) is used to characterize planar geometric overlap, formula (5) is used to suppress erroneous merging of heterogeneous instances, and formula (6) is used to stabilize instance semantic features under multiple observations.

[0058] Step S4: Construct a 3D hierarchical scene diagram; A 3D hierarchical scene graph is generated based on the merged object instances. This 3D hierarchical scene graph includes object nodes, component nodes, and the spatial relationships between objects. Spatial relationships include, but are not limited to, being on, being inside, belonging, and part (part-to-part composition). Relationships such as of, support, and include.

[0059] In this embodiment, object relationships are inferred based on directed bounding boxes or alpha-shape contours. For instance i, its point cloud's lower bound, upper bound, and center height in the vertical direction are respectively represented as... .

[0060] Instance i can be considered close to the support plane or root node when it meets the following conditions in the vertical direction with respect to the root node or support plane: (7) in, The vertical distance threshold. The vertical height of the supporting plane or root node; When instance i and instance j have sufficient overlap in the horizontal plane, and the bottom of instance i is close to the top of instance j, while the center height of instance i is above the top of instance j, it can be determined that instance i is above instance j. (8) in, Let be the projection overlap rate of instance i and instance j on the horizontal plane. The horizontal overlap rate threshold; When instance i and instance j have sufficient overlap in the horizontal plane, and the center height of instance i is below the upper bound of instance j, it can be determined that instance i belongs to or is located within the internal region of instance j. (9) in, This represents the horizontal overlap threshold.

[0061] Formula (8) can be used to generate (on) relations, and formula (9) can be used to generate (in) or (belong) relations.

[0062] Step S5: Update the voxel semantic feature map; For a three-dimensional point in the world coordinate system It can be based on the voxel side length Three-dimensional points in the current world coordinate system Discretize into voxel indexes: (10) in, Indicates the origin of the voxel map. This indicates the floor function.

[0063] The continuous three-dimensional coordinates are discretized into voxel index k, which is used to quantize the continuous space into a regular voxel grid. This enables semantic features observed at different times and from different perspectives to be accumulated and incrementally fused according to the same voxel index k, and supports voxel-based open vocabulary retrieval. For voxels If its historical semantic features are The cumulative confidence score is The current observed features are The current observation confidence score is Then the updated weights can be expressed as: (11) The voxel semantic features are updated as follows: (12) in, This is a normalization operation; The cumulative confidence score has been updated to: (13) in, The confidence score can be determined by the segmentation confidence score, detection confidence score, viewpoint quality, or a combination thereof. In this embodiment, the confidence score of the current observation can be expressed as: ;in, This represents the segmentation confidence level of the current observation. Indicates the detection confidence level. Indicates the quality of the viewpoint. These are the corresponding weighting coefficients.

[0064] When searching with open vocabulary, semantic features can be queried. With voxel semantic features Searching using cosine similarity: (14) Formula (10) is used to discretize continuous three-dimensional coordinates, formulas (11) to (13) are used for incremental fusion of voxel semantic features, and formula (14) is used to support open vocabulary target localization.

[0065] Step S6: Update the navigation occupancy map and generate a collision-free approach pose; A two-dimensional navigation occupancy map is generated based on point cloud projection, depth information, or an external grid map. Among them, the point cloud projection generation refers to projecting the three-dimensional point cloud vertically onto the horizontal plane, mapping it to a grid index according to the following formula (15), marking the grids falling into the obstacle height range as obstacles, and marking the remaining observable grids as passable; the depth information generation refers to obtaining the local point cloud by backprojection from the depth map and then projecting and rasterizing it in the above manner; the external grid map generation refers to directly reading the external SLAM or prior grid map and aligning and fusing it with the above occupancy results according to the grid; then, the obstacle grid is safely expanded according to formula (16), and the distance transformation is calculated according to formula (17) to obtain the final two-dimensional navigation occupancy map.

[0066] For two-dimensional position in the world coordinate system Its corresponding navigation grid index can be represented as: (15) in, 'c' indicates that the navigation occupies the origin of the map, and 'c' represents the raster resolution.

[0067] To ensure a safe distance between the mobile chassis and the robotic arm's overall envelope and the obstacle, the obstacle area can be expanded according to the safe radius r. The expansion core size can be expressed as: (16) in, This indicates the rounding up operation.

[0068] After obstacle expansion is completed, the distance from each passable grid cell to the nearest obstacle can be calculated using distance transformation: (17) in, Represents a set of obstacle grids. Index the obstacle grid. Select grids that satisfy the safe distance constraint as candidate approach poses and map them to robot poses in the world coordinate system.

[0069] Formula (15) is used to complete the mapping between world coordinates and grid coordinates, formula (16) is used to ensure the safety margin between the path and obstacles, and formula (17) is used to generate the safe distance index of the passable area.

[0070] Step S7: Perform visual cone occlusion determination; To reduce perception failures caused by target occlusion before the robotic arm executes, based on the target node center... Generate candidate viewpoints based on the target surface normal n: (18) in, Indicates the viewpoint offset distance; For a candidate viewpoint, if the area of ​​the intersection of the polygons of the candidate occluding node and the target node in the view frustum projection satisfies: (19) in, This is the threshold for the occlusion area. Let be the intersection area of ​​the polygons in the view cone projection. For the target node; If a candidate occluder is determined to pose a risk of obstructing the target, viewpoints that severely obstruct the target can be eliminated, or a new approach pose can be selected.

[0071] Step S8: Perform dynamic topology update based on robot operation results; After the robot completes operations such as grasping, placing, moving, or opening and closing, the dynamic update module updates the object positions, parent-child relationships, and spatial topology relationships in the scene graph according to the operation results.

[0072] For example, after the placement operation is complete, the object 'o' can be moved from its original parent node to the new target node 'target': (20) If the target node has the openable or containable property, the relationship between the manipulated object and the target node is updated to (in); otherwise, the relationship between them is updated to (on). (twenty one) in, Represents a collection of objects that have the property of being able to be opened or contained.

[0073] In this way, the robot's execution results can be used to update the three-dimensional scene map in reverse, so that the map state is consistent with the real environment.

[0074] Step S9: Perform cross-source confidence arbitration for the three-source map; When the 3D hierarchical scene map, the 3D voxel semantic feature map, and the 2D navigation occupancy map give different labels to the same world point p, a weighted voting method is used to determine the final label: (twenty two) in, For the candidate tag set; for ; For indicator functions; The weight of map source m at location p; A three-dimensional scene diagram; A three-dimensional voxel semantic feature map; For two-dimensional navigation occupancy map; In this embodiment, the map source weight can be expressed as: (twenty three) in, This represents the source-level prior weights of the map. This represents the confidence level of the corresponding map source at location p. This represents the sigmoid function.

[0075] For a 3D hierarchical scene map, the confidence level can be derived from the instance detection confidence level and the number of fusions; for a 3D voxel semantic feature map, the confidence level can be derived from the cumulative voxel score; for a 2D navigation occupancy map, the confidence level can be derived from the number of occupancy votes. This arbitration mechanism can prevent a single map source from dominating the final result. When the voxel semantic features are inconsistent with the instance semantics, or the occupancy grid state is inconsistent with the instance state, the target label is re-determined based on the cross-source confidence level, and instance updates, re-observations, or instance elimination are triggered. The cross-source confidence level refers to the degree of credibility of the labels given to the same world point p on the 3D hierarchical scene map M1, the 3D voxel semantic feature map M2, and the 2D navigation occupancy map M3, which is the map source weight in the above formula (23). It is determined by the map source-level prior weights. confidence level with respect to the map source at location p The target label is determined jointly; the target label here is the final label obtained by weighted voting according to formula (22). This aligns with the final label determined by the aforementioned weighted voting method.

[0076] Step S10: Jointly optimize navigation pose and view frustum visibility; To simultaneously satisfy both chassis collision-free and robotic arm visibility constraints, the joint cost is calculated for candidate navigation poses q, and the pose with the minimum cost is selected: (twenty four) ; The distance to the nearest obstacle; To prevent constants with a denominator of zero; This represents the robot's current or default pose. To observe the target node from the candidate pose The cost of occlusion at that time; and These are the weighting coefficients.

[0077] In this embodiment, the occlusion cost can be expressed as: (25) in, This represents the set of nodes that may occlude the target from the candidate viewpoint. This represents the area of ​​the intersection of the occluding node and the target node in the view frust projection; Let be the intersection area of ​​the polygons in the view cone projection. The target node.

[0078] Formula (24) unifies navigation safety distance, motion cost and view cone visibility into the same optimization goal, which can reduce the failure situation where "the chassis has been navigated to the correct position but the robotic arm cannot observe or operate the target".

[0079] Step S11: Select the capture anchor point based on the component node; For target instance i, its set of component nodes is represented as: (26) Candidate component nodes are scored based on instruction intent, component visibility, crawling feasibility, and occlusion status, and crawling anchor points are selected accordingly: (27) in, Semantic embeddings that represent user instructions or task intent; This indicates the semantic similarity between component nodes and instruction intents. Indicates the visibility ratio of component nodes from the current viewpoint; Indicates the feasibility of component grabbing; Indicates the area of ​​the component obstructed by the grasping view frustum; These are the weighting coefficients.

[0080] In this way, the (belong) end nodes in the scene graph are not only used to express the topological relationship of objects, but also serve as semantic anchors for robotic arm grasping, pressing, pulling or placing operations, enabling robots to perform more precise operations on parts such as handles, buttons, cup rims, and bottle caps.

[0081] Step S12: Perform three-source hierarchical asynchronous update and uncertainty propagation; In this implementation, the three-source maps employ a hierarchical asynchronous update strategy. A time window is defined. The update counts of the internal 3D hierarchical scene map, the 3D voxel semantic feature map, and the 2D navigation occupancy map are respectively... Then it can be expressed as: (28) (29) (30) in, Indicates the voxel feature map update cycle; Indicates the navigation occupancy map update cycle. Represents a set of perceived events. This represents a collection of placement or manipulation events. This represents the set of events that change due to obstacles.

[0082] This scheduling strategy enables high-frequency voxel updates, event-driven scene graph updates, and low-frequency navigation graph updates, reducing the overhead of ineffective high-frequency reconstruction.

[0083] To uniformly express instance semantic confidence, cumulative voxel score, number of votes, cross-source label consistency, and time freshness, instance-level uncertainty is defined: (31) in, Indicates the confidence level of instance detection. This indicates the cumulative number of times an instance has been merged. This indicates the maximum number of times an instance has been merged in history. Represents cross-source tag entropy. This represents the instance labels provided by the 3D hierarchical scene graph. This represents the search tags given by the voxel semantic feature map. This represents the distance from the candidate pose near the target to the obstacle. Indicates the elapsed time since the last observation. Represents the time decay constant. These are the weighting coefficients. To prevent extremely small positive numbers with a denominator of zero; when When the value exceeds a preset threshold, perform at least one of the following actions: reduce the weight of the corresponding map source in cross-source arbitration; trigger the robot to re-observe the target area; call higher resolution voxel retrieval or local reconstruction; reduce the grasping priority of the instance; suppress blind grasping operations on targets with high uncertainty.

[0084] By leveraging the aforementioned uncertainty propagation mechanism, we can comprehensively consider factors such as the amount of evidence, cross-source consistency, safe distance, and time freshness, thereby improving the reliability of map updates, target retrieval, and robotic arm operation decisions in dynamic environments.

[0085] In summary, this invention achieves unified maintenance and closed-loop updating of the three-dimensional hierarchical scene graph M1, the three-dimensional voxel semantic feature graph M2, and the two-dimensional navigation occupancy graph M3 through instance cross-frame fusion, voxel semantic feature incremental fusion, scene graph spatial relationship determination, navigation occupancy graph security constraints, view frustum occlusion determination, cross-source confidence arbitration, component-level grasping anchor point selection, and hierarchical asynchronous update mechanism. This enables embodied intelligent robots to complete target retrieval, navigation approach, robotic arm operation, and execution feedback updates in complex dynamic environments.

[0086] The present invention also provides a multi-source map fusion scene graph construction system for embodied intelligent robots. The multi-source map fusion scene graph construction system for embodied intelligent robots can be implemented by executing the process steps of the multi-source map fusion scene graph construction method for embodied intelligent robots. That is, those skilled in the art can understand the multi-source map fusion scene graph construction method for embodied intelligent robots as a preferred embodiment of the multi-source map fusion scene graph construction system for embodied intelligent robots.

[0087] Example 2 Example 2 is a preferred example of Example 1. This invention relates to an embodied intelligent robot system, which includes: a robot body, a robotic arm, an end effector, a vision sensor, and a computing unit. The robot body can be a mobile chassis or a fixed base; the robotic arm can be a six-degree-of-freedom, seven-degree-of-freedom, or other multi-degree-of-freedom robotic arm; the end effector can be a gripper, a suction cup, or a combination thereof; and the vision sensor can include one or more RGB-D cameras, depth cameras, or other three-dimensional perception sensors.

[0088] During operation, the robot acquires environmental images, depth information, camera intrinsic parameters, and camera pose through visual sensors, and identifies object instances in the environment through a perception module. Based on the above perception results, it constructs and maintains multiple complementary maps, including: A three-dimensional scene diagram is used to represent object nodes, spatial relationships, and component nodes; Three-dimensional voxel semantic feature maps are used to represent three-dimensional geometric structures and visual semantic features; A two-dimensional navigation occupancy map is used to represent passable areas, obstacle areas, and navigation safety distance information.

[0089] The aforementioned complementary maps are linked through a unified world coordinate system and dynamically updated based on new environmental observations and robot execution results, thereby supporting robots in completing target retrieval, navigation planning, and robotic arm operation tasks in scenarios such as home service, industrial picking, item sorting, grasping and placing.

[0090] A multi-source map fusion scene graph construction system for embodied intelligent robots, provided by the present invention, includes: a perception processing module, an instance pool module, a scene graph construction module, a voxel feature map module, a navigation occupancy map module, a dynamic update module, and an embodied execution module, such as... Figure 1 As shown.

[0091] The perception processing module receives sensor data such as RGB-D images, camera intrinsics, and camera pose, and generates object instance information in the environment. The object instance information includes the object's two-dimensional bounding box, segmentation mask, category label, visual semantic features, and optional six-degree-of-freedom pose information. This module provides input to the instance pool module, scene graph construction module, and voxel feature map module.

[0092] The instance pool module is used to maintain a cross-frame object instance set and to associate, fuse, and update object instances observed at different times, from different viewpoints, or from different sensors. Through instance matching and merging, the instance pool module outputs fused stable object instances and provides a unified object index for the 3D scene graph and voxel feature map.

[0093] The scene graph construction module is used to construct a three-dimensional hierarchical scene graph based on the fused object instances. The three-dimensional hierarchical scene graph includes object nodes, component nodes, and spatial relationships between objects. Spatial relationships include, but are not limited to, being on top of, inside, belonging to, supporting, occluding, and adjacent. Specifically, the scene graph construction module, based on the fused object instances and according to the directed bounding box or alpha-shape contour of each object instance, infers spatial relationships such as being on top of, inside, belonging to, supporting, occluding, and adjacent between objects, thereby constructing the three-dimensional hierarchical scene graph.

[0094] In this embodiment, the scene graph construction module can infer object relationships based on geometric information such as object vertices, center points, bounding boxes, convex and concave hulls, or directed bounding boxes; it can also load template priors to generate auxiliary nodes for representing components, containers, or functional areas.

[0095] The voxel feature map module is used to maintain voxel-level semantic features and geometric information in 3D space. This module can fuse visual semantic features to corresponding voxels and update the 3D geometric structure by combining depth information. The voxel feature map is complementary to the object instances in the scene map and can be used for open vocabulary retrieval, local spatial query, and target region localization. Specifically, updating the 3D geometric structure by combining depth information refers to discretizing the 3D points obtained by backprojecting the depth map into voxel indices and accumulating the occupancy observations and confidence scores of each voxel, thereby maintaining the 3D geometry expressed by the voxel occupancy structure online. The local spatial query and target region localization refer to retrieving candidate voxels based on the cosine similarity between the query semantic features and the voxel semantic features, and performing spatial clustering on adjacent voxels with similarity exceeding a threshold to obtain the 3D local region where the target is located. Maintaining geometric information specifically involves: discretizing continuous 3D coordinates into voxel indices, expressing the 3D geometry with the occupancy structure of a regular voxel grid; and continuously accumulating the occupancy status and confidence scores of each voxel with new depth observations, thereby maintaining and updating the 3D geometric structure online.

[0096] The navigation occupancy map module is used to generate a two-dimensional navigation occupancy map based on point clouds, depth information, or 3D maps. The two-dimensional navigation occupancy map includes passable areas, obstacle areas, safety expansion areas, and distance transformation indexes. This module provides a basis for mobile chassis navigation, approach pose selection, and safe position planning before robotic arm operations.

[0097] The dynamic update module is used to update the map online based on the robot's execution results. After the robot completes grasping, placing, moving, or other operations, this module updates the object positions, spatial relationships, and topology in the scene map based on feedback from the end effector, changes in object state, and new visual observations. This module can also perform occlusion detection and viewpoint evaluation to determine whether the current viewpoint meets the requirements for subsequent perception or operation.

[0098] The embodied execution module is used to generate and execute robot actions based on the 3D scene map, voxel feature map, and navigation occupancy map. After navigating to the target area, this module can trigger robotic arm grasping and placement skills, and write feedback information back to the map based on the robotic arm end-effector pose, execution status, and operation results, thus forming a closed loop of "perception-mapping-planning-execution-updating".

[0099] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0100] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for constructing a scene graph through multi-source map fusion for embodied intelligent robots, characterized in that, include: Step S1: Acquire RGB-D environmental perception data and corresponding camera parameters and pose information, and complete the conversion from 3D coordinates to world coordinates; Based on the acquired RGB-D environmental perception data, target detection, instance segmentation, and visual semantic feature extraction are performed on the environment. The set of object instances in the world coordinate system is generated by combining depth information, and invalid noise instances are filtered out. Step S2: Under the unified world coordinate system, perform spatial geometric matching and semantic feature fusion on object instances obtained from multi-frame and multi-view observations to eliminate duplicate instances and obtain globally unified stable object instances. Step S3: Based on the globally unified stable object instance, extract the geometric contour information of the object, infer the multi-class spatial topological relationships between objects, and construct a three-dimensional hierarchical scene graph containing object nodes and component nodes; Step S4: Perform voxelization discretization on the three-dimensional space, and fuse the three-dimensional geometric structure information and visual semantic features incrementally in units of voxels to form a three-dimensional voxel semantic feature map that supports target retrieval and spatial positioning. Step S5: Generate a two-dimensional planar grid map based on three-dimensional perception information, mark passable areas and obstacle areas, and after safety dilation and distance transformation processing, select robot candidate approach poses that meet safety distance constraints to construct a two-dimensional navigation occupancy map; Step S6: When the 3D hierarchical scene map, 3D voxel semantic feature map, and 2D navigation occupancy map output inconsistent representation information for the same spatial location, combine the prior weights and local confidence of each map source, and determine a unified environmental representation result through a weighted voting mechanism to correct the single-source perception bias. Step S7: Based on the unified environmental characterization results, construct a joint optimization objective that includes navigation safety distance, motion cost, and target occlusion cost, and solve for the optimal navigation pose that simultaneously satisfies the chassis collision-free constraint and the robot arm target visibility constraint from the candidate approach poses; Step S8: Based on the unified environmental representation results, the component nodes in the 3D hierarchical scene map are used as candidate operation points. The task intent, component visibility, grasping feasibility and occlusion are combined to make a comprehensive score, and the optimal component node is selected as the grasping anchor point for the robotic arm operation.

2. The method for constructing a multi-source map fusion scene graph for embodied intelligent robots according to claim 1, characterized in that, Step S1 includes: Step S1.1: Acquire environmental images and depth maps based on several RGB-D vision sensors; simultaneously, acquire camera intrinsic parameters. K and camera pose ; Step S1.2: For pixels ) Its depth value The corresponding 3D points in the camera coordinate system are: Among them, superscript For transpose; Step S1.3: Based on camera pose Transform 3D points to the world coordinate system: in, Represents a 3D point in the camera coordinate system. Represents a three-dimensional point in the world coordinate system; Step S1.4: Perform target detection and instance segmentation based on the RGB image in the RGB-D observation to obtain a two-dimensional detection box and segmentation mask; Step S1.5: Extract visual semantic features based on the image region corresponding to the segmentation mask in the RGB image to obtain visual semantic features; Step S1.6: Based on depth map and camera intrinsic parameters K and camera pose The pixels within the mask area are back-projected onto the world coordinate system to generate a 3D point cloud of the current object instance. Step S1.7: For point cloud numbers below a preset threshold Alternatively, instances located outside the preset robot workspace can be filtered.

3. The method for constructing a multi-source map fusion scene graph for embodied intelligent robots according to claim 1, characterized in that, Step S2 includes: Step S2.1: For the new instance obtained from the current observation i Historical examples j The three-dimensional intersection-union ratio of the two is expressed as: in, and Let i and j represent the 3D axis-aligned bounding box volumes of instance i and instance j, respectively. This represents the volume of the intersection of the two bounding boxes; when Greater than the preset threshold When it is determined that instance i and instance j satisfy the preset overlap condition in three-dimensional space, they are retained as candidate matching pairs; Calculate the projection overlap of the instance on the XY plane: in, and Let i and j represent the projected polygons of instance i and instance j on the XY plane, respectively. Indicates area; The visual semantic feature similarity between instances is represented by cosine similarity: in, and Let i and j represent the visual semantic feature vectors of instance i and instance j, respectively. For the set of real numbers, The dimension of the visual semantic feature vector; When the projection overlap rate Similarity to cosine If each value exceeds its respective preset threshold, it is determined that instance i and instance j correspond to the same object; The fused instance semantic features are represented as follows: in, and These represent the instance detection confidence scores, and These represent the cumulative number of fusions or the number of observations, respectively. This indicates a normalization operation.

4. The method for constructing a multi-source map fusion scene graph for embodied intelligent robots according to claim 1, characterized in that, The three-dimensional hierarchical scene graph includes: object nodes, component nodes, and spatial relationships between objects; the spatial relationships include: on, in, belong, part-of, support, and occlude relationships. Inferring object relationships based on directed bounding boxes or alpha-shape contours includes: For instance i, its point cloud's lower bound, upper bound, and center height in the vertical direction are respectively represented as: ; Instance i is considered to be close to the support plane or root node when it meets the following conditions in the vertical direction with respect to the root node or support plane: in, The vertical distance threshold. The vertical height of the supporting plane or root node; When instance i and instance j overlap on a horizontal plane and the degree of overlap meets a preset requirement, and the bottom of instance i is close to the top of instance j, while the center height of instance i is above the top of instance j, then instance i is determined to be above instance j. in, Let be the projection overlap rate of instance i and instance j on the horizontal plane. The horizontal overlap rate threshold; When instance i and instance j overlap on the horizontal plane and the degree of overlap meets the preset requirements, and the center height of instance i is below the upper limit of instance j, it is determined that instance i belongs to or is located within the internal region of instance j. in, Indicates the horizontal overlap threshold; After the robot completes the grasping and placement operations, it updates the position of objects, parent and child nodes, and spatial topological relationships in the 3D hierarchical scene map based on the operation results, so that the map state keeps synchronized with the real environment.

5. The method for constructing a multi-source map fusion scene graph for embodied intelligent robots according to claim 1, characterized in that, Step S4 includes: Step S4.1: For a 3D point in the world coordinate system According to the voxel side length Discretize it into voxel indexes: in, Indicates the origin of the voxel map. This indicates a round-down operation; Step S4.2: Index the semantic features observed at different times and from different perspectives according to the same voxel. To achieve incremental voxel feature updates, cumulative and incremental fusion is performed, including: For voxels If its historical semantic features are The cumulative confidence score is The current observation features are The current observation confidence score is The updated weights are then expressed as: The voxel semantic features are updated as follows: in, This is a normalization operation; The cumulative confidence score has been updated to: ； Step S4.3: When performing open vocabulary retrieval, query semantic features. With voxel semantic features Searching using cosine similarity: 。 6. The method for constructing a multi-source map fusion scene graph for embodied intelligent robots according to claim 1, characterized in that, Step S5 includes: Step S5.1: For the two-dimensional position in the world coordinate system The corresponding navigation grid index is represented as: in, This indicates that the navigation occupies the origin of the map. c Indicates raster resolution; Step S5.2: Mark the grid cells that fall within the obstacle height range as obstacle areas, and mark the remaining observable grid cells as passable areas; Step S5.3: Expand the obstacle region according to the safety radius r. The size of the expansion core is expressed as: in, This indicates the rounding up operation; Step S5.4: After completing obstacle expansion, calculate the distance from each passable grid cell to the nearest obstacle using distance transformation; in, Represents a set of obstacle grids; For obstacle grid index; Step S5.5: When the distance from the passable grid to the nearest obstacle is greater than or equal to a preset value, the safe distance constraint is satisfied; select the grid that satisfies the safe distance constraint as the candidate approach pose and map it to the robot pose in the world coordinate system.

7. The method for constructing a multi-source map fusion scene graph for embodied intelligent robots according to claim 1, characterized in that, Step S6 includes: when the 3D hierarchical scene map, the 3D voxel semantic feature map, and the 2D navigation occupancy map give different labels to the same world point p, a weighted voting method is used to determine the final label: in, For the candidate tag set; for ; For indicator functions; The weight of map source m at location p; A three-dimensional scene diagram; A three-dimensional voxel semantic feature map; For two-dimensional navigation occupancy map; in, This represents the source-level prior weights of the map. This represents the confidence level of the corresponding map source at location p. Represents the sigmoid function; The representation information of the corresponding map is corrected based on the final label, or an instance update or re-observation is triggered.

8. The method for constructing a multi-source map fusion scene graph for embodied intelligent robots according to claim 1, characterized in that, Step S7 includes: To simultaneously satisfy both chassis collision-free and robotic arm visibility constraints, the joint cost is calculated for candidate navigation poses q, and the pose with the minimum cost is selected: ; The distance to the nearest obstacle; To prevent constants with a denominator of zero; This represents the robot's current or default pose. To observe the target node from the candidate pose The cost of occlusion at that time; and These are the weighting coefficients; in, This represents the set of nodes that may occlude the target from the candidate viewpoint. This represents the area of ​​the intersection of the occluding node and the target node in the view frust projection; Let be the intersection area of ​​the polygons in the view cone projection. The target node.

9. The method for constructing a multi-source map fusion scene graph for embodied intelligent robots according to claim 1, characterized in that, Step S8 includes: For target instance i, its set of component nodes is represented as: Candidate component nodes are scored based on instruction intent, component visibility, crawling feasibility, and occlusion status, and crawling anchor points are selected accordingly: in, Semantic embeddings that represent user instructions or task intent; This indicates the semantic similarity between component nodes and instruction intents. Indicates the visibility ratio of component nodes from the current viewpoint; Indicates the feasibility of component grabbing; Indicates the area of ​​the component obstructed by the grasping view frustum; These are the weighting coefficients.

10. A multi-source map fusion scene graph construction system for embodied intelligent robots, characterized in that, include: Module M1 is used to acquire RGB-D environmental perception data and corresponding camera parameters and pose information, and to complete the conversion from 3D coordinates to world coordinates. Based on the acquired RGB-D environmental perception data, target detection, instance segmentation, and visual semantic feature extraction are performed on the environment. The set of object instances in the world coordinate system is generated by combining depth information, and invalid noise instances are filtered out. Module M2 is used to perform spatial geometric matching and semantic feature fusion on object instances obtained from multi-frame and multi-view observations in a unified world coordinate system, eliminate duplicate instances, and obtain globally unified stable object instances. Module M3 is used to extract the geometric contour information of objects based on the globally unified stable object instances, infer the multi-class spatial topological relationships between objects, and construct a three-dimensional hierarchical scene graph containing object nodes and component nodes. Module M4 is used to perform voxelization discretization processing on three-dimensional space, and to fuse three-dimensional geometric structure information and visual semantic features incrementally in units of voxels to form a three-dimensional voxel semantic feature map that supports target retrieval and spatial positioning. Module M5 is used to generate a two-dimensional planar grid map based on three-dimensional perception information, mark passable areas and obstacle areas, and after safety dilation and distance transformation processing, select robot candidate approach poses that meet safety distance constraints to construct a two-dimensional navigation occupancy map. Module M6 is used to determine a unified environmental representation result by combining the prior weights and local confidence of each map source when the three-dimensional hierarchical scene map, the three-dimensional voxel semantic feature map, and the two-dimensional navigation occupancy map output inconsistent representation information for the same spatial location, thereby correcting the single-source perception bias. Module M7 is used to construct a joint optimization objective based on unified environmental characterization results, including navigation safety distance, motion cost, and target occlusion cost, and solve for the optimal navigation pose that simultaneously satisfies the chassis collision-free constraint and the robot arm target visibility constraint from candidate approach poses; Module M8 is used to select the optimal component node as the gripping anchor point for the robotic arm operation based on the unified environmental representation results, using component nodes in the 3D hierarchical scene map as candidate operation points, and combining task intent, component visibility, grasping feasibility and occlusion situation for comprehensive scoring.

Images