Road topology analysis and scene hierarchical representation method under open set perception reasoning

Abstract

According to the road topology analysis and scene hierarchical representation method under environment open set perception reasoning, the image mask of the object instance is obtained, and the limitation of closed set environment perception is avoided; in combination with laser data, obtaining a three-dimensional instance of an object, and improving instance representation and open set understanding capability of a three-dimensional space of a complex open environment; based on object instances and large language model analysis and inference, artificially defined abstract objects or areas are obtained, an instance semantic map containing explicit and abstract objects is constructed, the environment cognition degree is improved, and the gap with the human cognition environment is shortened; in combination with an instance object map, point cloud geometric information and open vocabulary road semantics, environment obstacle information is extracted, a road Voronoi diagram is constructed, and the topology precision and generalization of an open scene are improved; finally, environment hierarchical representation containing geometry-semantics-topology is constructed, efficient understanding, reasoning and hierarchical representation of the environment are achieved, and an environment representation basis is provided for task application such as environment understanding, reasoning, query, positioning and planning.

Description

Technical Field

[0001] The present invention belongs to the technical field of hierarchical scene map representation, and specifically relates to a road topology analysis and scene hierarchical representation method under open set environmental perception reasoning. Background Art

[0002] Environmental perception, reasoning and hierarchical representation technologies play an important role in robots' understanding, cognition and memory of the environment, and are widely used in subsequent logistics, autonomous driving, home services, etc. Existing environmental perception, reasoning and hierarchical representation technologies face many challenges when dealing with complex open scenes, especially in semantic understanding and topological analysis, which are manifested in open set object instance representation, abstract object reasoning, drivable space representation, and reasonable generation of road topology.

[0003] In terms of semantic understanding, the current mainstream environmental map representation mostly relies on detection models based on a limited predefined vocabulary library. The detection effect of objects outside the vocabulary library is poor, which limits the system's understanding of open scene objects in complex environments and cannot flexibly adapt to diverse real-world open scenes. In addition, the current map structure also lacks analysis and reasoning of multi-level abstract objects. For example, a construction area composed of multiple cones. These abstract objects are usually artificially defined abstract objects (construction areas) based on several detected objects (cones), which affects road structure analysis and environmental understanding representation.

[0004] In terms of topological analysis, existing methods usually rely on lidar or depth cameras to obtain the geometric structure of the environment, and use geometric methods to extract the drivable areas of indoor and outdoor environments, without fully considering the impact of the semantic information of the driving space and abstract objects on the driving space; existing road topology analysis and intersection recognition methods are mostly concentrated in standard road environments, and their expansion and generalization capabilities are limited, and they perform poorly in non-standard road scenes (without road lines, etc.); some road topology methods only rely on their own dynamic trajectories to obtain road topology and lack environmental understanding.

[0005] In view of this situation, there is an urgent need for a road topology analysis and scene hierarchical representation method that integrates geometry, semantics and topology representation for open and complex scenes based on the open set perception and reasoning capabilities of the environment, so as to improve the environmental cognition and representation capabilities in open and complex scenes. Summary of the invention

[0006] In view of this, the present invention provides a road topology analysis and scene hierarchical representation method under open set environmental perception reasoning. This method adopts advanced open set environmental perception technology, combines forward-looking cameras and laser radars to achieve open set three-dimensional environmental perception, avoiding the limitations of closed set environmental perception; uses a large language model to reason about abstract objects, reasoning and cognizing the environment from the perspective of instance reasoning, shortening the gap with human cognitive environment; based on rich perception and reasoning data, environmental obstacle information is obtained, and road topology structure is extracted to promote the accuracy and generalization of road topology analysis technology in complex open scenes. Combined with the above-mentioned entire environmental perception and reasoning results, a hierarchical scene representation of a complex open environment is constructed to achieve efficient environmental expression.

[0007] A road topology analysis and scene hierarchical representation method based on open set perception reasoning, including:

[0008] Step 1: Build explicit and abstract object instance semantic maps:

[0009] Explicit objects are defined as object results directly detected by the perception model; abstract objects are defined as object results that are artificially defined and need to be obtained through semantic reasoning;

[0010] Based on the front view image, the open set object mask and features of the environment are obtained through the open set perception model. Combined with the lidar point cloud, the three-dimensional representation of the object instance is constructed through feature fusion technology, and an open vocabulary instance semantic map based on explicit objects is constructed; combined with the object instance set and the user language description, the large language model parses the user description, and retrieves and analyzes the instance relationship that meets the requirements from the object instance set to obtain the artificially defined abstract object instance, and constructs an open vocabulary instance semantic map based on abstract objects. Finally, a semantic map containing explicit and abstract object instances is constructed;

[0011] Step 2: Road topology analysis:

[0012] Based on the explicit and abstract object instance semantic maps established in step 1, combined with the point cloud geometric scale constraint information and the image road semantic mask constraint information, a comprehensive obstacle map of the environment is extracted from the instance object, geometric point cloud, and road semantics to construct a dense Voronoi map of the environment road, and the dense Voronoi map is processed by a sparse method to obtain a road backbone Voronoi map, and the road intersection type is obtained by analysis;

[0013] Step 3, perception-enhanced hierarchical scene representation: Based on the instance semantic map obtained in step 1 and the road backbone map and road intersection types obtained in step 2, a complex environment hierarchical environment representation is constructed, and a semantic point cloud layer, a semantic instance layer, a road topology layer, an intersection node map, and an environment layer are constructed in sequence; top-down hierarchical query, abstract object reasoning, global path planning, and instance dynamic update functions are provided for complex environment understanding, reasoning, and hierarchical dynamic maintenance; the meaning of each layer of map representation is as follows:

[0014] 3.1) Semantic point cloud layer: Combines laser geometry and image semantic perception through point cloud projection to provide environment geometry and semantic information;

[0015] 3.2) Semantic instance layer: Through step 1, an instance map containing explicit objects and abstract objects is constructed, and the two are classified and managed to support object re-reasoning and updating;

[0016] 3.3) Road topology layer: represented as the road backbone Voronoi diagram after removing redundant road branches;

[0017] 3.4) Intersection node layer: represents information of different types of road intersections;

[0018] 3.5) Environment layer: represents the overall description of each scene.

[0019] Preferably, the step 1 comprises the following steps:

[0020] 1.1) Image instance feature extraction:

[0021] The TagText model is used to obtain open vocabulary category labels for objects in the image. The Grounding DINO model is used to implement open set object detection, including detecting 2D bounding boxes, by combining the above category labels with the front view image. The TAP model is used to obtain image masks corresponding to the detection results by combining the above bounding boxes with the front view image. and title Afterwards, the SBERT model is used to encode the object titles obtained above to obtain the title feature f of each object i (t) ; Each image instance feature includes

[0022] 1.2) Global 3D instance representation: The extrinsic matrix is ​​obtained by automatic calibration of camera and laser radar extrinsic parameters, and the laser point cloud data is projected onto the image plane; according to the different instance masks in the image obtained in step 1.1), the 3D point cloud data corresponding to each mask is obtained, thereby obtaining the 3D space instance representation in the current position coordinate system, and these instances are converted to the global coordinate system through the current frame pose to obtain the global 3D instance representation of the current frame; each object instance detected in each current frame is represented as Point clouds representing each object instance title and title feature f i (t) ;

[0023] 1.3) Matching and fusing the object instances in each frame obtained in step 1.2) with the instance objects in the existing global instance map;

[0024] 1.4) Explicit instance map creation: Based on the instance fusion results of step 1.3), an explicit instance map is constructed. The instance map is obtained by detecting and segmenting through an open set perception model and fusion in the global three-dimensional space;

[0025] 1.5) Abstract instance map creation: Based on the explicit instance map obtained in step 1.4) and the abstract object representation provided by the user, the large language model extracts the composition categories and discrimination conditions of the abstract objects; retrieves all relevant instance objects corresponding to the composition categories of the abstract objects from the global map, and generates a relationship description between each relevant instance object and its nearby relevant instance objects; based on the relationship description and the abstract object discrimination conditions, the large language model identifies the object combinations that meet the conditions, constitutes an abstract instance representation, and obtains an abstract instance map.

[0026] Preferably, the step 2 specifically includes:

[0027] 2.1) Collect laser point cloud data in the environment through laser radar, and obtain the global geometric point cloud according to the pose of each frame; in the global geometric point cloud, take the local point cloud within a certain height range near the pose of each frame to include all ground points, which is used to extract the ground equation of each frame; select the point cloud within the set height range from the ground equation as the obstacle point cloud of the current frame; accumulate the obstacle point clouds of all frames as the total obstacle map G1;

[0028] 2.2) Obtaining the road mask ahead by comparing the instance-title cosine similarity with the instance semantic mask and title information obtained in step 1.1); Obtaining the non-road point cloud ahead of each frame by projecting the laser point cloud onto the image, accumulating the non-road point cloud ahead of each frame, and accumulating all frame data within the set height range to obtain the obstacle map G2;

[0029] 2.3) treating the abstract instance map as a potential obstacle map G3;

[0030] 2.4) Integrate the three types of obstacle data sources in steps 2.1), 2.2), and 2.3) above, and accumulate them to generate a global obstacle map G. Ignore its height representation and project it onto a two-dimensional plane to represent the free space and obstacle area; then directly use the Voronoi map algorithm to obtain a global dense Voronoi map representing the road structure for subsequent road topology analysis;

[0031] 2.5) At each frame pose in the obstacle map G in step 2.4), 360 rays of equal angles are emitted to the surroundings, and the intersection points with the obstacles are recorded. These intersection points are connected to obtain a local polygon representing the free space of each frame, and only the Voronoi graph nodes and edges within a certain range of the polygon are selected as the valid Voronoi graph part; the valid Voronoi graph parts of all poses are accumulated to obtain a sparse Voronoi graph after global validation processing;

[0032] 2.6) For the sparse Voronoi graph obtained in step 2.5), redundancy processing is performed;

[0033] 2.7) For the Voronoi diagram after redundancy processing in step 2.6), the type of road intersection is determined according to the number of connecting edges of the node and the shape of the connecting branches, and the adjacent nodes are regarded as the subsidiary nodes of the intersection.

[0034] Preferably, the 1.3) global 3D instance fusion: match and fuse the object instance of each frame obtained in step 1.2) with the instance object in the existing global instance map. The specific steps are as follows: for the object instance detected in the current frame, based on the point cloud, title and title features of the instance, calculate the point cloud geometric similarity Φ between each object instance in the global instance map geo (i,j), title similarity Φ cap (i, j) and the title feature cosine similarity Φ fea (i, j), where i represents the object instance index in the current frame and j represents the object instance index in the global instance map; then the overall weighted similarity score Φ(i, j) is calculated based on the prior weights of each similarity; the above scores are compared to see if they exceed the similarity threshold δ sim , and initialize or fuse instance objects; fusing instance objects requires updating object properties including point cloud, title, and title features. The update process is as follows: update the point cloud and corresponding bounding box by taking the union of the point cloud and re-denoising; update the object title features by weighted fusion of features based on the number of observations; update the object title based on the understanding ability of the visual language large model. The fused and updated instance map result is reused for the instance map update of the next frame.

[0035] Preferably, the redundancy processing includes:

[0036] For the sparse Voronoi graph obtained in step 2.5), the KDTree algorithm is used to calculate the distance from the leaf node to the nearest obstacle. If the distance is less than the distance threshold and the length of the independent branch to which the leaf node belongs is less than the given length threshold, the independent node branch in the Voronoi graph is deleted. Multiple iterations are performed to make all leaf nodes meet the above requirements and delete redundant short branches of the Voronoi graph.

[0037] Preferably, the redundant processing includes: for the Voronoi diagram with short branches deleted, traversing the nodes whose edge connection number is 2, calculating the length and angle between their connected segments, removing the nodes corresponding to the approximately horizontal angles and the nodes with too short lengths, and merging their connecting segments, iterating multiple times until the above-mentioned nodes no longer exist.

[0038] Preferably, the redundant processing includes: for the Voronoi diagram after deleting the nodes whose edge connection number is 2, recording the number of connected edges, the nearest obstacle distance, and the edge length information of each node, and deleting all edges and connections corresponding to the truncated nodes that are less than the length threshold in the Voronoi diagram, ensuring that the nodes maintain a safe distance from the obstacles and the edge lengths are greater than the threshold, thereby reducing node redundancy.

[0039] Preferably, in step 2.7), for the Voronoi diagram after reducing node redundancy, crossroads with 4 connections and angles close to 90 degrees, T-intersections with 3 connections and angles close to 90 degrees, and L-shaped corner nodes with 2 connections and angles close to 90 degrees are identified, and the adjacent nodes are treated as intersection subsidiary nodes.

[0040] Preferably, the set height range is 0.2m to 1.0m.

[0041] The present invention has the following beneficial effects:

[0042] The present invention proposes a road topology analysis and scene hierarchical representation method under open set perception reasoning of the environment, which improves the spatial understanding and hierarchical representation capabilities of open and complex scenes. After acquiring the laser point cloud data and front-view images of the environment, the open-set perception technologies of the environment, such as open vocabulary recognition, detection and segmentation, are used to obtain the image mask of the object instance, thus avoiding the limitations of closed-set environment perception. In combination with the laser data, the three-dimensional instance of the object is obtained, and inter-frame fusion is achieved based on the multi-dimensional instance similarity between frames, thereby improving the instance representation and open-set understanding capabilities of the three-dimensional space of the complex open environment. Based on the set of object instances, the large language model is analyzed and inferred to obtain artificially defined abstract objects or areas, and an instance semantic map containing explicit and abstract objects is constructed to improve the degree of environmental cognition and shorten the gap with the human cognitive environment. In combination with the instance object map, point cloud geometric information, and open vocabulary road semantics, environmental obstacle information is extracted, a road Voronoi diagram is constructed, and road backbones are obtained by sparseness, and road intersections are analyzed and identified to improve the topological accuracy and generalization of open scenes. Finally, a hierarchical representation of the environment containing geometry, semantics, and topology is constructed to achieve efficient understanding, reasoning, and hierarchical representation of the environment, providing an environmental representation basis for tasks such as environmental understanding, reasoning, query, positioning, and planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of the road topology analysis and scene hierarchical representation method under open set perception reasoning of the present invention;

[0044] Figure 2 A schematic diagram of an explicit instance map construction method of the present invention;

[0045] Figure 3 It is a schematic diagram of the road topology structure analysis method of the present invention;

[0046] Figure 4 It is a schematic diagram of the scene hierarchical representation result of the present invention. DETAILED DESCRIPTION

[0047] The present invention is described in detail below with reference to the accompanying drawings and embodiments, and those skilled in the art can understand the advantages and functions of the present invention based on the contents described in this specification. The present invention can also be implemented and applied in other different ways.

[0048] This implementation case uses a forward-looking camera and a 16-line laser radar to perceive the environment. The flow chart of the road topology analysis and scene hierarchical representation method under open set perception reasoning is as follows: Figure 1 As shown, the following steps are included:

[0049] 1) Construction of semantic maps of explicit and abstract object instances: Explicit objects (vehicles, pedestrians, etc.) are object results obtained by direct detection through perception models; abstract objects (construction areas, no-parking areas, parking spaces, etc.) are artificially defined object results that require semantic reasoning. Based on the front view image, the open set object mask and features of the environment are obtained through the open set perception model. Combined with the lidar point cloud, the three-dimensional representation of the object instance is constructed through feature fusion technology, and an open vocabulary instance semantic map based on explicit objects is constructed; combined with the object instance set and the user language description, the large language model parses the user description, and retrieves and analyzes the instance relationship that meets the requirements from the object instance set to obtain artificially defined abstract object instances, and construct an open vocabulary instance semantic map based on abstract objects. Finally, a semantic map containing explicit and abstract object instances is constructed.

[0050] Image perception and instance fusion algorithm process is as follows Figure 2 As shown, it specifically includes the following detailed steps:

[0051] 1.1) Image instance feature extraction: The TagText model, Grounding DINO model, and TAP (Tokenize Anything via Prompting) model are used in turn to extract object features in the image. The TagText model is used to obtain the open vocabulary category labels of the objects in the image; combined with the above category labels and the front view image, the Grounding DINO model is used to implement open set object detection, including detection of 2D bounding boxes, confidence, etc.; combined with the above bounding boxes and the front view image, the TAP model is used to obtain the image mask and title of the corresponding detection result. The overall process can be expressed as follows:

[0052]

[0053] Among them, Rec(.,.), Det(.,.), and SegCap(.,.) represent the TagText model, the Grounding DINO model, and the TAP model, respectively. (t) Represents the current image, Represent the mask and title of the final result respectively. Then, the SBERT model is used to encode the object title obtained above to obtain the title feature f of each object i (t) . Each image instance feature includes

[0054] 1.2) Global 3D instance representation: First, obtain the global 3D instance representation of the current frame. The specific steps are as follows: obtain the extrinsic matrix through automatic calibration of camera and laser radar extrinsic parameters, and project the laser point cloud data onto the image plane; obtain the 3D point cloud data corresponding to each mask according to the different instance masks in the image obtained in step 1.1), thereby obtaining the 3D space instance representation in the current position coordinate system, and convert these instances to the global coordinate system through the current frame pose to obtain the global 3D instance representation of the current frame; each object instance detected in each current frame is represented as Point clouds representing each object instance title and title feature f i (t) .

[0055] 1.3) Global 3D instance fusion: Match and fuse the object instances in each frame obtained in step 1.2) with the instance objects in the existing global instance map. The specific steps are as follows: For the object instance detected in the current frame, based on the point cloud, title and title features of the instance, calculate the point cloud geometric similarity Φ between each object instance in the global instance map. geo (i,j), title similarity Φ cap (i, j) and the title feature cosine similarity Φfea (i, j), where i represents the object instance index in the current frame and j represents the object instance index in the global instance map; then the overall weighted similarity score Φ(i, j) is calculated based on the prior weights of each similarity; the above scores are compared to see if they exceed the similarity threshold δ sim , and initialize or fuse instance objects; fusing instance objects requires updating object properties including point cloud, title, and title features. The update process is as follows: update the point cloud and corresponding bounding box by taking the union of the point cloud and re-denoising; update the object title features by weighted fusion of features based on the number of observations; update the object title based on the understanding ability of the visual language large model. The fused and updated instance map result is reused for the instance map update of the next frame.

[0056] 1.4) Explicit instance map creation: Based on the instance fusion results of step 1.3), an explicit instance map is constructed. The instance map is obtained by detecting and segmenting through an open set detection model and fusion in the global three-dimensional space.

[0057] 1.5) Abstract instance map creation: Based on the explicit instance map obtained in step 1.4) and the abstract object representation provided by the user, the large language model extracts the composition categories and discrimination conditions of the abstract objects; retrieves all relevant instance objects corresponding to the composition categories of the abstract objects from the global map, and generates a relationship description between each relevant instance object and its nearby relevant instance objects; based on the relationship description and the abstract object discrimination conditions, the large language model identifies the object combinations that meet the conditions, constitutes an abstract instance representation, and obtains an abstract instance map. The process is illustrated by taking the abstract object "the polygonal area composed of multiple adjacent cones on the road is a construction area" as an example. The large language model identifies the two object categories of "road" and "cone" and the "polygonal area composed of multiple cones on the road" as the discrimination condition, and then retrieves all "road" and "cone" objects from the global explicit instance map of step 1.4) according to the cosine similarity of the object categories, and then obtains the instance description of each object, for example, "Cone 1 is close to cone x and is on the road". The large language model identifies the "cone" instances that meet the requirements according to the discrimination conditions, and regards the polygon composed of these "cone" instances as the "construction area", and finally constructs the abstract instance map.

[0058] 2) Road topology structure analysis: Based on the explicit and abstract object instance semantic maps established in step 1), combined with the point cloud geometric scale constraint information and the image road semantic mask constraint information, the comprehensive obstacle map of the environment is extracted from the instance object, geometric point cloud, and road semantics to construct a dense Voronoi map of the environment road. The dense Voronoi map is processed by the sparse method to obtain the road backbone map, and the road intersection type is analyzed. The algorithm flow is as follows Figure 2 As shown, it specifically includes the following detailed steps:

[0059] 2.1) Collect laser point cloud data in the environment through laser radar, and obtain the global geometric point cloud according to the pose of each frame; in the global geometric point cloud, take the local point cloud within a certain height range near the pose of each frame to include all ground points, which is used to extract the ground equation of each frame; select the point cloud within the range of 0.2m to 1.0m above the ground equation as the obstacle point cloud of the current frame; accumulate the obstacle point clouds of all frames as the total obstacle map G1;

[0060] 2.2) The instance semantic mask and title information obtained in step 1.1) are used to obtain the road mask ahead by comparing the instance title cosine similarity; the non-road point cloud ahead of each frame is obtained by projecting the laser point cloud onto the image, and the non-road point cloud ahead of each frame is accumulated, with the height restricted to a range of 0.2m to 1.0m, and all frame data are accumulated to obtain the obstacle map G2;

[0061] 2.3) The abstract instance map created in step 1.5) generally represents a construction area consisting of cones, an empty parking space consisting of adjacent vehicles, etc., which is regarded as a potential obstacle map G3;

[0062] 2.4) Integrate the three types of obstacle data sources in steps 2.1), 2.2), and 2.3) above, and accumulate them to generate a global obstacle map G. Ignore its height representation and project it onto a two-dimensional plane to represent the free space and obstacle area; then directly use the Voronoi map algorithm to obtain a global dense Voronoi map representing the road structure for subsequent road topology analysis;

[0063] 2.5) At each frame pose in the obstacle map G in step 2.4), 360 rays of equal angles are emitted to the surroundings, and the intersection points with the obstacles are recorded. These intersection points are connected to obtain a local polygon representing the free space of each frame, and only the Voronoi graph nodes and edges within a certain range of the polygon are selected as the valid Voronoi graph part; the valid Voronoi graph parts of all poses are accumulated to obtain a sparse Voronoi graph after global validation processing;

[0064] 2.6) For the sparse Voronoi graph obtained in step 2.5), the KDTree algorithm is used to calculate the distance from the leaf node to the nearest obstacle. If the distance is less than the distance threshold and the length of the independent branch to which the leaf node belongs is less than the given length threshold, the independent node branch in the Voronoi graph is deleted. Multiple iterations are performed to make all leaf nodes meet the above requirements and to delete the redundant short branches of the Voronoi graph.

[0065] 2.7) For the Voronoi diagram processed in step 2.6), traverse the nodes whose edge connection number is 2, calculate the length and angle between their connected segments, remove the nodes corresponding to the approximately horizontal angle and the short length, and merge their connecting segments, and iterate multiple times until the above nodes no longer exist;

[0066] 2.8) For the Voronoi diagram processed in step 2.7), record the number of connected edges, the nearest obstacle distance, edge length and other information of each node, and delete all edges and connected corresponding truncated nodes that are less than the length threshold in the Voronoi diagram to ensure that the node maintains a safe distance from the obstacle and the edge length is greater than the threshold to reduce node redundancy;

[0067] 2.9) For the Voronoi diagram processed in step 2.8), the type of road intersection is determined based on the number of connected edges of the node and the shape of the connected branches. Crossroads with 4 connections and angles close to 90 degrees, T-intersections with 3 connections and angles close to 90 degrees, and L-shaped corner nodes with 2 connections and angles close to 90 degrees are identified, and the adjacent nodes are treated as intersection subsidiary nodes.

[0068] 3) Perception-enhanced hierarchical scene representation: Based on the instance map obtained in step 1) and the road backbone map and road intersection types obtained in step 2), a hierarchical environment representation of a complex environment is constructed, and the semantic point cloud layer, semantic instance layer, road topology layer, intersection node map, and environment layer are constructed in sequence. It provides top-down hierarchical query, abstract object reasoning, global path planning, instance dynamic update and other functions for complex environment understanding, reasoning and hierarchical dynamic maintenance. The meaning of each layer of map representation is as follows:

[0069] 3.1) Semantic point cloud layer: Combines laser geometry and image semantic perception through point cloud projection to provide environmental geometry and semantic information.

[0070] 3.2) Semantic instance layer: Through step 1), an instance map containing explicit objects and abstract objects is constructed, and the two are classified and managed to support object re-reasoning and updating.

[0071] 3.3) Road topology layer: represented as the road backbone Voronoi diagram after removing redundant road branches in step 2.6).

[0072] 3.4) Intersection node layer: represents the information of different types of road intersections obtained by step 2.9).

[0073] 3.5) Environment layer: represents the overall description of each scene.

[0074] This hierarchical scene representation is applicable to complex indoor and outdoor scenes, and can realize tasks such as environmental understanding, reasoning, query, positioning, and planning. Specifically: environmental understanding is achieved through open set environmental perception; the combination of objects that meet the user's judgment conditions is identified through a large language model to achieve environmental reasoning; the corresponding target object is retrieved and queried through the target object description or category; self-relocation is achieved through graph matching technology based on object instances; given the starting point and target end point, global path planning is achieved based on the road backbone Voronoi diagram. This scene representation method breaks the limitations of closed perception due to its own open set environmental perception capabilities and hierarchical structure that integrates geometry, semantics, and topology. It can respond to various task requirements more flexibly and freely in the face of ever-changing and complex indoor and outdoor scenes, providing an efficient, intelligent, and highly scalable solution for various scene applications, with broad application prospects and huge development potential.

[0075] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A road topology analysis and scene hierarchical representation method based on open set perceptual reasoning, characterized in that: include: Step 1: Build explicit and abstract object instance semantic maps: Explicit objects are defined as objects detected directly by the perception model; Abstract objects are defined as artificial objects that need to be obtained through semantic reasoning; Based on the front view image, the open set object mask and features of the environment are obtained through the open set perception model. Combined with the lidar point cloud, the three-dimensional representation of the object instance is constructed through feature fusion technology, and an open vocabulary instance semantic map based on explicit objects is constructed; combined with the object instance set and the user language description, the large language model parses the user description, and retrieves and analyzes the instance relationship that meets the requirements from the object instance set to obtain the artificially defined abstract object instance, and constructs an open vocabulary instance semantic map based on abstract objects. Finally, a semantic map containing explicit and abstract object instances is constructed; Step 2: Road topology analysis: Based on the explicit and abstract object instance semantic maps established in step 1, combined with the point cloud geometric scale constraint information and the image road semantic mask constraint information, a comprehensive obstacle map of the environment is extracted from the instance object, geometric point cloud, and road semantics to construct a dense Voronoi map of the environment road, and the dense Voronoi map is processed by a sparse method to obtain a road backbone Voronoi map, and the road intersection type is obtained by analysis; Step 3, perception-enhanced hierarchical scene representation: Based on the instance semantic map obtained in step 1 and the road backbone map and road intersection types obtained in step 2, a complex environment hierarchical environment representation is constructed, and a semantic point cloud layer, a semantic instance layer, a road topology layer, an intersection node map, and an environment layer are constructed in sequence; top-down hierarchical query, abstract object reasoning, global path planning, and instance dynamic update functions are provided for complex environment understanding, reasoning, and hierarchical dynamic maintenance; the meaning of each layer of map representation is as follows: 3.1) Semantic point cloud layer: Combines laser geometry and image semantic perception through point cloud projection to provide environment geometry and semantic information; 3.2) Semantic instance layer: Through step 1, an instance map containing explicit objects and abstract objects is constructed, and the two are classified and managed to support object re-reasoning and updating; 3.3) Road topology layer: represented as the road backbone Voronoi diagram after removing redundant road branches; 3.4) Intersection node layer: represents information of different types of road intersections; 3.5) Environment layer: represents the overall description of each scene.

2. The method for road topology analysis and scene hierarchical representation under open set perceptual reasoning as claimed in claim 1, characterized in that: The step 1 comprises the following steps: 1.1) Image instance feature extraction: The TagText model is used to obtain open vocabulary category labels for objects in the image. The Grounding DINO model is used to implement open set object detection, including detecting 2D bounding boxes, by combining the above category labels with the front view image. The TAP model is used to obtain image masks corresponding to the detection results by combining the above bounding boxes with the front view image. and title Afterwards, the SBERT model is used to encode the object titles obtained above to obtain the title feature f of each object i (t) ; Each image instance feature includes 1.2) Global 3D instance representation: The extrinsic matrix is ​​obtained by automatic calibration of camera and laser radar extrinsic parameters, and the laser point cloud data is projected onto the image plane; according to the different instance masks in the image obtained in step 1.1), the 3D point cloud data corresponding to each mask is obtained, thereby obtaining the 3D space instance representation in the current position coordinate system, and these instances are converted to the global coordinate system through the current frame pose to obtain the global 3D instance representation of the current frame; each object instance detected in each current frame is represented as Point clouds representing each object instance title and title feature f i (t) ; 1.3) Matching and fusing the object instances in each frame obtained in step 1.2) with the instance objects in the existing global instance map; 1.4) Explicit instance map creation: Based on the instance fusion results of step 1.3), an explicit instance map is constructed. The instance map is obtained by detecting and segmenting through an open set perception model and fusion in the global three-dimensional space; 1.5) Abstract instance map creation: Based on the explicit instance map obtained in step 1.4) and the abstract object representation provided by the user, the large language model extracts the component categories and discrimination conditions of the abstract object; All relevant instance objects corresponding to the category that constitutes the abstract object are retrieved from the global map, and a relationship description between each relevant instance object and its nearby relevant instance objects is generated; based on the relationship description and the abstract object discrimination conditions, the large language model identifies the object combination that meets the conditions, constitutes an abstract instance representation, and obtains an abstract instance map.

3. The method for road topology analysis and scene hierarchical representation under open set perceptual reasoning as claimed in claim 1 or 2, characterized in that: The step 2 specifically includes: 2.1) Collect laser point cloud data in the environment through laser radar, and obtain the global geometric point cloud according to the pose of each frame; in the global geometric point cloud, take the local point cloud within a certain height range near the pose of each frame to include all ground points, which is used to extract the ground equation of each frame; select the point cloud within the set height range from the ground equation as the obstacle point cloud of the current frame; accumulate the obstacle point clouds of all frames as the total obstacle map G1; 2.2) Obtaining the road mask ahead by comparing the instance-title cosine similarity with the instance semantic mask and title information obtained in step 1.1); Obtaining the non-road point cloud ahead of each frame by projecting the laser point cloud onto the image, accumulating the non-road point cloud ahead of each frame, and accumulating all frame data within the set height range to obtain the obstacle map G2; 2.3) treating the abstract instance map as a potential obstacle map G3; 2.4) Integrate the three types of obstacle data sources in steps 2.1), 2.2), and 2.3) above, and accumulate them to generate a global obstacle map G. Ignore its height representation and project it onto a two-dimensional plane to represent the free space and obstacle area; then directly use the Voronoi map algorithm to obtain a global dense Voronoi map representing the road structure for subsequent road topology analysis; 2.5) At each frame pose in the obstacle map G in step 2.4), 360 rays of equal angles are emitted to the surroundings, and the intersection points with the obstacles are recorded. These intersection points are connected to obtain a local polygon representing the free space of each frame, and only the Voronoi graph nodes and edges within a certain range of the polygon are selected as the valid Voronoi graph part; the valid Voronoi graph parts of all poses are accumulated to obtain a sparse Voronoi graph after global validation processing; 2.6) For the sparse Voronoi graph obtained in step 2.5), redundancy processing is performed; 2.7) For the Voronoi diagram after redundancy processing in step 2.6), the type of road intersection is determined according to the number of connecting edges of the node and the shape of the connecting branches, and the adjacent nodes are regarded as the subsidiary nodes of the intersection.

4. The method for road topology analysis and scene hierarchical representation under open set perceptual reasoning as claimed in claim 1, characterized in that: 1.3) Global 3D instance fusion: Match and fuse the object instance of each frame obtained in step 1.2) with the instance object in the existing global instance map. The specific steps are as follows: For the object instance detected in the current frame, based on the point cloud, title and title features of the instance, calculate the point cloud geometric similarity Φ between each object instance in the global instance map. geo (i,j), title similarity Φ cap (i, j) and the title feature cosine similarity Φ fea (i, j), where i represents the object instance index in the current frame and j represents the object instance index in the global instance map; then the overall weighted similarity score Φ(i, j) is calculated based on the prior weights of each similarity; the above scores are compared to see if they exceed the similarity threshold δ sim , and initialize or fuse instance objects; fusing instance objects requires updating object properties including point cloud, title, and title features. The update process is as follows: update the point cloud and corresponding bounding box by taking the union of the point cloud and re-denoising; update the object title features by weighted fusion of features based on the number of observations; update the object title based on the understanding ability of the visual language large model. The fused and updated instance map result is reused for the instance map update of the next frame.

5. The method for road topology analysis and scene hierarchical representation under open set perceptual reasoning as claimed in claim 3, characterized in that: The redundancy processing includes: For the sparse Voronoi graph obtained in step 2.5), the KDTree algorithm is used to calculate the distance from the leaf node to the nearest obstacle. If the distance is less than the distance threshold and the length of the independent branch to which the leaf node belongs is less than the given length threshold, the independent node branch in the Voronoi graph is deleted. Multiple iterations are performed to make all leaf nodes meet the above requirements and delete redundant short branches of the Voronoi graph.

6. The method for road topology analysis and scene hierarchical representation under open set perceptual reasoning as claimed in claim 5, characterized in that: The redundant processing includes: for the Voronoi diagram with short branches deleted, traversing the nodes whose edge connection number is 2, calculating the length and angle between the connected line segments, removing the nodes corresponding to the approximately horizontal angle and the too short length, and merging their connecting line segments, iterating multiple times until the above nodes no longer exist.

7. The method for road topology analysis and scene hierarchical representation under open set perceptual reasoning as claimed in claim 6, characterized in that: The redundant processing includes: for the Voronoi diagram after deleting the nodes whose edge connection number is 2, recording the number of connected edges, the nearest obstacle distance, and the edge length information of each node, and deleting all edges and connected corresponding truncated nodes that are less than the length threshold in the Voronoi diagram, ensuring that the nodes maintain a safe distance from the obstacles, and the edge lengths are all greater than the threshold, thereby reducing node redundancy.

8. The method for road topology analysis and scene hierarchical representation under open set perceptual reasoning as claimed in claim 3, characterized in that: In the step 2.7), for the Voronoi diagram after reducing node redundancy, identify the intersections with 4 connections and angles close to 90 degrees, the T-intersections with 3 connections and angles close to 90 degrees, and the L-shaped corner nodes with 2 connections and angles close to 90 degrees, and treat the adjacent nodes as intersection subsidiary nodes.

9. The method for road topology analysis and scene hierarchical representation under open set perceptual reasoning as claimed in claim 3, characterized in that: The set height range is 0.2m to 1.0m.

Images