A robot unknown environment mapping method based on visual scene understanding

Abstract

The application relates to a robot unknown environment mapping method based on visual scene understanding, which comprises the following steps: acquiring a multi-view visual image, and generating a depth map based on the multi-view visual image; determining the pose of an unmanned system based on the multi-view visual image and by using a visual positioning system; performing semantic segmentation on the multi-view visual image by using a semantic segmentation model; constructing a three-dimensional semantic map according to the semantic segmentation result and the depth map, splicing the three-dimensional semantic maps of continuous frames in combination with the pose of the unmanned system, and constructing a three-dimensional scene correlation topological map under the guidance of semantic information; constructing a fusion exploration cost cost function, and solving the fusion exploration cost cost function to obtain a next optimal viewpoint; generating an obstacle segmentation map according to the semantic segmentation result, and performing local path planning based on the next optimal viewpoint and the obstacle segmentation map. The application realizes efficient unknown environment autonomous exploration.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of autonomous navigation and mapping of unmanned systems, in particular to a robot unknown environment mapping method based on visual scene understanding. BACKGROUND

[0002] When facing unknown environments, unmanned systems need to model the environment to achieve autonomous navigation, patrol, target search and situation awareness in the region and other complex autonomous tasks. When the unmanned system is in a non-line-of-sight area, the operator will not be able to conveniently operate the unmanned system, and in a communication denial area, long-term and stable human intervention operations will not be possible, so it is crucial for unmanned systems to have the ability to autonomously explore and map unknown environments.

[0003] Existing autonomous exploration methods for unmanned systems mainly include boundary-based theory, next optimal view-based theory, information theory-based and reinforcement learning-based types. The sensors used are mainly laser radar sensors and visual sensors, among which laser radar sensors are more convenient to obtain map and positioning information and have therefore received widespread attention. With the development of visual sensors and three-dimensional perception technology based on depth maps, visual sensors have become a hot research and application field in recent years when facing scenarios where active detection sensors such as laser radars cannot be used. Existing visual-based autonomous exploration algorithms do not fully utilize high-level understanding of the scene, and are prone to local optimization and repeated exploration problems in unstructured complex environments. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a robot unknown environment mapping method based on visual scene understanding, which can realize efficient autonomous exploration of unknown environments.

[0005] The technical solution adopted by the present application to solve the technical problem is to provide a robot unknown environment mapping method based on visual scene understanding, comprising the following steps:

[0006] Obtain multi-view visual images and generate a depth map based on the multi-view visual images;

[0007] Determine the pose of the unmanned system based on the multi-view visual images and using a visual positioning system;

[0008] Perform semantic segmentation on the multi-view visual images using a semantic segmentation model;

[0009] Construct a three-dimensional semantic map according to the semantic segmentation results and the depth map, and combine the pose of the unmanned system to stitch the three-dimensional semantic maps of consecutive frames, and construct a three-dimensional scene correlation topology graph under the guidance of semantic information;

[0010] A fusion exploration cost function is constructed and solved to obtain a next optimal viewpoint;

[0011] An obstacle segmentation map is generated according to a semantic segmentation result, and local path planning is performed based on the next optimal viewpoint and the obstacle segmentation map.

[0012] The depth map is generated based on the multi-view images, specifically:

[0013] For a case where there is no repeated field of view between the multi-view images, monocular depth estimation is performed on the monocular camera using a monocular depth estimation network to generate a corresponding depth map, and three-dimensional point splicing and fusion are performed in combination with the extrinsic parameters of the visual sensor;

[0014] For a case where there is a repeated field of view between the multi-view images, disparity calculation is performed on the multi-view images after stereo rectification as input of a stereo matching algorithm, and a depth map is calculated in combination with the calibration parameters of the visual sensor.

[0015] The stereo matching algorithm divides the input multi-view images into overlapping image blocks of a preset size, and a preset number of pixels are reserved between adjacent image blocks to perform cost aggregation across the boundaries of the image blocks.

[0016] The pose of the unmanned system is determined based on the multi-view images and using a visual positioning system, specifically: a network architecture of HF-Net is used to extract and describe local and global visual features of the continuously input multi-view images; when the visual features of consecutive frames are associated, KLT optical flow is used to track the visual features to establish data association, a global descriptor is used for preliminary image screening and matching in the local map and the global map, and a local visual feature is used for fine matching.

[0017] When the semantic segmentation model is used to perform semantic segmentation on the multi-view images, a network pruning method based on convolution kernel rank information is used to prune and accelerate the semantic segmentation model.

[0018] A three-dimensional semantic map is constructed according to the semantic segmentation result and the depth map, and consecutive frames of three-dimensional semantic maps are spliced in combination with the pose of the unmanned system, and a three-dimensional scene correlation topological map is constructed under the guidance of semantic information, specifically including:

[0019] The semantic segmentation result and the depth map are mapped and aligned to obtain a three-dimensional semantic map;

[0020] The three-dimensional semantic map is initialized in combination with the pose of the unmanned system, the node of the initialized three-dimensional semantic map is the features of the semantic label, the point cloud structure and the spatial position of each instance, and the feature information of the edge of the initialized three-dimensional semantic map is the relationship features between different instances, and a spatial distance threshold is set to initialize whether the relationship exists or not.

[0021] The topological relationship between different instances is induced according to the relationship features and the spatial distance between different instances, each layer of instances is connected by the edges with the topological relationship, and a three-dimensional scene correlation topological map is obtained.

[0022] The fusion exploration cost cost function is represented as: Cost(x fp )=Nav(x fp )+Cov(x fp )+Info(x fp )+Res(x fp )+Vis(x fp ), wherein Cost(x fp ) is the exploration cost cost of the frontier point x fp , Nav(x fp ) is a target approaching navigation cost, which is the expected distance of the unmanned system to reach the identified exploration target, Cov(x fp ) is a scene coverage navigation cost, which is the expected distance of the unmanned system to reach the frontier point x fp , Info(x fp ) is an information gain cost, when the unmanned system is in a free exploration mode and no semantic object of interest is detected in the scene, the information gain cost is the number of unknown units within a user-defined radius around the frontier point x fp to be explored, when the unmanned system is in a free exploration mode and a semantic object of interest is detected in the scene, the information gain cost is the number of unknown units within a user-defined radius around the semantic point of interest; Res(x fp ) is a frontier point benefit cost, which is whether the line between the frontier point x fp and the current unmanned system position exists an obstacle to selectively plan a path; Vis(x fp ) is a visual feature richness cost, which is the inverse of the proportion of visual features in the scene.

[0023] The obstacle segmentation map is generated according to the semantic segmentation result, specifically: a three-dimensional semantic map is constructed according to the semantic segmentation result and the depth map, on the basis of the three-dimensional semantic map, the dynamics constraints of the unmanned system, the passability and size factors of the unmanned system are fused, the obstacle judgment basis is designed, and the obstacle segmentation map is generated.

[0024] The technical scheme adopted by the present application to solve its technical problems is: providing an electronic device, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the robot unknown environment mapping method based on visual scene understanding when executing the computer program.

[0025] The technical scheme adopted by the present application to solve its technical problems is: providing a computer readable storage medium, having a computer program stored thereon, wherein the computer program implements the steps of the robot unknown environment mapping method based on visual scene understanding when executed by a processor.

[0026] Advantages

[0027] Compared with the prior art, the present application has the following advantages and positive effects: the present application combines the establishment of a three-dimensional scene correlation topology graph based on semantic information guidance, simultaneously considers the construction of a cost function of multiple types of exploration costs, finally calculates a next exploration viewpoint, and generates a global exploration three-dimensional map, which realizes efficient unknown environment autonomous exploration, is particularly suitable for unknown target-oriented autonomous search tasks, and has a wide application prospect in the fields of scene reconstruction, article search and emergency rescue. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is the implementation block diagram of the robot unknown environment mapping method based on visual scene understanding of the first embodiment of the present application. DETAILED DESCRIPTION

[0029] The present application will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present application and not to limit the scope of the present application. In addition, it should be understood that after reading the content taught by the present application, those skilled in the art can make various modifications or modifications to the present application, and these equivalent forms also fall within the scope defined by the appended claims of the present application.

[0030] The first embodiment of the present application relates to a robot unknown environment mapping method based on visual scene understanding, and an unmanned system using the method uses a multi-view visual sensor (the number of cameras is greater than or equal to 2), such as Figure 1 As shown in the figure, specifically comprising the following steps:

[0031] Step 1, acquiring a multi-view visual image, and generating a depth map based on the multi-view visual image.

[0032] In this step, the internal and external parameters of the multi-view visual sensor can be calibrated first to obtain the internal parameters of each visual sensor and the external parameter relationship between the multi-view visual sensors. For the case where there is no repeated field of view between multi-view images, a monocular depth estimation network can be used for depth estimation of the monocular camera to generate a corresponding depth map, and the external parameters of the visual sensor are combined for splicing and fusion of three-dimensional points. For the case where there is a repeated field of view between multi-view images, the stereo corrected multi-view images are used as the input of the algorithm, which can use a semi-global stereo matching or deep learning stereo matching algorithm to calculate the disparity, and the depth map is calculated by combining the calibration parameters of the visual sensor. To meet the depth map calculation requirements of unmanned systems under limited resources, this step also proposes a semi-global stereo matching processing method based on overlapping blocks. When processing, the input multi-view images are divided into 50x50 pixel overlapping image blocks, and 8 pixels of overlap are reserved between adjacent image blocks to aggregate costs across image block boundaries.

[0033] Step 2, determining the pose of the unmanned system based on the multi-view images and using a visual positioning system.

[0034] To meet the visual positioning requirements for complex scenes, this step uses deep learning extracted visual features as the feature input of the visual positioning system. Specifically, the network architecture of HF-Net is used to extract and describe local and global visual features for continuous input multi-view images. When correlating visual features of consecutive frames, KLT optical flow is used to track visual features to establish data association, and global descriptors are used in local maps and global maps for preliminary image screening and matching, and local visual features are used for fine matching. The multi-view positioning system used in this step only constrains the local and global feature descriptors and correlation methods used, and uses a traditional optimization-based method to solve the positioning results.

[0035] Step 3, performing semantic segmentation on the multi-view images using a semantic segmentation model.

[0036] To meet the subsequent requirement of constructing a scene-related topological graph, this step uses a semantic segmentation model to perform semantic segmentation on the multi-view images. The semantic segmentation model uses a network pruning method based on convolution kernel rank information to prune and accelerate the model to meet the operational requirements of unmanned systems under limited resources.

[0037] Step 4, constructing a three-dimensional semantic map according to the semantic segmentation results and the depth map, splicing consecutive frames of three-dimensional semantic maps in combination with the pose of the unmanned system, and constructing a three-dimensional scene-related topological graph under the guidance of semantic information.

[0038] The method uses semantic information of a target scene for distribution information learning and reasoning, constructs an associated topology graph of a search target and scene distribution, reduces redundant instance targets, learns and understands the distribution information of the scene in the form of a knowledge graph, quickly analyzes and searches the scene according to the characteristics (node characteristics) of different instances and the relationship characteristics (edge characteristics) between the instances in the knowledge graph, quickly locates the range of a target to be detected, and meets the efficient target detection requirements of an unmanned system.

[0039] The specific process of this step is as follows: mapping and aligning the semantic segmentation result and the depth map to obtain a three-dimensional semantic map; initializing the three-dimensional semantic map in combination with the pose of the unmanned system, the node of the initialized three-dimensional semantic map being the semantic label, point cloud structure and spatial position features of each instance, the feature information of the edge of the initialized three-dimensional semantic map being the relationship characteristics between different instances and setting a spatial distance threshold to initialize whether there is a relationship; and finally, according to the relationship characteristics and spatial distance between different instances, the topological relationship between different instances is induced, each layer of instances is connected by edges with the topological relationship, and a three-dimensional scene associated topology graph is obtained.

[0040] This embodiment adopts a graph-based representation method, denoted as C g , for representing a three-dimensional scene and the topological relationship between scene entities. The topology graph C g =(pt,E) contains an analysis tree pt, which organizes scene entities and proximal relationships E between entities in a hierarchical structure, and these relationships are represented by undirected edges. There are two kinds of relationships represented by edges in this embodiment, which are support (spatially in an up-down relationship and in contact with each other) and proximity (objects are in adjacent positions in space).

[0041] The scene entity nodes V={v s}∪V L ∪V R ∪V A The scene node v s is the root node of the analysis tree pt, and the layout node set V L includes a floor, a ceiling and walls defining a three-dimensional scene. Each object in the rigid object set V R and the movable object set V A has components available for task interaction by an unmanned system. Each non-root node v i =<o i ,c i ,M i ,B i (p i ,q i ,s i ),Π i > encodes a unique instance label o iA semantic tag c i A complete geometric model M i A 3D bounding box B i (from its central position p) i , direction q i and size s i Parameterization, all in (in the middle), and a set of surface planes Among them, plane Given a homogeneous vector This means that, in the projection space, the normal vector of the unit plane is... Any point on a plane Satisfy constraints: Compared to other geometric primitives, planes have the advantage of being reliably extracted from damaged object meshes and can serve as effective features in subsequent computations. Support relations S are the set of directed edges from a parent node to its child nodes in the parse tree pt. Each edge S... p,c ∈S in parent node V p and child node V c This incorporates basic physics principles. These constraints ensure that the parent node V... p Support child node V in a physically reasonable way c The nearest neighbor relation E introduces links between entities in the parse tree pt. It imposes additional constraints by modeling the spatial relationship between two unsupported but physically adjacent objects v1 and v2, whose meshes should not intersect each other, i.e., Vol(M1∩M2). This implementation assigns nearest neighbor relations only when the bounding boxes of two objects overlap, i.e., when Vol(B1∩B2)>0, rather than between every pair of objects, in order to reduce computational costs.

[0042] Step 5: Construct the fusion exploration cost function and solve the fusion exploration cost function to obtain the next optimal viewpoint.

[0043] This step involves boundary clustering and target identification based on the current positioning data of the unmanned system and the results of the locally constructed scene association topology map. Boundary clustering mainly guides the unmanned system to explore unknown scenes. Based on the constructed fusion exploration cost function, it guides the unmanned system to explore in a more direct direction and get closer to the target of interest.

[0044] To improve exploration efficiency, construct a complete 3D scene map, and ensure high-quality visual positioning results, this step integrates the costs of approach navigation (Nav), scene coverage navigation (Cov), information gain (Info), front point gain cost (Res), and visual feature richness cost (Vis) for comprehensive evaluation. The candidate front point with the highest comprehensive score is selected as the exploration target point to obtain less overlapping paths, shorter movement time, smoother unmanned system movement trajectory, and higher-quality visual positioning results.

[0045] Among them, the target approach navigation cost Nav(x) fp ): Defined as the expected distance for the unmanned system to reach the identified exploration target; for simplicity, Euclidean distance is used here; Scene coverage navigation cost Cov(x) fp ): Defined as the unmanned system reaching the leading edge point x fp The expected distance is calculated using Euclidean distance for simplicity; the information gain cost Info(x) is calculated using Euclidean distance. fp When the unmanned system does not detect any semantically interesting objects in the scene during free exploration mode, the information gain cost is the expected frontier point x. fp The number of unknown units within the user-defined radius surrounding the semantic point of interest is calculated. When the unmanned system detects a semantic object of interest in the scene during free exploration mode, the information gain cost is the number of unknown units within the user-defined radius surrounding the semantic point of interest; the frontier point gain cost Res(x) is calculated. fp ): Defined as the leading edge point x fp Path planning is selectively performed based on whether there are obstacles in the connection between the current location of the unmanned system and the location of the system; the cost of visual feature richness Vis(x) fp ): Defined as the reciprocal of the proportion of visual features in a scene. When visual features are abundant, this cost is low; otherwise, this cost needs to be considered.

[0046] Therefore, the overall situation for the frontier point x fp The cost function for fusion exploration is expressed as:

[0047] Cost(x fp ) = Nav(x fp )+Cov(x fp )+Info(x fp )+Res(x fp )+Vis(x fp )

[0048] To determine the current perspective of the unmanned system covering more unknown areas, an exploration viewpoint is generated using a sampling algorithm based on the acquired boundary and target positions, and the viewpoint sequence with the highest exploration coverage is identified. The exploration cost between boundaries is determined using the viewpoint sequence, and a global coverage path is obtained through solving for this path, enabling efficient exploration of unknown areas. To further improve exploration efficiency and ensure target detection quality, the global coverage path is optimized, and finally, a local path planner transmits the local trajectory to the unmanned system controller for autonomous movement.

[0049] Step 6: Generate an obstacle segmentation map based on the semantic segmentation results, and perform local path planning based on the next optimal viewpoint and the obstacle segmentation map.

[0050] In this step, when generating the obstacle segmentation map, a three-dimensional semantic map is first constructed based on the semantic segmentation results and the depth map. Then, based on the three-dimensional semantic map, the dynamic constraints of the unmanned system, its own passability and size factors are integrated to design the obstacle judgment criteria and generate the obstacle segmentation map. The obstacle segmentation map is used to meet the local path planning needs of the unmanned system.

[0051] To enable unmanned systems to achieve rapid and smooth local target acquisition, multi-view stereo vision factors are fully considered, and a detection quality-driven local path and corresponding trajectory are generated. This step involves determining the position p of each viewpoint. i,j Uniform sampling is performed from cylindrical coordinates originating from the cluster center, and p is connected along a straight line. i,j Centered on the cluster center, multiple yaw angles {ξ} are uniformly sampled. i,j,1 ,ξ i,j,2 ,…,ξ i,j,m To reduce redundant computation, this step calculates the information gain along each slice and aggregates the gains within each FOV to obtain prediction information at different sampling yaw angles. Information during ray projection is better predicted by examining voxels in the current and predicted maps. After predicting the information gain g_(i,j,k) at each yaw angle ξ_(i,j,k), the one with the largest information gain is used as ξ_(i,j), and p_(i,j) and g_(i,j) are saved together. Finally, all viewpoints are arranged in descending order of prediction information, and the top N_vp viewpoints are saved during the planning process to maximize utility and achieve efficient and smooth local trajectory generation.

[0052] It is easy to see that this invention can use pure vision as the main sensor to complete the autonomous exploration of unknown environments and construct a high-level three-dimensional scene association topology map in an efficient manner that avoids local optima. This invention is not only suitable for autonomous exploration tasks that prioritize scene coverage, but also for target-level exploration in an efficient way, achieving rapid search for specified targets. This invention is not limited to the type of unmanned system and can be adapted to unmanned vehicles such as ground unmanned vehicles, quadruped robots and low-altitude quadcopter aircraft.

[0053] The second embodiment of the present invention relates to an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the robot unknown environment mapping method based on visual scene understanding of the first embodiment.

[0054] The third embodiment of the present invention relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the robot unknown environment mapping method based on visual scene understanding of the first embodiment.

[0055] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0056] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0057] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction methods implemented in a process. Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0058] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0059] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for mapping unknown environments of robots based on visual scene understanding, characterized in that, Includes the following steps: Acquire multi-view visual images and generate depth maps based on the multi-view visual images; The pose of the unmanned system is determined based on the multi-view visual images and using a visual positioning system. The multi-view visual image is semantically segmented using a semantic segmentation model. A 3D semantic map is constructed based on the semantic segmentation results and depth map, and a 3D semantic map of continuous frames is stitched together with the pose of the unmanned system. A 3D scene association topology map is constructed under the guidance of semantic information. Construct a fusion exploration cost function and solve it to obtain the next optimal viewpoint; wherein, the fusion exploration cost function is expressed as: Cost(x fp ) = Nav(x fp )+Cov(x fp )+Info(x fp )+Res(x fp )+Vis(x fp ), where Cost(x) fp For the frontier point x fp The exploration costs and consequences, Nav(x) fp Cov(x) represents the target approach navigation cost, which is the expected distance for the unmanned system to reach the identified target. fp The cost of navigation for scene coverage is x, which represents the cost of the unmanned system reaching the leading edge point. fp The expected distance, Info(x) fp The information gain cost is the estimated frontier point x when the unmanned system does not detect any semantically interesting objects in the scene during free exploration mode. fp The number of unknown units within the user-defined radius surrounding the semantic point of interest. When the unmanned system detects a semantic object of interest in the scene during free exploration mode, the information gain cost is the number of unknown units within the user-defined radius surrounding the semantic point of interest; Res(x) fp Let be the frontier point's revenue and cost, and let x be the frontier point's revenue and cost. fp Path planning is selectively performed based on whether there are obstacles in the connection between the current location of the unmanned system and the location of the system; Vis(x) fp ) represents the cost of visual feature richness, which is the reciprocal of the proportion of visual features in the scene; An obstacle segmentation map is generated based on the semantic segmentation results, and local path planning is performed based on the next optimal viewpoint and the obstacle segmentation map.

2. The robot unknown environment mapping method based on visual scene understanding according to claim 1, characterized in that, The generation of the depth map based on the multi-view visual image specifically involves: For cases where there is no overlapping field of view among multi-view visual images, a monocular depth estimation network is used to estimate the depth of the monocular camera and generate the corresponding depth map. The extrinsic parameters of the visual sensor are then combined to stitch and fuse the 3D points. For cases where there is overlapping field of view among multi-view visual images, the stereo-corrected multi-view visual images are used as input to the stereo matching algorithm for disparity calculation, and the depth map is calculated in combination with the calibration parameters of the visual sensor.

3. The robot unknown environment mapping method based on visual scene understanding according to claim 2, characterized in that, The stereo matching algorithm segments the input multi-view visual image into overlapping image blocks of a preset size, and retains the overlap of a preset number of pixels between adjacent image blocks in order to perform cost aggregation across the boundaries of the image blocks.

4. The robot unknown environment mapping method based on visual scene understanding according to claim 1, characterized in that, The process of determining the pose of the unmanned system based on the multi-view visual images and using a visual positioning system specifically involves: using the HF-Net network architecture to extract and describe the local and global visual features of the continuously input multi-view visual images; when associating visual features in consecutive frames, using KLT optical flow to track visual features to establish data association; using global descriptors in local and global maps for preliminary image filtering and matching; and using local visual features for refined matching.

5. The robot unknown environment mapping method based on visual scene understanding according to claim 1, characterized in that, When performing semantic segmentation on the multi-view visual image using a semantic segmentation model, a network pruning method based on convolution kernel rank information is used to accelerate the semantic segmentation model.

6. The robot unknown environment mapping method based on visual scene understanding according to claim 1, characterized in that, A 3D semantic map is constructed based on the semantic segmentation results and depth map. This map is then stitched together with the poses of the unmanned system to create a continuous 3D semantic map. Guided by semantic information, a 3D scene association topology map is constructed, specifically including: The semantic segmentation results are mapped and aligned with the depth map to obtain a three-dimensional semantic map; The three-dimensional semantic map is initialized by combining the pose of the unmanned system. The nodes of the initialized three-dimensional semantic map are the semantic labels, point cloud structures and spatial location features of each instance. The feature information of the edges of the initialized three-dimensional semantic map is the relationship features between different instances and a spatial distance threshold is set to initialize whether a relationship exists. Based on the relationship characteristics and spatial distance between different instances, the topological relationships between different instances are summarized. Each layer of instances is connected by edges with topological relationships to obtain a three-dimensional scene association topology graph.

7. The robot unknown environment mapping method based on visual scene understanding according to claim 1, characterized in that, The process of generating an obstacle segmentation map based on semantic segmentation results specifically involves: constructing a three-dimensional semantic map based on the semantic segmentation results and depth map; and on the basis of the three-dimensional semantic map, integrating the dynamic constraints of the unmanned system, its own passability, and size factors to design obstacle judgment criteria and generate an obstacle segmentation map.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the robot unknown environment mapping method based on visual scene understanding as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the robot unknown environment mapping method based on visual scene understanding as described in any one of claims 1-7.

Images