Robot unknown environment mapping method based on visual scene understanding

Abstract

The invention relates to a robot unknown environment mapping method based on visual scene understanding, and the method comprises the steps: obtaining a multi-view visual image, and generating a depth map based on the multi-view visual image; determining the pose of the unmanned system by using a visual positioning system based on the multi-view visual image; performing semantic segmentation on the multi-view visual image by adopting a semantic segmentation model; constructing a three-dimensional semantic map according to the semantic segmentation result and the depth map, splicing the three-dimensional semantic map of continuous frames in combination with the pose of the unmanned system, and constructing a three-dimensional scene association topological graph under the guidance of semantic information; constructing a fusion exploration cost-cost function, and solving the fusion exploration cost-cost function to obtain a next optimal viewpoint; and generating an obstacle segmentation map according to a semantic segmentation result, and performing local path planning based on the next optimal viewpoint and the obstacle segmentation map. According to the invention, efficient unknown environment autonomous exploration is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous navigation and mapping of unmanned systems, and in particular to a robot unknown environment mapping method based on visual scene understanding. Background Art

[0002] Unmanned systems need to model the environment when facing unknown environments in order to achieve complex autonomous tasks such as autonomous navigation, patrol, target search, and situational awareness in the area. When the unmanned system is in a non-line-of-sight area, the operator will not be able to operate the unmanned system conveniently, and long-term and stable human intervention operations will not be possible in the communication denial area. Therefore, it is crucial for the unmanned system to have the ability to autonomously explore and build maps in unknown environments.

[0003] The existing autonomous exploration methods for unmanned systems are mainly divided into types based on boundary theory, next best perspective theory, information theory, and reinforcement learning. The sensors used are mostly lidar sensors and visual sensors. Among them, lidar sensors are more convenient to obtain map and positioning information, so they have also received widespread attention. With the development of visual sensors and three-dimensional perception technology based on depth maps, visual sensors have become a hot research topic and a hot field for application in recent years when facing scenes where active detection sensors such as lidar cannot be used. Existing vision-based autonomous exploration algorithms do not fully utilize high-level scene understanding information, especially in unstructured complex environments, they are prone to local optimality and repeated exploration problems. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a robot unknown environment mapping method based on visual scene understanding, which can realize efficient autonomous exploration of unknown environments.

[0005] The technical solution adopted by the present invention to solve the technical problem is: to provide a robot unknown environment mapping method based on visual scene understanding, comprising the following steps:

[0006] Acquire a multi-eye visual image, and generate a depth map based on the multi-eye visual image;

[0007] Determine the position and posture of the unmanned system based on the multi-view visual image and using a visual positioning system;

[0008] Performing semantic segmentation on the multi-view visual image using a semantic segmentation model;

[0009] Construct a 3D semantic map based on the semantic segmentation results and depth map, and combine the 3D semantic maps of consecutive frames with the posture of the unmanned system, and construct a 3D scene association topology map under the guidance of semantic information;

[0010] Constructing a fusion exploration cost function, and solving the fusion exploration cost function to obtain the next optimal viewpoint;

[0011] An obstacle segmentation map is generated according to the semantic segmentation result, and local path planning is performed based on the next optimal viewpoint and the obstacle segmentation map.

[0012] The generating of the depth map based on the multi-view image is specifically:

[0013] When there is no overlap in the multi-view images, the monocular depth estimation network is used to estimate the depth of the monocular camera to generate the corresponding depth map, and the extrinsic parameters of the visual sensor are combined to perform the splicing and fusion of the 3D points.

[0014] In the case of repeated fields of view between multi-view images, the stereo-corrected multi-view images are used as the input of the stereo matching algorithm to calculate the disparity, and the depth map is calculated in combination with the calibration parameters of the visual sensor.

[0015] The stereo matching algorithm divides the input multi-view image into overlapping image blocks of a preset size and retains a preset pixel overlap between adjacent image blocks so as to perform cost aggregation across the boundaries of the image blocks.

[0016] The method of determining the position and posture of the unmanned system based on the multi-eye visual image and using the visual positioning system is specifically as follows: using the HF-Net network architecture to extract and describe the local and global visual features of the continuously input multi-eye visual image; when associating the visual features of consecutive frames, using KLT optical flow to track the visual features to establish data association, using global descriptors in local maps and global maps to perform preliminary image screening and matching, and using local visual features for refined matching.

[0017] When the semantic segmentation model is used to perform semantic segmentation on the multi-view visual image, a network pruning method based on convolution kernel rank information is used to accelerate the pruning of the semantic segmentation model.

[0018] A 3D semantic map is constructed based on the semantic segmentation results and the depth map. The 3D semantic maps of continuous frames are spliced ​​together in combination with the posture of the unmanned system. A 3D scene association topology map is constructed under the guidance of semantic information. Specifically, it includes:

[0019] Mapping and aligning the semantic segmentation result and the depth map to obtain a three-dimensional semantic map;

[0020] The three-dimensional semantic map is initialized in combination with the position and posture of the unmanned system. The nodes of the initialized three-dimensional semantic map are the semantic labels, point cloud structures and spatial position features of each instance. The feature information of the edges of the initialized three-dimensional semantic map is the relationship features between different instances and a spatial distance threshold is set to initialize whether there is a relationship.

[0021] The topological relationships between different instances are summarized according to the relationship features and spatial distances between them, and each layer of instances is connected with edges with topological relationships to obtain a three-dimensional scene association topology graph.

[0022] The fusion exploration cost function is expressed as: Cost(x fp )=Nav(x fp )+Cov(x fp )+Info(x fp )+Res(x fp )+Vis(x fp ), where Cost(x fp ) is for the frontier point x fp The exploration cost, Nav(x fp ) is the target approach navigation cost, which is the expected distance for the unmanned system to reach the identified exploration target, Cov(x fp ) is the scene coverage navigation cost, which is the unmanned system reaching the frontier point x fp The expected distance, Info(x fp ) is the information gain cost. When the unmanned system does not detect the existence of the semantic object of interest in the scene in the free exploration mode, the information gain cost is the frontier point x expected to be explored. fp The number of unknown units within the user-defined radius around the semantic point of interest. When the unmanned system detects the existence of a semantic object of interest in the scene in free exploration mode, the information gain cost is calculated as the number of unknown units within the user-defined radius around the semantic point of interest; Res(x fp ) is the frontier point benefit cost, which is the frontier point x fp The path planning is selectively performed based on whether there are obstacles on the line connecting the current unmanned system position; Vis(x fp ) is the visual feature richness cost, which is the inverse of the proportion of visual features in the scene.

[0023] The generating of the obstacle segmentation map according to the semantic segmentation result is specifically as follows: constructing a three-dimensional semantic map according to the semantic segmentation result and the depth map, integrating the unmanned system dynamic constraints, its own passability and size factors on the basis of the three-dimensional semantic map, designing an obstacle judgment basis, and generating the obstacle segmentation map.

[0024] The technical solution adopted by the present invention to solve its technical problem is: to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the above-mentioned robot unknown environment mapping method based on visual scene understanding are implemented.

[0025] The technical solution adopted by the present invention to solve its technical problem is: providing a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned robot unknown environment mapping method based on visual scene understanding are implemented.

[0026] Beneficial Effects

[0027] Due to the adoption of the above-mentioned technical scheme, the present invention has the following advantages and positive effects compared with the prior art: the present invention combines the establishment of a three-dimensional scene association topology map guided by semantic information, and simultaneously considers the construction of a cost function of various types of exploration costs, and finally calculates the next exploration viewpoint and generates a global exploration three-dimensional map, which realizes efficient autonomous exploration of unknown environments, is particularly suitable for autonomous search tasks for unknown targets, and has broad application prospects in scene reconstruction, object search, emergency rescue and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a block diagram of an implementation of a robot unknown environment mapping method based on visual scene understanding according to the first embodiment of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0030] The first embodiment of the present invention relates to a robot unknown environment mapping method based on visual scene understanding, and the unmanned system using the method uses a multi-eye visual sensor (the number of cameras is greater than or equal to 2), such as Figure 1 As shown, the specific steps include:

[0031] Step 1: Acquire a multi-eye visual image and generate a depth map based on the multi-eye visual image.

[0032] In this step, the internal and external parameters of the multi-view visual sensor can be calibrated first to obtain the internal parameters of each visual sensor and the external parameter relationship between the multi-view visual sensors. In the case where there is no repeated field of view between the multi-view visual images, the monocular depth estimation network can be used to estimate the depth of the monocular camera to generate the corresponding depth map, and the external parameters of the visual sensor can be combined to perform the splicing and fusion of three-dimensional points; in the case where there is a repeated field of view between the multi-view visual images, the stereo corrected multi-view visual image is used as the input of the algorithm. The algorithm can use a semi-global stereo matching or deep learning stereo matching algorithm, calculate the disparity through the algorithm, and calculate the depth map in combination with the calibration parameters of the visual sensor. In order to meet the depth map calculation requirements of unmanned systems under limited resources, this step also proposes a semi-global stereo matching processing method based on overlapping blocks. During the processing, the input multi-view visual image is divided into overlapping image blocks of 50×50 pixels, and 8 pixels of overlap are retained between adjacent image blocks to perform cost aggregation across the boundaries of the image blocks.

[0033] Step 2: Determine the position and posture of the unmanned system based on the multi-view visual image and using a visual positioning system.

[0034] In order to meet the visual positioning requirements for complex scenes, this step uses the visual features extracted by deep learning as the feature input of the visual positioning system, specifically: the HF-Net network architecture is used to extract and describe the local and global visual features of the continuously input multi-eye visual images. When associating the visual features of consecutive frames, KLT optical flow is used to track the visual features to establish data association, global descriptors are used in local maps and global maps for preliminary image screening and matching, and local visual features are used for refined matching. The multi-eye visual positioning system used in this step only constrains the local and global feature descriptors and association methods used, and the traditional optimization-based method is used to solve the positioning results.

[0035] Step 3: Use a semantic segmentation model to perform semantic segmentation on the multi-view visual image.

[0036] In order to meet the subsequent needs of constructing a scene-related topology map, this step uses a semantic segmentation model to perform semantic segmentation on the multi-view visual image. The semantic segmentation model adopts a network pruning method based on convolution kernel rank information to accelerate the pruning of the model to meet the operating capability requirements under resource-constrained unmanned systems.

[0037] Step 4: construct a 3D semantic map based on the semantic segmentation results and the depth map, and combine the 3D semantic maps of continuous frames with the posture of the unmanned system, and construct a 3D scene association topology map under the guidance of semantic information.

[0038] This method uses the semantic information of the target scene to learn and reason about distribution information, constructs a topological graph of the association between the search target and the scene distribution, reduces redundant instance targets, learns and understands the distribution information of the scene in the form of a knowledge graph, and quickly analyzes and searches the scene based on the characteristics of different instances themselves (node ​​characteristics) and the relationship characteristics between instances (edge ​​characteristics) in the knowledge graph, quickly locates the range of the target to be detected, and meets the needs of efficient target detection in unmanned systems.

[0039] The specific process of this step is as follows: the semantic segmentation result and the depth map are mapped and aligned to obtain a three-dimensional semantic map; the three-dimensional semantic map is initialized in combination with the posture of the unmanned system, and the nodes of the initialized three-dimensional semantic map are the semantic labels, point cloud structures and spatial position features of each instance. The feature information of the edges of the initialized three-dimensional semantic map is the relationship features between different instances and a spatial distance threshold is set to initialize whether there is a relationship; finally, the topological relationship between different instances is summarized according to the relationship features and spatial distances between different instances, and each layer of instances is connected with edges with topological relationships to obtain a three-dimensional scene association topology map.

[0040] This embodiment adopts a graph-based representation method, denoted as C g , used to represent the topological relationship between the three-dimensional scene and the scene entities. Topological graph C g =(pt,E) contains a parse tree pt, which organizes scene entities and proximal relationships E between entities in a hierarchical structure, and these relationships are represented by undirected edges. In this embodiment, there are two types of relationships represented by edges, namely support (being in a vertical relationship in space and touching each other) and neighbor (objects are in adjacent positions in space).

[0041] Scene entity node V = {v s}∪V L ∪V R ∪V A Scene node v in s As the root node of the parse tree pt, layout node set V L Includes the floor, ceiling, and walls that define the three-dimensional scene. Rigid Object Set V R And the movable body set V A Each object in has parts that can be used for unmanned system tasks to interact. Each non-root node v i =<o i ,c i ,M i ,B i (p i ,q i ,s i ),Π i > encodes a unique instance label o i, a semantic label c i , a complete geometric model M i , a 3D bounding box B i (by its center position p i , direction q i and size s i Parameterized, both in ), and a set of surface planes The plane By a homogeneous vector Indicates that in the projective space, the unit plane normal vector is Any point on the plane Satisfy the constraints: Compared with other geometric primitives, planes have the advantage of being able to be stably extracted from damaged object meshes and can be used as effective features in subsequent calculations. The support relation S is the set of directed edges from the parent node to its child nodes in the parse tree pt. Each edge S p,c ∈S in the parent node V p and child node V c These constraints ensure that the parent node V p Support the child nodes V in a physically reasonable way c The neighbor relation E introduces links between entities in the parse tree pt. It imposes an additional constraint by modeling the spatial relationship between two unsupported but physically adjacent objects v1 and v2, whose meshes should not penetrate each other, i.e., Vol(M1∩M2). This implementation only assigns neighbor relations when the bounding boxes of two objects overlap, i.e., when Vol(B1∩B2)>0, rather than between every pair of objects, to reduce computational cost.

[0042] Step 5: construct a fusion exploration cost function, and solve the fusion exploration cost function to obtain the next optimal viewpoint.

[0043] This step performs boundary clustering and target recognition based on the current positioning data of the unmanned system and the topological map results of the locally constructed scene association. Boundary clustering mainly guides the unmanned system to explore unknown scenes. Based on the constructed fusion exploration cost function, it guides the unmanned system to explore the direction and get closer to the target of interest.

[0044] In order to improve the exploration efficiency, build a complete three-dimensional scene map and ensure high-quality visual positioning results, this step conducts a comprehensive evaluation by integrating the approach navigation cost (Nav), scene coverage navigation cost (Cov), information gain cost (Info), frontier point benefit cost (Res), and visual feature richness cost (Vis), and selects a candidate frontier point with the highest comprehensive score as the exploration target point to obtain fewer overlapping paths, shorter movement time, smoother unmanned system motion trajectory and higher quality visual positioning results.

[0045] Among them, the target approach navigation cost Nav(x fp ): defined as the expected distance of the unmanned system to reach the identified exploration target. To simplify the calculation, the Euclidean distance is used here; the scene coverage navigation cost Cov(x fp ): defined as the unmanned system reaching the frontier point x fp The expected distance, to simplify the calculation, the Euclidean distance is used here; the information gain cost Info(x fp ): When the unmanned system does not detect the existence of the semantic object of interest in the scene in the free exploration mode, the information gain cost is the frontier point x expected to be explored fp The number of unknown units within the user-defined radius around the unmanned system. When the unmanned system detects the existence of a semantic object of interest in the scene in free exploration mode, the information gain cost is calculated as the number of unknown units within the user-defined radius around the semantic point of interest; the frontier point benefit cost Res(x fp ): defined as the frontier point x fp Whether there are obstacles in the line between the current unmanned system position and the current unmanned system position, the path planning is selectively performed; the visual feature richness cost Vis(x fp ): It is defined as the inverse of the proportion of visual features in the scene. When the visual features are richer, the cost is lower, otherwise it needs to be considered.

[0046] Therefore, for the overall frontier point x fp The fusion exploration cost function is expressed as:

[0047] Cost(x fp )=Nav(x fp )+Cov(x fp )+Info(x fp )+Res(x fp )+Vis(x fp )

[0048] In order to determine the current perspective of the unmanned system that covers more unknown areas, the exploration viewpoints are generated through a sampling algorithm based on the boundary and target positions, and the viewpoint sequence with the highest exploration coverage is determined. The viewpoint sequence is used to determine the exploration cost between boundaries, and the global coverage path is obtained by solving to achieve efficient exploration of unknown areas. In order to further improve the exploration efficiency and ensure the quality of target detection, the global coverage path is optimized, and finally the local trajectory is transmitted to the unmanned system controller through the local path planner for autonomous movement.

[0049] Step 6: Generate an obstacle segmentation map according to the semantic segmentation result, and perform local path planning based on the next optimal viewpoint and the obstacle segmentation map.

[0050] In this step, when generating the obstacle segmentation map, a three-dimensional semantic map is first constructed based on the semantic segmentation results and the depth map. Then, based on the three-dimensional semantic map, the unmanned system's dynamic constraints, its own passability and size factors are integrated to design an obstacle judgment basis and generate an obstacle segmentation map. The obstacle segmentation map is used to meet the local path planning requirements of the unmanned system.

[0051] In order to meet the requirements of unmanned systems to achieve local targets quickly and smoothly, the multi-view stereo vision factors are fully considered to generate a local path driven by detection quality and the corresponding trajectory. i,j Uniform sampling is done from cylindrical coordinates originating from cluster centers and connected along straight lines p i,j With the cluster center as the center, multiple yaw angles {ξ i,j,1 ,ξ i,j,2 ,…,ξ i,j,m}, in order to reduce redundant calculations, this step calculates the information gain along each slice and aggregates the gain within each FOV to obtain the predicted information under different sampling yaw angles, and better predicts the information in the ray casting process by checking the voxels in the current image and the predicted image. After predicting the information gain g_(i,j,k) at each yaw angle ξ_(i,j,k), the one with the largest information gain is saved as ξ_(i,j), p_(i,j) and g_(i,j). Finally, all viewpoints are sorted in descending order of predicted information, and the first N_vp viewpoints are saved during the planning process to maximize utility and achieve efficient and smooth local trajectory generation.

[0052] It is not difficult to find that the present invention can use pure vision as the main sensor to complete autonomous exploration of unknown environments and construct a high-level three-dimensional scene association topology map in an efficient and local optimal way; the present invention is not only suitable for autonomous exploration tasks with scene coverage priority, but also can perform target-level exploration in an efficient exploration manner to achieve rapid search for designated targets; the present invention is not limited to the type of unmanned system, and can be adapted to unmanned carriers such as ground unmanned vehicles, quadruped robots and low-altitude quadcopters.

[0053] A second embodiment of the present invention relates to an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the robot unknown environment mapping method based on visual scene understanding of the first embodiment when executing the computer program.

[0054] A third embodiment of the present invention relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the robot unknown environment mapping method based on visual scene understanding of the first embodiment.

[0055] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0056] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0057] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction method, which is implemented in the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0058] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0059] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A robot unknown environment mapping method based on visual scene understanding, characterized in that: The following steps are involved: Acquire a multi-eye visual image, and generate a depth map based on the multi-eye visual image; Determine the position and posture of the unmanned system based on the multi-view visual image and using a visual positioning system; Performing semantic segmentation on the multi-view visual image using a semantic segmentation model; Construct a 3D semantic map based on the semantic segmentation results and depth map, and combine the 3D semantic maps of consecutive frames with the posture of the unmanned system, and construct a 3D scene association topology map under the guidance of semantic information; Constructing a fusion exploration cost function, and solving the fusion exploration cost function to obtain the next optimal viewpoint; An obstacle segmentation map is generated according to the semantic segmentation result, and local path planning is performed based on the next optimal viewpoint and the obstacle segmentation map.

2. The method for mapping an unknown environment using a robot based on visual scene understanding according to claim 1, characterized in that: The generating of the depth map based on the multi-view image is specifically: When there is no overlap in the multi-view images, the monocular depth estimation network is used to estimate the depth of the monocular camera to generate the corresponding depth map, and the extrinsic parameters of the visual sensor are combined to perform the splicing and fusion of the 3D points. In the case of repeated fields of view between multi-view images, the stereo-corrected multi-view images are used as the input of the stereo matching algorithm to calculate the disparity, and the depth map is calculated in combination with the calibration parameters of the visual sensor.

3. The method for mapping an unknown environment using a robot based on visual scene understanding according to claim 2, characterized in that: The stereo matching algorithm divides the input multi-view image into overlapping image blocks of a preset size and retains a preset pixel overlap between adjacent image blocks so as to perform cost aggregation across the boundaries of the image blocks.

4. The method for mapping an unknown environment using a robot based on visual scene understanding according to claim 1, characterized in that: The method of determining the position and posture of the unmanned system based on the multi-eye visual image and using the visual positioning system is specifically as follows: using the HF-Net network architecture to extract and describe the local and global visual features of the continuously input multi-eye visual image; when associating the visual features of consecutive frames, using KLT optical flow to track the visual features to establish data association, using global descriptors in local maps and global maps to perform preliminary image screening and matching, and using local visual features for refined matching.

5. The method for mapping an unknown environment using a robot based on visual scene understanding according to claim 1, characterized in that: When the semantic segmentation model is used to perform semantic segmentation on the multi-view visual image, a network pruning method based on convolution kernel rank information is used to accelerate the pruning of the semantic segmentation model.

6. The method for mapping an unknown environment using a robot based on visual scene understanding according to claim 1, characterized in that: A 3D semantic map is constructed based on the semantic segmentation results and the depth map. The 3D semantic maps of consecutive frames are spliced ​​together in combination with the posture of the unmanned system. A 3D scene association topology map is constructed under the guidance of semantic information. Specifically, it includes: Mapping and aligning the semantic segmentation result and the depth map to obtain a three-dimensional semantic map; The three-dimensional semantic map is initialized in combination with the position and posture of the unmanned system. The nodes of the initialized three-dimensional semantic map are the semantic labels, point cloud structures and spatial position features of each instance. The feature information of the edges of the initialized three-dimensional semantic map is the relationship features between different instances and a spatial distance threshold is set to initialize whether there is a relationship. The topological relationships between different instances are summarized according to the relationship features and spatial distances between them, and each layer of instances is connected with edges with topological relationships to obtain a three-dimensional scene association topology graph.

7. The method for mapping an unknown environment using a robot based on visual scene understanding according to claim 1, characterized in that: The fusion exploration cost function is expressed as: Cost(x fp )=Nav(x fp )+Cov(x fp )+Info(x fp )+Res(x fp )+Vis(x fp ), where Cost(x fp ) is for the frontier point x fp The exploration cost, Nav(x fp ) is the target approach navigation cost, which is the expected distance for the unmanned system to reach the identified exploration target, Cov(x fp ) is the scene coverage navigation cost, which is the unmanned system reaching the frontier point x fp The expected distance, Info(x fp ) is the information gain cost. When the unmanned system does not detect the existence of the semantic object of interest in the scene in the free exploration mode, the information gain cost is the frontier point x expected to be explored. fp The number of unknown units within the user-defined radius around the semantic point of interest. When the unmanned system detects the existence of a semantic object of interest in the scene in free exploration mode, the information gain cost is calculated as the number of unknown units within the user-defined radius around the semantic point of interest; Res(x fp ) is the frontier point benefit cost, which is the frontier point x fp The path planning is selectively performed based on whether there are obstacles on the line connecting the current unmanned system position; Vis(x fp ) is the visual feature richness cost, which is the inverse of the proportion of visual features in the scene.

8. The method for building a robot map of an unknown environment based on visual scene understanding according to claim 1, characterized in that: The generating of the obstacle segmentation map according to the semantic segmentation result is specifically as follows: constructing a three-dimensional semantic map according to the semantic segmentation result and the depth map, integrating the unmanned system dynamic constraints, its own passability and size factors on the basis of the three-dimensional semantic map, designing an obstacle judgment basis, and generating the obstacle segmentation map.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the robot unknown environment mapping method based on visual scene understanding as described in any one of claims 1-8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the robot unknown environment mapping method based on visual scene understanding as described in any one of claims 1 to 8 are implemented.

Images