Map construction method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a map construction method and device, equipment and a storage medium. The method comprises the following steps: constructing a three-dimensional grid containing depth information; generating a position mapping matrix based on the three-dimensional grid, wherein the position mapping matrix comprises a first position mapping matrix corresponding to the depth map and a second position mapping matrix corresponding to the RGB image; traversing the point cloud in the three-dimensional grid, acquiring a depth value of the camera depth map corresponding to the traversed point cloud and a semantic tag of the RGB semantic segmentation map according to the position mapping matrix, and determining whether to add the semantic tag to the traversed point cloud based on the depth value; and generating a map containing semantic information according to the three-dimensional grid added with the semantic tag. According to the technical scheme provided by the embodiment of the invention, the process of pre-generating the position mapping matrix and projecting the two-dimensional semantic tag to the three-dimensional point cloud is optimized into a fixed memory access assignment process, so that the calculation amount during operation is reduced, and the calculation complexity is reduced.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and in particular to a map construction method, apparatus, device and storage medium. Background Art

[0002] When building an obstacle avoidance map, adding semantic information to the map can provide more basis for subsequent decision-making. In related technologies, the geometric projection method is generally used to project the point cloud onto the semantic segmentation image, and the semantic segmentation label and the point cloud are matched one by one, thereby giving the point cloud a semantic category and building a map with semantic information. However, the calculation amount of this projection transformation is relatively large, and when there are multiple RGB cameras, the calculation amount increases exponentially. Summary of the invention

[0003] The present disclosure provides a map construction method, apparatus, device and storage medium, which optimizes the process of projecting two-dimensional semantic labels to three-dimensional point clouds into a fixed memory access assignment process by pre-generating a position mapping matrix, thereby reducing the amount of runtime calculations and reducing the calculation complexity.

[0004] In a first aspect, an embodiment of the present disclosure provides a map construction method, including:

[0005] Construct a 3D mesh containing depth information;

[0006] Generate a position mapping matrix based on the three-dimensional grid, the position mapping matrix including a first position mapping matrix corresponding to the depth map and a second position mapping matrix corresponding to the RGB image;

[0007] Traverse the point cloud in the three-dimensional grid, obtain the depth value of the camera depth map corresponding to the traversed point cloud and the semantic label of the RGB semantic segmentation map according to the position mapping matrix, and determine whether to add a semantic label to the traversed point cloud based on the depth value;

[0008] Based on the 3D grid with semantic labels, a map containing semantic information is generated.

[0009] In a second aspect, the present disclosure also provides a map construction device, including:

[0010] A mesh construction module for constructing a 3D mesh containing depth information;

[0011] A position mapping module, configured to generate a position mapping matrix based on the three-dimensional grid, wherein the position mapping matrix includes a first position mapping matrix corresponding to the depth map and a second position mapping matrix corresponding to the RGB image;

[0012] A label adding module is used to traverse the point cloud in the three-dimensional grid, obtain the depth value of the camera depth map corresponding to the traversed point cloud and the semantic label of the RGB semantic segmentation map according to the position mapping matrix, and determine whether to add a semantic label to the traversed point cloud based on the depth value;

[0013] The map construction module is used to generate a map containing semantic information based on the three-dimensional grid with semantic labels added.

[0014] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0015] one or more processors;

[0016] a storage device for storing one or more programs,

[0017] When one or more programs are executed by one or more processors, the one or more processors implement the map construction method as in the embodiment of the present disclosure.

[0018] In a fourth aspect, the embodiments of the present disclosure further provide a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the map construction method of the embodiments of the present disclosure.

[0019] The disclosed embodiments construct a three-dimensional grid including depth information; generate a position mapping matrix based on the three-dimensional grid, the position mapping matrix including a first position mapping matrix corresponding to the depth map and a second position mapping matrix corresponding to the RGB image; traverse the point cloud in the three-dimensional grid, obtain the depth value of the camera depth map corresponding to the traversed point cloud and the semantic label of the RGB semantic segmentation map according to the position mapping matrix, and determine whether to add semantic labels to the traversed point cloud based on the depth value; generate a map including semantic information according to the three-dimensional grid with the semantic label added, thereby solving the problem of large amount of computation during runtime in the related art of constructing a map by projecting the point cloud onto the semantic segmentation image, and optimizing the process of projecting the two-dimensional semantic label onto the three-dimensional point cloud into a fixed memory access assignment process by pre-generating the position mapping matrix, thereby reducing the amount of computation during runtime and reducing the computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0021] Figure 1 It is a flowchart of a map construction method provided by an embodiment of the present disclosure;

[0022] Figure 2 is a flowchart of another map construction method provided by an embodiment of the present disclosure;

[0023] Figure 3 is a schematic diagram of adding semantic labels to a three-dimensional grid provided by an embodiment of the present disclosure;

[0024] Figure 4 is a schematic diagram of the structure of a map construction device provided by an embodiment of the present disclosure;

[0025] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0027] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0028] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0029] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0030] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0031] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0032] Figure 1 This is a flowchart of a map construction method provided by an embodiment of the present disclosure. The embodiment of the present disclosure is applicable to the case where a map is constructed by projecting semantic labels onto a point cloud of a three-dimensional grid through a pre-generated position mapping matrix. The method can be executed by a map construction device, which can be implemented in the form of software and / or hardware. Optionally, it can be implemented by an electronic device, which can be a mobile terminal, a PC or a server. Figure 1 As shown, the method includes:

[0033] S110: Construct a three-dimensional grid including depth information.

[0034] Among them, the depth information can be the distance of the object from the camera sensor. The three-dimensional grid can be constructed by superimposing multiple depth maps assigned different depth values ​​in a specified coordinate system. The specified coordinate system can be a virtual coordinate system in which the coordinate axis direction is different from the coordinate axis direction in the real-world coordinate system. Exemplarily, in the xyz coordinate system corresponding to the real world, the x coordinate axis faces forward, the y coordinate axis faces left, and the z coordinate axis faces right; in the specified coordinate system, such as the uvd coordinate system, the u coordinate axis can face right, the v coordinate axis can face down, and the d coordinate axis can face forward.

[0035] Optionally, construct a three-dimensional grid containing depth information, including: constructing a three-dimensional grid in a specified coordinate system based on the width and height of the depth map, and a custom depth value set according to the actual distance between two adjacent points; wherein the width of the depth map, the height of the depth map, and the custom depth value sequence correspond to the three dimensions of the specified coordinate system.

[0036] In this embodiment, the width W and height H of the depth map are obtained, and then the coordinate values ​​on the u coordinate axis are generated in the uvd coordinate system. i =1,2,3,…,W, the coordinate values ​​on the v axis are v i =1,2,3,…,H, the coordinate values ​​on the d axis are d i =i*S voxel , i = 1, 2, 3, ..., D, such a set of W*H*D points, thus forming a three-dimensional grid of size [W, H, D, 3]. Among them, the last dimension in [W, H, D, 3] represents the coordinates of each point cloud in 3D space, D*S voxel is a custom depth value, D is a modifiable constant, S voxel It is the distance between two adjacent points in the three-dimensional grid V in the real world.

[0037] S120, generating a position mapping matrix based on the three-dimensional grid.

[0038] The position mapping matrix includes a first position mapping matrix corresponding to the depth map and a second position mapping matrix corresponding to the RGB image. The first position mapping matrix is ​​used to represent the correspondence between the positions of each point in the first two coordinate dimensions of the three-dimensional grid and the positions of each point in the depth map. The second position mapping matrix is ​​used to represent the correspondence between the positions of each point in the first two coordinate dimensions of the three-dimensional grid and the positions of each point in the RGB image.

[0039] In this embodiment, after constructing the three-dimensional grid, the depth map in the three-dimensional grid can be mapped to the observation plane of the depth camera by performing internal and external parameter transformation on the three-dimensional grid, and the corresponding relationship between the position of each point in the u and v coordinate dimensions of the three-dimensional grid and the position of each point in the depth map observed by the depth camera is determined, and a first position mapping matrix corresponding to the depth map is generated. Also, the depth map in the three-dimensional grid is mapped to the observation plane of the RGB camera, and the corresponding relationship between the position of each point in the u and v coordinate dimensions of the three-dimensional grid and the position of each point in the RGB image taken by the RGB camera is determined, and a second position mapping matrix corresponding to the RGB image is generated. Thus, the process of projecting two-dimensional semantic labels to three-dimensional point clouds can be optimized into a fixed memory access assignment process, that is, the runtime operation is optimized into a single data read, which reduces the amount of computation at runtime and reduces the computational complexity.

[0040] S130, traversing the point cloud in the three-dimensional grid, obtaining the depth value of the camera depth map corresponding to the traversed point cloud and the semantic label of the RGB semantic segmentation map according to the position mapping matrix, and determining whether to add a semantic label to the traversed point cloud based on the depth value.

[0041] In this embodiment, in order to realize the projection of semantic labels onto the three-dimensional point cloud, taking a monocular depth camera and a monocular RGB camera as examples, first obtain the camera depth map captured by the depth camera and the RGB image captured by the RGB camera. Then, the monocular RGB image is semantically segmented through a semantic segmentation model based on deep learning to obtain an RGB semantic segmentation map of the same size as the monocular RGB image. Then, the point cloud in the three-dimensional grid is traversed, and for each traversed point cloud, it is determined whether to mark the point and add the corresponding semantic label.

[0042] In an optional embodiment, the depth values ​​of the camera depth map corresponding to the traversed point cloud and the semantic labels of the RGB semantic segmentation map are obtained according to the position mapping matrix, and it is determined whether to add the semantic labels to the traversed point cloud based on the depth values, including: according to the first position mapping matrix, the depth values ​​of the camera depth map corresponding to the traversed point cloud are obtained, and based on the depth values, it is determined whether to mark the traversed point cloud; in the case of marking the traversed point cloud, according to the second position mapping matrix, the semantic labels of the RGB semantic segmentation map corresponding to the traversed point cloud are obtained, and the semantic labels are added to the traversed point cloud.

[0043] In this embodiment, since a first position mapping matrix capable of representing the correspondence between the positions of each point in the first two coordinate dimensions of the three-dimensional grid and the positions of each point in the camera depth map is pre-generated, the depth value corresponding to the traversed point cloud in the camera depth map can be obtained by querying the first position mapping matrix, and whether the point cloud needs to be marked is determined according to the depth value, that is, whether the point cloud is a point visible to the eye. If the point cloud is marked, the semantic label corresponding to the traversed point cloud in the RGB semantic segmentation map is further obtained according to the pre-generated second position mapping matrix capable of representing the correspondence between the positions of each point in the first two coordinate dimensions of the three-dimensional grid and the positions of each point in the RGB image, and the semantic label is added to the point cloud. If the point cloud is not marked, there is no need to add a semantic label to the point cloud, so there is no need to query the second position mapping matrix to obtain the semantic label corresponding to the point cloud.

[0044] Among them, since the size of the RGB image and the RGB semantic segmentation map is consistent, the second position mapping matrix can also represent the correspondence between the position of each point in the first two coordinate dimensions in the three-dimensional grid and the position of each point in the RGB semantic segmentation map.

[0045] S140: Generate a map containing semantic information based on the three-dimensional grid with semantic labels added.

[0046] In this embodiment, a three-dimensional map containing semantic information or a two-dimensional map containing semantic information can be generated as required. In an optional implementation, when a three-dimensional map is to be constructed, the three-dimensional grid with semantic labels added is used as the three-dimensional map containing semantic information; when a two-dimensional map is to be constructed, the three-dimensional grid with semantic labels added is projected downward into a plane map, and the semantic labels of the corresponding point clouds in the three-dimensional grid are added to each point in the plane map to generate a two-dimensional map containing semantic information.

[0047] In this embodiment, after the three-dimensional grid is projected downward into a plane map, the three-dimensional point cloud corresponding to each point in the plane map can be determined based on the correspondence between each point cloud in the three-dimensional grid and each point in the plane map, and then the semantic label corresponding to the point in the plane map can be determined based on the semantic label of the three-dimensional point cloud. In an optional implementation, each point in the plane map is traversed to determine at least one target point cloud corresponding to the traversed point in the three-dimensional grid; from the semantic label of at least one target point cloud, the semantic label with the highest importance level is selected and added to the traversed point.

[0048] In this embodiment, since each point in the plan view corresponds to at least one three-dimensional point cloud, if there is no point cloud with a semantic label in the at least one three-dimensional point cloud, then the point in the plan view does not need to be added with a semantic label. If there is at least one point cloud with a semantic label in the at least one three-dimensional point cloud, then a semantic label with the highest importance level is selected from the at least one semantic label and added to the point in the plan view.

[0049] The technical solution of the embodiment of the present disclosure is to construct a three-dimensional grid containing depth information; generate a position mapping matrix based on the three-dimensional grid, the position mapping matrix including a first position mapping matrix corresponding to the depth map, and a second position mapping matrix corresponding to the RGB image; traverse the point cloud in the three-dimensional grid, obtain the depth value of the camera depth map corresponding to the traversed point cloud and the semantic label of the RGB semantic segmentation map according to the position mapping matrix, and determine whether to add semantic labels to the traversed point cloud based on the depth value; generate a map containing semantic information according to the three-dimensional grid with the semantic label added, so as to solve the problem of large amount of computation during runtime in the related art that the map is constructed by projecting the point cloud onto the semantic segmentation image, and optimize the process of projecting the two-dimensional semantic label to the three-dimensional point cloud into a fixed memory access assignment process by pre-generating the position mapping matrix, thereby reducing the amount of computation during runtime and reducing the computational complexity.

[0050] Figure 2 1 is a flow chart of another map construction method provided by an embodiment of the present disclosure. Based on the above embodiment, this embodiment further provides steps of generating a first position mapping matrix and a second position mapping matrix based on a three-dimensional grid, and specific steps of adding semantic labels to a point cloud of a three-dimensional grid. Figure 2 As shown, the method includes:

[0051] S210: Construct a three-dimensional grid including depth information.

[0052] S220. Generate, based on the three-dimensional grid, a first position mapping matrix corresponding to the depth map and a second position mapping matrix corresponding to the RGB image.

[0053] In this embodiment, in order to reduce the amount of calculation for projecting semantic labels onto three-dimensional point clouds, a first position mapping matrix for representing the correspondence between the positions of each point in the first two coordinate dimensions of the three-dimensional grid and the positions of each point in the depth map, and a second position mapping matrix for representing the correspondence between the positions of each point in the first two coordinate dimensions of the three-dimensional grid and the positions of each point in the RGB image can be generated in advance. When the semantic labels are subsequently projected onto the three-dimensional grid, the first position mapping matrix and the second position mapping matrix are directly loaded, and the semantic label projection process is converted into a memory access assignment process, thereby reducing end-side power consumption and improving runtime calculation speed.

[0054] Optionally, generating a first position mapping matrix based on the three-dimensional grid includes: converting the depth map contained in the three-dimensional grid into a point cloud according to the camera intrinsic parameters corresponding to the three-dimensional grid; converting the point cloud to the same camera extrinsic parameters as the depth camera through the camera extrinsic parameters corresponding to the three-dimensional grid; converting the point cloud into a target depth map in the same coordinate system as the depth map taken by the depth camera through the same camera intrinsic parameters as the depth camera; determining the correspondence between the positions of each point cloud in the three-dimensional grid in the first two dimensions and the positions of each point in the depth map according to the target depth map, and generating a first position mapping matrix.

[0055] The camera intrinsic parameters refer to the parameters related to the camera's own characteristics, such as the camera's focal length and pixel size. The camera extrinsic parameters refer to the parameters in the world coordinate system, such as the camera's position and rotation direction. The camera intrinsic parameters corresponding to the 3D grid can be custom settings or the intrinsic parameters of a specified depth camera.

[0056] In this embodiment, the point cloud in the UVD coordinate system is obtained by multiplying the depth map contained in the three-dimensional grid by the inverse of the intrinsic parameter matrix corresponding to the three-dimensional grid. Then, based on the difference between the camera extrinsic parameters corresponding to the three-dimensional grid and the extrinsic parameters of the depth camera, the point cloud is translated and rotated accordingly to convert the point cloud to the same camera extrinsic parameters as the depth camera. The point cloud is then multiplied by the intrinsic parameter matrix of the depth camera to convert the point cloud into a target depth map in the same coordinate system as the depth map observed by the depth camera. By comparing the target depth map with the observed depth map, the corresponding relationship between the position of each point cloud in the three-dimensional grid in the first two dimensions and the position of each point in the observed depth map is obtained to generate a first position mapping matrix.

[0057] Optionally, generating a second position mapping matrix based on the three-dimensional grid includes: converting the depth map contained in the three-dimensional grid into a point cloud according to the camera intrinsic parameters corresponding to the three-dimensional grid; converting the point cloud to the same camera extrinsic parameters as the RGB camera through the camera extrinsic parameters corresponding to the three-dimensional grid; converting the point cloud into a target depth map in the same coordinate system as the RGB image taken by the RGB camera through the same camera intrinsic parameters as the RGB camera; determining the correspondence between the position of each point cloud in the three-dimensional grid in the first two dimensions and the position of each point in the RGB image according to the target depth map, and generating a second position mapping matrix.

[0058] In this embodiment, the point cloud in the UVD coordinate system is obtained by multiplying the depth map contained in the three-dimensional grid by the inverse of the intrinsic parameter matrix corresponding to the three-dimensional grid. Then, based on the difference between the camera extrinsic parameters corresponding to the three-dimensional grid and the extrinsic parameters of the RGB camera, the point cloud is translated and rotated accordingly, and the point cloud is converted to the same camera extrinsic parameters as the RGB camera. The point cloud is then multiplied by the RGB camera intrinsic parameter matrix to convert the point cloud into a target depth map in the same coordinate system as the RGB image taken by the RGB camera. By comparing the target depth map with the RGB image, the corresponding relationship between the position of each point cloud in the three-dimensional grid in the first two dimensions and the position of each point in the RGB image is obtained, and a second position mapping matrix is ​​generated.

[0059] S230 . Convert the depth map contained in the three-dimensional grid into a point cloud according to the camera intrinsic parameters corresponding to the three-dimensional grid.

[0060] In an optional embodiment, points with the same depth value in the three-dimensional grid are regarded as a depth map; for each depth map in the three-dimensional grid, the point cloud coordinates corresponding to each point in the depth map are determined by multiplying the coordinates of each point in the depth map in the first two dimensions by the inverse of the intrinsic parameter matrix corresponding to the three-dimensional grid.

[0061] For example, for each depth value d i , the coordinate value of the third dimension in the three-dimensional grid can be the same as d i All points of i The depth map of the 3D grid can be regarded as i grids with different depths of d i For each depth map in the three-dimensional grid, the coordinates of each point in the depth map in the first two dimensions (u i , v i ) is multiplied by the inverse of the intrinsic parameter matrix corresponding to the three-dimensional grid to convert the coordinate values ​​of each point in the three-dimensional grid from the "depth map" to the "point cloud".

[0062] S240, traversing the point cloud in the three-dimensional grid, obtaining the depth value of the camera depth map corresponding to the traversed point cloud according to the first position mapping matrix, and determining whether to mark the traversed point cloud based on the depth value.

[0063] Among them, the first position mapping matrix is ​​used to represent the correspondence between the position of each point in the first two coordinate dimensions of the three-dimensional grid and the position of each point in the depth map. The camera depth map can be a depth map captured by a depth camera. In this embodiment, any number of depth cameras can be selected to obtain depth information, not limited to monocular depth cameras.

[0064] Optionally, according to the first position mapping matrix, the depth value of the camera depth map corresponding to the traversed point cloud is obtained, and based on the depth value, it is determined whether to mark the traversed point cloud, including: according to the coordinates of the traversed point cloud in the first two dimensions in the three-dimensional grid, indexing in the first position mapping matrix to obtain the target position of the corresponding camera depth map; obtaining the target depth value of the camera depth map at the target position; if the coordinate value of the traversed point cloud in the third dimension in the three-dimensional grid is equal to the target depth value, marking the traversed point cloud.

[0065] In this embodiment, Figure 3 As shown in the figure, in the process of marking the point cloud in the three-dimensional grid, it is necessary to traverse all the point clouds in the three-dimensional grid, and find the corresponding target position in the camera depth map according to the coordinates of the traversed point cloud in the first two dimensions in the three-dimensional grid. The target depth value of the camera depth map at the target position is obtained, and then the coordinate value of the traversed point cloud in the third dimension in the three-dimensional grid is compared with the target depth value. If the two are equal, the point cloud is marked, otherwise, the point cloud is not marked.

[0066] S250. In the case of marking the traversed point cloud, obtain the semantic label of the RGB semantic segmentation map corresponding to the traversed point cloud according to the second position mapping matrix, and add the semantic label to the traversed point cloud.

[0067] Among them, the second position mapping matrix is ​​used to represent the correspondence between the position of each point in the first two coordinate dimensions of the three-dimensional grid and the position of each point in the RGB image. The RGB semantic segmentation map is obtained by semantically segmenting the RGB image taken by the RGB camera. In this embodiment, any number of RGB cameras can be selected to obtain semantic information, not limited to monocular RGB cameras.

[0068] In this embodiment, Figure 3 As shown in the figure, when the traversed point cloud is not marked, there is no need to assign semantic labels to the point cloud. When the traversed point cloud is marked, it is necessary to assign semantic labels to the point cloud. At this time, according to the coordinates of the traversed point cloud in the first two dimensions in the three-dimensional grid, the corresponding position is found in the RGB semantic segmentation map, and the semantic label corresponding to the position in the RGB semantic segmentation map is assigned to the traversed point cloud.

[0069] S260: Generate a map containing semantic information based on the three-dimensional grid with semantic labels added.

[0070] The technical solution of the embodiment of the present disclosure is to construct a three-dimensional grid containing depth information; generate a position mapping matrix based on the three-dimensional grid, the position mapping matrix including a first position mapping matrix corresponding to the depth map, and a second position mapping matrix corresponding to the RGB image; traverse the point cloud in the three-dimensional grid, obtain the depth value of the camera depth map corresponding to the traversed point cloud and the semantic label of the RGB semantic segmentation map according to the position mapping matrix, and determine whether to add semantic labels to the traversed point cloud based on the depth value; generate a map containing semantic information according to the three-dimensional grid with the semantic label added, so as to solve the problem of large amount of computation during runtime in the related art that the map is constructed by projecting the point cloud onto the semantic segmentation image, and optimize the process of projecting the two-dimensional semantic label to the three-dimensional point cloud into a fixed memory access assignment process by pre-generating the position mapping matrix, thereby reducing the amount of computation during runtime and reducing the computational complexity.

[0071] Figure 4 is a schematic diagram of a map construction device provided by an embodiment of the present disclosure, such as Figure 4 As shown, the apparatus includes: a grid construction module 410 , a location mapping module 420 , a labeling module 430 , and a map construction module 440 .

[0072] A mesh construction module 410, for constructing a three-dimensional mesh including depth information;

[0073] A position mapping module 420, configured to generate a position mapping matrix based on the three-dimensional grid, the position mapping matrix comprising a first position mapping matrix corresponding to the depth map and a second position mapping matrix corresponding to the RGB image;

[0074] A label adding module 430 is used to traverse the point cloud in the three-dimensional grid, obtain the depth value of the camera depth map corresponding to the traversed point cloud and the semantic label of the RGB semantic segmentation map according to the position mapping matrix, and determine whether to add a semantic label to the traversed point cloud based on the depth value;

[0075] The map construction module 440 is used to generate a map containing semantic information according to the three-dimensional grid with semantic labels added.

[0076] The technical solution provided by the embodiment of the present disclosure constructs a three-dimensional grid containing depth information; generates a position mapping matrix based on the three-dimensional grid, the position mapping matrix includes a first position mapping matrix corresponding to the depth map, and a second position mapping matrix corresponding to the RGB image; traverses the point cloud in the three-dimensional grid, obtains the depth value of the camera depth map corresponding to the traversed point cloud and the semantic label of the RGB semantic segmentation map according to the position mapping matrix, and determines whether to add semantic labels to the traversed point cloud based on the depth value; generates a map containing semantic information according to the three-dimensional grid with the semantic label added, which solves the problem of large amount of computation during runtime in the related art that a map is constructed by projecting the point cloud onto the semantic segmentation image, and optimizes the process of projecting the two-dimensional semantic label to the three-dimensional point cloud into a fixed memory access assignment process by pre-generating the position mapping matrix, thereby reducing the amount of computation during runtime and reducing the computational complexity.

[0077] Optionally, the location mapping module 420 is used to:

[0078] According to the camera intrinsic parameters corresponding to the 3D mesh, the depth map contained in the 3D mesh is converted into a point cloud;

[0079] Through the camera extrinsics corresponding to the 3D grid, the point cloud is converted to the same camera extrinsics as the depth camera;

[0080] Using the same camera internal parameters as the depth camera, the point cloud is converted into a target depth map with the same coordinate system as the depth map taken by the depth camera;

[0081] According to the target depth map, the corresponding relationship between the position of each point cloud in the three-dimensional grid in the first two dimensions and the position of each point in the depth map is determined to generate a first position mapping matrix.

[0082] Optionally, the location mapping module 420 is used to:

[0083] According to the camera intrinsic parameters corresponding to the 3D mesh, the depth map contained in the 3D mesh is converted into a point cloud;

[0084] Through the camera extrinsics corresponding to the 3D grid, the point cloud is converted to the same camera extrinsics as the RGB camera;

[0085] Using the same camera internal parameters as the RGB camera, the point cloud is converted into a target depth map in the same coordinate system as the RGB image taken by the RGB camera;

[0086] According to the target depth map, the corresponding relationship between the position of each point cloud in the three-dimensional grid in the first two dimensions and the position of each point in the RGB image is determined to generate a second position mapping matrix.

[0087] Optionally, the tag adding module 430 includes:

[0088] A point cloud marking unit, configured to obtain a depth value of a camera depth map corresponding to the traversed point cloud according to the first position mapping matrix, and determine whether to mark the traversed point cloud based on the depth value;

[0089] The label adding unit is used to obtain the semantic label of the RGB semantic segmentation map corresponding to the traversed point cloud according to the second position mapping matrix when marking the traversed point cloud, and add the semantic label to the traversed point cloud.

[0090] Optional, point cloud labeling unit for:

[0091] According to the coordinates of the traversed point cloud in the first two dimensions of the three-dimensional grid, the target position of the corresponding camera depth map is indexed in the first position mapping matrix;

[0092] Get the target depth value of the camera depth map at the target position;

[0093] If the coordinate value of the traversed point cloud in the third dimension of the three-dimensional grid is equal to the target depth value, the traversed point cloud is marked.

[0094] Optionally, the map construction module 440 is used to:

[0095] The semantically labeled 3D grid is used as a 3D map containing semantic information; or

[0096] The 3D grid with semantic labels is projected downward into a plane map, and the semantic labels of the corresponding point clouds in the 3D grid are added to each point in the plane map to generate a 2D map containing semantic information.

[0097] Optionally, the map construction module 440 is used to:

[0098] Traversing each point in the plane graph, and determining at least one target point cloud corresponding to the traversed point in the three-dimensional grid;

[0099] From the semantic labels of at least one target point cloud, select the semantic label with the highest importance level and add it to the traversed point.

[0100] Optionally, the grid construction module 410 is used to:

[0101] Construct a 3D grid in a specified coordinate system based on the width and height of the depth map and a custom depth value set according to the real distance between two adjacent points;

[0102] Among them, the width of the depth map, the height of the depth map, and the custom depth value order correspond to the three dimensions of the specified coordinate system.

[0103] Optionally, the grid construction module 410 is further configured to:

[0104] After constructing a three-dimensional grid containing depth information, points with the same depth value in the three-dimensional grid are regarded as a depth map;

[0105] For each depth map in the three-dimensional grid, the point cloud coordinates corresponding to each point in the depth map are determined by multiplying the coordinates of each point in the depth map in the first two dimensions by the inverse of the intrinsic parameter matrix corresponding to the three-dimensional grid.

[0106] The map construction device provided in the embodiments of the present disclosure can execute the map construction method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0107] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0108] Figure 5 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Figure 5 , which shows an electronic device (eg, Figure 5 The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0109] like Figure 5 As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An edit / output (I / O) interface 505 is also connected to the bus 504.

[0110] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0111] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0112] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0113] The electronic device provided in the embodiment of the present disclosure and the map construction method provided in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0114] The embodiment of the present disclosure provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, the map construction method provided by the above embodiment is implemented.

[0115] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0116] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0117] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0118] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: constructs a three-dimensional grid containing depth information; generates a position mapping matrix based on the three-dimensional grid, the position mapping matrix including a first position mapping matrix corresponding to the depth map and a second position mapping matrix corresponding to the RGB image; traverses the point cloud in the three-dimensional grid, obtains the depth value of the camera depth map corresponding to the traversed point cloud and the semantic label of the RGB semantic segmentation map according to the position mapping matrix, and determines whether to add semantic labels to the traversed point cloud based on the depth value; generates a map containing semantic information according to the three-dimensional grid with the semantic label added.

[0119] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or combinations thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0120] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0121] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0122] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0123] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0124] According to one or more embodiments of the present disclosure, Example 1 provides a map construction method, including:

[0125] Construct a 3D mesh containing depth information;

[0126] Generate a position mapping matrix based on the three-dimensional grid, the position mapping matrix including a first position mapping matrix corresponding to the depth map and a second position mapping matrix corresponding to the RGB image;

[0127] Traverse the point cloud in the three-dimensional grid, obtain the depth value of the camera depth map corresponding to the traversed point cloud and the semantic label of the RGB semantic segmentation map according to the position mapping matrix, and determine whether to add a semantic label to the traversed point cloud based on the depth value;

[0128] Based on the 3D grid with semantic labels, a map containing semantic information is generated.

[0129] According to one or more embodiments of the present disclosure, Example 2 generates a first position mapping matrix based on a three-dimensional grid according to the method of Example 1, including:

[0130] According to the camera intrinsic parameters corresponding to the 3D mesh, the depth map contained in the 3D mesh is converted into a point cloud;

[0131] Through the camera extrinsics corresponding to the 3D grid, the point cloud is converted to the same camera extrinsics as the depth camera;

[0132] Using the same camera internal parameters as the depth camera, the point cloud is converted into a target depth map with the same coordinate system as the depth map taken by the depth camera;

[0133] According to the target depth map, the corresponding relationship between the position of each point cloud in the three-dimensional grid in the first two dimensions and the position of each point in the depth map is determined to generate a first position mapping matrix.

[0134] According to one or more embodiments of the present disclosure, Example 3 generates a second position mapping matrix based on the three-dimensional grid according to the method of Example 1, including:

[0135] According to the camera intrinsic parameters corresponding to the 3D mesh, the depth map contained in the 3D mesh is converted into a point cloud;

[0136] Through the camera extrinsics corresponding to the 3D grid, the point cloud is converted to the same camera extrinsics as the RGB camera;

[0137] Using the same camera internal parameters as the RGB camera, the point cloud is converted into a target depth map in the same coordinate system as the RGB image taken by the RGB camera;

[0138] According to the target depth map, the corresponding relationship between the position of each point cloud in the three-dimensional grid in the first two dimensions and the position of each point in the RGB image is determined to generate a second position mapping matrix.

[0139] According to one or more embodiments of the present disclosure, Example 4, according to the method of Example 1, obtains the depth value of the camera depth map corresponding to the traversed point cloud and the semantic label of the RGB semantic segmentation map according to the position mapping matrix, and determines whether to add a semantic label to the traversed point cloud based on the depth value, including:

[0140] According to the first position mapping matrix, obtain the depth value of the camera depth map corresponding to the traversed point cloud, and determine whether to mark the traversed point cloud based on the depth value;

[0141] In the case of marking the traversed point cloud, the semantic label of the RGB semantic segmentation map corresponding to the traversed point cloud is obtained according to the second position mapping matrix, and the semantic label is added to the traversed point cloud.

[0142] According to one or more embodiments of the present disclosure, Example 5, according to the method of Example 4, obtains the depth value of the camera depth map corresponding to the traversed point cloud according to the first position mapping matrix, and determines whether to mark the traversed point cloud based on the depth value, including:

[0143] According to the coordinates of the traversed point cloud in the first two dimensions of the three-dimensional grid, the target position of the corresponding camera depth map is indexed in the first position mapping matrix;

[0144] Get the target depth value of the camera depth map at the target position;

[0145] If the coordinate value of the traversed point cloud in the third dimension of the three-dimensional grid is equal to the target depth value, the traversed point cloud is marked.

[0146] According to one or more embodiments of the present disclosure, Example 6 generates a map containing semantic information according to the method of Example 1 based on the three-dimensional grid to which the semantic labels are added, including:

[0147] The semantically labeled 3D grid is used as a 3D map containing semantic information; or

[0148] The 3D grid with semantic labels is projected downward into a plane map, and the semantic labels of the corresponding point clouds in the 3D grid are added to each point in the plane map to generate a 2D map containing semantic information.

[0149] According to one or more embodiments of the present disclosure, Example 7, based on the method of Example 6, adds a semantic label of a corresponding point cloud in a three-dimensional grid to each point in the plane map, including:

[0150] Traversing each point in the plane graph, and determining at least one target point cloud corresponding to the traversed point in the three-dimensional grid;

[0151] From the semantic labels of at least one target point cloud, select the semantic label with the highest importance level and add it to the traversed point.

[0152] According to one or more embodiments of the present disclosure, Example 8 constructs a three-dimensional mesh containing depth information according to the method of Example 1, including:

[0153] Construct a 3D grid in a specified coordinate system based on the width and height of the depth map and a custom depth value set according to the real distance between two adjacent points;

[0154] Among them, the width of the depth map, the height of the depth map, and the custom depth value order correspond to the three dimensions of the specified coordinate system.

[0155] According to one or more embodiments of the present disclosure, Example 9, according to the method of Example 8, after constructing the three-dimensional mesh containing the depth information, further includes:

[0156] Treat points with the same depth value in the 3D grid as a depth map;

[0157] For each depth map in the three-dimensional grid, the point cloud coordinates corresponding to each point in the depth map are determined by multiplying the coordinates of each point in the depth map in the first two dimensions by the inverse of the intrinsic parameter matrix corresponding to the three-dimensional grid.

[0158] According to one or more embodiments of the present disclosure, Example 10 provides a map construction device, including:

[0159] A mesh construction module for constructing a 3D mesh containing depth information;

[0160] A position mapping module, configured to generate a position mapping matrix based on the three-dimensional grid, wherein the position mapping matrix includes a first position mapping matrix corresponding to the depth map and a second position mapping matrix corresponding to the RGB image;

[0161] A label adding module is used to traverse the point cloud in the three-dimensional grid, obtain the depth value of the camera depth map corresponding to the traversed point cloud and the semantic label of the RGB semantic segmentation map according to the position mapping matrix, and determine whether to add a semantic label to the traversed point cloud based on the depth value;

[0162] The map construction module is used to generate a map containing semantic information based on the three-dimensional grid with semantic labels added.

[0163] According to one or more embodiments of the present disclosure, Example 11 provides an electronic device, the electronic device including:

[0164] one or more processors;

[0165] a storage device for storing one or more programs,

[0166] When one or more programs are executed by one or more processors, the one or more processors implement the map construction method as described in any one of Examples 1-9.

[0167] According to one or more embodiments of the present disclosure, Example 12 provides a storage medium containing computer executable instructions, which are used to perform the map construction method as described in any one of Examples 1-9 when executed by a computer processor.

[0168] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0169] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0170] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A map construction method, characterized in that: include: Construct a 3D mesh containing depth information; generating a position mapping matrix based on the three-dimensional grid, the position mapping matrix comprising a first position mapping matrix corresponding to the depth map and a second position mapping matrix corresponding to the RGB image; Traversing the point cloud in the three-dimensional grid, obtaining the depth value of the camera depth map corresponding to the traversed point cloud and the semantic label of the RGB semantic segmentation map according to the position mapping matrix, and determining whether to add the semantic label to the traversed point cloud based on the depth value; Based on the 3D grid with semantic labels, a map containing semantic information is generated.

2. The method according to claim 1, characterized in that Generating a first position mapping matrix based on the three-dimensional grid includes: Converting a depth map contained in the three-dimensional grid into a point cloud according to camera intrinsic parameters corresponding to the three-dimensional grid; The point cloud is converted to the same camera extrinsic parameters as the depth camera by using the camera extrinsic parameters corresponding to the three-dimensional grid; The point cloud is converted into a target depth map having the same coordinate system as the depth map taken by the depth camera by using the same camera intrinsic parameters as the depth camera; According to the target depth map, the corresponding relationship between the position of each point cloud in the three-dimensional grid in the first two dimensions and the position of each point in the depth map is determined to generate a first position mapping matrix.

3. The method according to claim 1, characterized in that Generating a second position mapping matrix based on the three-dimensional grid includes: Converting a depth map contained in the three-dimensional grid into a point cloud according to camera intrinsic parameters corresponding to the three-dimensional grid; The point cloud is converted to the same camera extrinsic parameters as the RGB camera by using the camera extrinsic parameters corresponding to the three-dimensional grid; The point cloud is converted into a target depth map having the same coordinate system as the RGB image taken by the RGB camera by using the same camera intrinsic parameters as the RGB camera; According to the target depth map, the corresponding relationship between the position of each point cloud in the three-dimensional grid in the first two dimensions and the position of each point in the RGB image is determined to generate a second position mapping matrix.

4. The method according to claim 1, characterized in that Acquiring the depth value of the camera depth map corresponding to the traversed point cloud and the semantic label of the RGB semantic segmentation map according to the position mapping matrix, and determining whether to add the semantic label to the traversed point cloud based on the depth value, including: According to the first position mapping matrix, obtaining a depth value of a camera depth map corresponding to the traversed point cloud, and determining whether to mark the traversed point cloud based on the depth value; In the case of marking the traversed point cloud, the semantic label of the RGB semantic segmentation map corresponding to the traversed point cloud is obtained according to the second position mapping matrix, and the semantic label is added to the traversed point cloud.

5. The method according to claim 4, characterized in that The step of obtaining a depth value of a camera depth map corresponding to the traversed point cloud according to the first position mapping matrix, and determining whether to mark the traversed point cloud based on the depth value, includes: According to the coordinates of the traversed point cloud in the first two dimensions in the three-dimensional grid, indexing in the first position mapping matrix obtains the target position of the corresponding camera depth map; Obtain a target depth value of the camera depth image at the target position; If the coordinate value of the traversed point cloud in the third dimension in the three-dimensional grid is equal to the target depth value, the traversed point cloud is marked.

6. The method according to claim 1, characterized in that The generating a map containing semantic information according to the three-dimensional grid with semantic labels added includes: Using the three-dimensional grid with added semantic labels as a three-dimensional map containing semantic information; or The three-dimensional grid with added semantic labels is projected downward into a plane map, and the semantic label of the corresponding point cloud in the three-dimensional grid is added to each point in the plane map to generate a two-dimensional map containing semantic information.

7. The method according to claim 6, characterized in that The step of adding a semantic label of a corresponding point cloud in the three-dimensional grid to each point in the plan view includes: Traversing each point in the plane map, and determining at least one target point cloud corresponding to the traversed point in the three-dimensional grid; From the semantic labels of the at least one target point cloud, a semantic label with the highest importance level is selected and added to the traversed point.

8. The method according to claim 1, characterized in that The constructing of a three-dimensional grid containing depth information comprises: Construct a 3D grid in a specified coordinate system based on the width and height of the depth map and a custom depth value set according to the real distance between two adjacent points; The width of the depth map, the height of the depth map, and the custom depth value sequence correspond to the three dimensions of the specified coordinate system.

9. The method according to claim 8, characterized in that After constructing the three-dimensional grid containing the depth information, the method further includes: Considering points with the same depth value in the three-dimensional grid as a depth map; For each depth map in the three-dimensional grid, the point cloud coordinates corresponding to each point in the depth map are determined by multiplying the coordinates of each point in the depth map in the first two dimensions by the inverse of the intrinsic parameter matrix corresponding to the three-dimensional grid.

10. A map construction device, characterized in that: include: A mesh construction module for constructing a 3D mesh containing depth information; A position mapping module, configured to generate a position mapping matrix based on the three-dimensional grid, wherein the position mapping matrix includes a first position mapping matrix corresponding to the depth map and a second position mapping matrix corresponding to the RGB image; A label adding module is used to traverse the point cloud in the three-dimensional grid, obtain the depth value of the camera depth map corresponding to the traversed point cloud and the semantic label of the RGB semantic segmentation map according to the position mapping matrix, and determine whether to add the semantic label to the traversed point cloud based on the depth value; The map construction module is used to generate a map containing semantic information based on the three-dimensional grid with semantic labels added.

11. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the map construction method as described in any one of claims 1-9.

12. A storage medium comprising computer executable instructions, wherein the computer executable instructions are used to perform the map construction method according to any one of claims 1 to 9 when executed by a computer processor.

Images