Semantic-elevation map construction method for complex indoor scene

Abstract

The invention discloses a semantic-elevation map construction method for a complex indoor scene, and relates to the field of elevation map construction. The method comprises the following steps: firstly, acquiring image information of a surrounding environment by using a depth camera, and generating a semantic segmentation mask through a YOLACT semantic segmentation network; secondly, aligning point cloud data and image data of the depth camera through coordinate transformation by using internal and external parameters of the camera; thirdly, according to the semantic segmentation mask, modifying the RGB value of the point cloud into a corresponding category color, and obtaining the point cloud representing semantic information through color coding; and finally, calculating the height of each point relative to a camera plane by using the three-dimensional coordinate of each point in the semantic point cloud, generating an elevation map, introducing specific semantic information for the elevation map according to the RGB value of the point cloud, and constructing a semantic-elevation map. According to the method, the problem that effective navigation decision making and planning are difficult to carry out due to semantic information loss in a complex indoor environment is solved, and a safer and more accurate solution is provided for a navigation method based on an elevation map.

Description

Technical Field

[0001] The invention relates to the technical field of elevation map construction, and in particular to a semantic-elevation map construction method for complex indoor scenes. Background Art

[0002] The elevation map aims to generate a map description containing height data for a given terrain. It is a multimodal task based on sensor data and probabilistic models. Its model structure is based on the mapping frame of the center of the subject equipped with a camera, and the point cloud data of the surrounding environment collected by the depth sensor is used as input; these data are processed by probabilistic methods to generate an elevation map containing confidence intervals for subsequent navigation tasks.

[0003] Existing elevation map construction technologies perform well in ideal environments, but they still face challenges in practical applications, especially in complex indoor environments such as spaces containing corridors, rooms, stairs, and dynamic obstacles. These technologies mainly rely on prior 3D geometric information to identify traffic areas, but it is difficult to form accurate judgments when processing complex and changing spatial information, which affects navigation decisions and limits their applicability in reality.

[0004] Although elevation maps can provide data on the height changes of terrain, as a method that only relies on point cloud data, it is unable to capture the semantic information of the environment through visual information. When faced with the need to understand the semantic level of objects in the environment, such as distinguishing between static and dynamic obstacles or identifying object categories, the function is limited, which is particularly evident in complex scenes that require advanced cognitive processing. Therefore, it is particularly important to develop a multimodal method that integrates two-dimensional images and three-dimensional point clouds. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes a semantic-elevation map construction method for complex indoor scenes. The surrounding environment information is collected by a depth camera, and a point cloud representing semantic information through color coding is generated by fusing the two-dimensional image semantic segmentation mask and the three-dimensional point cloud spatial data, and a semantic-elevation map is constructed based on this.

[0006] The present invention proposes a semantic-elevation map construction method for complex indoor scenes, comprising the following steps:

[0007] Step 1: Use a depth camera to collect image information of the surrounding environment and input it into the YOLACT semantic segmentation network to output the semantic segmentation result of the image. According to the navigation task, the required category labels and their corresponding pixel position information are extracted, and different grayscale values ​​are used to fill them to generate a semantic segmentation mask;

[0008] Step 2: Use the internal and external parameters of the camera to align the depth camera point cloud data and image data through coordinate transformation. The aligned point cloud is fused with the semantic segmentation mask generated in step 1, and the point cloud RGB value is modified to the corresponding category color to obtain a point cloud that represents semantic information through color coding;

[0009] Step 3: Calculate the height of each point relative to the camera plane according to the 3D coordinates of the semantic point cloud generated in step 2 and generate an elevation map. Assign specific semantic information to the elevation map according to the RGB value of the point cloud to construct a semantic-elevation map.

[0010] Specifically, generating a semantic segmentation mask in step 1 includes the following steps:

[0011] After the RGB image captured by the depth camera is input into the YOLACT semantic segmentation network, the features are extracted by the ResNet backbone network to generate a global prototype mask, and a set of coefficient matrices are obtained by the coefficient matrix prediction module based on mask position regression; the coefficient matrix prediction module based on mask position regression first passes the prototype mask through two sets of 2D convolution layers with 3×3 convolution kernels in series, and then passes through three 2D convolution layers with 3×3 convolution kernels in parallel to generate a category confidence prediction matrix. Assume that each pixel position of the semantic segmentation image is (u s ,v s ), for each pixel in the semantic image The confidence that the pixel belongs to each category k∈Class can be obtained from the category confidence prediction matrix The SoftMax function is used to convert the confidence into a normalized label probability:

[0012]

[0013] Select the maximum label probability of all categories as the label of the pixel

[0014]

[0015] After determining the pixel labels, select the category labels that need to be retained according to the navigation task. For example, in an indoor navigation task, you can choose to retain the ground and stair labels, and classify the remaining categories as obstacle labels. Set different grayscale values ​​for the pixel position according to the label to generate semantic segmentation mask information.

[0016] Specifically, point cloud alignment and fusion in step 2 to generate semantic point cloud includes the following steps:

[0017] To achieve point cloud alignment, first obtain the internal parameters of the depth camera, including the focal length (f x ,f y ) and the principal point (c x ,cy ), as well as the external parameter rotation matrix R and translation t. The internal parameter matrix is ​​usually expressed as:

[0018]

[0019] Among them, f x , f y is the focal length of the X and Y axes, c x , c y is the optical center of the image.

[0020] After obtaining the camera intrinsic parameters, each pixel in the depth map needs to be converted into coordinates in three-dimensional space. Suppose the position of a pixel in the depth map is (u d ,v d ), the depth value of the point is d. The coordinates (X, Y, Z) of the point in three-dimensional space can be calculated by the following formula:

[0021]

[0022]

[0023] Z=d

[0024] The obtained 3D point cloud is then mapped to the RGB image. The 3D coordinates are converted back to pixel coordinates using the intrinsic parameters of the depth camera, and the extrinsic parameters are used to adjust the 3D coordinates. The formula is as follows:

[0025]

[0026] Among them, u ′ ,v ′ is the pixel coordinate after mapping. After this step, the aligned point cloud data is mapped to the semantic segmentation mask generated in step 1, and the RGB value of the point cloud is modified according to the mask category of the point cloud location to obtain a point cloud that represents semantic information in a color-coded manner.

[0027] Specifically, the steps of generating the semantic-elevation map in step 3 include:

[0028] First, four coordinate systems are defined: inertial coordinate system N, user coordinate system U, sensor coordinate system S, and map coordinate system M. The inertial coordinate system N is fixed in the environment, and it is assumed that the actual terrain is stationary relative to this coordinate system. The three-dimensional position of the grid cell i of the elevation map is given by Given, where x i and i is defined by the position of grid cell i, is the estimated height of the terrain at grid cell i.

[0029] In the map coordinate system M, the height measurements are approximated by a Gaussian probability distribution: The mean is p and the variance is The depth camera continuously collects point cloud data of the terrain. The new measurement data is processed into points in space and mapped into the elevation map. The measurement point in the sensor coordinate system S is represented as Sr SP , these points can be converted to corresponding height measurements p:

[0030]

[0031] Where P = [0,0,1] is a projection matrix that maps the 3D measurements to heights in the map coordinate system; Φ SM Represents the rotation matrix from the sensor coordinate system S to the map coordinate system M, Mr SM It represents the translation vector from the sensor coordinate system S to the map coordinate system M. To obtain the variance of the height measurement value We derive the Jacobian matrix J of the sensor measurement from the above formula S The Jacobian matrix J of the sensor coordinate system rotation Φ

[0032]

[0033]

[0034] Where C(Φ) represents the corresponding rotation matrix, and the parameters Denotes the position vector r from the sensor coordinate system S to the map frame M, and uses the cross product symbol × to indicate vector rotation. The error propagation is:

[0035]

[0036] where Σ S is the covariance matrix from the sensor, is the covariance matrix of the sensor rotation.

[0037] New altitude measurement The estimated value from the existing map is compared through a one-dimensional Kalman filter Fusion:

[0038]

[0039]

[0040] in, and are the height and variance before update, and are the updated height and variance. The initial value setting depends mainly on the specific scene and sensor resolution and accuracy. After converting the three-dimensional coordinates of the point cloud into the height information in the map, the RGB values ​​of the cell point cloud are counted, and the color that appears most frequently is recorded as the main color of the cell. The cell is mapped to a predefined semantic category according to different colors. For example, in an indoor environment, green can represent a passable road, blue can represent stairs, and red can represent obstacles and walls. For cells without valid point cloud data, or cells with a large number of mixed colors, that is, when no color accounts for more than 80%, the cell is marked as an obstacle category. Finally, a new "class" class is added to the elevation map to store the semantic category of each cell to realize the construction of a semantic-elevation map.

[0041] Compared with the existing technology, the beneficial technical effect of the present invention is: the depth camera collects the surrounding environment information, and generates a point cloud representing the semantic information through color coding by fusing the semantic segmentation mask of the two-dimensional image with the three-dimensional point cloud data, and constructs a semantic-elevation map based on this. The advantage is that it solves the problem that the traditional elevation map is difficult to make effective navigation decisions and planning due to the lack of semantic information in the face of complex indoor environments, and provides a safer and more accurate solution for navigation methods based on elevation maps. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present invention.

[0043] Figure 1 It is a flow chart of the semantic-elevation map construction method in the present invention;

[0044] Figure 2 This is a schematic diagram comparing the semantic-elevation map in the present invention with the ordinary elevation map. DETAILED DESCRIPTION

[0045] A preferred embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. Note that the aspects described below in conjunction with the accompanying drawings are only exemplary and should not be construed as limiting the scope of protection of the present invention in any way.

[0046] The semantic-elevation map construction process proposed in the present invention is as follows: Figure 1 As shown in the figure, it is mainly divided into three modules: (a) semantic segmentation module; (b) point cloud fusion module; (c) semantic-elevation map construction module.

[0047] (a) Semantic segmentation module: After the RGB image captured by the depth camera is input into the YOLACT semantic segmentation network, the features are extracted by the ResNet backbone network to generate a global prototype mask, and a set of coefficient matrices are obtained by the coefficient matrix prediction module based on mask position regression; the coefficient matrix prediction module based on mask position regression first passes the prototype mask through two sets of 2D convolution layers with 3×3 convolution kernels in series, and then passes through three 2D convolution layers with 3×3 convolution kernels in parallel to generate a category confidence prediction matrix. Assume that each pixel position of the semantic segmentation image is (u s ,v s ), for each pixel in the semantic image The confidence that the pixel belongs to each category k∈Class can be obtained from the category confidence prediction matrix The SoftMax function is used to convert the confidence into a normalized label probability:

[0048]

[0049] Select the maximum label probability of all categories as the label of the pixel

[0050]

[0051] After determining the pixel labels, select the category labels that need to be retained according to the navigation task. For example, in an indoor navigation task, you can choose to retain the ground and stair labels, and classify the remaining categories as obstacle labels. Set different grayscale values ​​for the pixel position according to the label to generate semantic segmentation mask information.

[0052] (b) Point cloud fusion module: To achieve point cloud alignment, we first obtain the internal parameters of the depth camera, including the focal length (f x ,f y ) and the principal point (c x ,c y ), as well as the external parameter rotation matrix R and translation t. The internal parameter matrix is ​​usually expressed as:

[0053]

[0054] Among them, f x , f y is the focal length of the X and Y axes, c x , c y is the optical center of the image.

[0055] After obtaining the camera intrinsic parameters, each pixel in the depth map needs to be converted into coordinates in three-dimensional space. Suppose the position of a pixel in the depth map is (u d ,v d), the depth value of the point is d. The coordinates (X, Y, Z) of the point in three-dimensional space can be calculated by the following formula:

[0056]

[0057]

[0058] Z=d

[0059] The obtained 3D point cloud is then mapped to the RGB image. The 3D coordinates are converted back to pixel coordinates using the intrinsic parameters of the depth camera, and the extrinsic parameters are used to adjust the 3D coordinates. The formula is as follows:

[0060]

[0061] Among them, u ′ ,v ′ is the pixel coordinate after mapping. After this step, the aligned point cloud data is mapped to the semantic segmentation mask generated in step 1, and the RGB value of the point cloud is modified according to the mask category of the point cloud location to obtain a point cloud that represents semantic information in a color-coded manner.

[0062] (c) Semantic-elevation map construction module: First, four coordinate systems are defined: inertial coordinate system N, user coordinate system U, sensor coordinate system S, and map coordinate system M. The inertial coordinate system N is fixed in the environment, and it is assumed that the actual terrain is stationary relative to this coordinate system. The three-dimensional position of the grid cell i of the elevation map is given by Given, where x i and i is defined by the position of grid cell i, is the estimated height of the terrain at grid cell i.

[0063] In the map coordinate system M, the height measurements are approximated by a Gaussian probability distribution: The mean is p and the variance is The depth camera continuously collects point cloud data of the terrain. The new measurement data is processed into points in space and mapped into the elevation map. The measurement point in the sensor coordinate system S is represented as Sr SP , these points can be converted to corresponding height measurements p:

[0064]

[0065] Where P = [0,0,1] is a projection matrix that maps the 3D measurements to heights in the map coordinate system; Φ SM Represents the rotation matrix from the sensor coordinate system S to the map coordinate system M, Mr SM It represents the translation vector from the sensor coordinate system S to the map coordinate system M. To obtain the variance of the height measurement value We derive the Jacobian matrix J of the sensor measurement from the above formula S The Jacobian matrix J of the sensor coordinate system rotation Φ

[0066]

[0067]

[0068] Where C(Φ) represents the corresponding rotation matrix, and the parameters Denotes the position vector r from the sensor coordinate system S to the map frame M, and uses the cross product symbol × to indicate vector rotation. The error propagation is:

[0069]

[0070] where Σ S is the covariance matrix from the sensor, is the covariance matrix of the sensor rotation.

[0071] New altitude measurement The estimated value from the existing map is compared through a one-dimensional Kalman filter Fusion:

[0072]

[0073]

[0074] in, and are the height and variance before update, and are the updated height and variance. The initial value setting depends mainly on the specific scene and sensor resolution and accuracy. After converting the three-dimensional coordinates of the point cloud into the height information in the map, the RGB values ​​of the cell point cloud are counted, and the color that appears most frequently is recorded as the main color of the cell. The cell is mapped to a predefined semantic category according to different colors. For example, in an indoor environment, green can represent a passable road, blue can represent stairs, and red can represent obstacles and walls. For cells without valid point cloud data, or cells with a large number of mixed colors, that is, when no color accounts for more than 80%, the cell is marked as an obstacle category. Finally, a new "class" class is added to the elevation map to store the semantic category of each cell, so as to realize the construction of a semantic-elevation map. The comparison effect between the generated semantic elevation map and the ordinary elevation map is shown below. Figure 2 shown.

[0075] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0076] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A semantic-elevation map construction method for complex indoor scenes, wherein a depth camera collects surrounding environment information, generates a point cloud representing semantic information through color coding by fusing a two-dimensional image semantic segmentation mask with three-dimensional point cloud data, and constructs a semantic-elevation map based on the point cloud; the method is characterized in that: The following steps are involved: Step 1: Use a depth camera to collect image information of the surrounding environment and input it into the YOLACT semantic segmentation network to output the semantic segmentation result of the image; extract the required category labels and their corresponding pixel position information according to the navigation task, fill them with different grayscale values, and generate a semantic segmentation mask; Step 2: Using the internal and external parameters of the camera, align the depth camera point cloud data and the image data through coordinate transformation; fuse the aligned point cloud with the semantic segmentation mask generated in step 1, modify the point cloud RGB value to the corresponding category color, and obtain a point cloud that represents semantic information through color coding; Step 3: Calculate the height of each point in the semantic point cloud generated in step 2 relative to the camera plane based on its 3D coordinates and generate an elevation map. Assign specific semantic information to the elevation map based on the RGB value of the point cloud to construct a semantic-elevation map.

2. A semantic-elevation map construction method for complex indoor scenes according to claim 1, characterized in that: Generating the semantic segmentation mask in step 1 includes the following steps: After the RGB image captured by the depth camera is input into the YOLACT semantic segmentation network, the features are extracted by the ResNet backbone network to generate a global prototype mask, and a set of coefficient matrices are obtained by the coefficient matrix prediction module based on mask position regression; the coefficient matrix prediction module based on mask position regression first passes the input through two sets of 2D convolution layers with 3×3 convolution kernels in series, and then passes through three 2D convolution layers with 3×3 convolution kernels in parallel to generate a category confidence prediction matrix; let each pixel position of the semantic segmentation image be (u s ,v s ), for each pixel in the semantic image The confidence that the pixel belongs to each category k∈Class is obtained from the category confidence prediction matrix The SoftMax function is used to convert the confidence into a normalized label probability: Select the maximum label probability of all categories as the label of the pixel After determining the pixel label, the category label that needs to be retained is selected according to the navigation task, and different grayscale values ​​are set for the pixel position according to the label to generate semantic segmentation mask information.

3. The method for constructing a semantic-elevation map for complex indoor scenes according to claim 1, characterized in that: In step 2, point cloud alignment and fusion to generate semantic point cloud includes the following steps: To achieve point cloud alignment, first obtain the internal parameters of the depth camera, including the focal length (f x ,f y ) and the principal point (c x ,c y ), as well as the external parameter rotation matrix R and translation t; the internal parameter matrix is ​​usually expressed as: Among them, f x , f y is the focal length of the X and Y axes, c x , c y is the optical center of the image; After obtaining the camera intrinsic parameters, each pixel in the depth map needs to be converted into coordinates in three-dimensional space; let the position of a pixel in the depth map be (u d ,v d ), the depth value of the point is d; the coordinates (X, Y, Z) of the point in three-dimensional space are calculated by the following formula: Z=d The obtained 3D point cloud is then mapped to the RGB image; the 3D coordinates are converted back to pixel coordinates using the intrinsic parameters of the depth camera, and the extrinsic parameters are used to adjust the 3D coordinates; the formula is as follows: Among them, u′, v′ are the pixel coordinates after mapping; after this step, the aligned point cloud data is mapped to the semantic segmentation mask generated in step 1, and the RGB value of the point cloud is modified according to the mask category of the point cloud location to obtain a point cloud that represents semantic information in a color-coded manner.

4. The method for constructing a semantic-elevation map for complex indoor scenes according to claim 1, characterized in that: The steps of generating the semantic-elevation map in step 3 include: First, four coordinate systems are defined: the inertial coordinate system N, the user coordinate system U, the sensor coordinate system S, and the map coordinate system M. The inertial coordinate system N is fixed in the environment, and it is assumed that the actual terrain is stationary relative to this coordinate system. The three-dimensional position of the grid cell i in the elevation map is given by Given, where x i and i is defined by the position of grid cell i, is the estimated height of the terrain at grid cell i; In the map coordinate system M, the height measurements are approximated by a Gaussian probability distribution: The mean is p and the variance is The depth camera continuously collects point cloud data of the terrain. The new measurement data is processed into points in space and mapped into the elevation map. The measurement point in the sensor coordinate system S is represented as Sr SP ,These points are converted to corresponding height measurements p; Where P = [0,0,1] is a projection matrix that maps the three-dimensional measurement value to the height in the map coordinate system; Φ SM Represents the rotation matrix from the sensor coordinate system S to the map coordinate system M, Mr SM It represents the translation vector from the sensor coordinate system S to the map coordinate system M; to obtain the variance of the height measurement value The Jacobian matrix J of the sensor measurement is derived from the above formula: S The Jacobian matrix J of the sensor coordinate system rotation Φ Where C(Φ) represents the corresponding rotation matrix, and the parameters represents the position vector r from the sensor coordinate system S to the map frame M, and uses the cross product symbol × to represent vector rotation; variance The error propagation is: where Σ S is the covariance matrix from the sensor, is the covariance matrix of the sensor rotation; New altitude measurement The estimated value from the existing map is compared through a one-dimensional Kalman filter Fusion: in, and are the height and variance before update, and are the updated height and variance; for The initial value setting depends mainly on the specific scene and sensor resolution and accuracy. After converting the 3D coordinates of the point cloud into height information in the map, the RGB values ​​of the cell point cloud are counted, and the color with the most occurrences is recorded as the main color of the cell; the cell is mapped to a predefined semantic category according to different colors; for cells without valid point cloud data, or cells with a large number of mixed colors, that is, when no color accounts for more than 80%, the cell is marked as an obstacle category; finally, a new "class" class is added to the elevation map to store the semantic category of each cell, realizing the construction of a semantic-elevation map.

Images