A semantic map construction method based on deep learning

Abstract

The present invention discloses a semantic map construction method based on deep learning, which is specifically as follows: step 1: calibrate the depth camera installed above the robot, calculate the pose by RGBD image registration, and solve the pose by using ICP algorithm according to the input point cloud; step 2: after calculating the pose, the depth camera collects the image information of the surrounding environment in real time, and uses openGL to fuse the point cloud of the current frame with the reconstructed part to establish a three-dimensional map; step 3: match the current moment image acquired by the RGBD camera with the features of the previous key frame to determine whether the current moment image is a key frame; step 4: use the three-dimensional map established in step 2 as input, and send the two-dimensional image semantic segmentation result in step 3 to the Elastic fusion model to reconstruct the three-dimensional semantic segmentation map. This method solves the problem of large error in the SLAM mapping method in the prior art.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a semantic map construction method based on deep learning. Background Art

[0002] With the rapid development of technologies such as robots, computers, sensors, automated control, and artificial intelligence, the technology of autonomous mobile robots building semantic maps has become one of the most cutting-edge areas of scientific and technological development.

[0003] In traditional SLAM technology, the scene perception sensors commonly used by mobile robots are optical sensors and radar sensors. Radar sensors are large and expensive, and are easily affected by smoke and dust, which causes scattering, affecting recognition accuracy. Their scene recognition capabilities are limited to contour features and cannot fully utilize the rich semantic features of the scene. On the contrary, optical sensors are small in size, light in weight, and cost-effective. They can fully utilize the rich semantic features of the scene and are ideal recognition media.

[0004] However, as visual information becomes richer, the scenes to be identified become more and more complex, the global and local features of the images in the scene are very mixed, and the image processing needs to be completed simultaneously during the movement, and the real-time requirements are high, which brings challenges to the traditional indoor precision navigation and mapping tasks. The emergence of deep learning provides new ideas for image description, and the application of deep learning to improve the mapping and environment perception algorithms, and the use of semantic information to improve the accuracy and robustness of SLAM. At present, there is no mature technology in this field in China. Summary of the invention

[0005] The purpose of the present invention is to provide a semantic map construction method based on deep learning, which solves the problem of large errors in the SLAM mapping method in the prior art.

[0006] The technical solution adopted by the present invention is a method for constructing a semantic map based on deep learning, which specifically includes the following steps:

[0007] Step 1: Calibrate the depth camera installed on the robot, calculate the pose through RGBD image registration, and solve the pose using the ICP algorithm based on the input point cloud;

[0008] Step 2: After the pose is calculated in step 1, the depth camera collects the surrounding image information in real time, and uses OpenGL to fuse the point cloud of the current frame with the reconstructed part to build a three-dimensional map;

[0009] Step 3: Match the current image acquired by the RGBD camera with the features of the previous key frame to determine whether the current image is a key frame;

[0010] If it is not a key frame, find a new picture and determine a new key frame;

[0011] Assuming it is a key frame, the key frame is sent to the improved PSP-Net model to extract the two-dimensional semantic information, obtain the two-dimensional semantic information of the real-time image, and then perform closed-loop detection to obtain the closed-loop result;

[0012] Step 4: The 3D map created in step 2 is used as input, and the 2D image semantic segmentation result in step 3 is sent to the Elastic fusion model to reconstruct the 3D semantic segmentation map.

[0013] The present invention is also characterized in that

[0014] In step 1, the specific process of solving the pose using the ICP algorithm based on the input point cloud is as follows;

[0015] Step 1.1: Perform 3-layer sampling on the two depth images obtained by the RGBD depth camera, and then filter the two sampled depth images. The point cloud registration is iterated in a coarse-to-fine manner;

[0016] Step 1.2: Given the intrinsic parameters of the depth image, the 3D point cloud coordinates of the points are calculated through the original unsampled depth image for point cloud registration and fusion. The 3D point cloud coordinates of the two filtered depth images are also calculated for the calculation of the normal vector.

[0017] Step 1.3: According to the point cloud coordinates of the two depth images obtained in step 1.2, the projection pixel coordinates of one image in the other image are calculated by a projection algorithm;

[0018] Step 1.4: According to the matching points calculated in step 1.3, calculate the distance from the minimum point of the matching points to the plane to calculate the pose, so that the objective function error is less than the set minimum value, or the iteration is stopped when the set number of iterations is reached, otherwise go to step 1.3.

[0019] In step 1.3, the projection pixel coordinate function expression is as follows:

[0020] k(u d ,v d ,1)=KT ji p i (1)

[0021] Among them, k(u d ,v d ,1): Projection pixel coordinate function,

[0022] p i : The three-dimensional coordinates of the midpoint i in the depth image,

[0023] T ji : The pose from depth image i to depth image j,

[0024] K: camera intrinsic parameters,

[0025] (u d ,v d ): p i The projected pixel coordinates in depth image j.

[0026] In step 1.4, the algorithm process for calculating the pose by minimizing the distance from the point to the plane is as follows:

[0027] (1) Set the objective function as follows:

[0028]

[0029] In the above expression, the rotation matrix R and the translation matrix t are the poses to be solved. R and t can be expressed by the following formulas:

[0030]

[0031] t=[t x ,t y ,t z ] T (4)

[0032] Where α, β, and γ represent the rotation angles along the x, y, and z axes, respectively. In each iteration, the actual value is a small change in R ΔR and a small change in t Δt. i and q i are a point in the original point cloud and a point in the target point cloud, n i Refers to the normal vector of the target point;

[0033] (2) Linearize R. When the pose change between two adjacent frames is small, the following approximation is obtained:

[0034] sin(θ)=θ,cos(θ)=1 (5)

[0035] (3) The rotation angle of R in three directions is r = (α, β, γ) for

[0036] R i +t≈p i +r×p i +t, and r×p i ·n i = r·(p i ×n i ), based on the above approximation, the objective function is written as:

[0037]

[0038] (4) The above objective function is derived for the six-dimensional pose parameters and the derivative is set to 0, and Ax+b=0 is obtained, where the expression of A is:

[0039]

[0040] The expression of b is:

[0041]

[0042] The expression of the unknown parameter x is:

[0043] x=(α,β,γ,t x ,t y ,t z ) (9)

[0044] By calculating and solving Ax+b=0, R and t, that is, the calculated pose matrix, can be obtained.

[0045] In step 3, the process of sending the key frame into the improved PSP-Net model to extract the two-dimensional semantic information is as follows:

[0046] (1) By fusing and integrating the contextual information of different regions, a pyramid pooling module (PPM) is constructed to give the model the ability to analyze global prior knowledge, so that the model has the ability to understand global contextual information;

[0047] In order to make full use of the multi-scale features contained in the encoder part of PSP-Net, multiple bottom-up paths are added to PSP-Net. First, under the action of bottom-up paths and skip-layer connections, the features of each layer in the encoder are gradually integrated to obtain features with multi-scale semantic information. Then, the features are sent to the decoder for the next convolution operation.

[0048] (2) The encoding network of the improved PSP-Net network is expressed as follows:

[0049] f n =C n (f n-1 )=C n (C n-1 (...C1(I))) (10)

[0050] Where: I represents the input image; C n Indicates that the nth convolution module consists of a convolution layer and a pooling layer; f n Represents the feature map of the nth layer;

[0051] (3) The fusion process of substituting the deep feature map into the shallow feature map is described by the following mathematical expression:

[0052] f' n =f n ,f' n-1 =G n (f' n ,f n-1 ),...f' n-k =G n-k (f' n-k+1 ,f n-k ) (11)

[0053] where f n-k is the feature map of the nkth layer, 0<k<n. When k is closer to n, the feature map is closer to the shallow layer of the network, that is, the resolution of the feature map is getting higher and higher, and the detailed information contained is also increasing; f' n-k Represents the fused feature map; G n-k Indicates that the feature map f n-k And the feature map f' n-k+1 blend together;

[0054] (4) The process of fusing the obtained multi-level feature maps is described by the following mathematical expression:

[0055] D={P n-k (f' n-k ),...P n (f' n )} (12)

[0056] Among them, P is the refinement operation of the feature map; D represents the fusion operation of all feature maps to obtain the final two-dimensional image semantic information.

[0057] The beneficial effects of the present invention are as follows: a semantic map construction method based on deep learning of the present invention builds a mobile robot software and hardware platform according to the visual SLAM principle of the mobile robot, calculates the pose by RGBD image registration, solves the pose by using the ICP algorithm according to the input point cloud, uses openGL to fuse the point cloud of the current frame with the reconstructed part, constructs a three-dimensional map, extracts semantic features of the scene image by using the improved PSP-Net model by judging the key frame, and finally fuses the image semantic feature information with the three-dimensional reconstruction algorithm to obtain a global semantic segmentation map. The semantic map construction method based on deep learning of the present invention can use vision to quickly and accurately construct a map, making the autonomous navigation of the mobile robot more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1It is a flow chart of a semantic map construction method based on deep learning of the present invention;

[0059] Figure 2 It is a schematic diagram of the distance from a point to a plane in the method of the present invention;

[0060] Figure 3 Schematic diagram of an improved PSP-Net semantic segmentation model based on deep learning in the method of the present invention;

[0061] Figure 4 The images collected by the RGBD sensor in the method of the present invention include RGB two-dimensional images and depth images. DETAILED DESCRIPTION

[0062] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0063] The present invention provides a semantic map construction method based on deep learning, such as Figure 1 As shown, the specific steps include:

[0064] Step 1: Calibrate the depth camera installed on the robot, calculate the pose through RGBD image registration, and solve the pose using the ICP algorithm based on the input point cloud;

[0065] In step 1, the specific process of solving the pose using the ICP algorithm based on the input point cloud is as follows;

[0066] Step 1.1: Perform 3-layer sampling on the two depth images obtained by the RGBD depth camera, and then filter the two sampled depth images. The point cloud registration is iterated in a coarse-to-fine manner;

[0067] Step 1.2: Given the intrinsic parameters of the depth image, the 3D point cloud coordinates of the points are calculated through the original unsampled depth image for point cloud registration and fusion. The 3D point cloud coordinates of the two filtered depth images are also calculated for the calculation of the normal vector.

[0068] Step 1.3: According to the point cloud coordinates of the two depth images obtained in step 1.2, the projection pixel coordinates of one image in the other image are calculated by a projection algorithm;

[0069] like Figure 2 The schematic diagram of the point-to-plane distance shown in the figure, the projection method calculates the matching points. Since the pose transformation between the two frames is relatively small, the projection algorithm is used to calculate the matching points. The projection algorithm calculates the matching points faster than the kd-tree-based matching algorithm. For a depth image 3D point coordinate p i , the projected pixel coordinates in another image can be calculated. The projected pixel coordinate function expression is as follows:

[0070] k(u d ,v d ,1)=KT ji p i (1)

[0071] Among them, k(u d ,v d ,1): Projection pixel coordinate function,

[0072] p i : The three-dimensional coordinates of the midpoint i in the depth image,

[0073] T ji : The pose from depth image i to depth image j,

[0074] K: camera intrinsic parameters,

[0075] (u d ,v d ): p i The projected pixel coordinates in the depth image j;

[0076] Step 1.4: According to the matching points calculated in step 1.3, calculate the distance from the minimum point of the matching point to the plane to calculate the pose, so that the objective function error is less than the set minimum value, or the iteration number is reached, stop the iteration, otherwise go to step 1.3;

[0077] In step 1.4, the algorithm process for calculating the pose by minimizing the distance from the point to the plane is as follows:

[0078] (1) Set the objective function as follows:

[0079]

[0080] In the above expression, the rotation matrix R and the translation matrix t are the poses to be solved. R and t can be expressed by the following formulas:

[0081]

[0082] t=[t x ,t y ,t z ] T (4)

[0083] Where α, β, and γ represent the rotation angles along the x, y, and z axes, respectively. In each iteration, the actual value is a small change in R ΔR and a small change in t Δt. i and q i are a point in the original point cloud and a point in the target point cloud, n i Refers to the normal vector of the target point;

[0084] (2) Since the rotation matrix R is nonlinear, the objective function is also nonlinear. Here, R is linearized. When the pose change between two adjacent frames is small, the following approximation is obtained:

[0085] sin(θ)=θ,cos(θ)=1 (5)

[0086] (3) The rotation angle of R in three directions is r = (α, β, γ) for Rp i +t≈p i +r×p i +t, and r×p i ·n i = r·(p i ×n i ), based on the above approximation, the objective function is written as:

[0087]

[0088] (4) The above objective function is derived for the six-dimensional pose parameters and the derivative is set to 0, and Ax+b=0 is obtained, where the expression of A is:

[0089]

[0090] The expression of b is:

[0091]

[0092] The expression of the unknown parameter x is:

[0093] x=(α,β,γ,t x ,t y ,t z ) (9)

[0094] By calculating and solving Ax+b=0, R and t, that is, the calculated pose matrix, can be obtained.

[0095] Step 2: After calculating the R and t of the pose matrix in step 1, the depth camera collects the surrounding environment image information in real time, and uses OpenGL to fuse the point cloud of the current frame with the reconstructed part to build a three-dimensional map;

[0096] Step 3: Match the current image acquired by the RGBD camera with the features of the previous key frame to determine whether the current image is a key frame;

[0097] If it is not a key frame, find a new picture and determine a new key frame;

[0098] Assuming it is a key frame, the key frame is sent to the improved PSP-Net model to extract the two-dimensional semantic information, obtain the two-dimensional semantic information of the real-time image, and then perform closed-loop detection to obtain the closed-loop result;

[0099] In step 3, the process of sending the key frame into the improved PSP-Net model to extract the two-dimensional semantic information is as follows:

[0100] like Figure 3 As shown in the figure, the improved PSP-Net semantic segmentation model based on deep learning. The model is divided into two parts: encoding and decoding. The f1, f2, f3, f4 and f5 in the figure are feature maps of 1 / 2, 1 / 4, 1 / 8, 1 / 16 and 1 / 32 sizes of the input image in the encoding network. In the improved model, feature maps of 1 / 8, 1 / 16 and 1 / 32 sizes are used to fuse features with specific layers in the decoding network, and then pyramid pooling is performed to finally output the semantic segmentation results.

[0101] (1) By fusing and integrating the contextual information of different regions, a pyramid pooling module (PPM) is constructed to give the model the ability to analyze global prior knowledge, so that the model has the ability to understand global contextual information;

[0102] In order to make full use of the multi-scale features contained in the encoder part of PSP-Net, multiple bottom-up paths are added to PSP-Net. First, under the action of bottom-up paths and skip-layer connections, the features of each layer in the encoder are gradually integrated to obtain features with multi-scale semantic information. Then, the features are sent to the decoder for the next convolution operation.

[0103] (2) The encoding network of the improved PSP-Net network is expressed as follows:

[0104] f n =C n (f n-1 )=C n (C n-1 (...C1(I))) (10)

[0105] Where: I represents the input image; C n Indicates that the nth convolution module consists of a convolution layer and a pooling layer; f n Represents the feature map of the nth layer;

[0106] (3) The fusion process of substituting the deep feature map into the shallow feature map is described by the following mathematical expression:

[0107] f' n =f n,f' n-1 =G n (f' n ,f n-1 ),...f' n-k =G n-k (f' n-k+1 ,f n-k ) (11)

[0108] where f n-k is the feature map of the nkth layer, 0<k<n. When k is closer to n, the feature map is closer to the shallow layer of the network, that is, the resolution of the feature map is getting higher and higher, and the detailed information contained is also increasing; f' n-k Represents the fused feature map; G n-k Indicates that the feature map f n-k And the feature map f' n-k+1 From formula (11), we can see that the deep feature information is continuously integrated with the shallow feature information level by level, enriching the shallow feature information.

[0109] (4) The process of fusing the obtained multi-level feature maps is described by the following mathematical expression:

[0110] D={P n-k (f' n-k ),...P n (f' n )} (12)

[0111] Among them, P is the refinement operation of the feature map; D represents the fusion operation of all feature maps to obtain the final two-dimensional image semantic information.

[0112] Step 4: The 3D map created in step 2 is used as input, and the 2D image semantic segmentation result in step 3 is sent to the Elastic fusion model to reconstruct the 3D semantic segmentation map.

[0113] like Figure 4 As shown in the figure, the images collected by the RGBD sensor include RGB two-dimensional images and depth images. For RGB images, the improved PSP-Net model obtains accurate two-dimensional image segmentation maps. Then the two-dimensional segmentation image and the depth image are sent to the Elastic fusion model for three-dimensional semantic map reconstruction.

Claims

1. A semantic map construction method based on deep learning, characterized in that: The specific steps include: Step 1: Calibrate the depth camera installed on the robot, calculate the pose through RGBD image registration, and solve the pose using the ICP algorithm based on the input point cloud; In step 1, the specific process of solving the pose using the ICP algorithm based on the input point cloud is as follows; Step 1.1: Perform 3-layer sampling on the two depth images obtained by the RGBD depth camera, and then filter the two sampled depth images. The point cloud registration is iterated in a coarse-to-fine manner; Step 1.2: Given the intrinsic parameters of the depth image, the 3D point cloud coordinates of the points are calculated through the original unsampled depth image for point cloud registration and fusion. The 3D point cloud coordinates of the two filtered depth images are also calculated for the calculation of the normal vector. Step 1.3: According to the point cloud coordinates of the two depth images obtained in step 1.2, the projection pixel coordinates of one image in the other image are calculated by a projection algorithm; In step 1.3, the projection pixel coordinate function expression is as follows: k(u d ,v d ,1)=KT ji p i (1) Among them, k(u d ,v d ,1): Projection pixel coordinate function, p i : The three-dimensional coordinates of the midpoint in the depth image i, T ji : The pose from depth image i to depth image j, K: camera intrinsic parameters, (u d ,v d ): p i The projected pixel coordinates in the depth image j; Step 1.4: According to the matching points calculated in step 1.3, calculate the distance from the minimum point of the matching point to the plane to calculate the pose, so that the objective function error is less than the set minimum value, or the iteration number is reached, stop the iteration, otherwise go to step 1.3; In step 1.4, the algorithm process for calculating the pose by minimizing the distance from the point to the plane is as follows: (1) Set the objective function as follows: In the above expression, the rotation matrix R and the translation matrix t are the poses to be solved. R and t can be expressed by the following formulas: t=[t x ,t y ,t z ] T (4) Where α, β, and γ represent the rotation angles along the x, y, and z axes, respectively. In each iteration, the actual value is a small change in R ΔR and a small change in t Δt. i and q i are a point in the original point cloud and a point in the target point cloud, n i Refers to the normal vector of the target point; (2) Linearize R. When the pose change between two adjacent frames is small, the following approximation is obtained: sin(θ)=θ, cos(θ)=1 (5) (3) The rotation angle of R in three directions is r = (α, β, γ) for Rp i +t≈p i +r×p i +t, and r×p i ·n i = r·(p i ×n i ), based on the above approximation, the objective function is written as: (4) The above objective function is derived for the six-dimensional pose parameters and the derivative is set to 0, and Ax+b=0 is obtained, where the expression of A is: The expression of b is: The expression of the unknown parameter x is: x=(α,β,γ,t x ,t y ,t z ) (9) By calculating and solving Ax+b=0, R and t can be obtained, that is, the calculated pose matrix; Step 2: After the pose is calculated in step 1, the depth camera collects the surrounding image information in real time, and uses OpenGL to fuse the point cloud of the current frame with the reconstructed part to build a three-dimensional map; Step 3: Match the current image acquired by the RGBD camera with the features of the previous key frame to determine whether the current image is a key frame; If it is not a key frame, find a new picture and determine a new key frame; Assuming it is a key frame, the key frame is sent to the improved PSP-Net model to extract the two-dimensional semantic information, obtain the two-dimensional semantic information of the real-time image, and then perform closed-loop detection to obtain the closed-loop result; In step 3, the process of sending the key frame into the improved PSP-Net model to extract the two-dimensional semantic information is as follows: (1) By fusing and integrating the contextual information of different regions, a pyramid aggregation module is constructed to give the model the ability to analyze global prior knowledge, so that the model has the ability to understand global contextual information; In order to make full use of the multi-scale features contained in the encoder part of PSP-Net, multiple bottom-up paths are added to PSP-Net. First, under the action of bottom-up paths and skip-layer connections, the features of each layer in the encoder are gradually integrated to obtain features with multi-scale semantic information. Then, the features are sent to the decoder for the next convolution operation. (2) The encoding network of the improved PSP-Net network is expressed as follows: f n =C n (f n-1 )=C n (C n-1 (...C1(I))) (10) Where: I represents the input image; C n Indicates that the nth convolution module consists of a convolution layer and a pooling layer; f n Represents the feature map of the nth layer; (3) The fusion process of substituting the deep feature map into the shallow feature map is described by the following mathematical expression: f n '=f n ,f n ' -1 =G n (f n ',f n-1 ),...f n ' -k =G n-k (f n ' -k+1 ,f n-k ) (11) where f n-k is the feature map of the nkth layer, 0<k<n. When k is closer to n, the feature map is closer to the shallow layer of the network, that is, the resolution of the feature map is getting higher and higher, and the detailed information contained is also increasing. n ' -k Represents the fused feature map; G n-k Indicates that the feature map f n-k And the feature map f n ' -k+1 blend together; (4) The process of fusing the obtained multi-level feature maps is described by the following mathematical expression: D={P n-k (f n ' -k ),...P n (f n ')} (12) Where P is the refinement operation of the feature map; D represents the fusion operation of all feature maps to obtain the final two-dimensional image semantic information; Step 4: The 3D map created in step 2 is used as input, and the 2D image semantic segmentation result in step 3 is sent to the Elastic fusion model to reconstruct the 3D semantic segmentation map.

Images