An object-level semantic SLAM method for autonomous positioning in underground environments

Abstract

The present invention discloses an object-level semantic SLAM method for autonomous positioning in underground garage environments. The present invention integrates object-level scene semantics obtained based on deep learning into the SLAM system. In view of the difficulties brought about by weak texture and poor illumination in underground garage scenes for conventional feature point extraction, it is proposed to use a monocular 3D target detection network to obtain the position and posture information of static objects such as pillars in the underground garage, establish it as a semantic map, and realize real-time positioning in the semantic map. Compared with the monocular SLAM method based on feature points, the present invention effectively improves the robustness of the SLAM system. The established map has clear semantics and good interpretability, can be reused over a long span, has low computational cost, and has strong real-time performance.

Description

Technical Field

[0001] The present invention relates to a mapping and positioning method for a mobile robot or intelligent vehicle, and in particular to a mapping and positioning method for a mobile robot or intelligent vehicle based on semantic information in a specific scenario of an underground garage. Background Art

[0002] With the development of autonomous driving technology, demand for autonomous parking in underground garages is growing. Underground garages lack GPS, relying solely on maps for positioning. A common approach is to create a high-precision 3D map of the underground garage and use sensors like LiDAR for positioning. However, LiDAR is relatively expensive. Using images captured by cameras, purely visual SLAM can be used to achieve navigation and positioning in the garage environment, reducing costs.

[0003] SLAM, short for Simultaneous Localization And Mapping, is a technology that starts from an unknown location in an unknown environment, locates its position and posture through repeated observations of map features during movement, and then incrementally constructs a map based on its posture, achieving both simultaneous positioning and mapping. Monocular vision SLAM algorithms have a long history of research and are widely used due to their low sensor requirements. For pose estimation, the front-end primarily uses image frames captured by a moving camera at different times to determine the camera pose transformation between adjacent frames through feature matching. The back-end primarily optimizes and fine-tunes the inaccurate camera pose and reconstructed map obtained by the vision front-end.

[0004] Based on the density of the constructed map, traditional visual SLAM frameworks mainly include sparse methods, direct methods, and semi-direct methods. Sparse methods, such as the ORB-SLAM series, use image feature points as features. They offer high stability and mapping accuracy and are currently the mainstream visual SLAM frameworks. Direct methods have certain advantages in environments with relatively weak textures, but they are significantly affected by lighting variations, resulting in low positioning accuracy and map reusability. Semi-direct methods fall somewhere in between these two approaches, but are computationally more complex. Although the SLAM feature point method based on feature points has achieved great success, its principle is to extract feature points based on the grayscale changes of the image and use them to build a map. In the basement environment, this type of method has several disadvantages: (1) The extraction of feature points is sensitive to the grayscale changes of the image. If the scene has a significant change in illumination, the extraction of feature points will produce large differences, affecting the performance of SLAM; (2) There are a large number of weak texture scenes in the basement, which makes it difficult to extract effective features, affecting the reliability of positioning; (3) Feature point extraction does not take semantic information into account, which makes the extracted feature points distributed on dynamic objects such as pedestrians or moving vehicles. This in turn affects the camera positioning accuracy. With the development of deep learning, the acquisition of high-level object-level semantic information of the scene, such as monocular 3D target detection technology, has made significant progress. It has made it possible to directly obtain the position and posture information of fixed 3D objects such as common pillars from the image and build a SLAM system based on this information. The use of this semantic information also helps to improve the robustness of SLAM. Summary of the Invention

[0005] In order to solve the problems existing in the background technology, the purpose of the present invention is to provide an object-level semantic SLAM method for autonomous positioning in underground garage environments, which is suitable for visual mapping and positioning problems in underground garage environments.

[0006] The present invention firstly obtains the category, 3D position and 3D posture results of objects in the scene in real time through an efficient monocular 3D target detection network, and designs a corresponding loss function for underground garage scenes for supervised network training, thereby greatly improving the detection effect in scenes such as underground garages. Furthermore, based on the traditional 3D point cloud SLAM framework, only 3D objects are used for mapping and positioning calculations, completing a SLAM solution based on high-level semantic information, and improving the robustness of SLAM in underground garage scenes.

[0007] This invention improves upon the traditional SLAM framework, no longer using feature points and their feature descriptions for map construction and self-positioning. Instead, it uses a monocular 3D target detection network to obtain object categories, three-dimensional positions, and three-dimensional pose information. This information is then used to construct an object-based semantic map, while simultaneously positioning the camera's own pose in real time. During optimization, object information is used for current frame pose adjustment, loop detection, relative pose calculation between the current frame and the looped frame, optimization of local maps, and optimization of global maps, resulting in a semantic map that consumes fewer resources and improves positioning robustness.

[0008] The steps of the technical solution adopted by the present invention are as follows:

[0009] 1) The mobile camera captures and obtains multiple underground garage scene images, which are recorded as frame images. Each frame image is input into the monocular 3D object detection network for 3D object detection, and the 3D detection boxes of all static objects in each frame image are obtained.

[0010] 2) The first frame of the scene graph captured by the mobile camera is taken as the initial frame and recorded as the initial keyframe. All 3D detection boxes in the initial keyframe are then added to the initial object-level map. The next frame of the scene graph after the initial frame is recorded as the current frame.

[0011] 3) Based on the current object-level local map and the 3D detection boxes of all static objects in the current frame, perform inter-frame pose estimation on the current frame to obtain the pose of the current frame in the world coordinate system, and then obtain the pose of all static objects in the world coordinate system. The pose of the current frame in the world coordinate system and the pose of all static objects in the world coordinate system are recorded as the total pose of the current frame;

[0012] 4) Determine whether the current frame is a key frame based on the motion distance and angle change. If it is a key frame, update the current object-level map according to the current frame, then perform local detection and loop detection on the current key frame. If the conditions corresponding to the local detection and / or loop detection are met, perform local optimization and / or loop optimization to obtain the optimized total pose of the current continuous key frame and update it, and then optimize the current object-level map and update it, and update the next frame scene graph of the current frame to the current frame. Otherwise, update the next frame scene graph of the current frame to the current frame;

[0013] 5) Repeat 3)-4) to obtain the overall pose and object-level map of each frame in real time.

[0014] The monocular 3D object detection network adds a homography loss function, a rotation consistency loss function, and an alignment loss function. After weighted summation of the homography loss function, the rotation consistency loss function, the alignment loss function, and the original loss function, a total loss function is obtained.

[0015] The supervision target of the homography loss function is that the coordinates of the corner points and the center point of the bottom surface of the 3D detection frame of the object after transformation on the ground plane are equal to the real coordinates of the corner points and the center point of the bottom surface of the 3D detection frame of the object after transformation on the ground plane. The coordinates of the corner points and the center point of the bottom surface of the 3D detection frame of the object after transformation on the ground plane are obtained by calculating the homography matrix between the coordinates of the corner points and the center point of the bottom surface of the 3D detection frame of the static object corresponding to the detection on the ground plane and the real coordinates of the 2D projection of the corner points and the center point of the bottom surface of the 3D detection frame of the static object corresponding to the detection, and then the real coordinates of the corner points and the center point of the bottom surface of the 3D detection frame of the static object corresponding to the detection are transformed by the homography matrix.

[0016] The supervision target of the rotation consistency loss function is that the orientation angle of the 3D detection frame corresponding to each static object in the scene graph of each frame is equal to the average orientation angle of the 3D detection frame corresponding to all static objects in the scene graph of the current frame.

[0017] The supervision target of the arrangement neatness loss function is the arrangement direction angle and orientation angle between all static objects in each frame scene graph. If the preset conditions are met, the arrangement direction between the two static objects is parallel or perpendicular to their own orientation.

[0018] In the above 4), if the current continuous key frames meet the sliding window condition, local optimization is performed.

[0019] In the above 4), if a loop is detected, loop optimization is performed to obtain the optimized total pose of all key frames in the loop interval and update it, thereby optimizing the object-level semantic map in the current loop interval.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] (1) The present invention can achieve good SLAM positioning accuracy by relying on a cheap monocular camera.

[0022] (2) The present invention can reduce the impact of objects with changing ambient lighting on SLAM estimation and improve the robustness of the SLAM system in environments with significant lighting changes.

[0023] (3) The present invention can reduce the impact of dynamic objects on SLAM estimation and improve the robustness of the SLAM system in an environment containing many dynamic objects.

[0024] (4) The present invention adopts a more efficient monocular 3D target detection network, which can achieve real-time system operation and is more suitable for use in SLAM scenarios compared to the more complex 3D target detection network based on radar point cloud.

[0025] (5) The present invention makes targeted improvements to the design and training of monocular 3D object detection networks, enabling them to achieve higher detection accuracy in complex scenarios such as underground garages.

[0026] (6) The semantic map constructed by the present invention occupies less computing resources and has higher robustness to environmental changes, and can achieve map reuse over a longer time span.

[0027] In summary, the present invention effectively improves the detection accuracy of the monocular 3D target detection network in the underground garage scenario, and uses its detection results to design a visual semantic SLAM solution, which makes it more robust to changes in ambient lighting and situations involving dynamic objects. The constructed semantic map has the characteristics of occupying less computing resources. At the same time, the semantic map can realize map reuse over a longer time span and has better real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is the basic framework of the semantic SLAM method based on 3D object information.

[0029] Figure 2 Schematic diagram of the structure of the monocular 3D target detection network used.

[0030] Figure 3 This is a schematic diagram of the operation of the present invention based on a sequence collected and annotated in the underground garage of a shopping mall.

[0031] Figure 4 The semantic map established by the present invention is based on the sequences collected and labeled in the underground garage of a shopping mall. DETAILED DESCRIPTION

[0032] The embodiments of the complete method according to the present invention are as follows:

[0033] like Figure 1 As shown, the present invention includes the following steps:

[0034] 1) The mobile camera captures and obtains multiple underground garage scene images (RGB images), recorded as each frame scene image. Each frame scene image is input into the monocular 3D object detection network for 3D object detection, and the 3D detection frame of all static objects (such as pillars) in each frame scene image is obtained. The 3D detection frame includes the category, position, size and posture of each static object. The categories include square pillars and round pillars. The position is the coordinates of the center point of the 3D detection frame relative to the camera in the x, y, and z directions. The size is the length, width, and height of the 3D detection frame. Since the objects are placed on the ground, only the rotation angle around the axis perpendicular to the bottom surface, that is, the orientation angle, is estimated. The attributes of the 3D static objects form a set Di, which is recorded.

[0035] During the training of the monocular 3D target detection network, the homography loss function, the rotation consistency loss function and the alignment loss function are added to the loss function. The homography loss function, the rotation consistency loss function and the alignment loss function are weightedly summed with the original loss function to obtain the total loss function. In the specific implementation, the monocular 3D target detection network uses the FCOS3D network as the baseline, and based on it, a specially designed loss function is added to improve the accuracy of monocular 3D target detection based on the characteristics of the basement scene. The FCOS3D network consists of three parts: backbone, neck and head. Figure 2 shown.

[0036] The backbone part is used to extract features that can be used by the network from the image. It adopts the ResNet101 structure, which is composed of: first, it passes through a 3*3 convolutional layer with a stride of 2 and a maximum pooling layer (MaxPooling) with a stride of 2, and the feature resolution is reduced to 1 / 4; then it passes through 4 ResBlocks, and the resolution of the feature is reduced to 1 / 2 of the original after each ResBlock; each ResBlock is composed of several BottleNeck modules, and each BottleNeck module is composed of 3 convolutional layers, namely 1*1 convolution, 3*3 convolution and 1*1 convolution. The first two convolutional layers are each followed by a batch normalization layer (Batch Normalization) and a ReLU activation function, and the last convolutional layer is followed by only one batch normalization layer. There is a skip connection (SkipConnection) at the beginning and end of each BottleNeck module so that its input is added to the output; the last average pooling layer and fully connected layer are removed, so that its output is the features of each level.

[0037] The role of the Neck part is to further fuse features at different levels to obtain more complete features. It adopts the Feature Pyramid Network (FPN) structure, and its input is the three features of different resolutions output by the backbone part. It uses 2x upsampling to convert the low-resolution features into higher-resolution features and fuses them with the backbone features of the same resolution to obtain fused features that contain both high-level semantic information and low-level texture features. At the same time, two convolution modules are applied to the features with the lowest original resolution to obtain two features with resolutions reduced to 1 / 2 and 1 / 4, resulting in a total of five features of different scales.

[0038] The head component uses features to regress the required object attributes. Objects are divided into different scales based on their projected size onto the image: larger objects are assigned to low-resolution features, while smaller objects are assigned to high-resolution features. A set of heads is designed for regressing object attributes based on the features at each scale, and the parameters of the five head groups are shared. Each head group consists of the following: a classification head, which predicts the object's category; a centerness head, which predicts the proximity of each pixel to the center of an object; a center offset head, which predicts the offset from the 2D projection of the object's 3D center to the pixel; a depth head, which predicts the distance from the camera to the 3D center of the object at that pixel; a size head, which predicts the length, width, and height of the object's 3D bounding box; and a rotation head, which predicts the object's rotation angle around an axis perpendicular to the ground. After obtaining the 2D projection position and depth value of the object's 3D center, combined with the camera's intrinsic parameters, the 3D position of the object's center is obtained, and thus the object's category, position, and pose information are obtained.

[0039] When training the network, the prediction value of each head is supervised by a corresponding loss function. The supervision value of the classification head is the category of the object where the pixel is located. If it is not on the object, it is classified as the background class. The loss function uses Focal Loss, that is,

[0040]

[0041] in is the predicted value of the probability of the cth category classification at the i-position pixel, The supervision value of the center head is the distance from the pixel to the center of the object. It is generated in advance based on the 2D projection position of the center of the object. The center is 1.0, and the surrounding area decreases according to the Gaussian distribution. The part outside the 2D box is 0. The loss function is the cross entropy loss, that is,

[0042]

[0043] where p i is the predicted value of the center at pixel i, y i The supervision value of the center point offset head is the position offset of the 2D projection point of the center point of the object where the pixel is located to the pixel; the supervision value of the depth head is the depth value of the 3D center point of the object where the pixel is located; the supervision value of the size head is the length, width, and height of the 3D detection box of the object where the pixel is located; the loss function of the above three heads is Smooth L1 Loss, that is

[0044]

[0045] where p i is the predicted value at pixel i, y i The supervision value of the head rotation is the actual rotation angle of the 3D box under the camera's perspective. The loss function is as follows:

[0046]

[0047] in is the angle prediction value at pixel i, is the corresponding true value.

[0048] In addition, the present invention designs three loss functions for detecting 3D objects such as pillars in underground garage scenes to improve detection performance. They are: (a) homography loss function, (b) angle consistency loss function, and (c) arrangement loss function, which can be selected according to the actual situation of the garage.

[0049] The supervision target of the homography loss function is that the coordinates of the corner points and the center point of the bottom surface of the 3D detection frame of the transformed object on the ground plane are equal to the real coordinates of the corner points and the center point of the bottom surface of the 3D detection frame of the object on the ground plane. The coordinates of the corner points and the center point of the bottom surface of the 3D detection frame of the transformed object on the ground plane are based on the prior information that all objects are placed above the ground plane and the bottom surface is closely fitted with the ground plane. The coordinates of the corner points and the center point of the bottom surface of the 3D detection frame of the static object obtained by calculation are obtained by calculating the homography matrix between the coordinates of the corner points and the center point of the bottom surface of the 3D detection frame of the static object on the ground plane and the real coordinates of the 2D projection of the corner points and the center point of the bottom surface of the 3D detection frame of the static object, and then the real coordinates of the corner points and the center point of the bottom surface of the 3D detection frame of the static object on the ground plane are transformed by the homography matrix.

[0050] Specifically: The starting point of the design of the homography loss function is to take into account the flat ground of the underground garage scene. During the entire process of the vehicle or robot driving, the ground can be considered as a plane, and the bottom surfaces of the column objects to be detected coincide with the ground. Therefore, all points on the bottom surfaces of all columns in the same image are distributed on the same plane, and the four corner points and one center point of the bottom surface of each column are taken as representatives; at the same time, the 2D projections of these points on the image are also distributed on the same plane, that is, the image plane; in this way, if the height coordinates of the bottom 3D points are ignored and regarded as 2D points, the transformation relationship between these points and their 2D projection points can be described by a homography matrix. After obtaining the 3D detection frame of the object, the 3D coordinates of its bottom corner points and center points can be calculated. Knowing the image coordinates of these points, the homography matrix H is estimated according to the following formula:

[0051] P pred =Hp gt

[0052] Among them, P pred The predicted values ​​of the 3D coordinates of these points, the z-axis coordinate is set to 1, and p gt is the corresponding 2D point pixel coordinate value, and the z axis is set to 1. After obtaining the homography matrix H, it is applied to the loss function:

[0053] L homo =SmoothL1(P gt -Hp gt )

[0054] Among them, P gt is the true value of the 3D coordinate, p gt is the true value of the pixel coordinate. The meaning of this loss function is that the coordinate value obtained according to the homography matrix transformation should be consistent with the actual coordinate value. SmoothL1() represents the SmoothL1 loss function.

[0055] The supervision goal of the rotation consistency loss function is to ensure that the pose of the 3D detection box corresponding to each static object in the scene graph of each frame is equal to the average pose of the 3D detection box corresponding to all static objects in the scene graph of the current frame, based on the prior information that the objects of all pillar categories in the same frame image are oriented in the same direction.

[0056] Specifically, the starting point of the angle consistency loss function is to consider that the rotation angles of the pillars observed in the same frame of the underground garage scene should be consistent, so the angles of all objects detected in the same frame should be the same as their average angles; for pillar objects, they are 90° rotationally symmetric, so the angle should be normalized to the interval (-45°, 45°) and its value should be the same as the normalized average, that is,

[0057]

[0058] Among them, L rot_consistency represents the angle consistency loss function, θ p is the normalized angle value, is its average value, || means taking the absolute value

[0059] The supervision target of the arrangement neatness loss function is to use the prior information that all objects of the column category in the same frame image are neatly arranged and their arrangement direction is parallel or perpendicular to their own direction, and calculate the arrangement direction angle and orientation angle between all columns. The arrangement direction angle is the angle between the straight line connecting the center points of the two columns in the bird's-eye view and the straight line where the camera optical axis is located. If the arrangement direction angle and orientation angle between all static objects in each frame scene image meet the preset conditions, the arrangement direction between the two static objects is parallel or perpendicular to their own direction. In a specific implementation, if the difference between the arrangement direction angle and the orientation angle between the two columns satisfies [-n°, n°] or [k*90°-m°, k*90°+m°], it means that the two columns are on the same actual inherent straight line, where n, m<90, and k are positive integers. The n and m here are both about 10 in actual experiments, which is consistent with the following Score pattern It is related to the value of the threshold.

[0060] Specifically, the starting point for the design of the alignment loss function is that the columns in underground garages are usually arranged in rows rather than randomly. The rotation angle of each column in each row should be the same as the angle of the line. Considering the 90° rotational symmetry of the columns and the relative positions of the columns in two rows, the rotation angle of each column in each row should be parallel or perpendicular to the angle of the line. The specific implementation method is to determine whether all detected columns are arranged in the same row:

[0061]

[0062] Among them, Score pattern Indicates the probability score of whether two pillars are on the same preset straight line. The preset straight line is the straight line when two pillars are arranged in a row. θ p Represents the predicted value of the column's own orientation angle, θ line Indicates the arrangement direction angle between the two pillars, Score pattern That is, it reflects whether the connecting angle is parallel or perpendicular to the angle itself. The higher the value, the higher the possibility of parallelism or perpendicularity. If the value exceeds a certain threshold, it is considered that the two columns are arranged on the same preset straight line, making them more neatly arranged, even if the Score pattern Higher values, i.e.

[0063] L pattern =-∑[Score pattern >thresh]*Score pattern

[0064] Among them, L pattern Represents the alignment loss function, thresh represents the Score patternThe threshold is 0.9 in the experiment, which is related to the values ​​of n and m above.

[0065] Finally, the overall loss function of the network is the weighted sum of all the above loss functions. According to the above method, the network model is designed and trained, and finally a network is obtained that can quickly and accurately detect objects in the underground garage scene.

[0066] 2) The first frame of the scene graph captured by the mobile camera is taken as the initial frame and recorded as the initial keyframe. All 3D detection boxes in the initial keyframe are then added to the initial object-level map. The next frame of the scene graph after the initial frame is recorded as the current frame.

[0067] 3) Based on the current object-level local map and the 3D detection frames of all static objects in the current frame, the inter-frame pose estimation is performed on the current frame. In the specific implementation, the inter-frame pose estimation is to perform data association and calculate the inter-frame pose based on the relative pose of the associated object pairs. Data association is specifically achieved by projecting the previous and next frames into the world coordinate system and using the nearest neighbor method to perform 3D detection frame matching, that is, searching for the object of the same category with the closest distance to its center point in the detection result of the previous frame. If the difference in the center point distance and the rotation angle orientation angle does not exceed the respective thresholds, the two are considered to be matched successfully, and then the pose of the current frame in the world coordinate system is obtained, and then the pose of all static objects in the world coordinate system is obtained. The pose of the current frame in the world coordinate system and the pose of all static objects in the world coordinate system are recorded as the total pose of the current frame;

[0068] 4) Determine whether the current frame is a keyframe based on the movement distance and angle change. If the distance from the previous keyframe exceeds 1m or the posture change exceeds 5°, the current frame is judged to be a keyframe. If it is a keyframe, update the current object-level map based on the current frame, then perform local detection and loop detection on the current keyframe. If the conditions corresponding to local detection and / or loop detection are met, perform local optimization and / or loop optimization to obtain the optimized total pose of the current continuous keyframe and update it. Then optimize the current object-level map and update it, and update the next frame scene graph of the current frame to the current frame. Otherwise, update the next frame scene graph of the current frame to the current frame.

[0069] In step 4), if the current consecutive keyframes meet the sliding window conditions, specifically, if the current keyframe is the first sliding window optimization, the sum of the current keyframe and the previous keyframes is equal to the sliding window length, which is 20 frames. If this is the second or subsequent sliding window optimization, the overlapping frames between the two sliding window optimizations are 5 frames, and local optimization is performed. That is, the current consecutive keyframes are optimized every 15 frames. The map composed of the 3D detection boxes observed in the frames within the sliding window is called the local object-level semantic map.

[0070] In step 4), if a loop is detected, loop optimization is performed to obtain the optimized total pose of all key frames in the loop interval and update it, thereby optimizing the object-level semantic map in the current loop interval.

[0071] Loop closure detection is to match the 3D detection frames of all static objects in the current frame with the 3D detection frames in the current object-level map. If at least one pair of 3D detection frames is successfully matched and the interval between the two frames of the matching pair exceeds a certain threshold, loop closure optimization is performed. For the successfully matched 3D detection frame, its pose is updated to the pose of the 3D detection frame with the highest confidence; for the unmatched 3D detection frame, if its confidence exceeds a certain threshold, it is added to the object-level map to obtain the optimized total pose of all key frames in the entire loop closure interval, and then the object-level map is updated.

[0072] 5) Repeat 3)-4) to obtain the overall pose and object-level map of each frame in real time. The final map is called the global object-level semantic map.

[0073] The loop optimization and sliding window optimization methods for the pose error optimization of the successfully matched 3D detection frame in the present invention are the same, specifically: using a graph optimization method, in which the pose of the current frame and the 3D detection frame are set as nodes in the graph, the constructed optimization term (i.e., the objective function, as shown in the following formula) is used as the edge in the graph, the error of the 3D detection frame is used as the information matrix, the confidence of the detection result is set as a weight, and the least squares method is used to solve the problem to obtain the optimized pose;

[0074]

[0075] Among them, T c * The SE(3) of the optimized camera pose is expressed as T c is the SE(3) representation of the original camera pose, w i is the weight of the i-th object, which is determined by the confidence of the target detection result; E se3，i is the pose error of the i-th object under the current camera pose, T i，o is the pose of the map object matched to the i-th object in the world coordinate system, T i，oc is the pose of the i-th object in the current frame camera coordinate system obtained by detection, and is expressed in the form of SE(3). The pose error of the i-th object in the current camera pose is the Lie algebra of the multiplication of the pose of the map object in the current frame camera coordinate system calculated based on the current frame camera pose and the world coordinate system pose of the matching object in the map and the pose inverse matrix of the i-th object in the current frame camera coordinate system obtained based on the target detection result, which is a vector of length 6; E shape，i is the 3D detection box shape error of the i-th object, d i，ocis the length, width and height of the 3D box detected by the i-th object, d i，w is the length, width and height of the 3D box of the map object it matches. The error is a vector of length 3. Connecting the two results in a vector of length 9, which is the error of the i-th object. N represents the total number of objects. Represents the mapping operation from Lie groups to Lie algebras.

[0076] The monocular 3D object detection network is trained using a dataset collected in an underground garage scene, and supervised using the loss function provided by the present invention to obtain a trained monocular 3D object detection network. The evaluation index of monocular 3D detection is the mean average precision (mAP) under 40 uniformly distributed recall rates with IoU exceeding 0.5 and 0.7 as the standard, calculated in 3D space and bird's eye view (BEV) perspectives respectively. The experimental results are shown in Tables 1 and 2. The visualization of 3D object detection results is shown in Table 1. Figure 3 As shown in the right half, you can see that the predicted 3D detection box tightly surrounds the pillar in the scene.

[0077] According to the requirements in the steps, add a trained monocular 3D object detection thread to the front-end feature extraction module of the traditional LiDAR SLAM framework. When the camera receives an image, call the monocular 3D object detection network to obtain 3D object information in the scene. Modify the visual odometry component and use this information for map construction and localization. Modify the back-end optimization part of the SLAM framework to use 3D object information for optimization.

[0078] This part of the experiment is based on a self-collected dataset in an underground garage scene, with a total of 13,272 images. The network takes an average of about 25ms per frame on the RTX2080Ti platform, which meets the conditions for real-time operation. The runtime visualization is as follows Figure 3 As shown in the figure, the right half shows the current frame and the 3D detection box obtained by detection, and the left half shows the object-level semantic map established at the current moment.

[0079] Get the experimental trajectory, and for each point in the sequence, calculate its error with the true value to get the APE (absolute pose error) of each point, then do statistics, calculate the RMSE (root mean square error), and get the experimental results. Repeat the experiment multiple times on the same sequence to get the average RMSE for comparison. The comparison method is ORB-SLAM, which is also based on monocular vision information but does not use object-level semantic information, and performs scale alignment. The experimental results are shown in Table 3. The experimental distance length is about 470 meters. The final global object-level semantic map is shown in Figure 4 shown.

[0080] Table 1 Experimental comparison of the monocular 3D detection baseline model and the proposed method on the self-collected dataset (3D average accuracy mAP)

[0081] IoU Baseline The present invention Relative improvement 0.5 57.69％ 62.42％ 8.2％ 0.7 16.1％ 23.39％ 45.3％

[0082] Table 2 Experimental comparison of the monocular 3D detection baseline model and the proposed method on the self-collected dataset (BEV average accuracy mAP)

[0083] IoU Baseline The present invention Relative improvement 0.5 61.56％ 68.68％ 11.6％ 0.7 23.84％ 32.49％ 36.3％

[0084] Table 3 Experimental comparison between the original SLAM and the present invention on self-collected sequences (average APE RMSE, unit: meter)

[0085] ORB-SLAM The present invention Relative improvement 1.007 0.910 9.63％

[0086] It can be seen that the present invention can effectively improve the detection accuracy of the monocular 3D detection model in the underground garage scenario, and then apply it to the semantic SLAM system. Compared with the traditional SLAM, the positioning accuracy is effectively improved, especially when the lighting conditions change significantly, there are fewer textures or more dynamic objects. The present invention can significantly improve the robustness of the system while having a lower computational cost.

Claims

1. An object-level semantic SLAM method for autonomous positioning in underground environments, characterized by: The following steps are involved: 1) The mobile camera captures and obtains multiple underground garage scene images, which are recorded as frame images. Each frame image is input into the monocular 3D object detection network for 3D object detection, and the 3D detection boxes of all static objects in each frame image are obtained. 2) The first frame of the scene graph captured by the mobile camera is taken as the initial frame and recorded as the initial keyframe. All 3D detection boxes in the initial keyframe are then added to the initial object-level map. The next frame of the scene graph after the initial frame is recorded as the current frame. 3) Based on the current object-level local map and the 3D detection boxes of all static objects in the current frame, perform inter-frame pose estimation on the current frame to obtain the pose of the current frame in the world coordinate system, and then obtain the pose of all static objects in the world coordinate system. The pose of the current frame in the world coordinate system and the pose of all static objects in the world coordinate system are recorded as the total pose of the current frame; 4) Determine whether the current frame is a key frame based on the motion distance and angle change. If it is a key frame, update the current object-level map according to the current frame, then perform local detection and loop detection on the current key frame. If the conditions corresponding to the local detection and / or loop detection are met, perform local optimization and / or loop optimization to obtain the optimized total pose of the current continuous key frame and update it, and then optimize the current object-level map and update it, and update the next frame scene graph of the current frame to the current frame. Otherwise, update the next frame scene graph of the current frame to the current frame; 5) Repeat 3)-4) to obtain the overall pose and object-level map of each frame in real time.

2. The object-level semantic SLAM method for autonomous positioning in a basement environment according to claim 1, characterized in that: The monocular 3D object detection network adds a homography loss function, a rotation consistency loss function, and an alignment loss function. After weighted summation of the homography loss function, the rotation consistency loss function, the alignment loss function, and the original loss function, a total loss function is obtained.

3. The object-level semantic SLAM method for autonomous positioning in a basement environment according to claim 2, characterized in that: The supervision target of the homography loss function is that the coordinates of the corner points and the center point of the bottom surface of the 3D detection frame of the object after transformation on the ground plane are equal to the real coordinates of the corner points and the center point of the bottom surface of the 3D detection frame of the object after transformation on the ground plane. The coordinates of the corner points and the center point of the bottom surface of the 3D detection frame of the object after transformation on the ground plane are obtained by calculating the homography matrix between the coordinates of the corner points and the center point of the bottom surface of the 3D detection frame of the static object corresponding to the detection on the ground plane and the real coordinates of the 2D projection of the corner points and the center point of the bottom surface of the 3D detection frame of the static object corresponding to the detection, and then the real coordinates of the corner points and the center point of the bottom surface of the 3D detection frame of the static object corresponding to the detection are transformed by the homography matrix.

4. The object-level semantic SLAM method for autonomous positioning in a basement environment according to claim 2, characterized in that: The supervision target of the rotation consistency loss function is that the orientation angle of the 3D detection frame corresponding to each static object in the scene graph of each frame is equal to the average orientation angle of the 3D detection frame corresponding to all static objects in the scene graph of the current frame.

5. The object-level semantic SLAM method for autonomous positioning in a basement environment according to claim 2 is characterized in that: The supervision target of the arrangement neatness loss function is the arrangement direction angle and orientation angle between all static objects in each frame scene graph. If the preset conditions are met, the arrangement direction between the two static objects is parallel or perpendicular to their own orientation.

6. The object-level semantic SLAM method for autonomous positioning in a basement environment according to claim 1 is characterized in that: In the above 4), if the current continuous key frames meet the sliding window condition, local optimization is performed.

7. The object-level semantic SLAM method for autonomous positioning in a basement environment according to claim 1, characterized in that: In the above 4), if a loop is detected, loop optimization is performed to obtain the optimized total pose of all key frames in the loop interval and update it, thereby optimizing the object-level semantic map in the current loop interval.

Images