A large scene lane mapping method and electronic device based on monocular vision

Abstract

The present invention uses a monocular camera to build lane maps in large scenes and an electronic device. It uses the road plane properties in an outdoor vehicle environment, combined with the prior information of the vehicle camera calibration, to restore the monocular mapping scale and suppress scale drift, and further uses the road plane properties to reversely project the camera data to build a lane-based map of an outdoor large scene. It uses a single consumer-grade sensor and combines a unique mapping method to achieve stable mapping, high mapping accuracy, and low cost.

Description

Technical Field

[0001] The present invention is used in the field of autonomous driving of intelligent unmanned vehicles, and more specifically relates to map construction in unmanned driving perception. Background Art

[0002] With the upgrading of sensors, the development of artificial intelligence, navigation positioning, intelligent control and other related technologies, autonomous driving is no longer so far away. Navigation positioning is an indispensable and very important part of autonomous driving technology. Currently, positioning on pre-made maps is still the mainstream solution.

[0003] With the rapid development of simultaneous localization and mapping (SLAM) technology, visual SLAM is widely used in AR, robots, drones and other fields. However, due to the lack of scale faced by monocular visual SLAM and the fact that the map data created by SLAM is too single, it is difficult to apply monocular visual mapping in large scene environments.

[0004] Patent document CN201710645663.2 A method and system for autonomous positioning and map construction of unmanned vehicles, using a variety of sensors such as wheel odometers, IMU inertial measurement units, panoramic cameras, and three-dimensional lidars for mapping. This method uses SLAM technology to fuse data from multiple sensors, which requires a variety of professional high-precision acquisition equipment, which is costly and difficult to promote to consumer-grade passenger cars. Patent document CN202010710365.9 provides a semi-dense map construction method for mobile robots based on monocular vision, which uses monocular vision for semi-dense mapping, aiming to add semantic information of semantic recognition to the traditional feature point SLAM semi-dense mapping. This method is only applicable to small and medium-sized indoor scenes and is difficult to apply in large outdoor scene environments. Summary of the invention

[0005] The purpose of the present invention is to provide a method and electronic device for mapping the road on which a vehicle is traveling in a large scene using monocular vision, and to use the road plane attributes in an outdoor vehicle environment, combined with the prior information of the vehicle camera calibration, to restore the monocular mapping scale and suppress scale drift, and further use the road plane attributes to reversely project the camera data, so as to construct a lane-based map of a large outdoor scene.

[0006] The technical solution of the present invention is as follows:

[0007] The present invention proposes a method for constructing lane maps of large scenes using a monocular camera, which includes the following main steps:

[0008] Step 1: Calibrate the vehicle-mounted front-view camera to obtain the camera's internal and external parameters relative to the vehicle.

[0009] Step 2: Use a monocular camera to collect data while the vehicle is driving, and use a neural network to identify ground elements, including lane lines, zebra crossings, and arrows.

[0010] Step 3: Use monocular ORB_SLAM to track the odometer and construct point cloud of the collected data.

[0011] Step 4: Use the RANSANC method to fit the ground plane to the point cloud and obtain the ground plane expression equation.

[0012] Step 5: Use the camera pose and ground plane equation to calculate the distance between the camera and the ground, associate the camera external parameters to restore the map scale, and insert optimization to prevent scale drift.

[0013] Step 6: Adaptive IPM projection: project the recognition results of the neural network onto the ground plane to generate a ground plane element point cloud, with one point cloud produced for each frame.

[0014] Step 7: Overlay the point cloud sub-images according to the ORB_SLAM tracking pose, and perform probability statistics on the overlapping parts of the point cloud to determine whether they are background or elements.

[0015] Step 8: After the monocular ORB_SLAM tracking fails, reinitialize and repeat steps 2 to 7 to build a new point cloud sub-map. Use NDT point cloud matching to stitch multiple point cloud sub-maps into a large scene road map.

[0016] The present invention further provides an electronic device in another aspect, comprising:

[0017] One or more processors.

[0018] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the electronic device implements the steps of the above-mentioned method for large-scene lane mapping using a monocular camera.

[0019] By adopting the above technical solution, the present invention has at least the following advantages:

[0020] 1. The present invention utilizes the road plane properties in an outdoor vehicle environment, combined with the prior information of the vehicle camera calibration, to restore the monocular mapping scale and suppress scale drift, and further utilizes the road plane properties to reversely project the camera data, thereby constructing a lane-based map of an outdoor large scene. It uses a single consumer-grade sensor and combines a unique mapping method to achieve stable mapping, high mapping accuracy, and low cost.

[0021] 2. The present invention utilizes the pre-calibrated distance between the camera and the ground, constructs a sparse point cloud of a three-dimensional environment online, and fits the ground plane to calculate the ratio between the real physical distance and the calculated distance. This can overcome the problems faced by monocular vision SLAM, such as scale loss and scale drift over time, and the problem that SLAM creates map data that is too single, and can realize monocular vision mapping in large scene environments.

[0022] 3. The present invention uses a single consumer-grade sensor (such as a front-view camera) for stable mapping. The prior information of the camera's external parameters is used to solve the monocular scale loss and scale drift problems, and the monocular visual odometer and the plane constraint of the ground are used to realize the ground lane element mapping. It can be used on consumer-grade passenger cars, which overcomes the existing technology that uses multiple sensors in sensor configuration, which is costly and requires the fusion of multiple sensor data, and the data processing is complex and inaccurate.

[0023] 4. The present invention utilizes the road plane attributes in the outdoor vehicle environment and projects the semantic information onto the ground plane using the IPM algorithm to realize the construction of ground elements. It uses multiple constructed sub-graphs and uses point cloud NDT registration to realize large-scene ground mapping. It is significantly superior to the existing technology in terms of algorithm principle, structure, and mapping format. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flowchart of the steps of the method of the present invention;

[0025] Figure 2 An example diagram of the vehicle body and camera coordinate system in the method of the present invention;

[0026] Figure 3 Schematic diagram of the semantic recognition mask (top) and IPM projection (bottom) in the method of the present invention;

[0027] Figure 4 A sparse point cloud image constructed by the ORB_SLAM algorithm according to the method of the present invention;

[0028] Figure 5 The result diagram of ground plane fitting for point cloud in the method of the present invention;

[0029] Figure 6 This is the result image of the multi-sub-image NDT registration in the method of the present invention. DETAILED DESCRIPTION

[0030] The following will illustrate the implementation of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific implementations, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0031] See also Figure 1 This embodiment describes the detailed steps of a method for building lane maps in a large scene using a monocular camera:

[0032] Step 1: Calibrate the vehicle-mounted front-view camera and obtain the internal and external parameters of the camera:

[0033] The vehicle body sensor (in this embodiment, a vehicle-mounted front-view camera is used) is calibrated to obtain the camera's internal parameters and the camera's external parameters relative to the vehicle.

[0034] like Figure 2 As shown in the figure, the camera internal parameters obtained by calibration are: focal length (fx, fy), principal point coordinates (cx, cy). Camera external parameters: coordinate values ​​of the camera optical center in the vehicle coordinate system (X, Y, Z), and deflection angles of the camera coordinate system in the vehicle coordinate system (Roll, Pitch, Yaw).

[0035] Step 2: Collect image data and use convolutional neural network to identify ground elements:

[0036] The monocular camera collects data and uses a convolutional neural network to identify ground elements.

[0037] like Figure 3 As shown in the figure above, the semantic segmentation of the convolutional neural network is used to semantically recognize the ground elements on the front view image collected by the monocular camera to obtain the pixel mask of the ground elements. The ground elements here include lane lines, zebra crossings, arrows, etc.

[0038] Step 3: Input image data into ORB_SLAM for odometer tracking and point cloud construction:

[0039] Use ORB_SLAM to track odometer and construct point cloud based on data collected by monocular camera.

[0040] Specifically, the continuous video stream data collected by the front-view camera is used as input and input into the ORB_SLAM algorithm for mapping and tracking. The ORB_SLAM algorithm obtains the real-time camera position and posture (x, y, z, roll, pitch, yaw), sparse feature point cloud and three-dimensional coordinates of each point (xi, yi, zi), such as Figure 4As shown in Figure 2, there is an unknown scaling ratio s between the camera position and the position of the point in the point cloud obtained in this step and the real physical world.

[0041] ORB_SLAM is an open source SLAM algorithm module. This algorithm module is used to calculate the relative displacement and relative posture change of the camera when collecting continuous video stream data. This is to track the mileage trajectory of the data. While calculating the mileage, the ORB_SLAM algorithm generates a sparse point cloud representation of the feature points of the three-dimensional world, that is, the point cloud map is constructed, which is mapping. ORB_SLAM mapping tracking is to construct a surrounding environment map while positioning each frame, which is simultaneous positioning and mapping SLAM.

[0042] Step 4: Fit the ground plane to the point cloud:

[0043] Use the RANSANC method to fit the ground plane on the point cloud and obtain the ground plane expression equation, such as Figure 5 As shown:

[0044] 4.1. Project the 3D coordinates of the point cloud onto the current image to obtain the pixel coordinates, and determine whether it is a ground point according to the ground semantic mask to obtain the ground point cloud set Pg[p0, p1, p2…pn];

[0045] 4.2. Randomly select five points from Pg, calculate the mean of the three-dimensional coordinates [x_, y_, z_] of these five points and the covariance matrix ∑ of the three-dimensional coordinates. Perform eigendecomposition on ∑ to obtain three eigenvalues ​​from large to small, namely λ0, λ1, λ2, and the corresponding eigenvectors v0, v1, v2. If ①10*λ2<λ1, ②λ0<2*λ1, it is considered that these five points are approximately located on the same plane, proceed to the next step, and return to step 2).

[0046] In this step, the mean of the three-dimensional coordinates of the five points is calculated, aiming to represent all dimensions of this point cloud with the least number of points. Four points or less are not sufficient to stably calculate the feature dimensions of this point cloud.

[0047] 4.3. Use the three-dimensional coordinates of these five points to calculate the least squares solution of the plane equation;

[0048] 4.4. Calculate the distance d from each point in Pg to the plane. If d<6*λ0, the point is considered to be an interior point on the plane, and the interior point set Pin[pin0, pin1, pin2…pinn] is obtained;

[0049] 4.5. Repeat steps 2), 3), and 4) ten times to obtain the calculation result with the maximum number of internal points.

[0050] Here, it is determined empirically that the more times, the greater the amount of calculation, and the more accurate the result. Of course, other values ​​are also possible, but the best value is around ten times.

[0051] 4.6. Use the three-dimensional coordinates of all internal points to calculate the least squares solution of the plane equation and obtain the fitted ground equation parameters (a, b, c, d). The ground plane equation is a*x+b*y+c*z+d=0.

[0052] Step 5: Restore the map scale

[0053] The camera pose and ground plane equation are used to calculate the distance between the camera and the ground, the camera external parameters are associated to restore the map scale, and optimization is inserted to prevent scale drift.

[0054] In this step, the average distance D between the camera position (x, y, z) and the ground plane (a, b, c, d) of ten consecutive frames is calculated. The distance between the camera and the ground in the real physical world is Z in the camera external parameter, and the scale s=Z / D is calculated based on this. In this way, the ORB_SLAM calculation result is scaled. The scale mentioned in monocular camera SLAM is the scaling ratio between the constructed map and the real world.

[0055] Step 6: Adaptive IPM projection:

[0056] Adaptive IPM projection projects the recognition results of the neural network onto the ground plane to generate a ground plane element point cloud sub-image, producing a point cloud image for each frame.

[0057] In this step, the camera position and posture and the ground plane equation are used to calculate the position and posture T of the camera relative to the ground plane. This posture T is used to perform IPM projection on the semantically recognized ground element mask, and the recognition result of the neural network is projected onto the ground plane to generate a ground plane element point cloud. One point cloud is produced for each frame. IPM projection is based on the principle of camera imaging model, simulating the imaging process of the ground plane on the image, and reversely calculating the pixel points on the image onto the ground plane. This process can be summarized as a projection transformation. See Figure 3 The picture below.

[0058] Step 7: Probability statistics of overlapping parts

[0059] The point cloud sub-images are superimposed according to the ORB_SLAM tracking pose, and the overlapping parts of the point cloud are statistically determined to be background or elements.

[0060] Since the IPM projection fields of different frames overlap, the overlapping parts of the projections are superimposed using the camera position and posture. The number of times each projected grid mask falls here as a ground element or background is counted, and the label with the largest number of times is taken as the grid label to obtain the grid map, which is then converted into a point cloud.

[0061] Step 8: Matching of NDT point clouds from multiple maps

[0062] After the monocular ORB_SLAM tracking fails, reinitialize and build a new point cloud map, and use NDT point cloud matching to stitch multiple point cloud maps into a large scene road map.

[0063] In this step, the calculation result from the successful initialization of ORB_SLAM to the failure of ORB_SLAM tracking is used as a point cloud sub-map. Steps 2 to 8 are executed multiple times to obtain multiple sub-maps with different coverage ranges. Finally, the NDT algorithm is used to align and merge different sub-maps to obtain a large scene point cloud semantic map. The NDT algorithm is a point cloud matching algorithm that can calculate the relative position and rotation between two slightly overlapping point clouds. After obtaining the relative displacement and rotation, the two point clouds can be spliced. Figure 6 shown.

[0064] In another embodiment, an electronic device is provided, comprising:

[0065] One or more processors.

[0066] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the electronic device implements the steps of the method for large-scene lane mapping using a monocular camera as described in the previous embodiment.

[0067] The preferred embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. However, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, a variety of simple modifications can be made to the technical solution of the present invention, and these simple modifications all belong to the protection scope of the present invention.

[0068] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations. In addition, the various different embodiments of the present invention can also be combined arbitrarily, as long as they do not violate the ideas of the present disclosure, they should also be regarded as the contents disclosed by the present invention.

Claims

1. A method for constructing lane maps in a large scene using a monocular camera, characterized in that: The steps include: Step 1: calibrate the on-board front-view camera to obtain the internal parameters of the camera and the external parameters of the camera relative to the vehicle; Step 2: Use a monocular camera to collect data while the vehicle is driving, and use a neural network to identify ground elements, including lane lines, zebra crossings, and arrows; Step 3: Use monocular ORB_SLAM to track the odometer and construct point cloud of the collected data; Step 4: Use the RANSANC method to fit the ground plane to the point cloud and obtain the ground plane expression equation; Step 5: Use the camera pose and ground plane equation to calculate the distance between the camera and the ground, associate the camera external parameters to restore the map scale, and insert optimization to prevent scale drift; Step 6: Adaptive IPM projection, projecting the recognition results of the neural network onto the ground plane to generate a ground plane element point cloud, and generating a point cloud corresponding to each frame; Step 7: Overlay the point clouds corresponding to multiple frames according to the ORB_SLAM tracking posture, perform probability statistics on the overlapping parts of the point clouds to determine whether they are background or elements, and generate a point cloud sub-image; Step 8: After the monocular ORB_SLAM tracking fails, reinitialize and repeat steps 2 to 7 to build a new point cloud sub-map. Use NDT point cloud matching to stitch multiple point cloud sub-maps into a large scene road map.

2. The method for constructing lane maps of large scenes using a monocular camera according to claim 1, characterized in that: In the step 1, the camera intrinsic parameters obtained by calibration include: focal length (fx, fy), principal point coordinates (cx, cy); the camera extrinsic parameters include: coordinate values ​​of the camera optical center in the vehicle body coordinate system (X, Y, Z), and deflection angles (Roll, Pitch, Yaw) of the camera coordinate system in the vehicle body coordinate system.

3. The method for constructing lane maps of large scenes using a monocular camera according to claim 1, characterized in that: The step 3 uses monocular ORB_SLAM to track the odometer and construct a point cloud for the collected data, including: taking the continuous video stream data collected by the front-view camera as input, inputting the ORB_SLAM algorithm for map tracking, and the ORB_SLAM algorithm obtains the real-time camera position and posture (x, y, z, roll, pitch, yaw), sparse feature point cloud and three-dimensional coordinates of each point (xi, yi, zi).

4. The method for constructing lane maps of large scenes using a monocular camera according to claim 1, characterized in that: The step 4 uses the RANSANC method to fit the ground plane to the point cloud, and obtains the ground plane expression equation specifically including: Step 4.1, project the three-dimensional coordinates of the point cloud onto the current image to obtain the pixel coordinates, and determine whether it is a ground point according to the ground semantic mask to obtain the ground point cloud set Pg[p0, p1, p2…pn]; Step 4.2, randomly select five points from Pg, calculate the mean of the three-dimensional coordinates [x_, y_, z_] of these five points and the covariance matrix ∑ of the three-dimensional coordinates; Performing eigendecomposition on ∑, we get three eigenvalues ​​from large to small, namely λ0, λ1, λ2, and the corresponding eigenvectors v0, v1, v2; If ①10*λ2<λ1, ②λ0<2*λ1, then it is considered that these five points are approximately located on the same plane, proceed to the next step, and return to step 4.2; Step 4.3, use the three-dimensional coordinates of these five points to calculate the least squares solution of the plane equation; Step 4.4, calculate the distance d from each point in Pg to the plane. If d<6*λ0, the point is considered to be an interior point on the plane, and the interior point set Pin[pin0, pin1, pin2…pinn] is obtained; Step 4.5, repeat steps 4.2, 4.3, and 4.4; Step 4.6: Use the three-dimensional coordinates of all interior points to calculate the least squares solution of the plane equation and obtain the fitted ground equation parameters (a, b, c, d). The ground plane equation is a*x+b*y+c*z+d=0.

5. The method for constructing lane maps of large scenes using a monocular camera according to claim 1, characterized in that: In step 5, the average distance D between the camera position (x, y, z) of multiple consecutive frames and the ground plane (a, b, c, d) is calculated. The distance between the camera and the ground in the real physical world is Z in the camera extrinsic parameter. The calculation scale s=Z / D, where s is the scaling ratio between the camera position and the position of the midpoint in the point cloud and the real physical world, is used to scale the ORB_SLAM calculation result.

6. The method for constructing lane maps of large scenes using a monocular camera according to claim 1, characterized in that: In step 6, the position and posture T of the camera relative to the ground plane is calculated using the camera position and posture and the ground plane equation, and the ground element recognition result of semantic recognition is subjected to IPM projection using this posture T to obtain the ground plane element point cloud of this frame, and a point cloud is produced for each frame.

7. The method for constructing lane maps of large scenes using a monocular camera according to claim 1, characterized in that: In step 8, the calculation result from a successful ORB_SLAM initialization to a failed ORB_SLAM tracking is a point cloud sub-map, which is a superposition of multiple frame point clouds. Steps 2 to 8 are executed multiple times to obtain multiple point cloud sub-maps with different coverage ranges. Finally, the NDT algorithm is used to align and merge different sub-maps to obtain a large scene point cloud semantic map.

8. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the electronic device implements the steps of the method for large-scene lane mapping using a monocular camera as described in any one of claims 1 to 7.

Images