A semi-dense mapping method, storage medium and device for monocular camera imaging

Abstract

The present invention discloses a semi-dense mapping method for monocular camera imaging, which comprises the following steps: S1: screening key frames: gradually screening key frames from the imaging of a monocular camera to obtain a key frame set; S2: information processing: processing the image information in the key frame set to obtain pixel inverse depth related information; S3: semi-dense mapping: combining the related information of inverse depth distribution with the key frame set to obtain a semi-dense point cloud map. Compared with the prior art, the semi-dense mapping method for monocular camera imaging provided in the present invention uses a monocular camera as a front-end image collector, and after screening key frames from the collected images by a set means, semi-dense mapping is performed based on the pixel inverse depth related information of the images in the key frame set. Compared with the common dense mapping in traditional visual SLAM systems, it is convenient to obtain faster, more efficient, and more lightweight mapping effects.

Description

Technical Field

[0001] The invention belongs to the technical field of visual SLAM, and in particular relates to a visual SLAM technology based on monocular camera imaging. Background Art

[0002] SLAM (Simultaneous Localization and Mapping) technology, that is, positioning and mapping technology, refers to a subject equipped with specific sensors, which builds a model of the environment during movement and estimates its own movement without prior information about the environment. Among them, the positioning and mapping technology that uses cameras as the main sensor is called visual SLAM technology. Compared with other SLAM technologies, visual SLAM technology has a large amount of information and a wide range of applications. In recent years, it has been increasingly used in specific scenarios such as robot guides, augmented reality, and unmanned driving.

[0003] Taking the unmanned driving scene as an example, the visual SLAM technology applied in the unmanned driving scene often uses three types of cameras: monocular cameras, binocular cameras, and depth cameras, among which binocular cameras and depth cameras are the most commonly used. The binocular camera uses two lenses to observe the same target from different viewpoints, calculates the position deviation between image pixels, and performs stereoscopic imaging of the target within the field of view. The depth camera often has its own infrared structured light ranging unit, which actively emits light to the target and receives the returned light, measures the distance between the target object and the camera while obtaining the image on the target side, and directly obtains the depth information.

[0004] It should be pointed out that unmanned driving scenes using binocular cameras or depth cameras often need to face the problems of high cost, large amount of calculation and low mapping efficiency: first, the binocular cameras and depth cameras used in unmanned driving scenes are often industrial-grade, large in size and high in price; secondly, for unmanned vehicles using binocular cameras, the configuration and calibration of the binocular cameras are relatively complex, and their depth range and accuracy are limited by the binocular baseline and resolution. In addition, the calculation of parallax consumes a lot of computing resources, and often requires the use of GPU and FPGA devices for acceleration before the distance information of the entire image can be output in real time. The amount of calculation is very large. Although the depth camera solves the depth problem from its sensing principle and directly obtains depth information by physical means, the amount of image information is still very large. In addition, when the binocular camera and the depth camera are used to collect environmental images and the SLAM technology is used for positioning and synchronous mapping, the images collected by the camera are often combined with the depth information. After further processing, the three-dimensional point cloud data can be obtained, and the dense mapping form is used to realize the reconstruction from two-dimensional to three-dimensional. The dense mapping process requires not only a huge storage space, but also a huge amount of computing power. The mapping process has a huge amount of calculation and the mapping efficiency is also very low.

[0005] In unmanned driving scenarios using monocular cameras, the monocular cameras used are small in size and low in cost, which can to a certain extent solve the problems of high cost and limited application derived from the use of binocular cameras and depth cameras mentioned above; but it is inevitable that monocular camera imaging has depth and scale uncertainty. Therefore, the unmanned driving car / unmanned driving platform equipped with a monocular camera must change the angle to observe the same target from multiple positions, that is, multi-angle imaging, and its image processing is also very complicated. The problems of huge computational complexity and low mapping efficiency in the above-mentioned mapping process exist.

[0006] In summary, when the existing visual SLAM technology is applied to specific scenarios, whether using a monocular camera, a binocular camera or a depth camera, it cannot simultaneously solve the problems of high cost, large amount of calculation and low mapping efficiency. It is urgent for technical personnel in this field to propose an effective solution to this problem. Summary of the invention

[0007] To solve the above problems, the purpose of the present invention is to provide a monocular camera imaging semi-dense mapping method, storage medium and device. Compared with the traditional dense mapping method, this method can achieve faster and more efficient mapping effects with lighter calculations.

[0008] To achieve the above object, the technical solution of the present invention is as follows:

[0009] A semi-dense mapping method for monocular camera imaging, the method is:

[0010] S1: Filter key frames: gradually filter out key frames from the imaging of the monocular camera to obtain a key frame set;

[0011] S2: Information processing: Process the image information in the key frame set to obtain pixel inverse depth information;

[0012] S3: Semi-dense mapping: Combine the relevant information of the inverse depth distribution with the key frame set to obtain a semi-dense point cloud map.

[0013] Furthermore, S1 is specifically as follows: setting the number of feature points detected in a single frame image, the co-viewing ratio range of adjacent key frame map points, and the total number of key frames as constraints, and gradually screening out key frames from the imaging of the monocular camera to obtain a key frame set.

[0014] The number of feature points detected in a single frame image reflects the amount of feature information in the external environment. If there is a large amount of feature information and a more complex environment, the value can be adjusted accordingly to relax the constraint. The common view ratio of adjacent key frames is greatly affected by the movement speed of the monocular camera and the complexity of the environmental features, and should be adjusted according to the actual environment. When the monocular camera moves at a high speed, the surrounding environment changes significantly, and the common view ratio of adjacent key frames is small. At this time, a common view ratio range with a smaller median should be set to adapt to the current faster movement speed, otherwise a common view ratio range with a larger median should be set to adapt to its corresponding slower movement speed; and similarly, if there is an environment with more complex features, the common view ratio between adjacent key frames is often larger. At this time, a common view ratio range with a smaller median should be set to adapt to the current complexity of environmental features, otherwise a common view ratio range with a larger median should be set to adapt to its corresponding ordinary complexity of environmental features.

[0015] Furthermore, S2 is specifically:

[0016] S21: Setting pixel gradient threshold: and calculating the pixel gradient of each pixel in each key frame;

[0017] S22: Compare the pixel points with the pixel gradient threshold in order. If the pixel gradient of a certain pixel point is greater than the pixel gradient threshold, jump to S23;

[0018] S23: Mark the pixel as a target pixel, and mark the key frame containing the target pixel as a target key frame;

[0019] S24: setting a search range around the target key frame, and marking other key frames entering the search range as matching key frames;

[0020] S25: searching for a matching point corresponding to the target pixel point in the matching key frame, and making a corresponding matching inverse depth estimation for the target pixel point based on the searched matching point;

[0021] S26: Check the compatibility of the two estimates in sequence. If the compatibility passes, the two estimates are merged into one estimate. If the compatibility fails, the latter estimate is discarded.

[0022] S27: loop S25-S26 until the inverse depth estimation information of the target pixel is obtained;

[0023] S28: Return to S22 until the inverse depth information of all target pixels is obtained.

[0024] Among them, S24 is specifically: determine the expected value range of the inverse depth in the environment, and set the expected value range as the search range; since the expected value of the inverse depth obeys the Gaussian distribution, its upper and lower limits can be calculated according to the probability model, from which the expected value range of the inverse depth in the environment can be known, and the expected value range is set as the search range. The key frames that have a common view relationship with the target key frame often observe the same map point, so for each target key frame, other key frames that have a common view relationship with it can be set as search objects, and stereo search constraints are established for the search process.

[0025] Furthermore, S25 is specifically:

[0026] S251: Calculate the basic matrix of the target key frame and the matching key frame based on the camera internal parameters and the matched point pairs;

[0027] S252: Determine the epipolar line by the basic matrix;

[0028] S253: searching for a pixel point that matches the target pixel point along the epipolar line in terms of brightness value, gradient modulus and direction as a matching point;

[0029] S254: Make an inverse depth estimation corresponding to the target pixel point based on the searched matching point.

[0030] However, due to certain similarities between pixels and possible occlusion, when searching, the key frames may have too large a spacing, which may increase the search range of the epipolar lines and lead to mismatching. To avoid this, the method provided in this application also involves point pair matching verification, specifically: S26 includes the following sub-steps:

[0031] S261: Setting compatibility threshold;

[0032] S262: Calculate compatibility parameters of two successive inverse depth estimates;

[0033] S263: Verify the compatibility of the two inverse depth estimates based on the compatibility parameters and compatibility threshold of the two inverse depth estimates. If the compatibility passes the verification, the two inverse depth estimates are fused into one estimate; otherwise, the latter estimate is discarded.

[0034] As the matching point search process continues to advance, matching points are found one by one. Every time a matching point appears, an inverse depth estimation matching the matching point is made according to the target pixel corresponding to the matching point. When two matching points appear, the inverse depth estimations made for the above two matching points need to be verified. If the verification is passed, it proves that the above two matching points are normally matched with the target pixel, and the two estimates are fused into one estimate. While verifying the compatibility of the inverse depth estimation of the matching point, the matching point search process is also synchronously continued. New matching points are continuously searched out, and new inverse depth estimates made for the target pixel corresponding to the new matching points are continuously obtained. An estimate formed by the fusion of the previous two estimates is verified and fused again with the new inverse depth estimation that appears later, or the verification is discarded. In this form of sequential and incremental fusion, following the search process of the matching point, a new matching point appears, that is, the verification fusion or verification is discarded once, and the inverse depth estimation for the same target pixel is finally fused into one estimate, and finally the inverse depth estimation information of the target pixel is obtained.

[0035] Furthermore, S3 is specifically:

[0036] S31: forming a semi-dense depth map from pixel inverse depth information;

[0037] S32: Combine the semi-dense depth map with the key frame set to generate a semi-dense point cloud map.

[0038] The present application also provides a storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the method described above are implemented.

[0039] The present application also provides a device, which includes:

[0040] Monocular camera: used to record monocular images;

[0041] Mobile device: used to carry a monocular camera and move in a specified environment;

[0042] a memory having a computer program stored thereon;

[0043] and a processor, configured to execute the computer program in the memory to implement the steps of the method described above.

[0044] The advantage of the present invention is that compared with the prior art, the monocular camera imaging semi-dense mapping method provided in the present invention uses a monocular camera as a front-end image collector, and after filtering out key frames from the collected images by a set means, semi-dense mapping is performed based on pixel inverse depth-related information of the images in the key frame set. Compared with the dense mapping commonly used in traditional visual SLAM systems, faster, more efficient and lighter mapping effects can be easily achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 3 is a system framework diagram of an improved ORB-SLAM 3 system when the monocular camera imaging semi-dense mapping method provided in the specific implementation manner is applied in the ORB-SLAM 3 system.

[0046] Figure 2 yes Figure 1 Flowchart of the specific steps of the method implemented in the semi-dense point cloud mapping frame. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0048] To achieve the above object, the technical solution of the present invention is as follows:

[0049] See also Figure 1-2 .

[0050] In this specific embodiment, a device is provided, which includes:

[0051] Monocular camera: used to record monocular images;

[0052] Unmanned vehicle: used to carry a monocular camera and move in a specified environment;

[0053] a memory having a computer program stored thereon;

[0054] and a processor for executing the computer program in the memory.

[0055] Based on the above device, a monocular camera imaging semi-dense mapping method is also provided in this specific embodiment, and the method is specifically applied in the ORB-SLAM 3 system.

[0056] ORB-SLAM3 is a SLAM system that supports vision, vision plus inertial navigation, and hybrid maps. It can run on a monocular camera, a binocular camera, or a depth camera (RGB-D camera) using a pinhole or fisheye model. The monocular camera imaging semi-dense mapping method provided in this specific embodiment will be inserted into the ORB-SLAM3 system in the form of an integral module, as an additional semi-dense mapping module added to the monocular vision part of the system to achieve the expansion of the SLAM mapping function.

[0057] Specifically, in this specific implementation, the monocular camera imaging semi-dense mapping method is:

[0058] S1: Filter key frames: According to the default strategy of the original system in the ORB-SLAM3 system, after the monocular camera is imaged, its image frame is input into the system. After the system is tracked and locally mapped, the monocular vision part of the system will initially filter out the original key frames from the image frames input by the monocular camera, and mark the original key frame set as {k r-a ,...,k r ,...,k r+b}, further screening is performed in the original key frame set, and the obtained key frame set is a subset of the original key frame set, denoted as {K s-c ,...,K s ,...,K s+d};

[0059] S2: Information processing: Process the image information in the key frame set to obtain pixel inverse depth related information;

[0060] S3: Semi-dense mapping: Combine the relevant information of the inverse depth distribution with the key frame set to obtain a semi-dense point cloud map.

[0061] In this specific implementation, further, S1 is specifically:

[0062] On the one hand, set the number of feature points N detected in a single frame image Feature point As a constraint, search each key frame image in the key frame set to find that the number of feature points detected in a single frame image is not less than N Feature point For general environments, set N Feature points ≥80, for environments with complex feature information, set N Feature points ≥100.

[0063] On the other hand, set the common view scale range of adjacent keyframe map points and The value of needs to be adjusted according to the actual environment. Specifically, we can consider the speed of the unmanned vehicle. When the unmanned vehicle is in a low-speed motion state of less than 10km / h, the changes in the surrounding environment will not be too obvious. At this time, the similarity between frames is high, and the common view ratio range is Set to (70%, 80%); When the speed of the unmanned vehicle is high, the rapid movement will cause the environmental information detected by the camera to change rapidly. At this time, the common view ratio range is set Alternatively, the complexity of the environmental feature information can be considered. When the complexity of the environmental feature information is high, the common view ratio range is set to (70%, 80%), when the complexity of the feature information in the environment is relatively general, set the common view ratio range is (70%, 90%).

[0064] On the other hand, an upper limit of the total number of key frames is set as a constraint condition, and the upper limit of the total number of key frames is represented by a proportional value σ. In this specific implementation, σ satisfies: In this formula, N represents the quantity, that is, the numerator represents the total number of key frames contained in the key frame set, and the denominator represents the total number of key frames in the original key frame set. The upper limit ratio value σ of the total number of key frames is obtained by comparing the two. As the unmanned vehicle continues to move, the monocular camera will record more and more image frames, the original key frame set will be continuously updated, and the number of key frames will continue to increase. The key frame set will also be continuously updated accordingly, and the number of key frames will continue to increase. The upper limit of the number of key frames can be controlled in the form of the ratio value σ, and the dynamic changes of the original key frame set and the key frame set can be followed in real time. When the total number reaches the threshold, the screening process ends and the set stops updating. The smaller the value, the higher the limit, the fewer the total number of key frames, and the less the amount of calculation in the later stage. In this specific embodiment, the ratio value σ is set between 75% and 90%.

[0065] Therefore, the monocular vision part of the system first filters the imaging input of the monocular camera to obtain the original key frame set: {k r-a ,...,k r ,...,k r+b}, the method provided in this specific implementation mode is to obtain the original key frame set {k r-a ,...,k r ,...,k r+b} in accordance with the above constraints to obtain the key frame set {K s-c ,...,K s ,...,K s+d}, complete S1.

[0066] Further, in this specific implementation, S2 is specifically:

[0067] S21: Set pixel gradient threshold λ G :And calculate the pixel gradient of each pixel in each key frame;

[0068] S22: Compare the pixel points with the pixel gradient threshold in order. If there is a pixel point with a pixel gradient greater than the pixel gradient threshold λ G , then jump to S23;

[0069] S23: Mark the pixel as the target pixel p i , marking the target pixel p iThe key frame is the target key frame K i ;

[0070] S24: Around the target key frame K i Set the search range and mark other keyframes that enter the search range as matching keyframes K j ;

[0071] S25: After matching key frame K j Inside is the target pixel p i Search for its corresponding matching point P j , the matching point P found by the search j is the target pixel p i making an inverse depth estimate of the corresponding match;

[0072] S26: Check the compatibility of the two estimates in sequence. If the compatibility passes, the two estimates are merged into one estimate. If the compatibility fails, the latter estimate is discarded.

[0073] S27: loop S25-S26 until the target pixel point p is obtained i Inverse depth estimation information;

[0074] S28: Return to S22 until the inverse depth information of all target pixels is obtained.

[0075] Wherein, S24 specifically includes: determining an expected value range of the inverse depth in the environment, and setting the expected value range as the search range;

[0076] Furthermore, S25 is specifically:

[0077] S251: Calculate the target key frame K from the camera internal parameters and the matched point pairs i Matching key frame K j The basis matrix of

[0078] S252: Determine the epipolar line l from the basic matrix j ;

[0079] S253: Along the polar line l j Search the target pixel p from the brightness value, gradient modulus and direction i The pixel point with matching relationship is the matching point P j ;

[0080] S254: The matching point P found by the search j is the target pixel p i An inverse depth estimate of the corresponding match is made.

[0081] Specifically: S26 includes the following sub-steps:

[0082] S261: Setting a compatibility threshold ξ, setting the threshold range to 5.5<ξ<6; waiting for two inverse depth estimates E1 and E2 to appear;

[0083] S262: Calculate compatibility parameters η1 and η2 of two successive inverse depth estimates E1 and E2, where η1 = (ρ1-ρ2) / σ1, η2 = (ρ1-ρ2) / σ2;

[0084] S263: Take ξ=5.99, and verify the compatibility of the two estimates using the compatibility parameters η1 and η2 of the above two inverse depth estimates and the compatibility threshold ξ: When there is 12 <ξ, that is When the two inverse depth estimates E1 and E2 can pass the verification, E1 and E2 are fused into one estimate (The corresponding compatibility parameter is ), otherwise discard E2.

[0085] Specifically, in this specific implementation, S27 is also included: loop S25-S26, then each time a matching point P3, P4, P5, etc. is searched out, P3 is used as the target pixel point p i Make the corresponding matching inverse depth estimate E3, calculate its compatibility parameter η3 based on E3, and merge η3 with the previous E1 and E2 to form Phase verification: When there is 23 <ξ, that is When , the two inverse depth estimates and E3 can pass the verification, Combined with E3 to form an estimate Otherwise, discard E3. And so on, until the inverse depth estimation information of the target pixel is obtained;

[0086] S28: Return to S22 until the inverse depth information of all target pixels is obtained.

[0087] In this specific embodiment, a storage medium is also provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the above method are implemented.

[0088] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A semi-dense mapping method for monocular camera imaging, characterized in that: The method is: S1: Filter key frames: gradually filter out key frames from the imaging of the monocular camera to obtain a key frame set; S2: Information processing: Process the image information in the key frame set to obtain pixel inverse depth information; S3: Semi-dense mapping: Combine the relevant information of the inverse depth distribution with the key frame set to obtain a semi-dense point cloud map; S1 specifically includes: setting the number of feature points detected in a single frame image, the co-viewing ratio range of adjacent key frame map points, and the total number of key frames as constraint conditions, and gradually filtering out key frames from the imaging of the monocular camera to obtain a key frame set; The S2 is specifically: S21: setting a pixel gradient threshold and calculating the pixel gradient of each pixel in each key frame; S22: Compare the pixel points with the pixel gradient threshold in order. If the pixel gradient of a certain pixel point is greater than the pixel gradient threshold, jump to S23; S23: Mark the pixel as a target pixel, and mark the key frame containing the target pixel as a target key frame; S24: setting a search range around the target key frame, and marking other key frames entering the search range as matching key frames; S25: searching for a matching point corresponding to the target pixel point in the matching key frame, and making a corresponding matching inverse depth estimation for the target pixel point based on the searched matching point; S26: Check the compatibility of the two estimates in sequence. If the compatibility passes, the two estimates are merged into one estimate. If the compatibility fails, the latter estimate is discarded. S27: loop S25-S26 until the inverse depth estimation information of the target pixel is obtained; S28: Return to S22 until the inverse depth information of all target pixels is obtained.

2. The monocular camera imaging semi-dense mapping method according to claim 1, characterized in that: The S24 is specifically as follows: determining an expected value range of the inverse depth in the environment, and setting the expected value range as a search range; and at the same time, for each target key frame, setting other key frames having a co-viewing relationship with it as search objects.

3. The monocular camera imaging semi-dense mapping method according to claim 1, characterized in that: The S25 is specifically: S251: Calculate the basic matrix of the target key frame and the matching key frame based on the camera internal parameters and the matched point pairs; S252: Determine the epipolar line by the basic matrix; S253: searching for a pixel point that matches the target pixel point along the epipolar line in terms of brightness value, gradient modulus and direction as a matching point; S254: Make an inverse depth estimation corresponding to the target pixel point based on the searched matching point.

4. The monocular camera imaging semi-dense mapping method according to claim 3, characterized in that: The S26 is specifically: S261: Setting compatibility threshold; S262: Calculate compatibility parameters of two successive inverse depth estimates; S263: Verify the compatibility of the two inverse depth estimates based on the compatibility parameters and compatibility threshold of the two inverse depth estimates. If the compatibility passes the verification, the two inverse depth estimates are fused into one estimate; otherwise, the latter estimate is discarded.

5. The monocular camera imaging semi-dense mapping method according to claim 1, characterized in that: The S3 is specifically: S31: forming a semi-dense depth map from pixel inverse depth information; S32: Combine the semi-dense depth map with the key frame set to generate a semi-dense point cloud map.

6. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 5 are implemented.

7. A device, characterized in that: The device includes: Monocular camera: used to record monocular images; Mobile device: used to carry a monocular camera and move in a specified environment; a memory having a computer program stored thereon; and a processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 5.

Images