A sparse mapping method for drainage pipelines in weak light environment based on pipeline robot

Abstract

The present invention relates to the field of surveying technology, and in particular to a sparse mapping method for drainage pipelines in a weak light environment based on a pipeline robot, comprising: using a drainage pipeline surveying robot to enter a drainage pipeline to collect a video stream; preprocessing images in the video stream frame by frame using an image defogging method to obtain an enhanced image; processing the enhanced image using a frame processing thread, performing single-image composition, and obtaining image parameters; inputting the image parameters into a tracking thread, calculating the matching of image feature points, obtaining a homography matrix and an essential matrix, and restoring the spatial position according to the position of the image feature points in the image; inserting the image feature points after the spatial position is restored into a local mapping thread, and using closed-loop detection to reduce the offset error generated by visual SLAM to complete the construction of a pipeline map, thereby solving the problem that the existing sparse mapping of drainage pipelines cannot meet the surveying needs of drainage pipelines.

Description

Technical Field

[0001] The present invention relates to the field of surveying technology, and in particular to a sparse mapping method for drainage pipelines in a weak light environment based on a pipeline robot. Background Art

[0002] A robot is a machine device that can automatically perform tasks. Its task is to assist or replace human work, especially for dangerous work that is beyond human ability and requires extremely high precision. A drainage pipeline survey robot is a robot device that can move inside a drainage pipe and identify, measure, record and analyze the pipe through high-definition cameras, detection sensors and other equipment. It usually consists of a robot body, control equipment and data transmission equipment. The main part usually uses wireless remote control to perform real-time detection inside the pipeline through high-definition cameras, detection sensors and other equipment, avoiding the danger of manual entry into the pipeline.

[0003] Sparse composition technology is a technology based on advanced computer vision and other technical means to collect, process and analyze certain spatial structure information, so as to draw a comprehensive, three-dimensional map of the spatial structure. The characteristic of sparse composition technology is that it integrates the position, length, depth, state and other information of the spatial structure through the analysis and processing of the detection data, and presents it as a digital map or three-dimensional model through digital technology. However, the existing sparse composition technology is generally used in scenes with sufficient lighting. In underground drainage pipelines, although there are lights for supplementary lighting, the lighting effect is poor, and sparse composition technology has the problem of poor composition effect.

[0004] In summary, the existing sparse mapping of drainage pipelines cannot meet the survey needs of drainage pipelines, and there is an urgent need for a sparse mapping method of drainage pipelines in a weak light environment based on a pipeline robot. Summary of the invention

[0005] The purpose of the present invention is to provide a sparse mapping method for drainage pipelines in a weak light environment based on a pipeline robot, aiming to solve the problem that the existing sparse mapping of drainage pipelines cannot meet the survey needs of drainage pipelines.

[0006] To achieve the above object, the present invention provides a sparse mapping method for drainage pipelines in a weak light environment based on a pipeline robot, comprising the following steps:

[0007] Use a drainage pipeline survey robot to enter the drainage pipeline to collect video streams;

[0008] Preprocessing the images in the video stream frame by frame using an image defogging method to obtain an enhanced image;

[0009] The enhanced image is processed by a frame processing thread, a single picture is composed, and image parameters are obtained;

[0010] Input the image parameters into the tracking thread, calculate the matching of the image feature points, obtain the homography matrix and the intrinsic matrix, and restore the spatial position according to the position of the image feature points in the image;

[0011] The image feature points after the spatial position is restored are inserted into the local mapping thread, and closed-loop detection is used to reduce the offset error generated by visual SLAM to complete the pipeline map construction.

[0012] The step of preprocessing the images in the video stream frame by frame using an image defogging method to obtain an enhanced image includes:

[0013] Convert the images in the video stream into grayscale images frame by frame, and obtain the length, width and pixel values ​​at each position of the images;

[0014] Traversing the pixel values ​​at each position of the image, and negating them to obtain negated pixels;

[0015] Traverse the inverted pixels and mark the maximum value;

[0016] Traverse the pixel values ​​at each position of the image and obtain the pixel ratio;

[0017] Calculating the illumination distribution ratio at each position according to the pixel ratio;

[0018] An enhanced image is obtained according to a calculation formula based on the illumination distribution ratio, the pixel ratio and the pixel value at each position.

[0019] The enhanced image is processed by a frame processing thread, a single picture is composed, and image parameters are obtained, including:

[0020] The enhanced image is input into the frame processing thread of the composition framework, an image pyramid is calculated for each input image, and ORB features and descriptors are extracted to obtain image parameters.

[0021] The image feature points after the spatial position restoration are inserted into the local map construction thread, and closed-loop detection is used to reduce the offset error generated by visual SLAM to complete the pipeline map construction, including:

[0022] Establish an initiator to determine whether the parameters of the frame meet the requirements. If they meet the requirements, establish a tracking initiator.

[0023] In the tracking initiator, eight groups of feature points are randomly obtained, and the homography matrix and the basic matrix are calculated. The homography matrix describes the transformation relationship of the same feature points in different images, and the essential matrix describes the transformation relationship of the same feature points in different spaces. The rotation matrix and displacement vector of the camera are restored according to the transformation relationship between the two, and the rotation matrix and displacement vector are scored to obtain the target rotation matrix and displacement vector as the actual route of the camera;

[0024] Select the spatial positions of four pairs of feature points, and use the BA optimization algorithm to optimize the spatial positions of the feature points to obtain the local mapping thread;

[0025] Matching constant speed model tracking, reference frame model tracking and relocation tracking modes for the local mapping thread;

[0026] Inputting the image after restoring the spatial position into the local mapping thread to obtain mapping feature points;

[0027] Determine whether a keyframe needs to be inserted based on the mapping situation and the distance and ID of each camera frame, and then insert a new keyframe;

[0028] The 3D positions of the mapping feature points are normalized to complete the pipeline map construction.

[0029] The step of inputting the restored spatial position image into the local mapping thread to obtain mapping feature points includes:

[0030] Input the image after restoring the spatial position into the local mapping thread, and remove the 3D points within a preset range according to the observation of the pixel 3D points;

[0031] Generate new 3D points using the triangulation algorithm using the current keyframe and adjacent keyframes;

[0032] Fuse the repeated 3D points of the current keyframe with those of the adjacent keyframes;

[0033] Use the BA algorithm to optimize the 3D points, and remove the redundant key frames of the current key frame based on the optimized 3D points;

[0034] Add the current frame to the closed-loop detection thread to obtain the mapping feature points.

[0035] The step of adding the current frame to the closed-loop detection thread to obtain the mapping feature points includes:

[0036] Add the current frame to the closed-loop detection thread, and take out the head key frame of the buffer queue as the current closed-loop detection key frame;

[0037] Calculate the maximum similarity between the current keyframe and the common view keyframe, where the common view keyframe is the weight relationship between the images;

[0038] Find closed-loop candidate key frames of the current key frame based on similarity relationships;

[0039] Find a match between the current keyframe group and the previous consecutive keyframe group;

[0040] The loop variables are maintained, and the closed-loop key frame group of the current frame is used as the closed-loop candidate key frame group before the next frame to obtain the mapping feature points.

[0041] The present invention discloses a sparse mapping method for drainage pipelines in a weak light environment based on a pipeline robot. The method comprises the following steps: using a drainage pipeline survey robot to enter a drainage pipeline to collect a video stream; preprocessing the images in the video stream frame by frame using an image defogging method to obtain an enhanced image; processing the enhanced image using a frame processing thread to perform single-image composition to obtain image parameters; inputting the image parameters into a tracking thread to calculate the matching of image feature points, obtaining a homography matrix and an essential matrix, and restoring the spatial position according to the position of the image feature points in the image; inserting the image feature points after the spatial position is restored into a local mapping thread, and using closed-loop detection to reduce the offset error generated by visual SLAM to complete the construction of a pipeline map. The present invention adopts an image defogging preprocessing method to optimize the effect of image feature extraction, and finally optimizes the effect of sparse composition of drainage pipelines; adopts an image acquisition method based on pipeline survey robots to avoid the embarrassing situation that surveyors themselves need to enter the drainage pipelines, and protects the lives of surveyors; adopts an automatic composition method to avoid the errors that may be caused by manual composition of drainage pipelines, and makes full use of the advantages of automatic image acquisition and automatic composition by robots; adopts loop detection to maximize the reduction of offset errors caused by visual SLAM, optimizes the final composition effect, and solves the problem that the existing sparse composition of drainage pipelines cannot meet the survey needs of drainage pipelines. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 This is an overall flow chart of a sparse mapping method for drainage pipelines in a weak light environment based on a pipeline robot provided by the present invention.

[0044] Figure 2 It is the preprocessing flow chart.

[0045] Figure 3 It is a flow chart for restoring spatial position.

[0046] Figure 4 It is a local mapping flowchart.

[0047] Figure 5 It is a closed-loop detection flow chart.

[0048] Figure 6 This is a flow chart of a sparse mapping method for drainage pipelines in a weak light environment based on a pipeline robot provided by the present invention. DETAILED DESCRIPTION

[0049] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0050] See also Figures 1 to 6 The present invention provides a drainage pipeline sparse mapping method in a weak light environment based on a pipeline robot, comprising the following steps:

[0051] S1 uses a drainage pipeline survey robot to enter the drainage pipeline to collect video streams;

[0052] Specifically, the sparse mapping method of the drainage pipeline in a weak light environment uses a pipeline survey robot equipped with an image acquisition device to enter the drainage pipeline to realize the video stream acquisition of the method.

[0053] In this embodiment, the ground personnel remotely control the pipeline survey robot through the umbilical cable, and turn on the LED light for fill light operation at the same time, so as to collect a clearer image. It should be noted that although the LED light is used for fill light, the image collected in the drainage pipeline is still a weak light environment for the conventional sparse composition method, so the conventional sparse composition method still cannot achieve sparse composition in this environment.

[0054] S2 preprocesses the images in the video stream frame by frame using an image defogging method to obtain an enhanced image;

[0055] Specifically, the preprocessing process includes obtaining image information, taking and marking the maximum value of pixel values, then calculating the illumination distribution ratio, and finally obtaining the preprocessed image based on the pixel values ​​at each position adjusted by the illumination distribution.

[0056] The specific method is:

[0057] S21 converts the images in the video stream into grayscale images frame by frame, and obtains the length, width and pixel values ​​at each position of the images;

[0058] S22 traverses the pixel values ​​of each position of the image, and inverts them to obtain inverted pixels;

[0059] Specifically, the inverted pixel=255-the pixel value at each position.

[0060] S23 traverses the inverted pixels and marks the maximum value;

[0061] S24 traverses the pixel values ​​at each position of the image to obtain the pixel ratio;

[0062] Specifically, the pixel at each position is divided by the inverted pixel at each position.

[0063] S25, calculating the illumination distribution ratio of each position according to the pixel ratio;

[0064] Specifically, the illumination distribution ratio T=1-W*pixel ratio, where W is obtained based on experience.

[0065] S26 obtains an enhanced image according to a calculation formula based on the illumination distribution ratio, the pixel ratio and the pixel value at each position.

[0066] Specifically, pixel at each position = 255 - (pixel value at each position in the first step - pixel ratio) / T + pixel ratio, and the obtained picture composed of pixels at each position is the picture pre-processed for image enhancement in a low-light environment.

[0067] S3 processes the enhanced image using a frame processing thread, composes a single picture, and obtains image parameters;

[0068] Specifically, the enhanced image is input into the frame processing thread of the composition framework, an image pyramid is calculated for each input image, and ORB features and descriptors are extracted to obtain image parameters.

[0069] The feature sub-calculation process is as follows: for a pixel value, obtain the pixels at each position on the circle with a radius of 3 around it. If 12 pixel values ​​are greater than a certain parameter, the pixel value of the point is judged to be a feature sub-calculation; the descriptor calculation process is as follows: for the feature sub-calculation, obtain the pixel values ​​of the 31*31 square around it, randomly select 256 pairs of pixel values ​​for comparison, and use 01 to express it as a descriptor.

[0070] S4 inputs the image parameters into the tracking thread, calculates the matching of the image feature points, obtains the homography matrix and the intrinsic matrix, and restores the spatial position according to the position of the image feature points in the image;

[0071] S5 inserts the image feature points after the spatial position is restored into the local mapping thread, and uses closed-loop detection to reduce the offset error generated by visual SLAM to complete the pipeline map construction.

[0072] The specific method is:

[0073] S51 establishes an initiator, determines whether various parameters of the frame meet the requirements, and establishes a tracking initiator if the parameters meet the requirements;

[0074] S52 randomly obtains eight groups of feature points in the tracking initiator, calculates the homography matrix and the basic matrix, the homography matrix describes the transformation relationship of the same feature points in different images, and the essential matrix describes the transformation relationship of the same feature points in different spaces, restores the rotation matrix and displacement vector of the camera according to the transformation relationship between the two, and scores the rotation matrix and displacement vector to obtain the target rotation matrix and displacement vector as the actual route of the camera;

[0075] Specifically, the best rotation matrix and displacement vector are obtained as the actual route of the camera.

[0076] S53 selects the spatial positions of four pairs of feature points, and uses the BA optimization algorithm to optimize the spatial positions of the feature points to obtain a local mapping thread;

[0077] S54 matches the constant speed model tracking, reference frame model tracking and relocation tracking modes for the local mapping thread;

[0078] Specifically, after the initializer is established, three tracking modes are adopted according to the actual situation, namely constant speed model tracking, reference frame model tracking and repositioning tracking mode. The repositioning tracking mode is used for the situation where the sparse composition framework initializer fails to initialize or there are few tracking points and matching points.

[0079] S55: inputting the image after the spatial position is restored into the local mapping thread to obtain mapping feature points;

[0080] The specific method is:

[0081] S551: input the image after the spatial position is restored into the local mapping thread, and remove the 3D points within a preset range according to the observation of the pixel 3D points;

[0082] Specifically, some bad 3D points are removed according to the observation conditions of the pixel 3D points.

[0083] S552 generates a new 3D point using a triangulation algorithm using the current key frame and adjacent key frames;

[0084] S553 fuses the repeated 3D points of the current key frame and the adjacent key frame;

[0085] S554 optimizes the 3D points using the BA algorithm, and removes redundant key frames of the current key frame according to the optimized 3D points;

[0086] S555 adds the current frame to the closed-loop detection thread to obtain mapping feature points.

[0087] Specifically, the closed loop detection algorithm is mainly aimed at maps with loops, that is, "U"-shaped corridors, which can greatly reduce the offset error caused by visual SLAM. The specific method is:

[0088] S5551 adds the current frame to the closed-loop detection thread, and takes out the head key frame of the buffer queue as the current closed-loop detection key frame;

[0089] Specifically, if the time since the last loop detection is too short, no loop detection is performed.

[0090] S5552 calculates the maximum similarity between the current key frame and the common view key frame, where the common view key frame is a weight relationship between the pictures;

[0091] S5553 searches for closed-loop candidate key frames of the current key frame based on the similarity relationship;

[0092] S5554 searches for a matching relationship between the current key frame group and the previous continuous key frame group;

[0093] S5555 maintains the loop variable, and the closed-loop key frame group of the current frame is used as the closed-loop candidate key frame group before the next frame to obtain the mapping feature points.

[0094] S56 determines whether a key frame needs to be inserted according to the mapping situation and the distance and ID of each frame of the camera, thereby inserting a new key frame;

[0095] Specifically, a new key frame is inserted to facilitate the suggestion of an initializer for the next frame of the picture.

[0096] S57 normalizes the 3D positions of the mapping feature points to complete pipeline map construction.

[0097] Specifically, the 3D position of the normalized feature point, that is, because of the uncertainty of the scale in the single camera case, needs to be determined by a normalization algorithm, and some parameters need to be updated, such as: the weight ratio between images, the pixel 3D points observed in the frame image, the scale of the pixel 3D points, the observation direction of the 3D points, etc.

[0098] The present invention provides a sparse mapping method for drainage pipelines in a weak light environment based on a pipeline robot. The algorithm consists of two parts: image preprocessing and visual SLAM. In the image preprocessing stage, the idea of ​​image defogging is borrowed, and the grayscale image is inverted to calculate the illumination ratio coefficient, and the pixel value of each position of the image is recalculated according to the illumination ratio coefficient; the visual SLAM consists of three parts: tracking, local mapping and loop detection. The tracking part extracts feature points and descriptors from the image, and calculates the homography matrix and the essential matrix according to the feature points, so as to calculate the camera rotation and displacement; compared with the original algorithm, the local mapping further optimizes the 3D space points by using the size of the sewage pipe, and removes the honorary redundant key frames; the closed loop detection reduces the cumulative drift phenomenon caused by the visual SLAM by detecting whether there is a loop. Compared with the unimproved sparse composition algorithm, the improved sparse composition algorithm performs well on the data set with fewer low-light feature points, and improves the positioning and mapping accuracy.

[0099] Beneficial effects:

[0100] 1. The image defogging preprocessing method is used to optimize the effect of image feature extraction, and finally optimize the effect of sparse composition of drainage pipelines;

[0101] 2. The image acquisition method based on the pipeline survey robot is adopted to avoid the embarrassing situation that the surveyors need to enter the drainage pipeline, thus protecting the lives of the surveyors;

[0102] 3. The automatic composition method is adopted to avoid the errors that may be caused by manual composition of drainage pipelines, and fully utilize the advantages of automatic image acquisition and automatic composition of robots;

[0103] 4. Loop detection is used to minimize the offset error caused by visual SLAM and optimize the final composition effect.

[0104] What is disclosed above is only a preferred embodiment of the sparse mapping method of drainage pipelines in a weak light environment based on a pipeline robot of the present invention. Of course, it cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiments and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A sparse mapping method for drainage pipelines in a weak light environment based on a pipeline robot, characterized in that: The following steps are involved: Use a drainage pipeline survey robot to enter the drainage pipeline to collect video streams; Preprocessing the images in the video stream frame by frame using an image defogging method to obtain an enhanced image; The enhanced image is processed by a frame processing thread, a single picture is composed, and image parameters are obtained; Input the image parameters into the tracking thread, calculate the matching of the image feature points, obtain the homography matrix and the intrinsic matrix, and restore the spatial position according to the position of the image feature points in the image; Insert the image feature points after restoring the spatial position into the local mapping thread, and use closed-loop detection to reduce the offset error generated by visual SLAM to complete the pipeline map construction; The preprocessing of the images in the video stream frame by frame using an image defogging method to obtain an enhanced image includes: converting the images in the video stream frame by frame into grayscale images, and obtaining the length, width and pixel values ​​at each position of the image; Traversing the pixel values ​​at each position of the image, and negating them to obtain negated pixels; Traverse the inverted pixels and mark the maximum value; Traverse the pixel values ​​at each position of the image and obtain the pixel ratio; Calculating the illumination distribution ratio at each position according to the pixel ratio; Obtaining an enhanced image according to a calculation formula based on the illumination distribution ratio, the pixel ratio and the pixel value at each position; The image feature points after the spatial position restoration are inserted into the local map construction thread, and closed-loop detection is used to reduce the offset error generated by visual SLAM to complete the pipeline map construction, including: Establish an initiator to determine whether the parameters of the frame meet the requirements. If they meet the requirements, establish a tracking initiator. In the tracking initiator, eight groups of feature points are randomly obtained, and the homography matrix and the basic matrix are calculated. The homography matrix describes the transformation relationship of the same feature points in different images, and the essential matrix describes the transformation relationship of the same feature points in different spaces. The rotation matrix and displacement vector of the camera are restored according to the transformation relationship between the two, and the rotation matrix and displacement vector are scored to obtain the target rotation matrix and displacement vector as the actual route of the camera. Select the spatial positions of four pairs of feature points, and use the BA optimization algorithm to optimize the spatial positions of the feature points to obtain the local mapping thread; Matching constant speed model tracking, reference frame model tracking and relocation tracking modes for the local mapping thread; inputting the image after the spatial position is restored into the local mapping thread to obtain mapping feature points; Determine whether a keyframe needs to be inserted based on the mapping situation and the distance and ID of each camera frame, and then insert a new keyframe; The 3D positions of the mapping feature points are normalized to complete the pipeline map construction.

2. The sparse mapping method for drainage pipelines in a weak light environment based on a pipeline robot according to claim 1, characterized in that: The enhanced image is processed by a frame processing thread, a single picture is composed, and image parameters are obtained, including: The enhanced image is input into the frame processing thread of the composition framework, an image pyramid is calculated for each input image, and ORB features and descriptors are extracted to obtain image parameters.

3. The sparse mapping method for drainage pipelines in a weak light environment based on a pipeline robot according to claim 2, characterized in that: The step of inputting the image after restoring the spatial position into the local mapping thread to obtain mapping feature points includes: Input the image after restoring the spatial position into the local mapping thread, and remove the 3D points within a preset range according to the observation of the pixel 3D points; Generate new 3D points using triangulation algorithm using current keyframe and adjacent keyframes; fuse repeated 3D points of current keyframe and adjacent keyframes; The BA algorithm is used to optimize the 3D points, and the redundant key frames of the current key frame are removed according to the optimized 3D points; the current frame is added to the closed-loop detection thread to obtain the mapping feature points.

4. The sparse mapping method for drainage pipelines in a weak light environment based on a pipeline robot according to claim 3, characterized in that: The current frame is added to the closed-loop detection thread to obtain the mapping feature points, including: Add the current frame to the closed-loop detection thread, and take out the head key frame of the buffer queue as the current closed-loop detection key frame; Calculate the maximum similarity between the current keyframe and the common view keyframe, where the common view keyframe is the weight relationship between the images; Find closed-loop candidate key frames of the current key frame based on similarity relationships; Find a match between the current keyframe group and the previous consecutive keyframe group; The loop variables are maintained, and the closed-loop key frame group of the current frame is used as the closed-loop candidate key frame group before the next frame to obtain the mapping feature points.

Images