Pipeline three-dimensional model reconstruction method, device and equipment based on multi-ring laser

Abstract

The embodiment of the invention provides a pipeline three-dimensional model reconstruction method, device and equipment based on multi-ring laser. The method comprises the steps that under the condition that a laser transmitter transmits multi-ring laser, corresponding image data are obtained, and the image data are used for indicating a plurality of laser stripes obtained through reflection based on the multi-ring laser under modulation of the morphology of the inner wall of a pipeline; extracting light stripe centers of the plurality of laser stripes, and determining a light plane coefficient corresponding to each light stripe center based on the plurality of light stripe centers, a camera internal reference matrix and a position relationship between a laser plane formed by the camera and the multi-ring laser; determining three-dimensional point cloud data corresponding to the image data based on the plurality of light strip centers and the light plane coefficient corresponding to each light strip center; and reconstructing a three-dimensional model of the pipeline based on the three-dimensional point cloud data. According to the method, the efficiency of pipeline detection is improved, high-precision local point cloud reconstruction and splicing are realized, and the accuracy of overall three-dimensional reconstruction is ensured.

Description

Technical Field

[0001] The present application relates to the technical field of three-dimensional model reconstruction, and in particular to a method, device and equipment for reconstructing a three-dimensional model of a pipeline based on multi-ring lasers. Background Art

[0002] Drainage pipes play a vital role in urban water environment construction. Through their construction and maintenance, water resources can be effectively protected, sewage treatment efficiency can be improved, water pollution can be prevented, and ecological safety can be guaranteed, thereby promoting the sustainable development of the city. The drainage pipe system has always been a shortcoming in water environment construction due to its strong concealment and difficulty in detection.

[0003] At present, the three-dimensional point cloud of the pipeline is spliced ​​according to the inertial measurement unit carried by the pipeline robot to realize the three-dimensional reconstruction of the pipeline; or the coordinate system transformation of the measurement sensor is obtained by using attitude sensors combined with laser ranging devices to realize the overall three-dimensional reconstruction.

[0004] However, the 3D reconstruction of pipelines using the inertial measurement unit carried by the pipeline robot has the problem of low accuracy; the 3D reconstruction of pipelines using the coordinate system transformation of the sensor has the problem of complex equipment and is difficult to implement in drainage pipes. In the prior art, there is no further explanation of the defect assessment of pipelines based on the 3D model of the pipeline, making it difficult to effectively form opinions and guidance on the subsequent work required for the pipeline. Summary of the invention

[0005] The embodiments of the present application provide a pipeline three-dimensional model reconstruction method, device and equipment based on multi-circular laser, which improves the efficiency of pipeline detection, realizes high-precision local three-dimensional point cloud reconstruction, realizes high-precision point cloud stitching, and ensures the accuracy of overall three-dimensional reconstruction.

[0006] In a first aspect, an embodiment of the present application provides a pipeline three-dimensional model reconstruction method based on multi-ring lasers, comprising:

[0007] When the laser transmitter emits a multi-ring laser, corresponding image data is obtained, wherein the image data is used to indicate a plurality of laser stripes obtained by reflection of the multi-ring laser under modulation of the inner wall morphology of the pipeline;

[0008] Extracting the light stripe centers of the plurality of laser stripes, and determining the light plane coefficient corresponding to each light stripe center based on the positional relationship between the plurality of light stripe centers, the camera internal parameter matrix, and the laser plane formed by the camera and the multi-ring laser;

[0009] Determining three-dimensional point cloud data corresponding to the image data based on the centers of the plurality of light strips and the light plane coefficient corresponding to the center of each light strip;

[0010] Based on the three-dimensional point cloud data, a three-dimensional model of the pipeline is reconstructed.

[0011] In a possible implementation, extracting light stripe centers of a plurality of laser stripes includes:

[0012] For any laser stripe among the plurality of laser stripes, determining a laser profile pixel point set of the laser stripe;

[0013] The laser contour pixel point set is traversed, and a plurality of pixel points satisfying a preset condition are deleted from the laser contour pixel point set to obtain the light stripe center of the laser stripe.

[0014] In a possible implementation, determining three-dimensional point cloud data corresponding to the image data based on the centers of the plurality of light strips and the light plane coefficient corresponding to the center of each light strip includes:

[0015] For any one of the plurality of light stripe centers, determining a pixel coordinate of the light stripe center in the image data;

[0016] Determining the camera normalized coordinates of the center of the light strip based on the camera intrinsic parameter matrix and the pixel coordinates;

[0017] The three-dimensional point cloud data corresponding to the image data is determined according to the normalized camera coordinates of the centers of the plurality of light strips and the light plane coefficient corresponding to the center of each light strip.

[0018] In a possible implementation, the image data is multi-frame image data, and reconstructing a three-dimensional model of the pipeline based on the three-dimensional point cloud data includes:

[0019] Extracting feature points from the multiple frames of image data to obtain feature points corresponding to each frame of image data;

[0020] For any two frames of image data in the multiple frames of image data, matching processing is performed on feature points corresponding to the two frames of image data to obtain a feature point set corresponding to the two frames of image data, wherein the feature point set is used to indicate positions of the same feature point at different viewing angles;

[0021] Determining the relative position and posture corresponding to the two frames of image data according to the feature point set;

[0022] The three-dimensional model of the pipeline is reconstructed based on the three-dimensional point cloud data and the multiple relative postures corresponding to the multiple frames of image data.

[0023] In a possible implementation, determining the relative position and posture corresponding to the two frames of image data according to the feature point set includes:

[0024] Determine a basic matrix corresponding to the feature point set, wherein the basic matrix is ​​used to indicate an epipolar geometric relationship between the two frames of image data;

[0025] Determine multiple pose solutions according to the basic matrix and the camera intrinsic parameter matrix;

[0026] Back-projecting a first feature point in the feature point set according to the multiple pose solutions to obtain a position of the first feature point in three-dimensional space;

[0027] Based on the position of the first feature point in the three-dimensional space, a target pose solution is determined from the multiple pose solutions, and the relative pose is determined.

[0028] In a possible implementation, after reconstructing the three-dimensional model of the pipeline based on the three-dimensional point cloud data, the method further includes:

[0029] Performing closed-loop detection on the three-dimensional model, and when it is determined that a closed loop is detected, performing geometric consistency verification to obtain a verification result;

[0030] When the verification result indicates that the three-dimensional model has a reprojection error, the three-dimensional model is optimized based on the reprojection error to obtain an optimized three-dimensional model.

[0031] In a possible implementation, after reconstructing the three-dimensional model of the pipeline based on the three-dimensional point cloud data, the method further includes:

[0032] The three-dimensional model is subjected to geometric deformation analysis and surface defect detection to obtain a health result of the three-dimensional model, wherein the health result is used to indicate whether the three-dimensional model has geometric deformation and / or surface defects.

[0033] In a second aspect, an embodiment of the present application provides a pipeline three-dimensional model reconstruction device based on multi-ring lasers, comprising:

[0034] An acquisition module, used for acquiring corresponding image data when the laser transmitter emits multiple circular lasers, wherein the image data is used for indicating multiple laser stripes obtained by reflection of the multiple circular lasers under the modulation of the inner wall morphology of the pipeline;

[0035] A processing module, used to extract the light stripe centers of the multiple laser stripes, and determine the light plane coefficient corresponding to each light stripe center based on the multiple light stripe centers, the camera internal parameter matrix and the positional relationship between the camera and the laser plane formed by the multi-ring laser;

[0036] The processing module is further used to determine the three-dimensional point cloud data corresponding to the image data based on the centers of the multiple light strips and the light plane coefficient corresponding to the center of each light strip;

[0037] A reconstruction module is used to reconstruct the three-dimensional model of the pipeline based on the three-dimensional point cloud data.

[0038] In a possible implementation, the device includes: an extraction module;

[0039] The extraction module is used to determine the laser contour pixel point set of any laser stripe among multiple laser stripes; traverse the laser contour pixel point set, delete multiple pixel points that meet preset conditions from the laser contour pixel point set, and obtain the light stripe center of the laser stripe.

[0040] In a possible implementation, the processing module is also used to determine the pixel coordinates of the center of any one of the multiple light strip centers in the image data; determine the camera normalized coordinates of the center of the light strip based on the camera intrinsic parameter matrix and the pixel coordinates; and determine the three-dimensional point cloud data corresponding to the image data based on the camera normalized coordinates of the multiple light strip centers and the light plane coefficient corresponding to the center of each light strip.

[0041] In a possible implementation, the image data is multiple frames of image data, and the reconstruction module is further used to extract feature points from the multiple frames of image data to obtain feature points corresponding to each frame of image data; for any two frames of image data in the multiple frames of image data, matching processing is performed on the feature points corresponding to the two frames of image data to obtain a set of feature points corresponding to the two frames of image data, wherein the set of feature points is used to indicate the position of the same feature point under different viewing angles; based on the set of feature points, a relative posture corresponding to the two frames of image data is determined; and based on the three-dimensional point cloud data and multiple relative postures corresponding to the multiple frames of image data, a three-dimensional model of the pipeline is reconstructed.

[0042] In a possible implementation, the reconstruction module is also used to determine a basic matrix corresponding to a set of feature points, wherein the basic matrix is ​​used to indicate an epipolar geometric relationship between the two frames of image data; multiple pose solutions are determined based on the basic matrix and the camera intrinsic parameter matrix; based on the multiple pose solutions, a first feature point in the set of feature points is back-projected to obtain a position of the first feature point in three-dimensional space; based on the position of the first feature point in three-dimensional space, a target pose solution is determined from the multiple pose solutions, and the relative pose is determined.

[0043] In a possible implementation, the device includes: an optimization module;

[0044] The optimization module is used to perform closed-loop detection on the three-dimensional model, and when it is determined that a closed loop is detected, perform geometric consistency verification to obtain a verification result; when the verification result indicates that the three-dimensional model has a reprojection error, the three-dimensional model is optimized based on the reprojection error to obtain an optimized three-dimensional model.

[0045] In a possible implementation, the device includes: an evaluation module;

[0046] The evaluation module is used to perform geometric deformation analysis and surface defect detection on the three-dimensional model to obtain a health result of the three-dimensional model, and the health result is used to indicate whether the three-dimensional model has geometric deformation and / or surface defects.

[0047] In a third aspect, an embodiment of the present application provides a pipeline three-dimensional model reconstruction device based on multi-ring lasers, including: a memory, a processor;

[0048] Memory stores computer-executable instructions;

[0049] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and various possible implementations of the first aspect.

[0050] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the above first aspect and various possible implementation methods of the first aspect.

[0051] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and various possible implementation methods of the first aspect.

[0052] The embodiments of the present application provide a method, device and equipment for reconstructing a three-dimensional model of a pipeline based on multi-ring lasers. The method, device and equipment extract the light stripe centers of multiple laser stripes from the image data of the pipeline acquired by a camera. The internal parameter matrix and light plane coefficients in the calibration results are obtained by calibrating according to the preset positions of the camera and the multi-ring laser emitter, so as to realize the coordinate transformation of the light stripe centers and determine the three-dimensional point cloud data corresponding to the image data. The multi-ring laser technology is used to simultaneously obtain information of multiple cross-sections of the pipeline, thereby improving the efficiency of pipeline detection. High-precision local three-dimensional point cloud reconstruction is realized through precise laser plane calibration and fast parallel image centerline extraction algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0054] Figure 1 A schematic diagram of a scenario of a pipeline 3D model reconstruction method based on multi-ring lasers provided in an embodiment of the present application;

[0055] Figure 2 A schematic diagram of a pipeline 3D model reconstruction method based on multi-ring laser provided in an embodiment of the present application Figure 1 ;

[0056] Figure 3 A schematic diagram of a pipeline 3D model reconstruction method based on multi-ring laser provided in an embodiment of the present application Figure 2 ;

[0057] Figure 4 A schematic diagram of camera viewing angles at different times provided in an embodiment of the present application;

[0058] Figure 5 A schematic diagram of the structure of a pipeline three-dimensional model reconstruction device based on multi-ring lasers provided in an embodiment of the present application;

[0059] Figure 6 A schematic diagram of the structure of a pipeline three-dimensional model reconstruction device based on multi-ring lasers provided in an embodiment of the present application.

[0060] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] "Multiple" means two or more, and other quantifiers are similar. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0063] In the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0064] Drainage pipes are an important part of urban water environment construction, and their efficient operation is crucial to urban health. However, the drainage pipe system has always been a shortcoming in water environment construction due to its strong concealment and difficulty in detection.

[0065] At present, the three-dimensional point cloud of the pipeline is usually spliced ​​according to the inertial measurement unit carried by the pipeline robot to achieve three-dimensional reconstruction of the pipeline; or the coordinate system transformation of the measurement sensor is obtained by using a posture sensor combined with a laser ranging device to achieve overall three-dimensional reconstruction.

[0066] Figure 1 A schematic diagram of a scene of a pipeline 3D model reconstruction method based on multi-ring lasers provided in an embodiment of the present application, such as Figure 1 As shown, the pipeline reconstruction system 10 is applied to a pipeline 11 , and the pipeline reconstruction system 10 includes a pipeline robot 101 , a camera 102 , a lighting device 103 , and a multi-ring laser emitter 104 .

[0067] The pipeline robot 101 is equipped with a camera 102, a lighting device 103 and a laser emitter 104 and moves forward inside the pipeline 11. The lighting device 103 is responsible for providing lighting of a certain brightness. The laser emitter 104 projects multi-circular lasers 12 onto the inner wall of the pipeline. The camera 102 obtains image data of the pipeline 11 and performs three-dimensional point cloud computing based on the image data to reconstruct a three-dimensional model of the pipeline 10.

[0068] Combining the above scenarios, it can be seen that in the existing technology, the three-dimensional reconstruction of the pipeline through the inertial measurement unit carried by the pipeline robot has the problem of low accuracy; the three-dimensional reconstruction of the pipeline through the coordinate system transformation of the sensor has the problem of complex equipment and difficult to achieve in the drainage pipeline. In addition, there are currently few three-dimensional models based on pipelines that further explain the defect assessment of the pipeline. Therefore, it is difficult to effectively form opinions and guidance on the subsequent work required for the pipeline.

[0069] The present application provides a pipeline three-dimensional model reconstruction method based on multi-circular laser. Through the laser transmitter, camera and lighting device carried by the pipeline robot, coordinate system transformation is realized according to the camera's posture information, thereby solving the problems of low accuracy and / or complex device in pipeline three-dimensional reconstruction.

[0070] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0071] Figure 2 A schematic diagram of a pipeline 3D model reconstruction method based on multi-ring laser provided in an embodiment of the present application Figure 1 The execution subject of this embodiment may be, for example, Figure 1 The camera shown, Figure 2 As shown, the method includes:

[0072] S201. When a laser transmitter emits a multi-ring laser, corresponding image data is acquired, where the image data is used to indicate a plurality of laser stripes obtained by reflection of the multi-ring laser under modulation of the inner wall morphology of a pipeline.

[0073] Among them, the laser transmitter projects multiple beams of annular light, i.e., multi-ring lasers, on the pipe cross section to illuminate the pipe cross section; the image data is the contour information of the inner wall of the pipe.

[0074] like Figure 1 As shown, when the pipeline robot moves in the pipeline, the laser transmitter it carries projects multiple beams of annular light, that is, multi-circular laser, on the cross section of the pipeline. The multi-circular laser will form multiple laser stripes when reflected under the modulation of the pipeline inner wall morphology; the camera it carries will obtain the contour information of the pipeline inner wall.

[0075] The laser emitter can be, for example, a slit-type generator, a multi-beam interference-type generator, or a conical reflection-type generator. The quality of the annular light is one of the direct factors affecting the measurement accuracy. The thinner and sharper the line width, the more accurate the light projection, the smaller the feedback distortion of the cross-sectional rifling information, and the more accurate the measurement. Considering the range requirements, space limitations, cost control, and other aspects of the pipeline measurement scenario, the conical reflection-type generator is preferred.

[0076] The camera can be, for example, a camera of a CMOS image sensor, a CCD image sensor. A camera of a CMOS image sensor is preferred here, because it has lower cost, power consumption, higher resolution, frame rate, sensitivity, wider dynamic range, and a higher degree of system integration.

[0077] S202, extracting the light stripe centers of the multiple laser stripes, and determining the light plane coefficient corresponding to each light stripe center based on the multiple light stripe centers, the camera internal parameter matrix, and the positional relationship between the camera and the laser plane formed by the multi-ring laser.

[0078] Among them, the light stripe center is the center line of the laser stripe; the camera intrinsic parameter matrix describes the relationship between the object point and the image point; the light plane coefficient describes the position of the circular laser plane.

[0079] Before extracting the center of the light strip, the relative position of the circular laser transmitter and the camera needs to be adjusted to a suitable position to ensure that the multiple circular lasers projected by the circular laser transmitter can be received by the receiver. After determining the position, the Zhang Zhengyou calibration algorithm is preferably used for calibration to obtain the calibration results of the camera, including the intrinsic parameter matrix, rotation matrix, and translation vector. The following formula is then used to further calibrate the laser plane of the system composed of the camera and multiple circular lasers to determine the light plane coefficients of multiple circular laser planes:

[0080]

[0081] Among them, r1, r2, ..., r n represents the equation of each ring laser plane, x c ,y c ,z c represents the camera coordinates of any point on the circular laser plane. The camera coordinates of this point are determined according to the rotation matrix and translation matrix of the camera; a1,...,a n ,b1,...,b n ,c1,...,c n ,d1,...,d n Represents the light plane coefficient, which is obtained by circular laser plane calibration.

[0082] The center lines of multiple laser stripes are extracted to obtain the corresponding centers of multiple light stripes. Based on the multi-line laser matching method of reprojection and reverse reprojection, the position of the circular laser plane where each light stripe center is located is determined, thereby determining the light plane coefficient corresponding to each light stripe center.

[0083] In a possible implementation, extracting the light stripe center of a plurality of laser stripes is described in detail, including: determining a laser contour pixel point set of any laser stripe among the plurality of laser stripes; traversing the laser contour pixel point set, deleting a plurality of pixel points that meet preset conditions from the laser contour pixel point set, and obtaining the light stripe center of the laser stripe.

[0084] The laser contour pixel point set represents all the circular laser contour pixel points.

[0085] Preferably, the light strip center is determined by a fast parallel image centerline extraction iterative algorithm, and the preset conditions are as follows:

[0086] ①2≤B≤6;

[0087] ②A=1;

[0088] ③P1*P3*P5=0 and P3*P5*P7=0;

[0089] ④P1*P3*P7=0 and P1*P5*P7=0

[0090] Among them, a 3*3 sized structural element is specified, the pixel in the upper left corner of the structural element is taken as P1, and the other 7 pixels are named clockwise to obtain P n , n=1,2,...,8 are the values ​​of 8 neighboring pixels. Traverse all the ring laser contour pixels and delete the pixels that meet the above conditions ①, ②, ③ or ①, ②, ④.

[0091] For any one of the multiple laser stripes, all the circular laser contour pixel points are traversed according to preset conditions, and multiple pixel points that meet the preset conditions are deleted from the laser contour pixel point set until there are no pixels to be deleted. The loop is terminated, and a uniform, refined, single-pixel-width light strip center is obtained.

[0092] Preferably, before extracting the light stripe centers of the plurality of laser stripes, image enhancement processing and image noise reduction processing may be performed on the image data.

[0093] The image data is enhanced through the top-hat operation. Through the opening operation of the top-hat operation, that is, corrosion followed by expansion, the smaller bright details are removed while maintaining the overall grayscale of the image and the characteristics of the larger bright area. The darker background area is weakened in different grayscale distributions, the influence of uneven lighting is corrected, and the edge details are guaranteed.

[0094] Threshold processing and median filtering are used to perform image denoising on image data. The average grayscale of the image after image enhancement is used as the initial value, and two groups of pixels are generated by segmentation. The average grayscale of the two groups is calculated respectively, and the mean of the two groups of average grayscale is used as the updated threshold. The method is continuously iterated until the change value is less than the preset value to further process the background noise.

[0095] S203: Determine three-dimensional point cloud data corresponding to the image data based on the centers of the multiple light strips and the light plane coefficient corresponding to the center of each light strip.

[0096] The three-dimensional point cloud data is used to represent the coordinate points in the three-dimensional space of the inner wall of the pipeline corresponding to the image data.

[0097] According to the light strip center and the light plane coefficient corresponding to each light strip center, the coordinate point in the three-dimensional space of the inner wall of the pipeline corresponding to the image data is determined.

[0098] In a possible implementation, determining the three-dimensional point cloud data corresponding to the image data based on the centers of the plurality of light strips and the light plane coefficient corresponding to the center of each light strip is described in detail, including:

[0099] For any one of the multiple light strip centers, determine the pixel coordinates of the light strip center in the image data; based on the camera intrinsic parameter matrix and the pixel coordinates, determine the camera normalized coordinates of the light strip center; and according to the camera normalized coordinates of the multiple light strip centers and the light plane coefficient corresponding to each light strip center, determine the three-dimensional point cloud data corresponding to the image data.

[0100] The pixel coordinates are the positions of the pixels in the image data; the camera normalized coordinates are the coordinates in the normalized coordinate system after the camera coordinate system is normalized.

[0101] After the light strip center is extracted, the camera normalized coordinates of the light strip center are determined using the following formula based on the pixel coordinates (u, v) of the light strip center:

[0102]

[0103] Among them, A is the camera internal parameter matrix, (X n ,Y n ) are the coordinates in the normalized coordinate system.

[0104] The following formula is used to determine the three-dimensional coordinates corresponding to the center of each light strip according to the light plane coefficient corresponding to the center of each light strip and the normalized coordinates of the camera:

[0105]

[0106] Among them, x c ,y c 、z c , is the three-dimensional coordinate corresponding to the center of the light bar, a i , b i 、c i ,d i is the light plane coefficient of the circular laser plane i where the light strip is located.

[0107] S204: Reconstruct a three-dimensional model of the pipeline based on the three-dimensional point cloud data.

[0108] Based on the three-dimensional point cloud data corresponding to the image data acquired by the camera, the three-dimensional model of the pipeline is reconstructed.

[0109] The embodiment of the present application provides a method for reconstructing a three-dimensional model of a pipeline based on multi-ring lasers. By adopting multi-ring laser technology according to the image data of the pipeline acquired by a camera, it is possible to simultaneously acquire information of multiple cross-sections of the pipeline, thereby improving the efficiency of pipeline detection. The light stripe centers of multiple laser stripes are extracted by a fast parallel image centerline extraction algorithm, and the light plane coefficients of the light stripe centers are determined in combination with the results of accurate laser plane calibration, thereby achieving high-precision local three-dimensional point cloud reconstruction. Based on the local three-dimensional point cloud, the three-dimensional model of the pipeline is reconstructed, thereby achieving high-precision point cloud splicing and ensuring the accuracy of the overall three-dimensional reconstruction.

[0110] Figure 3 A schematic diagram of a pipeline 3D model reconstruction method based on multi-ring laser provided in an embodiment of the present application Figure 2 In this embodiment Figure 2 Based on the embodiment, a possible implementation method of reconstructing the three-dimensional model of the pipeline based on the three-dimensional point cloud data is described in detail, such as Figure 3 As shown, the method includes:

[0111] S301 . The image data is multiple frames of image data. Feature points are extracted from the multiple frames of image data to obtain feature points corresponding to each frame of image data.

[0112] Among them, the multi-frame image data is the multi-frame image data of the inner wall of the pipeline acquired at different times when the pipeline robot moves in the pipeline; the feature point is the point with unique local features in the image data.

[0113] The feature points in each frame are extracted by the ORB algorithm (Oriented FAST and Rotated BRIEF). The ORB algorithm combines FAST corner detection (Features from Accelerated Segment Test) with binary robust independent basic feature descriptors, which can efficiently detect significant features in the image and generate rotation-invariant binary descriptors for each feature point.

[0114] S302 . For any two frames of image data among the multiple frames of image data, matching processing is performed on feature points corresponding to the two frames of image data to obtain a set of feature points corresponding to the two frames of image data.

[0115] Among them, the feature point set is the corresponding feature point pairs under different viewing angles.

[0116] Figure 4 A schematic diagram of a camera viewing angle at different times provided in an embodiment of the present application.

[0117] For example, Figure 4As shown in the figure, at point O in the inner wall of the pipeline, at time T1, the camera acquires image data A. After the pipeline robot moves for a period of time, at time T2, the camera acquires image data B. The feature points in image data A and image data B are extracted by the ORB algorithm, and the binary descriptors corresponding to the feature points are generated. The similarity between the descriptors is calculated to find the corresponding feature point sets under different viewing angles. The FLANN (Fast Library for Approximate Nearest Neighbors) algorithm can be selected to accelerate the matching process, which is especially suitable for matching large-scale feature point data.

[0118] S303: Determine a basic matrix corresponding to the feature point set.

[0119] The basic matrix is ​​used to indicate the epipolar geometric relationship between the two frames of image data.

[0120] For a pair of corresponding points x1 and x2 on two frames of image data, the basic matrix satisfies the following equation:

[0121]

[0122] Among them, x1 and x2 are the pixel coordinates of a pair of corresponding points in two frames of image data; F is the basic matrix.

[0123] The basic matrix is ​​solved by combining the eight-point method with the Random Sample Consensus (RANSAC) algorithm. By randomly selecting feature point pairs for model fitting multiple times, the RANSAC algorithm can identify the inliers that meet the geometric constraints and remove those incorrect matching points (outliers) that do not meet the geometric constraints. The algorithm iterates continuously until the best basic matrix is ​​found and the number of inliers is maximized. First, 8 pairs of matching points are randomly selected to calculate the basic matrix, and the epipolar constraints of other matching point pairs are evaluated to filter out the inliers. Then, this step is repeated multiple times, and the basic matrix with the most inliers is selected as the final result, and the matrix is ​​re-estimated to improve accuracy.

[0124] S304: Determine a plurality of pose solutions according to the basic matrix and the camera intrinsic parameter matrix.

[0125] Among them, the pose solution indicates the position and posture of the pipeline robot, including the rotation matrix R and the translation vector t.

[0126] The following formula is used to calculate the essential matrix E based on the internal parameter matrix A:

[0127] E=A T ·F·A

[0128] Perform singular value decomposition (SVD) on the essential matrix to further solve the rotation matrix R and translation vector t:

[0129] E=U·Σ·V T

[0130] Among them, U and V are orthogonal matrices, and Σ is a diagonal matrix (whose non-zero singular values ​​are 1, 1, 0).

[0131] It can be understood that the singular value decomposition of the essential matrix may produce four different pose solutions.

[0132] S305 . Back-project a first feature point in the feature point set according to the multiple pose solutions to obtain a position of the first feature point in three-dimensional space.

[0133] The first feature point is a feature point used as a reference in a feature point set for determining a posture solution.

[0134] According to the multiple posture solutions, the first feature point is back-projected to deduce the position of the first feature point in the three-dimensional space obtained according to the posture solution.

[0135] The depth scalar is obtained by triangulation using the following formula:

[0136] P=λ1A -1 ·x1=λ2R·A -1 ·x2+t

[0137] Among them, λ1 and λ2 are depth scalars that reflect the distance between the feature point and the camera, and x1 and x2 are the pixel coordinates under two viewing angles.

[0138] S306: Based on the position of the first feature point in the three-dimensional space, determine a target pose solution from the multiple pose solutions, and determine the relative pose.

[0139] Among them, the target pose solution is the correct pose solution for the pose transformation of the feature points in the feature point set; the relative pose indicates the pose change of the camera when acquiring different frame image data.

[0140] The depth scalar is obtained by triangulation, and the pose solution that makes all depth scalars positive is selected and used as the target pose solution.

[0141] Two feature points in the feature point set correspond to two frames of image data respectively. The camera has a different posture when acquiring each frame of image data. According to the posture solution, the relative posture of the camera is further obtained.

[0142] S307: Reconstruct the three-dimensional model of the pipeline based on the three-dimensional point cloud data and the multiple relative postures corresponding to the multiple frames of image data.

[0143] Based on the three-dimensional point cloud data and multiple relative poses corresponding to the multi-frame image data, the local point cloud is converted to the global coordinate system, the local point cloud is spliced, and the three-dimensional model of the pipeline is reconstructed.

[0144] In order to speed up the registration process of point cloud data, a space segmentation algorithm such as a KD tree (K-Dimension tree) or an octree can be used. For example, a weighted average method or a voxel gridding method can be used to fuse the point clouds in overlapping areas to reduce noise and redundant data and improve the accuracy of the final point cloud.

[0145] In a possible implementation, after reconstructing the three-dimensional model of the pipeline based on the three-dimensional point cloud data, the optimization process of the reconstructed three-dimensional model is described in detail:

[0146] A closed-loop detection is performed on the three-dimensional model, and when a closed loop is determined to be detected, a geometric consistency verification is performed to obtain a verification result; when the verification result indicates that a reprojection error exists in the three-dimensional model, the three-dimensional model is optimized based on the reprojection error to obtain an optimized three-dimensional model.

[0147] Among them, the main purpose of closed-loop detection is to detect whether the pipeline robot revisits a place it has visited before, that is, to detect whether the camera repeatedly obtains image data of the same scene.

[0148] A closed-loop detection is performed on the three-dimensional model of the pipeline to detect whether the camera repeatedly obtains image data of the same scene, that is, to detect whether there are repeated pipe sections corresponding to the same scene that are repeatedly spliced. Based on the closed-loop detection, the closed-loop detection result is obtained.

[0149] When a closed loop is detected, a further geometric consistency verification is performed, the purpose of which is to eliminate possible false matches in the closed loop detection and obtain a verification result.

[0150] When the verification results show that the 3D model has reprojection errors, the 3D model is optimized using a pose graph optimization algorithm.

[0151] In one possible implementation, after reconstructing the 3D model of the pipeline based on the 3D point cloud data, the process of health assessment of the 3D pipeline is described in detail:

[0152] The three-dimensional model is subjected to geometric deformation analysis and surface defect detection to obtain a health result of the three-dimensional model, wherein the health result is used to indicate whether the three-dimensional model has geometric deformation and / or surface defects.

[0153] The following is a detailed description of the process of geometric deformation analysis:

[0154] The RANSAC algorithm can be used to robustly fit the centerline of the pipeline in the presence of a large amount of noise and outliers. Based on the centerline of the pipeline, the cross-sectional point cloud is extracted at fixed intervals, and the following formula is used to perform circular fitting on each cross-sectional point cloud:

[0155]

[0156] Among them, (x i ,y i ) is the coordinate of the cross-section point cloud, (x0, y0) is the coordinate of the fitting circle center, and r is the radius of the fitting circle. These fitted circular cross-section data are an important basis for geometric deformation analysis.

[0157] Ovality is used to determine the degree of deformation of the pipe. The following formula is used to score the ovality:

[0158]

[0159] Where, ΔE max It represents the maximum allowable ellipticity error, preferably set to 0.5; E is the ellipticity; D max ,D min They represent the maximum and minimum curvature diameters of the ellipse respectively.

[0160] The cross-sectional area change rate is used to identify sudden changes or blocked areas in the pipeline. The cross-sectional area change rate is scored using the following formula:

[0161]

[0162] Where ΔA max represents the maximum allowable cross-sectional area change rate, preferably set to 20%; ΔA is the cross-sectional area change rate; A i , A i+1 is the area of ​​two adjacent cross sections.

[0163] The following is a detailed description of the surface defect detection process:

[0164] Surface defect detection is used to identify surface defects with complex shapes such as cracks and holes. The following formula is used to score surface defects:

[0165]

[0166] Among them, A d is the defect area; ΔA d is the maximum allowable defect area; H d is the defect depth; ΔH d The maximum allowable defect depth is the maximum allowable defect area that can be set based on the pipe diameter, and the maximum allowable defect depth that can be set based on the pipe wall thickness. No special requirements are made here.

[0167] Health outcomes are calculated using the following formula:

[0168] Score=w1·S ellipticity +w2·S area +w3·S defect

[0169] Among them, w1, w2, and w3 represent weight coefficients, which can be preferably set to 0.4, 0.3, and 0.3, for example, and can be adjusted according to the specific use environment and maintenance requirements of the pipeline.

[0170] Through the above scoring criteria and formulas, the health results are obtained and a comprehensive assessment of the health status of the pipeline is conducted.

[0171] An embodiment of the present application provides a method for reconstructing a three-dimensional model of a pipeline based on multi-circular lasers. The method extracts feature points of multiple frame images, calculates the similarity between binary descriptors of the feature points, matches feature points of different frame images, determines a feature point set, calculates the relative position and posture of the camera based on the feature point set, and reconstructs the three-dimensional model of the pipeline based on multiple relative postures corresponding to the three-dimensional point cloud data and the multi-frame image data, thereby achieving high-precision point cloud stitching and ensuring the accuracy of the overall three-dimensional reconstruction.

[0172] Figure 5 A schematic diagram of the structure of a pipeline 3D model reconstruction device based on multi-ring laser provided in this application, such as Figure 5 As shown, a pipeline 3D model reconstruction device 50 based on multi-ring lasers provided in this embodiment includes:

[0173] An acquisition module 501 is used to acquire corresponding image data when the laser transmitter emits multiple ring lasers, wherein the image data is used to indicate multiple laser stripes obtained by reflection of the multiple ring lasers under the modulation of the inner wall morphology of the pipeline;

[0174] A processing module 502 is used to extract the light stripe centers of the multiple laser stripes, and determine the light plane coefficient corresponding to each light stripe center based on the multiple light stripe centers, the camera internal parameter matrix, and the positional relationship between the camera and the laser plane formed by the multi-ring laser;

[0175] The processing module 502 is further used to determine the three-dimensional point cloud data corresponding to the image data based on the centers of the multiple light strips and the light plane coefficient corresponding to the center of each light strip;

[0176] The reconstruction module 503 is used to reconstruct the three-dimensional model of the pipeline based on the three-dimensional point cloud data.

[0177] In a possible implementation, the device includes: an extraction module;

[0178] The extraction module is used to determine the laser contour pixel point set of any laser stripe among multiple laser stripes; traverse the laser contour pixel point set, delete multiple pixel points that meet preset conditions from the laser contour pixel point set, and obtain the light stripe center of the laser stripe.

[0179] In a possible implementation, the processing module 502 is also used to determine the pixel coordinates of the center of any one of the multiple light strip centers in the image data; determine the camera normalized coordinates of the center of the light strip based on the camera intrinsic parameter matrix and the pixel coordinates; and determine the three-dimensional point cloud data corresponding to the image data based on the camera normalized coordinates of the multiple light strip centers and the light plane coefficient corresponding to the center of each light strip.

[0180] In a possible implementation, the image data is multi-frame image data, and the reconstruction module 503 is also used to extract feature points from the multi-frame image data to obtain feature points corresponding to each frame of image data; for any two frames of image data in the multi-frame image data, matching processing is performed on the feature points corresponding to the two frames of image data to obtain a feature point set corresponding to the two frames of image data, and the feature point set is used to indicate the position of the same feature point under different viewing angles; according to the feature point set, the relative posture corresponding to the two frames of image data is determined; based on the three-dimensional point cloud data and the multiple relative postures corresponding to the multi-frame image data, the three-dimensional model of the pipeline is reconstructed.

[0181] In one possible implementation, the reconstruction module 503 is also used to determine a basic matrix corresponding to the feature point set, wherein the basic matrix is ​​used to indicate the epipolar geometric relationship between the two frames of image data; multiple pose solutions are determined based on the basic matrix and the camera intrinsic parameter matrix; based on the multiple pose solutions, a first feature point in the feature point set is back-projected to obtain a position of the first feature point in three-dimensional space; based on the position of the first feature point in three-dimensional space, a target pose solution is determined from the multiple pose solutions, and the relative pose is determined.

[0182] In a possible implementation, the device includes: an optimization module;

[0183] The optimization module is used to perform closed-loop detection on the three-dimensional model, and when it is determined that a closed loop is detected, perform geometric consistency verification to obtain a verification result; when the verification result indicates that the three-dimensional model has a reprojection error, the three-dimensional model is optimized based on the reprojection error to obtain an optimized three-dimensional model.

[0184] In a possible implementation, the device includes: an evaluation module;

[0185] The evaluation module is used to perform geometric deformation analysis and surface defect detection on the three-dimensional model to obtain a health result of the three-dimensional model, and the health result is used to indicate whether the three-dimensional model has geometric deformation and / or surface defects.

[0186] The present embodiment provides a pipeline three-dimensional model reconstruction device based on multi-ring lasers, which can execute the method provided by the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0187] Figure 6 This is a schematic diagram of the structure of a pipeline 3D model reconstruction device based on multi-ring laser provided in this application. Figure 6 As shown, the electronic device 60 provided in this embodiment includes: at least one processor 601 and a memory 602. Optionally, the device 60 also includes a communication component 603. The processor 601, the memory 602 and the communication component 603 are connected via a bus 604.

[0188] In a specific implementation process, at least one processor 601 executes the computer execution instructions stored in the memory 602, so that at least one processor 601 executes the above-mentioned pipeline three-dimensional model reconstruction method based on multi-ring laser.

[0189] The specific implementation process of the processor 601 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0190] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0191] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0192] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0193] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned pipeline three-dimensional model reconstruction method based on multi-ring laser.

[0194] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above-mentioned pipeline three-dimensional model reconstruction method based on multi-ring laser is implemented.

[0195] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0196] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0197] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0198] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0199] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0200] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0201] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0202] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A pipeline three-dimensional model reconstruction method based on multi-ring laser, characterized in that: The method comprises: When the laser transmitter emits a multi-ring laser, corresponding image data is obtained, wherein the image data is used to indicate a plurality of laser stripes obtained by reflection of the multi-ring laser under modulation of the inner wall morphology of the pipeline; Extracting the light stripe centers of the plurality of laser stripes, and determining the light plane coefficient corresponding to each light stripe center based on the positional relationship between the plurality of light stripe centers, the camera internal parameter matrix, and the laser plane formed by the camera and the multi-ring laser; Determining three-dimensional point cloud data corresponding to the image data based on the centers of the plurality of light strips and the light plane coefficient corresponding to the center of each light strip; Based on the three-dimensional point cloud data, a three-dimensional model of the pipeline is reconstructed.

2. The method according to claim 1, characterized in that The step of extracting the light stripe centers of the plurality of laser stripes comprises: For any laser stripe among the plurality of laser stripes, determining a laser profile pixel point set of the laser stripe; The laser contour pixel point set is traversed, and a plurality of pixel points satisfying a preset condition are deleted from the laser contour pixel point set to obtain the light stripe center of the laser stripe.

3. The method according to claim 1, characterized in that The determining of the three-dimensional point cloud data corresponding to the image data based on the centers of the plurality of light strips and the light plane coefficient corresponding to the center of each light strip comprises: For any one of the plurality of light stripe centers, determining a pixel coordinate of the light stripe center in the image data; Determining the camera normalized coordinates of the center of the light strip based on the camera intrinsic parameter matrix and the pixel coordinates; The three-dimensional point cloud data corresponding to the image data is determined according to the normalized camera coordinates of the centers of the plurality of light strips and the light plane coefficient corresponding to the center of each light strip.

4. The method according to claim 1, characterized in that: The image data is multi-frame image data, and the three-dimensional model of the pipeline is reconstructed based on the three-dimensional point cloud data, including: Extracting feature points from the multiple frames of image data to obtain feature points corresponding to each frame of image data; For any two frames of image data in the multiple frames of image data, matching processing is performed on feature points corresponding to the two frames of image data to obtain a feature point set corresponding to the two frames of image data, wherein the feature point set is used to indicate positions of the same feature point at different viewing angles; Determining the relative position and posture corresponding to the two frames of image data according to the feature point set; The three-dimensional model of the pipeline is reconstructed based on the three-dimensional point cloud data and the multiple relative postures corresponding to the multiple frames of image data.

5. The method according to claim 4, characterized in that Determining the relative position and posture corresponding to the two frames of image data according to the feature point set includes: Determine a basic matrix corresponding to the feature point set, wherein the basic matrix is ​​used to indicate an epipolar geometric relationship between the two frames of image data; Determine a plurality of pose solutions according to the basic matrix and the camera intrinsic parameter matrix; Back-projecting a first feature point in the feature point set according to the multiple pose solutions to obtain a position of the first feature point in three-dimensional space; Based on the position of the first feature point in the three-dimensional space, a target pose solution is determined from the multiple pose solutions, and the relative pose is determined.

6. The method according to claim 1, characterized in that After reconstructing the three-dimensional model of the pipeline based on the three-dimensional point cloud data, the method further includes: Performing closed-loop detection on the three-dimensional model, and when it is determined that a closed loop is detected, performing geometric consistency verification to obtain a verification result; When the verification result indicates that the three-dimensional model has a reprojection error, the three-dimensional model is optimized based on the reprojection error to obtain an optimized three-dimensional model.

7. The method according to claim 1, characterized in that After reconstructing the three-dimensional model of the pipeline based on the three-dimensional point cloud data, the method further includes: The three-dimensional model is subjected to geometric deformation analysis and surface defect detection to obtain a health result of the three-dimensional model, wherein the health result is used to indicate whether the three-dimensional model has geometric deformation and / or surface defects.

8. A pipeline three-dimensional model reconstruction device based on multi-ring laser, characterized in that: include: An acquisition module, used for acquiring corresponding image data when the laser transmitter emits multiple circular lasers, wherein the image data is used for indicating multiple laser stripes obtained by reflection of the multiple circular lasers under the modulation of the inner wall morphology of the pipeline; A processing module, used to extract the light stripe centers of the multiple laser stripes, and determine the light plane coefficient corresponding to each light stripe center based on the multiple light stripe centers, the camera internal parameter matrix and the positional relationship between the camera and the laser plane formed by the multi-ring laser; The processing module is further used to determine the three-dimensional point cloud data corresponding to the image data based on the centers of the multiple light strips and the light plane coefficient corresponding to the center of each light strip; A reconstruction module is used to reconstruct the three-dimensional model of the pipeline based on the three-dimensional point cloud data.

9. A pipeline three-dimensional model reconstruction device based on multi-ring laser, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the pipeline three-dimensional model reconstruction method based on multi-ring laser as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the pipeline three-dimensional model reconstruction method based on multi-ring laser as described in any one of claims 1 to 7.

Images