A method and system for stitching pipeline images using dual cameras

Abstract

The present invention discloses a method and system for stitching pipeline images by using two cameras. The method includes: determining whether the original images acquired by the two cameras are excessively deformed; if the original images are not excessively deformed, aligning the two original images according to the triangles in each original image; scaling the aligned images according to a set ratio; intercepting the largest regions of the two images so that three points in each image overlap with the three points of the constructed triangle; calculating the overlapping regions to obtain the final image. By setting a threshold during image stitching, the present invention screens out images with relatively large distortions to a certain extent, and at the same time uses the geometric characteristics of the triangular reference objects to align the images, which can solve common basic stitching problems and will not cause great damage to the images themselves; the present invention effectively reduces the overall complexity of image stitching, leaves room for subsequent image processing work, and reduces the overall processing time cost.

Description

Technical Field

[0001] The present invention belongs to the field of image stitching, and particularly relates to a method and system for stitching pipeline images using dual cameras. Background Art

[0002] Image stitching is a technology that combines several overlapping images into a seamless panoramic image or a high-resolution image. Currently, image stitching is still an important prerequisite for processing large-scale scene images. With the help of deep learning and neural networks, image processing technology has developed by leaps and bounds, with higher and higher accuracy, but the complexity has also increased accordingly.

[0003] In the prior art, there are generally two methods for processing the image stitching part in the field of image processing. One method is to extract the feature points of each image, perform operations such as feature point matching and elimination between multiple images, and finally perform weighted fusion to obtain the stitched image. The method was elaborated in the "Review of the Development of Image Stitching Algorithms" (Luo Yun, He Xiang, Ding Shijie. Review of the Development of Image Stitching Algorithms [J]. Modern Computer, 2021(8):5.) published in 2021. The patent with the publication number CN112017114A uses this feature point matching method to match the feature points in the overlapping area of two half-frame images, but this method is relatively complex in operation.

[0004] Another processing method is to rely on deep learning to obtain the stitching parameters for a specific scenario. The method was elaborated in detail in the "Large Disparity Image Stitching Algorithm Based on the APAP Model" (Qu Daihui, Xie Yiwu. Large Disparity Image Stitching Algorithm Based on the APAP Model [J]. Measurement & Control Technology, 2021.) published in 2020. However, this method has quite limitations and requires a certain amount of training data to obtain a relatively accurate data model to complete the stitching.

[0005] When processing pipeline images taken of a thermal pipeline network, since the accuracy requirements for the pipeline images taken of the thermal pipeline network are not very strict, if a relatively complex image stitching algorithm is used, it will increase the additional computational burden and cannot meet the short-delay image stitching and subsequent processing operations. Moreover, in a relatively harsh environment of the pipeline, there are many unknown variables that may interfere with the selection of feature points in the algorithm, affecting the processing results. Summary of the Invention

[0006] In view of the above problems, the present invention discloses a method for stitching pipeline images using dual cameras, and the method includes the following steps:

[0007] Determine whether the original images obtained by the dual cameras are excessively deformed;

[0008] If the original image is not overly deformed, align the two original images according to the triangles in each original image;

[0009] Scale the aligned images according to the set value ratio;

[0010] Crop the largest regions of the two images so that the three points in each image overlap with the three points of the constructed triangle;

[0011] Calculate the overlapping regions to obtain the final image.

[0012] Further, the alignment of the two original images includes horizontal alignment and vertical alignment;

[0013] Among them,

[0014] The method of horizontal alignment is based on the base of the triangle in each original image;

[0015] The method of vertical alignment is to construct a triangle using the vertex of the triangle in one original image and the two bottom points in the other original image to complete the vertical alignment of the two original images.

[0016] Further, the method also includes adjusting the original images before aligning the two original images, and the adjustment includes the following steps:

[0017] Obtain the original images captured by the upper camera and the lower camera, and preset a desired side length l target ;

[0018] Set a threshold N according to the on-site camera angle and the pipeline curvature;

[0019] Obtain the coordinates of three landmark points in each original image;

[0020] Connect the two bottom points in each original image to obtain the bottom side length l;

[0021] Compare the bottom side length l with the desired side length l target and adjust the bottom side length l according to the desired side length l target ;

[0022] Further, adjust the bottom side length according to the following formula:

[0023]

[0024] where K is the set value ratio, l is the bottom side length of the triangle in the original image, and l target is the desired side length.

[0025] Further, if the original image is excessively deformed, the selected original image is discarded and a new one is acquired.

[0026] Further, according to the threshold N, the bottom side length l, and the expected side length l target Determine whether the original image is deformed;

[0027] When the bottom side length l satisfies (1 - N)l target ≤ l ≤ (1 + N)l target Then it is determined that the image is not excessively deformed;

[0028] When the bottom side length l > (1 + N)l target Or the bottom side length l < (1 - N)l target Then it is determined that the image is excessively deformed.

[0029] Further, the constructed triangle is preferably an equilateral triangle.

[0030] The present invention also discloses a system for splicing pipeline images by two cameras. The system includes a judgment module for judging whether the original images acquired by the two cameras are excessively deformed;

[0031] An alignment module for aligning two original images according to the triangular fiducial points in each original image when the original images are not excessively deformed;

[0032] A scaling module for scaling the aligned images according to a set ratio;

[0033] A cropping module for cropping the largest area of the two images so that the three points in each image overlap with the three points of the constructed triangle;

[0034] A calculation module for calculating the overlapping area to obtain the final image.

[0035] Further, an adjustment module for adjusting the original images before aligning the two original images; the adjustment module includes:

[0036] A first acquisition unit for acquiring the original images taken by the upper camera and the lower camera and presetting an expected side length;

[0037] A threshold unit for setting the threshold N according to the angles of the on-site cameras and the curvature of the pipeline;

[0038] A second acquisition unit for acquiring the coordinates of the three fiducial points in each original image;

[0039] A third acquisition unit for connecting two points at the bottom of each original image to obtain the bottom side length;

[0040] An adjustment unit for comparing the length of the bottom side with the desired length and adjusting the length of the bottom side according to the desired length.

[0041] Further, the alignment module includes

[0042] A horizontal alignment unit for horizontally aligning two original images according to the bottom side of the triangle in each original image;

[0043] A vertical alignment unit for vertically aligning two original images by constructing a triangle using the vertex of the triangle in one original image and the two bottom points in the other original image.

[0044] Advantages of the present invention:

[0045] By setting a threshold during image stitching, the present invention screens out some images with large distortions to a certain extent, avoiding the impact on subsequent processing;

[0046] By using the geometric characteristics of the triangular reference object for horizontal and vertical alignment; for local deformations caused by the pipeline curved surface and the camera perspective, measuring the side length of the triangle to detect and performing reverse stretching or compression can solve common basic stitching problems without causing great damage to the image itself;

[0047] The present invention effectively reduces the overall complexity of image stitching, leaving room for subsequent image processing work and reducing the overall processing time cost.

[0048] Other features and advantages of the present invention will be described in the following description of the specification, and some of them will be obvious from the description of the specification, or understood by implementing the present invention. The objectives and other advantages of the present invention can be achieved and obtained through the structures pointed out in the specification, claims and drawings. Description of the Drawings

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0050] Figure 1 Shows the flowchart of the method for stitching pipeline images using dual cameras proposed in the embodiments of the present invention;

[0051] Figure 2 Shows the conceptual diagram of the actual application scenario in the embodiments of the present invention;

[0052] Figure 3 Shows a schematic diagram of the pipeline image captured by the lower camera in the embodiment of the present invention;

[0053] Figure 4 Shows a schematic diagram of the pipeline image captured by the upper camera in the embodiment of the present invention;

[0054] Figure 5 Shows a schematic diagram of the horizontal correction of the splicing process in the embodiment of the present invention;

[0055] Figure 6 Shows a schematic diagram of the vertical correction of the splicing process in the embodiment of the present invention;

[0056] Figure 7 Shows a schematic diagram of stretching the first area in the embodiment of the present invention;

[0057] Figure 8 Shows a schematic diagram of stretching the second area in the embodiment of the present invention;

[0058] Figure 9 Shows a schematic diagram of the image after overall stretching in the embodiment of the present invention. Detailed implementation manners

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0060] The main objective of the present invention is to simplify the splicing process. For the problems caused by the translation and tilt of the image, the geometric characteristics of the triangular reference objects are used for alignment; for the local deformations caused by the pipeline curved surface and the camera perspective, the side lengths of the triangles are measured to detect, and reverse stretching or compression is performed to solve. It can solve common basic splicing problems and will not cause great damage to the image itself. By setting reference objects to splice the original images from a geometric perspective, it is to minimize the burden on the equipment carried by the inspection robot increased by complex processing algorithms as much as possible and speed up the processing speed.

[0061] Specifically, the splicing method includes the following steps, as Figure 1 shown:

[0062] Step 1: Obtain the original images of the upper camera and the lower camera, which contain triangular marker points, and calculate the position information and side length information of each Figure 3 point and perform preliminary scaling;

[0063] Step 2: Horizontally align the two images using the bottom edge;

[0064] Step 3: Use the vertex of one image to make the perpendicular bisector of the bottom edge of the other image, and meet the geometric requirements of an equilateral triangle to complete the alignment in the vertical direction;

[0065] Step 4: Filter out the images with excessive distortion by setting a threshold, and scale the images within the threshold according to the set ratio;

[0066] Step 5: Crop the largest image area, and stretch or compress each image so that the three points of the original image coincide with the selected three points, thus completing all the preprocessing operations of image stitching;

[0067] Step 6: Use a suitable existing algorithm to perform weighted calculation on the overlapping area to obtain the final image.

[0068] In the embodiment of the present invention, taking Figure 2 the conceptual diagram shown in Figures 2 - 7 as an example, the above stitching method will be described in detail in combination with

[0069] According to Figure 2 the conceptual diagram of the actual application scenario shown in, arrange the shooting scene, install the upper and lower cameras on the inspection robot, ensure that the distances from the upper camera and the lower camera to the pipeline are as consistent as possible, and the device for emitting three laser beams should be placed as close as possible to the middle of the upper camera and the lower camera, and the three points emitted should be arranged on the pipeline in the manner of the three vertices of an equilateral triangle.

[0070] Step 1: Obtain the pipeline images captured by the upper camera and the lower camera, as shown in Figure 4 and Figure 3 respectively. Assume that both cameras are slightly tilted due to the robot's movement. Here, we obtain the coordinates of three points in each image and specify a desired side length l target . Connect the two points at the bottom edge of each image to obtain the bottom edge length, and compare it with l target respectively, and adjust the size of the original image based on this value.

[0071] Let Figure 3 the coordinates of the three points from left to right in 1 be A1(X 1 , Y 1 ), A2(X 2 , Y 2 , Z 2 ), A3(X 3 , Y 3 , Z 3 ); where A2 is the vertex.

[0072] LetFigure 4 The three - point coordinates from left to right are B1(X 4 , Y 4 , Z 4 ), B2(X 5 , Y 5 , Z 5 ), B3(X 6 , Y 6 , Z 6 ), where B2 is the vertex.

[0073] Figure 3 The base - side length in

[0074]

[0075] Figure 4 The base - side length in

[0076]

[0077] To ensure that the base - side length l of the triangle in each image satisfies the expected side length l target , the base - side length of each image is multiplied by the corresponding 1 / K. Figure 3 The base - side length of the triangle in 1 is l Figure 4 The base - side length of the triangle in 2 is l

[0078]

[0079]

[0080] Step 2: Use the base - line connection of the triangle in Figure 3 and Figure 4 for horizontal calibration. Specifically, as shown in Figure 5 , make this connection perpendicular to the boundary of the splicing area (the newly created rectangular blank area P).

[0081] The newly created blank area is the container for holding the spliced image. If one of the images is used as the base, then this blank area P does not need to be newly created.

[0082] Step 3: Using the processed Figure 3 base - side length l of the triangle in 1 as the reference line, use the vertex B2 in Figure 4 to make the perpendicular bisector of l 1 such that the three points A1, A3, and B2 form a triangle with l target as the side length. Specifically, as shown in Figure 6 . Preferably, it can be an equilateral triangle.

[0083] When the three points A1, A3, and B2 form a triangle with ltarget When it is an equilateral triangle with side length

[0084] The length of the perpendicular bisector should satisfy:

[0085]

[0086] So that points A1, A3, and B2 form an equilateral triangle with side length l target as the side length.

[0087] Step 4: Consider in special cases, if the camera angle rotates and the pipeline curvature is too large, resulting in image deformation between three points in the image, a threshold N is set here to filter out images with large deformation.

[0088] The value of the threshold N needs to consider the following aspects:

[0089] The requirements of the image algorithm adopted after image stitching for the image; the distance between the camera and the pipeline and the ratio of the side length of the projected three points to the pipeline diameter, etc. If the image algorithm is only for images with small / large-amplitude deformation, the value of N should also be reduced / increased accordingly, and the specific value should be based on the best final effect after applying the image algorithm.

[0090] The deformation of the image is reflected in the deviation of the distance between two points from l target .

[0091] N = |the length of the line connecting any two points in the actual image - l target | / l target

[0092] The purpose of setting the threshold N is to filter out images with large deformation so as not to mislead the subsequent image algorithm, so the images should be screened by the threshold.

[0093] If Figure 3 , Figure 4 the length of the line connecting any two points l < (1 - N)l target or l > (1 + N)l target , it can be regarded that the deformation degree of the image is too large, and the stitching of the current two images is abandoned.

[0094] If Figure 3 , Figure 4 the length of the line connecting any two points l satisfies (1 - N)l target ≤ l ≤ (1 + N)l target When, compress or stretch the image by 1 / K times according to the direction of the abnormal side length.

[0095] In the embodiment of the present invention, the threshold N is taken as 0.2, and the specific threshold is determined on site.

[0096] Step 5: Scale the newly created blank area P so that at least one vertex of the blank area P lies on Figure 3 or Figure 4 at the boundary, specifically as shown in Figure 7 shown.

[0097] Figure 7 In [Figure 1], the first area and the second area are images of the same area taken by the upper camera and the lower camera. Due to the shooting angle and the curvature of the pipeline, there are different compressions or stretches and they do not completely overlap (the deformation effect is not shown in the figure); only need to stretch the vertex A2 in Figure 3 to the position of B2 in Figure 4 , and linear stretching can be used (only stretch the first area), and the stretching result is as shown in Figure 8 shown; similarly, stretch B1 and B3 in Figure 4 to the positions of A1 and A3 in Figure 3 (only stretch the second area), and the stretching result is as shown in Figure 9 shown. Then, intercept the stretched image to obtain the image we need to intercept.

[0098] Step 6: Select a suitable algorithm according to the computing and processing capabilities of the inspection robot or the backend to calculate the local homography matrix and map it onto the panoramic canvas A, and finally perform weighted fusion to obtain the final image.

[0099] The intercepted area obtained in the previous step is a partial overlap of the two processed images. Only need to perform weighted fusion on these two processed and intercepted images using a suitable algorithm to obtain the final image.

[0100] In the embodiments of the present invention, Steps 1 - 5 have completed operations such as the preliminary docking of images. Therefore, only the overlapping area needs to be processed. Specifically, the APAP (image stitching using moving DLT projection) algorithm can be used. The APAP algorithm uses sift (scale - invariant feature transform) descriptors to establish the relationship between similar feature points of two images, and uses RANSAC (random sample consensus algorithm) to screen the relationship set and remove outliers. Then, calculate the local homography matrix and map it onto the panoramic canvas A, and finally perform weighted fusion to obtain the final image.

[0101] Based on the above method for stitching pipeline images with two cameras, a system for stitching pipeline images with two cameras is proposed. The system is used to execute the above method for stitching pipeline images with two cameras. The system includes:

[0102] An adjustment module for adjusting the acquired original image;

[0103] Further, the adjustment module includes:

[0104] The first acquisition unit is configured to acquire the original images captured by the upper camera and the lower camera, and preset an expected side length l target ;

[0105] The threshold unit sets a threshold N according to the on-site camera angle and the pipeline curvature;

[0106] The second acquisition unit is configured to acquire the coordinates of three landmark points in each original image;

[0107] The third acquisition unit is configured to connect two bottom points in each original image to obtain the bottom side length l;

[0108] The adjustment unit is configured to compare the bottom side length l with the expected side length l target and adjust the bottom side length l according to the expected side length l target

[0109] The judgment module is configured to judge whether the original images acquired by the dual cameras are excessively deformed.

[0110] The alignment module is configured to complete the alignment of two original images according to the triangular landmark points in each original image when the original images are not excessively deformed.

[0111] The alignment module includes

[0112] The horizontal alignment unit is configured to perform horizontal alignment of two original images according to the triangular bottom sides in each original image;

[0113] The vertical alignment unit is configured to complete the vertical alignment of two original images by constructing a triangle using the triangular vertex in one original image and the two bottom points in the other original image.

[0114] The scaling module is configured to scale the aligned images according to a set ratio.

[0115] The cropping module is configured to crop the largest regions of two images so that the three points in each image overlap with the three points of the constructed triangle.

[0116] The calculation module is configured to calculate the overlapping regions to obtain the final image.

[0117] Common image stitching needs to complete the matching and region selection of two images according to feature points, and it takes a long time cost in the selection and matching of feature points. Based on the above method and the corresponding system for dual-camera pipeline image stitching, a relatively simple method is used to replace this process and quickly enter the weighted fusion of overlapping images, effectively reducing the overall complexity of image stitching, leaving room for subsequent image processing work, reducing the overall processing time cost, and achieving a certain degree of screening function to prevent the influence of individual images with large distortions on subsequent processing.​

[0118] Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for stitching pipeline images with dual cameras, characterized in that, the method comprises the following steps: judging whether the original images acquired by the dual cameras are excessively deformed; if the original images are not excessively deformed, aligning the two original images according to the triangles in each original image; the alignment of the two original images includes horizontal alignment and vertical alignment; wherein, the method of horizontal alignment is carried out according to the bottom edges of the triangles in each original image; the method of vertical alignment is to construct a triangle by using the vertex of a triangle in one original image and the two bottom points in the other original image to complete the vertical alignment of the two original images; scaling the aligned images according to a set ratio so that the three points in each image overlap with the three points of the constructed triangle; cropping the largest regions of the two images; calculating the overlapping regions to obtain the final image.

2. The method for stitching pipeline images with dual cameras according to claim 1, characterized in that, the method further comprises adjusting the original images before aligning the two original images, and the adjustment comprises the following steps: Obtain the original images captured by the upper camera and the lower camera, and preset a desired side length l target ; setting a threshold N according to the on-site camera angle and the pipeline curvature; acquiring the coordinates of three landmark points in each original image; connecting the two bottom points in each original image to obtain the bottom edge length l; Compare the base side length \(l\) with the expected side length \(l\). target Based on the expected side length \(l\), target adjust the base side length \(l\).

3. The method for stitching pipeline images with dual cameras according to claim 2, characterized in that, adjusting the bottom edge length l according to the following formula: Among them, K is the set value ratio, l is the length of the base side of the triangle in the original image, and l target is the expected side length.

4. The method for stitching pipeline images with dual cameras according to claim 1, characterized in that, if the original images are excessively deformed, discard the selected original images and acquire them again.

5. The method for stitching pipeline images with dual cameras according to claim 2, characterized in that, Based on the threshold N, the bottom side length l, and the expected side length l target judge whether the original image is deformed according to the length; When the length l of the bottom side satisfies (1 - N)l target ≤ l ≤ (1 + N)l target it is determined that the image is not overly deformed; When the bottom side length \(l > (1 + N)l\) target or the bottom side length \(l < (1 - N)l\) target it is determined that the image is excessively deformed.

6. The method for stitching pipeline images with dual cameras according to claim 1, characterized in that, the constructed triangle is an equilateral triangle.

7. A system for stitching pipeline images with dual cameras, characterized in that, the system comprises, a judging module for judging whether the original images acquired by the dual cameras are excessively deformed; an aligning module for aligning the two original images according to the triangle landmark points in each original image when the original images are not excessively deformed; the alignment of the two original images includes horizontal alignment and vertical alignment; wherein, the method of horizontal alignment is carried out according to the bottom edges of the triangles in each original image; the method of vertical alignment is to construct a triangle by using the vertex of a triangle in one original image and the two bottom points in the other original image to complete the vertical alignment of the two original images; a scaling module for scaling the aligned images according to a set ratio so that the three points in each image overlap with the three points of the constructed triangle; a cropping module for cropping the largest regions of the two images; a calculating module for calculating the overlapping regions to obtain the final image.

8. The system for stitching pipeline images with dual cameras according to claim 7, characterized in that, the system further comprises, Adjustment module, configured to adjust the original images before aligning two original images; The adjustment module includes: A first acquisition unit, configured to acquire original images captured by an upper camera and a lower camera, and preset an expected side length l target ; Threshold unit, configured to set a threshold N according to the on-site camera angle and the pipeline curvature; Second acquisition unit, configured to acquire the coordinates of three landmark points in each original image; Third acquisition unit, configured to connect two bottom points in each original image to acquire the bottom side length l; Adjustment unit for comparing the base side length \(l\) with the desired side length \(l\) target and adjusting the base side length \(l\) according to the desired side length \(l\). target ​

Images