Large-parallax large-angle rotation infrared image splicing method

Abstract

The invention discloses a large-parallax large-angle rotation infrared image splicing method, and relates to the technical field of infrared image splicing, the large-parallax large-angle rotation infrared image splicing method comprises the following steps: detecting a vanishing line in a to-be-spliced image, using the vanishing line and affine transformation to construct an integral homography matrix, and using the integral homography matrix to construct an integral homography matrix; and according to the overall homography matrix, recovering the parallel relationship of the straight line segments in the to-be-spliced image, correcting the orthogonal relationship of the straight line segments, and splicing and fusing the corrected image to obtain an infrared spliced image. The method can find the horizon line of the scene in the image and correct the horizon line to the horizontal direction, greatly reduces the difficulty of image splicing, has a strong correction capability for the image shot at a certain elevation angle, has a good effect of the spliced image obtained by the method, conforms to the visual habit of human eyes, and has a good application prospect. The requirements of large field of view and high resolution are met, and the system is particularly suitable for application scenes during infrared image shooting.

Description

Technical Field

[0001] This invention belongs to the field of infrared image stitching technology, specifically relating to an infrared image stitching method with large parallax and large-angle rotation. Background Technology

[0002] When shooting with cameras equipped with medium or long telephoto lenses, a common challenge is that the captured images often lack a grand enough composition (significantly different from human vision, failing to capture the scene the photographer intends to show). However, using a wide-field-of-view fisheye lens introduces severe image distortion, which differs greatly from human visual perception, resulting in unsatisfactory images. Infrared images, compared to visible light images, have weaker details and textures, making image stitching even more difficult.

[0003] When using infrared imaging systems to capture images, the large pixel size of infrared detectors results in low resolution. Furthermore, large infrared detectors are very expensive due to manufacturing limitations. To achieve both high resolution and a wide field of view, infrared systems typically use digital image processing algorithms to stitch the captured images together. However, in practice, the following problems arise: In many cases, the photographer cannot easily obtain a direct view of the subject and must resort to tilted shooting. Additionally, handheld shooting (which is not stable and prone to slight camera shake or rotation during shooting, resulting in only a certain overlap between the two images to be stitched) is also common. While relatively convenient and quick, random tilting, rotation, and vertical projection distortion all increase the difficulty of image stitching.

[0004] Traditional image stitching algorithms are mainly divided into global homography transformation algorithms (which convert the target image to the perspective of the reference image during stitching, resulting in severe perspective projection distortion for images with large parallax) and local homography transformation stitching algorithms. When using a global homography matrix for image stitching, the stitched image undergoes perspective distortion to achieve a good fusion effect in the overlapping area. Due to the large parallax between the two images involved in the stitching, the image to be stitched is overstretched, resulting in an unnatural stitched image. Currently, for image stitching with a certain degree of parallax, the most widely used stitching algorithms are based on local homography transformation. Examples include As-Projective-As-Possible (APAP, a non-rigid registration algorithm for image stitching, primarily used to solve local alignment problems caused by traditional homography transformation image stitching) and Shape-Preserving Half-Projective (SPHP, a semi-projective distortion algorithm for image stitching, primarily used to solve perspective distortion and artifact problems in traditional image stitching. This algorithm optimizes the transformation matrix to reduce misalignment caused by the global homography matrix while maintaining the shape of the stitched image). The disadvantages of local homography transformation algorithms are their complexity, slow computation speed, and the resulting unnatural distortion in the stitched image. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides an infrared image stitching method with large parallax and large-angle rotation. The technical problem to be solved by this invention is achieved through the following technical solution: This invention provides a method for stitching infrared images with large parallax and large-angle rotation, comprising: Step 1: Perform edge detection and line detection on the image to be stitched to obtain the line segments in the image to be stitched, and sort all line segments in descending order of length; Step 2: Obtain two vanishing point models based on the line segment, and perform precise calculations on the two vanishing point models to obtain the two vanishing points; Step 3: Calculate the vanishing line based on the two vanishing points, construct the overall homography matrix based on the vanishing line and affine transformation, restore the parallelism of the line segments in the image to be stitched based on the overall homography matrix, and correct the orthogonality of the line segments to obtain the corrected image; Step 4: Register and fuse the corresponding calibration images of the two images to be stitched together using trigonometric function weights to obtain the infrared stitched image.

[0006] Compared with the prior art, the beneficial effects of the present invention are as follows: The infrared image stitching method with large parallax and large-angle rotation of the present invention first detects vanishing lines in the images to be stitched, constructs an overall homography matrix using the vanishing lines and affine transformations, restores the parallelism of line segments in the images to be stitched based on the overall homography matrix, and corrects the orthogonality of the line segments. The corrected images are then stitched and fused to obtain the infrared stitched image. This method can find the horizon line of the scene in the image and correct it to a horizontal direction, greatly reducing the difficulty of image stitching. It has a strong correction capability for images captured at a certain elevation angle. The stitched image obtained by this method has good quality, the object-image relationship conforms to human visual habits, and meets the requirements of a large field of view and high resolution, making it particularly suitable for applications involving infrared image capture.

[0007] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0008] Figure 1 This is a flowchart of an infrared image stitching method with large parallax and large-angle rotation provided by an embodiment of the present invention; Figure 2 This is a schematic diagram of the infrared image before stitching provided in an embodiment of the present invention; Figure 3 It is using the method of the present invention to Figure 2 A schematic diagram of image point matching pairs; Figure 4 yes Figure 3 A schematic diagram of the image after cropping; Figure 5 It utilizes traditional global homography transformation to... Figure 2 The stitching result of the images in the image; Figure 6 It uses Autostitch software to... Figure 2 The stitching result of the images in the image; Figure 7 It uses the APAP algorithm to Figure 2 The stitching result of the images in the image; Figure 8 It uses Panorama Maker 4 software to... Figure 2 The stitched image before rotation; Figure 9 It uses Panorama Maker 4 software to... Figure 2 The result of stitching the rotated images in the image; Figure 10 It uses the SPHP algorithm to Figure 2 The stitched image before rotation; Figure 11 It uses the SPHP algorithm to Figure 2 The result of stitching the rotated images in the image. Detailed Implementation

[0009] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following describes in detail, with reference to the accompanying drawings and specific embodiments, a method for stitching infrared images with large parallax and large-angle rotation proposed according to the present invention.

[0010] The foregoing and other technical contents, features, and effects of the present invention will be clearly presented in the following detailed description of specific embodiments in conjunction with the accompanying drawings. Through the description of the specific embodiments, a more in-depth and concrete understanding can be gained of the technical means and effects adopted by the present invention to achieve its intended purpose. However, the accompanying drawings are for reference and illustration only and are not intended to limit the technical solutions of the present invention.

[0011] This invention provides a flowchart of an infrared image stitching method with large parallax and large-angle rotation. Please refer to [link / reference]. Figure 1 , Figure 1 This is a flowchart of an infrared image stitching method with large parallax and large-angle rotation provided by an embodiment of the present invention, as follows: Figure 1 As shown, the infrared image stitching method with large parallax and large-angle rotation according to an embodiment of the present invention includes the following steps: Step 1: Perform edge detection and line detection on the images to be stitched to obtain the line segments in the images to be stitched, and sort all the line segments in descending order of length.

[0012] In an optional embodiment, step 1 includes: Step 1.1: Obtain significant edges in the images to be stitched using Canny edge detection; Step 1.2: Perform hough line detection on the salient edges in the images to be stitched to obtain the line segments in the images to be stitched. Sort all line segments in descending order of length. A line segment is represented as:

[0013] In the formula, For the first A straight line segment, Homogeneous coordinates representing the position of a line segment. Homogeneous coordinates representing the direction of a line segment. Let be the length of the line segment.

[0014] In this embodiment, it can be based on straight line segments Establish the corresponding straight line , The line number is the line number. pass And with Parallel, will According to straight line segments Sort and store the data by length.

[0015] Step 2: Obtain two vanishing point models based on the line segment, and perform precise calculations on the two vanishing point models to obtain the two vanishing points.

[0016] In an optional embodiment, step 2 includes: Step 2.1: Based on the line segments in the image to be stitched, use the RANSAC algorithm to establish a vanishing point model, obtain the first vanishing point model, and record the line segments that satisfy the first vanishing point model; Step 2.2: Based on the remaining line segments in the image to be stitched after removing the line segments that satisfy the first vanishing point model, use the RANSAC algorithm to establish the vanishing point model to obtain the second vanishing point model, and record the line segments that satisfy the second vanishing point model; Step 2.3: Calculate the first and second vanishing points by using the least squares method for the line segments that satisfy the first vanishing point model and the line segments that satisfy the second vanishing point model, respectively.

[0017] In this embodiment, the process of establishing the vanishing point model using the RANSAC (Random Sample Consensus) algorithm is as follows: First, two line segments are randomly selected to calculate the intersection point of the line vectors (candidate vanishing point model), and then the remaining lines... Substitute the intersection points to calculate whether the intersection points lie on the straight line. If the intersection points lie on the straight line... The above is counted as one vote for the intersection point. The remaining line segments are substituted into the intersection point, and the total number of votes for the candidate vanishing point model is recorded. Then, the line segments that have participated in the candidate vanishing point model are removed. Two line segments are randomly selected from the remaining line segments to continue calculating the candidate vanishing point model. It is verified whether the remaining lines pass through the intersection point. The number of votes for each candidate vanishing point model is counted, and the candidate vanishing point model with the most votes is the vanishing point model.

[0018] It is understandable that two line segments are randomly selected. , The candidate vanishing point model is obtained by calculating the intersection of the two line segments. , Corresponding straight lines and The intersection of the lines. With a straight line Since both are represented in homogeneous coordinates, their intersection point can be obtained using the cross product of homogeneous coordinate vectors. When the selected lines coincide or the same line is selected twice, the intersection point cannot be calculated, indicating degeneration. Each time a new candidate vanishing point model is generated, it needs to be checked for degeneration. If it is degenerate, the candidate vanishing point model is rejected, and two line segments are randomly selected to construct a new candidate vanishing point model. To ensure a sufficiently low computation time, the line segments used to generate the candidate vanishing point model are not selected completely randomly. The first line segment for calculating the intersection point is chosen from the top 20% of line segments by length, and the second line segment is chosen from the top 50% of line segments by length. This process is used to filter longer line segments to participate in determining the candidate vanishing point model.

[0019] It should be noted that when calculating the votes for the candidate vanishing point model, all remaining line segments are considered. Substitute them sequentially into the candidate vanishing point model Calculations yielded and The angle between cosine value, In the formula, It is a straight line segment Location point and candidate vanishing point model Vectors between a straight line segment The angle between the two vectors can be determined using the calculated cosine value of the corresponding straight line. In this embodiment, the angle threshold is set to 3°; if the angle between the two vectors is less than 3°, it is considered... It is an interior point, and the line segment The candidate vanishing point model has voting rights; if the angle between two vectors is greater than 3°, it is considered... It is an exterior point, and the line segment The candidate vanishing point model has no voting rights. To give longer line segments higher weight, the length of the line segment is used as the number of votes for the candidate vanishing point model.

[0020] Optionally, establishing a vanishing point model using the RANSAC algorithm may include the following steps: S1: Initialize RANSAC parameters, including the number of iterations and the angle threshold; S2: Randomly select one line segment from the top 20% of line segments by length, and randomly select one line segment from the top 50% of line segments by length. S3: If the two selected line segments are parallel or coincident, skip the current iteration and return to S2; otherwise, calculate the intersection of the two selected line segments as a candidate vanishing point model. S4: Calculate the angle between all remaining line segments and the intersection-position line. The intersection-position line is the line connecting the intersection point and the position point of the currently calculated line segment. Line segments with angles not exceeding the angle threshold are taken as interior points of the candidate vanishing point model. The length of the line segment of the interior point is used as the vote to vote on the candidate vanishing point model. Record the number of votes for the candidate vanishing point model. S5: Repeat S2-S4 until the number of iterations is reached, and select the candidate vanishing point model with the most votes as the vanishing point model.

[0021] Based on the vanishing point model obtained above, least squares regression can be used to calculate the vanishing point more accurately. That is, the vanishing point is re-estimated from all interior points of the vanishing point model using the least squares method, let... Represents the vanishing point model The set of line segments whose interior points are interior points. This represents the vanishing point obtained from the re-estimation, theoretically representing a straight line segment. Corresponding straight line They will all pass through the vanishing point ,and .set up ,but Furthermore, each equation is weighted so that longer line segments bring the vanishing point model closer to those segments. Thus, each line segment will produce a model of the form... The formula, It is a straight line segment The voting values ​​for the current vanishing point model. The system of equations formed by all interior points can be represented in matrix form as follows: ; The above system of equations is a homogeneous linear system, which can be solved using singular value decomposition to obtain the vanishing point. .

[0022] Most images contain two vanishing points, one horizontal and one vertical. The first and second vanishing points obtained in this embodiment can be understood as the horizontal and vertical vanishing points, respectively. To prevent lines intersecting at different vanishing points from interfering with each other, this embodiment employs a method of first detecting one vanishing point, and then removing all interior points of the first vanishing point when calculating the second vanishing point, thus improving detection efficiency.

[0023] Step 3: Calculate the vanishing line based on the two vanishing points, construct the overall homography matrix based on the vanishing line and affine transformation, restore the parallelism of the line segments in the image to be stitched based on the overall homography matrix, and correct the orthogonality of the line segments to obtain the corrected image.

[0024] Understandably, to restore the parallelism of line segments, in a two-dimensional homogeneous space, the line at infinity represents a line containing all points at infinity, and all points at infinity are represented by... This indicates that a straight line at infinity is formed by... Indicate, then This means that points at infinity lie on straight lines at infinity. From a solid geometry perspective, an image taken by an arbitrarily rotating camera can be considered as a projection from the orthographic image plane onto other image planes. Through affine transformations, parallel lines in the orthographic image can be made parallel in images taken by arbitrarily rotating cameras.

[0025] In an optional embodiment, step 3 includes: Step 3.1: Calculate the vanishing line by cross product of the two vanishing points.

[0026] set up , It is a pair of vanishing points (i.e., the first vanishing point and the second vanishing point), through , Cross product yields vanishing lines , , The coordinates of the vanishing line are homogeneous.

[0027] Step 3.2: Construct an initial homography matrix based on the vanishing line, so that the vanishing line is mapped to a straight line at infinity. .

[0028] The initial homography matrix can restore the vanishing point to infinity. Therefore, the initial homography matrix is ​​expressed as: For the vanishing point , .

[0029] If any two vectors in a scalar triple product are equal, then the scalar triple product is equal to zero. As shown in the equation above, this represents the representation of a point at infinity in a homogeneous space. Similarly... The result also satisfies this representation; for vanishing lines If the point lies on a straight line, then the point transformation... The straight line is transformed as follows: ,therefore As can be seen, the vanishing line is restored to an infinity line after transformation using the initial homography matrix described above. Therefore, using this initial homography matrix, intersecting lines can be restored to parallel lines through transformation.

[0030] Step 3.3: Determine the angle between the horizontal parallel line and the x-axis of the image to be stitched, and the angle between the vertical parallel line and the y-axis of the image to be stitched, based on the two vanishing points. Determine the affine transformation matrix based on the angle between the horizontal parallel line and the x-axis of the image to be stitched, and the angle between the vertical parallel line and the y-axis of the image to be stitched.

[0031] As mentioned above, for images with intersecting parallel lines captured under non-orthographic conditions, the initial homography matrix can be used to restore them to parallel lines. However, the resulting image is a sheared image. Since the positions of the horizontal and vertical vanishing points (i.e., the first and second vanishing points) have been obtained, and the direction vectors of these two points are restored to the horizontal and vertical lines respectively, it is necessary to analyze the angle between the vertical parallel line and the image y-axis, and the angle between the horizontal parallel line and the x-axis. Using the angle between the vertical parallel line and the image y-axis... The angle between the horizontal parallel line and the x-axis The perpendicular relationship between vertical and horizontal lines in an image can be restored by distorting the image "in reverse" through affine transformation.

[0032] In this embodiment, the affine transformation matrix is ​​represented as: ,in, , , The angle between the vertical parallel line and the y-axis of the images to be stitched together. The angle between the horizontal parallel line and the x-axis of the images to be stitched together is denoted as . It can be calculated using the vertical vanishing point. , The coordinates of the vertical vanishing point. It can be calculated using the horizontal vanishing point. , These are the coordinates of the horizontal vanishing point.

[0033] Step 3.4: Obtain the global homography matrix based on the initial homography matrix and the affine transformation matrix.

[0034] In this embodiment, the global homography matrix is ​​represented as: ,Right now ,in, Let be the initial homography matrix. Let be the affine transformation matrix.

[0035] Step 3.5: Transform the coordinates of the image to be stitched using the global homography matrix, restore the parallelism of the line segments in the image to be stitched, and correct the orthogonality of the line segments to obtain the corrected image.

[0036] Specifically, the coordinates in the images to be stitched With scriptures After transformation, we get , .

[0037] Step 4: Register and fuse the corresponding calibration images of the two images to be stitched together using trigonometric function weights to obtain the infrared stitched image.

[0038] In an optional embodiment, step 4 includes: Step 4.1: Use the SIFT feature detection algorithm to extract feature points from the corresponding corrected images of the two images to be stitched. Based on the extracted feature points, use the FLANN matching algorithm to perform coarse matching of feature points to obtain initial matching point pairs. Step 4.2: Use the RANSAC algorithm to remove mismatched points from the initial matching point pairs to obtain optimized matching point pairs. Estimate the homography transformation matrix based on the optimized matching point pairs and use the homography transformation matrix to align the two corrected images. Step 4.3: The two aligned and corrected images obtained in Step 4.2 are stitched together as a stitching sub-image. The pixels in the overlapping area are weighted and fused to obtain an infrared stitched image.

[0039] It is understandable that if the two images to be stitched have a rotation relationship, for example, if the right image is rotated 45° counterclockwise relative to the left image, then during the stitching process, with the left image as the reference image, the right image must be rotated 45° clockwise. After the rotation, black gaps (blocky areas without pixel information) will appear at the four corners of the left image. During the fusion process, using the traditional blocky area fade-in / fade-out fusion method will cause black blocky shadows in local areas of the stitched image, affecting the image quality of the stitched image. Therefore, this embodiment proposes a fusion method based on overlapping area weight allocation, which performs weight allocation fusion on the pixels in the overlapping area to obtain an infrared stitched image.

[0040] Specifically, the process of weighting and fusing pixels in the overlapping areas to obtain the infrared stitched image is as follows: First, calculate the intersection between the two stitched sub-images to obtain the set of pixels in the overlapping area.

[0041] Specifically, after image correction and image stitching are completed, the sets of non-zero pixel positions in the stitched extended image of the reference image and the stitched image are recorded as set A and set B, respectively. The intersection of set A and set B is recorded as set C. The pixels in set C are the pixels in the overlapping area.

[0042] Then, the grayscale weight of each pixel in the overlapping region in the two stitched sub-images is calculated; the calculation process of the grayscale weight includes: Find the maximum column coordinate value of the row containing the pixel in the set of pixels. and minimum column coordinate value Calculate the weight parameters based on the maximum and minimum column coordinate values. , The column coordinates of the pixels are represented by the weight parameters, which determine the grayscale weights of the pixels in the two stitched sub-images. and .

[0043] In this embodiment, the grayscale weight of each pixel in the overlapping region is... This allows us to obtain infrared stitched images that meet the visual habits of the human eye.

[0044] The infrared image stitching method with large parallax and large-angle rotation in this invention can find the horizon of the scene in the image and correct it to a horizontal direction, greatly reducing the difficulty of image stitching. It has a strong correction capability for images taken at a certain elevation angle. The stitched image obtained by this method has good effect, the object-image relationship conforms to human visual habits, and meets the requirements of large field of view and high resolution, making it particularly suitable for application scenarios when shooting infrared images. At the same time, the method used to obtain the stitched image when there is a large relative rotation between two stitched images has been optimized to address the black shadow in the neighborhood of the overlapping area. The fusion method using the weight allocation of the overlapping area effectively solves the problem of black shadow in the fusion area.

[0045] Furthermore, specific examples are used to illustrate the effectiveness of the infrared image stitching method with large parallax and large-angle rotation of the present invention.

[0046] Please see Figure 2 , Figure 2 This is a schematic diagram of the infrared image before stitching provided in an embodiment of the present invention. An infrared camera was used to photograph the building structure. Due to the height of the building, the image was taken at a certain upward angle. Figure 2 As shown in the left image, the building appears tilted due to perspective projection. Rotating the camera horizontally by approximately 45° and using the camera's optical axis as the rotation axis by approximately 45°, resulting in a slight upward tilt, yielded the infrared image before stitching. Figure 2 The image on the right in the text.

[0047] The infrared image stitching method with large parallax and large-angle rotation of this invention is used to... Figure 2 The infrared images shown are stitched together, and the specific process is as described in the above embodiments, which will not be repeated here. It is understood that when utilizing... Figure 2 Before stitching the infrared images shown, side window filtering can be applied to denoise the images, and the DDE algorithm can be used to enhance the images.

[0048] In addition, embodiments of the present invention also employ several feature matching and splicing methods commonly used in the prior art to... Figure 2 The infrared images shown were stitched together using methods including traditional global homography transformation, Autostitch software, APAP algorithm, PanoramaMaker 4 software, and SPHP algorithm.

[0049] Please see Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 8 , Figure 9 , Figure 10 and Figure 11 As can be seen from the figure, the stitched image obtained by the method of the present invention has the best image straightness and best conforms to the human eye's habit of observing real scenes.

[0050] Furthermore, three objective image quality evaluation metrics were calculated for various stitching results, including Natural Image Quality Estimation (NIQE), RMS contrast, and local standard deviation. The NIQE evaluation results are shown in Table 1, and the RMS contrast and local standard deviation contrast are shown in Table 2.

[0051] Table 1

[0052] Table 2

[0053] As can be seen from Table 1, the Panorama Maker4_before method, which failed to stitch the image, had the highest NIQE value; the Autostitch method had an NIQE value close to 10, indicating that the image quality was low; the other methods all obtained medium to high quality stitching results.

[0054] As shown in Table 2, SPHP_before achieved the highest RMS contrast ratio of 0.271, while the method of this invention achieved 0.251. For local contrast, the vanishing line correction stitching algorithm and its cropped stitched image yielded the two best evaluation results, reflecting a high level of detail.

[0055] To address the poor performance of Panorama Maker4 software and the SPHP algorithm in relative rotation correction, the image on the right was rotated 45° clockwise before comparison. The stitched image obtained by inputting the original image is referred to as "before" (marked "_before" in the table), and the stitched image obtained by inputting the clockwise rotated image is referred to as "after" (marked "_after" in the table). The stitched image after rotation was also included in the comparison of stitching effects of various stitching methods to analyze whether manual rotation adjustment can effectively reduce the difficulty of image stitching. By comparing the NIQE and RMS contrast ratios before and after rotation restoration, two objective evaluation indicators, the results show that after manual rotation restoration, the NIQE value of the stitched image increased significantly, while the RMS contrast ratio decreased significantly. This phenomenon indicates a significant decline in the quality of the stitched image. To further verify this conclusion, we conducted a subjective evaluation analysis of the quality of the stitched images before and after rotation restoration. Figure 9 As shown, even after the image on the right has been rotated, Panorama Maker 4 software still cannot successfully stitch the images together. More specifically, from... Figure 11 As can be clearly observed, after the right-hand image was rotated and stitched together, significant ghosting appeared in the overlapping areas of the buildings. This experimental result fully demonstrates that rotating and restoring images does not improve the stitching success rate or the quality of the stitched image using PanoramaMaker4 software and the SPHP algorithm.

[0056] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that an article or device comprising a list of elements includes not only those elements but also other elements not expressly listed. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or device comprising said element. Terms such as "connected" or "linked" are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect. The orientations or positional relationships indicated by terms such as "upper," "lower," "left," and "right" are based on the orientations or positional relationships shown in the accompanying drawings and are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as limiting the invention.

[0057] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0058] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for stitching infrared images with large parallax and large-angle rotation, characterized in that, include: Step 1: Perform edge detection and line detection on the image to be stitched to obtain the line segments in the image to be stitched, and sort all line segments in descending order of length; Step 2: Obtain two vanishing point models based on the line segment, and perform precise calculations on the two vanishing point models to obtain the two vanishing points; Step 3: Calculate the vanishing line based on the two vanishing points, construct the overall homography matrix based on the vanishing line and affine transformation, restore the parallelism of the line segments in the image to be stitched based on the overall homography matrix, and correct the orthogonality of the line segments to obtain the corrected image; Step 4: Register and fuse the corresponding calibration images of the two images to be stitched together using trigonometric function weights to obtain the infrared stitched image.

2. The infrared image stitching method with large parallax and large-angle rotation according to claim 1, characterized in that, Step 1 includes: Step 1.1: Obtain significant edges in the image to be stitched using Canny edge detection; Step 1.2: Perform hough line detection on the significant edges in the image to be stitched to obtain the line segments in the image to be stitched. Sort all line segments in descending order of length, where each line segment is represented as: In the formula, For the first A straight line segment, Homogeneous coordinates representing the position of a line segment. Homogeneous coordinates representing the direction of a line segment. Let be the length of the line segment.

3. The infrared image stitching method with large parallax and large-angle rotation according to claim 1, characterized in that, Step 2 includes: Step 2.1: Based on the line segments in the image to be stitched, establish a vanishing point model using the RANSAC algorithm to obtain the first vanishing point model, and record the line segments that satisfy the first vanishing point model; Step 2.2: Based on the remaining line segments in the image to be stitched after removing the line segments that satisfy the first vanishing point model, establish a vanishing point model using the RANSAC algorithm to obtain a second vanishing point model, and record the line segments that satisfy the second vanishing point model; Step 2.3: Calculate the first vanishing point and the second vanishing point by using the least squares method for the line segments that satisfy the first vanishing point model and the line segments that satisfy the second vanishing point model, respectively.

4. The infrared image stitching method with large parallax and large-angle rotation according to claim 3, characterized in that, A vanishing point model is established using the RANSAC algorithm, including: S1: Initialize RANSAC parameters, including the number of iterations and the angle threshold; S2: Randomly select one line segment from the top 20% of line segments by length, and randomly select one line segment from the top 50% of line segments by length. S3: If the two selected line segments are parallel or coincident, skip the current iteration and return to S2; otherwise, calculate the intersection of the two selected line segments as a candidate vanishing point model. S4: Calculate the angle between all remaining line segments and the intersection-position line, where the intersection-position line is the connecting line between the intersection point and the position point of the currently calculated line segment. Line segments with angles not exceeding the angle threshold are taken as interior points of the candidate vanishing point model. The length of the line segment of the interior point is used as the vote count to vote on the candidate vanishing point model. Record the number of votes for the candidate vanishing point model. S5: Repeat S2-S4 until the number of iterations is reached, and select the candidate vanishing point model with the most votes as the vanishing point model.

5. The infrared image stitching method with large parallax and large-angle rotation according to claim 1, characterized in that, Step 3 includes: Step 3.1: Calculate the vanishing line by cross product of the two vanishing points; Step 3.2: Construct an initial homography matrix based on the vanishing line, such that the vanishing line is mapped to a straight line at infinity. ; Step 3.3: Determine the angle between the horizontal parallel line and the x-axis of the image to be stitched, and the angle between the vertical parallel line and the y-axis of the image to be stitched, based on the two vanishing points. Determine the affine transformation matrix based on the angle between the horizontal parallel line and the x-axis of the image to be stitched, and the angle between the vertical parallel line and the y-axis of the image to be stitched. Step 3.4: Obtain the global homography matrix based on the initial homography matrix and the affine transformation matrix; Step 3.5: Transform the coordinates of the image to be stitched using the overall homography matrix, restore the parallelism of the line segments in the image to be stitched, and correct the orthogonality of the line segments to obtain the corrected image.

6. The infrared image stitching method with large parallax and large-angle rotation according to claim 5, characterized in that, The vanishing line is represented as: , Let the homogeneous coordinates of the vanishing line be the initial homography matrix, which is expressed as: .

7. The infrared image stitching method with large parallax and large-angle rotation according to claim 5, characterized in that, The affine transformation matrix is ​​represented as follows: ,in, , , The angle between the vertical parallel line and the y-axis of the images to be stitched together. The angle between the horizontal parallel line and the x-axis of the image to be stitched together.

8. The infrared image stitching method with large parallax and large-angle rotation according to claim 5, characterized in that, The overall homography matrix is ​​represented as: ,in, Let be the initial homography matrix. Let be the affine transformation matrix.

9. The infrared image stitching method with large parallax and large-angle rotation according to claim 1, characterized in that, Step 4 includes: Step 4.1: Use the SIFT feature detection algorithm to extract feature points from the corresponding corrected images of the two images to be stitched. Based on the extracted feature points, use the FLANN matching algorithm to perform coarse matching of feature points to obtain initial matching point pairs. Step 4.2: Use the RANSAC algorithm to remove mismatched points from the initial matching point pairs to obtain optimized matching point pairs. Estimate the homography transformation matrix based on the optimized matching point pairs and use the homography transformation matrix to align the two corrected images. Step 4.3: The two aligned and corrected images obtained in step 4.2 are stitched together as a stitched sub-image, and the pixels in the overlapping area are weighted and fused to obtain the infrared stitched image.

10. The infrared image stitching method with large parallax and large-angle rotation according to claim 9, characterized in that, The step of weighting and fusing pixels in the overlapping region to obtain the infrared stitched image includes: Calculate the intersection between the two stitched sub-images to obtain the set of pixels in the overlapping region; The grayscale weight of each pixel in the overlapping region in the two stitched sub-images is calculated; wherein, the calculation process of the grayscale weight includes: Find the maximum column coordinate value of the row containing the pixel in the set of pixels. and minimum column coordinate value Calculate the weight parameter based on the maximum column coordinate value and the minimum column coordinate value. , The column coordinates of the pixel are represented, and the grayscale weights of the pixel in the two stitched sub-images are determined according to the weight parameters. and .

Images