A real-time infrared binocular image stitching method and system

Abstract

The present invention discloses a real-time infrared binocular image stitching method and system, the method comprising the following steps: step A: pre-stitching; step B: calculating the current frame stitching information; step C: calculating the homography matrix inside and outside the overlapping area; step D: fusing to obtain the stitching result image. The present invention is a fusion of the global homography matrix and the block homography matrix, which takes advantage of the speed and stability of the global homography matrix, and also takes advantage of the block homography matrix to ensure the alignment of targets in the overlapping area. The use of historical feature points also makes up for the shortcomings of insufficient feature points in certain scenes of infrared images; the present invention greatly improves the accuracy, real-time performance and inter-frame stability of infrared binocular image stitching.

Description

Technical Field

[0001] The present invention relates to the field of infrared image processing technology, and in particular to a real-time infrared binocular image stitching method and system. Background Art

[0002] Image stitching technology is based on the contradiction between the size of the image scene and the resolution of the imaging device. When people need to obtain an image with the largest field of view, they must adjust the focal length of the imaging device. Since the resolution of the imaging device is fixed, the larger the field of view, the lower the resolution. Conversely, the higher the resolution, the smaller the field of view. Image stitching technology was born to meet people's demand for scene images with both high resolution and large field of view. With the continuous improvement of image stitching related theories and the continuous development of computer applications, image stitching technology has become one of the hot spots in image processing.

[0003] Infrared image stitching technology is the process of stitching two or more scene images with small fields of view into a seamless target image with wide fields of view while ensuring that the resolution remains unchanged. The stitching process not only reduces the redundancy between images, but also the scene image with wide fields of view allows the observer to observe richer scene information at a stable image resolution. Infrared stitching technology can be applied to infrared early warning, panoramic scanning, target recognition and exploration.

[0004] The existing infrared binocular stitching technology mainly includes the following technical modules: feature extraction, matching, purification, homography matrix calculation and fusion.

[0005] For feature extraction, matching, and purification modules, they can be classified into two categories according to feature type: point feature-based and line feature-based. Among them: Based on point features: including feature point positioning, feature representation, feature point pair matching, and purification. Among them, orb+size relationship feature representation+brute force matching+ransac / Bian Jiawang version of GMS purification is a commonly used method with high computational efficiency and low time consumption; surf point+gradient feature representation+nearest neighbor distance matching+ransac purification is a method with good feature point performance and stable registration effect, but it is more time-consuming; based on line features: LCD / LSD straight line segment detection and edge extraction methods (canny, etc.) are commonly used methods for line segment extraction.

[0006] Obtaining the homography matrix: For the coordinates of the point pairs after the point features are purified, the homography matrix is ​​divided into the following methods according to the mathematical calculation method: least squares method, grid neighborhood DLT and other methods. Among them, DLT is more resistant to noise than the least squares method. The homography matrix can be divided into global homography matrix and block homography matrix according to whether it is global. Among them, the global homography matrix uses all the point pairs to obtain an H; the block homography matrix is ​​obtained based on the image block, and the H of each block is calculated, and the H of each block is gradually changed on the whole image; for the line feature extraction, Fourier Mellin transform matching is often used: the transformation relationship between the line segments of the two images is obtained in the frequency domain + logarithmic polar coordinate system. Compared with the point feature directly obtaining each element of H, the Fourier Mellin transform obtains the rotation, scale, and translation information, and then constitutes H. It also shows that the H obtained by the point feature can contain more information besides rotation, scale, and translation.

[0007] Fusion: According to the fusion method, it is divided into direct fusion, fade-in and fade-out fusion, dividing line fusion, etc. The ghosting suppression ability and time consumption increase in turn.

[0008] The difficulty of real-time infrared binocular stitching technology lies in the balance between stitching accuracy, real-time performance and inter-frame stability. In point feature extraction and matching algorithms, orb is fast, but the number of points located at corners and edges is small; surf feature point is slow, but the number of points located at corners and edges is large. In addition, in infrared images with sparse scene targets, the number of feature points is sparse, which is easy to cause mismatching. In line feature extraction and matching algorithms, the LSD region growing method is time-consuming and not conducive to hardware implementation; the edge extraction method is prone to ignore weak edge information and is easily interfered by noise; affected by the edge features involved in the calculation, the result obtained by Fourier Mellin transform is not robust enough, and the result is a global homography matrix, which is not conducive to the alignment of various image regions. At the same time, multiple Fourier transforms and complex number calculations have a large memory overhead; the global homography matrix follows the minimum overall matching error, so there is a large local matching error; and the block homography matrix can adapt to different transformation relationships at different depths of field of the image, which is easy to meet the matching alignment everywhere, but the calculation is time-consuming. How to make the registration result of a single frame visually stable in the registration results of multiple consecutive frames is a difficult point in real-time stitching processing. Affected by the changes in scene content: the depth of field, the position of feature points, and the homography matrix all change, and the registration results will jitter between frames. Some scholars have proposed stitching first and then anti-shake, or anti-shake first and then stitching, or using video context for stitching and anti-shake at the same time, which undoubtedly increases the complexity of stitching and reduces real-time performance. Summary of the invention

[0009] The first object of the present invention is to provide a real-time infrared binocular image stitching method to solve the technical problems of low stitching accuracy, poor real-time performance and poor inter-frame stability of existing infrared binocular image stitching methods.

[0010] The first object of the present invention is achieved by adopting the following technical solution: a real-time infrared binocular image stitching method, comprising the following steps:

[0011] Step A: Pre-stitching, obtaining the stitching information of N consecutive frames of image data, the N frames of image data are recorded as in, are the i-th frame image data obtained by the left and right cameras respectively;

[0012] Step B: Calculate the current frame stitching information and obtain the coordinate frame information rect of the feature points of the current frame convas And the outer rectangular frame

[0013] Step C: Calculate the homography matrix {H ij}, i = 1, ..., C1, j = 1, ..., C2, where C1 and C2 are the number of vertical and horizontal blocks;

[0014] Step D: Fusion to obtain the stitching result image.

[0015] Furthermore, the step A includes the following sub-steps:

[0016] Step A1: extract, match and purify feature points of each frame image;

[0017] Step A2: purifying feature points of N frames of images;

[0018] Step A3: Calculate the global homography matrix Hg using the feature points obtained in step A2 last , historical overlapping areas, historical feature points Kpt last .

[0019] Furthermore, in the step A1, the method for extracting feature points from each frame of image includes orb, the matching method includes brute force matching, and the purification method includes the ransac method.

[0020] Furthermore, the step A2 includes the following sub-steps:

[0021] Step A21: Fusing the purified feature point pair information of N frames;

[0022] Step A22: Purify the feature point pairs obtained in step A21.

[0023] Furthermore, the step A3 includes the following sub-steps:

[0024] Step A31: Calculate the global homography matrix Hg from the feature point pair Kpt2 obtained in step A2 last ;

[0025] Step A32: The global homography matrix Hg obtained in step A31 last Get the coordinate frame information of the overlapping area in the left image

[0026] Step A33: Hg obtained from step A31 last The overlapping area obtained in step A32 is in the coordinate box of the left figure Generate historical feature point pair Kpt last .

[0027] Furthermore, the step B includes the following sub-steps:

[0028] Step B1: Get the feature point pair Kpt of the current frame cur ;

[0029] Step B2: Kpt is obtained from the feature points of the current frame cur Calculate the current global homography matrix Hg cur ;

[0030] Step B3: Get the coordinate frame information rect of the overlapping area stitching result image convas ;

[0031] Step B4: Get the current feature point pair Kpt cur In the outer rectangle of the left picture

[0032] Furthermore, the step C includes the following sub-steps:

[0033] Step C1: Filter historical feature point pairs to obtain feature point pairs within the historical coordinate frame and outside the current coordinate frame

[0034] Step C2: Obtained from step C1 And the current frame feature point pair Kpt cur Fusion, get fusion feature point pair

[0035] Step C3: Obtained from step C2 According to the overlapping area, calculate the homography matrix {H ij},i=1,...,C1,j=1,...,C2.

[0036] Furthermore, the step D comprises the following sub-steps:

[0037] Step D1: Left Image Y L according to Mapping is performed to obtain a mapping diagram Right picture Y R according to Mapping is performed to obtain a mapping diagram

[0038] Step D2: and The fusion results in a stitched image.

[0039] The second object of the present invention is to provide a real-time infrared binocular image stitching system to solve the technical problems of low stitching accuracy, poor real-time performance and poor inter-frame stability of existing infrared binocular image stitching methods.

[0040] The second object of the present invention is achieved by the following technical means: a real-time infrared binocular image stitching system, used to implement the above-mentioned real-time infrared binocular image stitching method, including a preprocessing module, a stitching information calculation module, an overlap region inner and outer homography matrix calculation module and a fusion module, the preprocessing module is used to obtain stitching information of continuous N frames of image data, the N frames of image are recorded as in, The i-th frame image data obtained by the left and right cameras respectively; the splicing information calculation module is used to calculate the splicing information of the current frame and obtain the coordinate frame information rect of the feature point of the current frame convas And the outer rectangular frame The homography matrix calculation module inside and outside the overlap region is used to calculate the homography matrix {H ij}, i = 1, ..., C1, j = 1, ..., C2, wherein C1 and C2 are the numbers of longitudinal and transverse blocks; the fusion module is used to fuse and obtain a spliced ​​result image.

[0041] The beneficial effects of the present invention are as follows: the present invention takes into account both target alignment in the overlapping area and inter-frame stability outside the overlapping area by means of "blocking in the overlapping area and stabilization outside the overlapping area" and "gradual transition inside and outside the overlapping area is achieved by means of historical point features and current point features". It is a fusion of the global homography matrix and the block homography matrix, which gives full play to the speed and stability of the global homography matrix, while also giving full play to the block homography matrix to ensure target alignment in the overlapping area. The use of historical feature points also makes up for the disadvantage of insufficient feature points in certain scenes of infrared images. The present invention greatly improves the accuracy, real-time performance and inter-frame stability of time-infrared binocular image stitching. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0043] Figure 1 It is the overall flow chart of the present invention;

[0044] Figure 2 It is a pre-joining flow chart;

[0045] Figure 3 Flowchart of feature extraction, matching and purification for each frame of image;

[0046] Figure 4 It is the flow chart of purifying feature point pairs of N frames of images;

[0047] Figure 5 This is the flow chart of feature point pair deduplication and fusion;

[0048] Figure 6 Flowchart for computing global homography matrix, overlapping area, and historical features;

[0049] Figure 7 The flowchart for calculating the coordinate box of the overlapping area in the left figure;

[0050] Figure 8 Generate a flow chart for historical feature point pairs;

[0051] Fig. 9 Get the flow chart for the current frame splicing information;

[0052] Fig.10 A flow chart for obtaining the coordinate frame information of the overlapping area in the stitching result image;

[0053] Fig.11 Flowchart for calculating homography matrix inside and outside the overlapping area;

[0054] Fig.12 Calculate the flow chart of homography matrix for each block;

[0055] Fig.13 Flowchart for finding the homography matrix for block midpoints that are within the overlap region;

[0056] Fig.14 This is the image fusion flow chart;

[0057] Fig.15 This is a system block diagram of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0059] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0060] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0061] Embodiment 1:

[0062] See also Figure 1 , a real-time infrared binocular image stitching method, comprising the following steps:

[0063] Step A: Pre-stitching, obtaining the stitching information of N consecutive frames of image data, the N frames of image data are recorded as in, are the i-th frame image data obtained by the left and right cameras respectively;

[0064] Step B: Calculate the current frame stitching information and obtain the coordinate frame information rect of the feature points of the current frame convas And the outer rectangular frame

[0065] Step C: Calculate the homography matrix {H ij}, i = 1, ..., C1, j = 1, ..., C2, where C1 and C2 are the number of vertical and horizontal blocks;

[0066] Step D: Fusion to obtain the stitching result image.

[0067] See also Figure 2 In step A, pre-stitching: obtain the stitching information of N consecutive frames of image data, and the N frames of image data are recorded as in, The i-th frame image data obtained by the left and right cameras are respectively. The left and right cameras collect data at the same time, and the fields of view overlap to a certain extent. The images have the same size, with height and width being denoted as H×W. Further, the step A includes the following sub-steps:

[0068] Step A1: extract, match and purify feature points of each frame image;

[0069] Step A2: purifying feature points of N frames of images, where N may be 4;

[0070] Step A3: Calculate the global homography matrix Hg using the feature points obtained in step A2 last , historical overlapping areas, historical feature points Kpt last .

[0071] For further information, see Figure 3 , the step A1 specifically includes the following sub-steps:

[0072] Step A11, Perform point feature extraction to obtain feature points Each feature point Contains location information x, y and feature descriptor;

[0073] Step A12, Perform point feature extraction to obtain feature points Each feature point Contains location information x, y and feature descriptor;

[0074] Step A13, the feature points of the i-th frame image of the left and right cameras obtained in steps A11 and A12 are obtained. Perform initial matching to obtain feature matching point pairs Each matching point pair Contains the coordinate information of the point pair in the left and right images [x L ,y L ,x R ,y R ];

[0075] Step A14, matching point pairs of the i-th frame image of the left and right cameras obtained in step A13 After purification, remove the wrong matching points and get Among them, the feature point extraction method in steps A11 and A12 can be but not limited to orb; the initial matching method in step A13 can be but not limited to brute force matching; the feature point pair purification method in step A14 can be but not limited to the ransac method.

[0076] In this embodiment, refer to Figure 4 , the step A2 comprises the following sub-steps:

[0077] Step A21: Fuse the feature point pair information after N frames of purification, denoted as Kpt = {{kpt i}}, i = 1, ... N, Kpt is an array of length ne, where ne is the number of feature point pairs, and each array element contains the coordinate information of the point pair in the left and right images [x L ,y L ,x R ,y R ];

[0078] Step A22: purify the feature point pairs obtained in step A21. The purification method may be but is not limited to the ransac method to obtain point pairs Kpt2. The length of Kpt2 is nf, where nf is the number of feature point pairs. Each array element contains the coordinate information of the point pair in the left and right images [x L ,y L ,x R ,y R ].

[0079] In this embodiment, refer to Figure 5 , the step A21 further comprises the following sub-steps:

[0080] The fusion method of step A21 may be but is not limited to de-duplication fusion, and the de-duplication fusion process is:

[0081] Step A211, calculate point pairs The distance dist from each pair of points in Kpt;

[0082] Step A212, judging whether the point pair is adopted by Kpt based on the distance:

[0083] If the distance is less than the threshold thr_dist, the point pair is regarded as a coincidence point and is not adopted; otherwise, the point pair is adopted and put into the Kpt array.

[0084] In this embodiment, refer to Figure 6 , the step A3 comprises the following sub-steps:

[0085] Step A31: Calculate the global homography matrix Hg using the feature point pair Kpt2 obtained in step A2, using but not limited to the DLT method last , Hg last is a 3×3 matrix, Hg last Indicates: coordinates of the feature points in the left image (x L ,y L ) to the coordinates of the feature points on the right (x R ,y R ) relationship: Hg last *[x L ,y L ,1] T =[x R ,y R ,1] T ;

[0086] Step A32: Hg obtained from step A31 last , image size H, W to get the coordinate frame information of the overlapping area in the left image The coordinate frame contains [rect_x,rect_y,rect_w,rect_h], where rect_x and rect_y are the horizontal and vertical coordinates of the upper left vertex, and rect_w and rect_h are the frame width and frame height;

[0087] Step A33: Hg obtained from step A31 last The overlapping area obtained in step A32 is in the coordinate box of the left figure Generate historical feature point pair Kpt last .

[0088] For further information, see Figure 7 , the step A32 further comprises the following sub-steps:

[0089] Step A321: Initialization Can be initialized to [1,1,W,H];

[0090] Step A322: traverse the left image pixel coordinates (x, y), x∈[1, W], y∈[1, H], and calculate the mapping point coordinates (u, v):

[0091]

[0092] If (u,v) is within the coordinate range of the right figure, update

[0093]

[0094] Step A323: After step A322 is completed, update the coordinate frame Height and width:

[0095] rect_h=rect_h-rect_y+1;

[0096] rect_w=rect_w-rect_x+1.

[0097] For further information, see Figure 8 , the step A33 further comprises the following sub-steps:

[0098] Step A331, obtain the coordinate frame External expansion information, nx and ny are the horizontal and vertical expansion amounts respectively, and nx and ny can be 60:

[0099] x0 = rect_x - nx;

[0100] x1=rect_x+rect_w-1+nx+nx;

[0101] y0=rect_y-ny;

[0102] y1=rect_y+rect_h-1+ny+ny;

[0103] Step A332, generate sampling point coordinates X, Y:

[0104] In the interval [x0,x1], with equal spacing xstep, generate X=x0:xstep:x1;

[0105] In the interval [y0,y1], with equal spacing ystep, generate Y=y0:ystep:y1; where the spacing xstep and ystep can be 10;

[0106] Step A333, traverse the points composed of X and Y, using Hg last , get the coordinates of the corresponding point in the right figure, and add the historical feature point pair Kpt last , each element in the historical feature point pair contains [x i ,y i ,u i ,v i ], the calculation process is as follows:

[0107] u i =(x i ×h 11 +y i ×h 12 +h 13 ) / s

[0108] v i =(x i ×h 21 +y i ×h 22 +h 23 ) / s

[0109] s=x i ×h 31 +y i ×h 32 +h 33 ;

[0110] Among them, {x i ,y i} is the point composed of X and Y, {u i ,v i} is {x i ,y i} last After mapping,

[0111] hij is Hg last The element in the i-th row and j-th column of the matrix.

[0112] In this embodiment, refer to Fig. 9Step B: Calculate the current frame splicing information, including obtaining the current frame feature point pair Kpt cur , global mapping matrix Hg cur , the coordinate frame information rect of the overlapping area in the stitching result image convas 、Current feature point pair Kpt cur The outer rectangular box distributed in the left image Furthermore, the step B includes the following sub-steps:

[0113] Step B1: Get the feature point pair Kpt of the current frame cur ; Same as step A1, for the current frame image {Y L ,Y R}Get the feature point pair, and record the purified point pair as Kpt cur ={kpt j}, j = 1, ..., nd, where nd is the number of feature point pairs, and each array element contains the coordinate information of the point pair in the left and right images [x L ,y L ,x R ,y R ]. Current frame image {Y L ,Y R} is the real-time image data after N frames of pre-stitched data;

[0114] Step B2: As described in step A31, the feature points of the current frame are used to obtain Kpt cur Calculate the current global homography matrix Hg cur ;

[0115] Step B3: Get the coordinate frame information rect of the overlapping area stitching result image convas ;

[0116] Step B4: Get the current feature point pair Kpt cur In the outer rectangle of the left picture The specific method is to traverse Kpt cur The left coordinate point [x L ,y L ], the range of the horizontal and vertical coordinates is counted, forming

[0117] For further information, see Fig.10 , the step B3 further comprises the following sub-steps:

[0118] Step B31, Hg cur Calculate the stitched image size, nH_com, nW_com;

[0119] The four boundary points of the left image are curAfter mapping, nW_com is the difference between the maximum horizontal coordinate and the minimum horizontal coordinate of the mapping result, and nH_com is the difference between the maximum vertical coordinate and the minimum vertical coordinate of the mapping result;

[0120] Step B32, traverse the stitching image coordinates (i, j), i∈[min(1, minimum vertical coordinate of the mapping result), max(1, maximum vertical coordinate of the mapping result)], j∈[min(1, minimum horizontal coordinate of the mapping result), max(1, maximum horizontal coordinate of the mapping result)], if the following two conditions are met at the same time, (i, j) is the coordinate frame information rect convas Inside a point and update rect convas information:

[0121] Condition 1: i∈[1,H] and j∈[1,W];

[0122] Condition 2: u∈[1,H] and v∈[1,W], (u,v) is the value of (i,j) The mapped coordinates.

[0123] In this embodiment, refer to Fig.11 , the step C calculates the homography matrix {H ij}, i = 1, ..., C1, j = 1, ..., C2, C1, C2 are the numbers of vertical and horizontal blocks, C1, C2 can be integers between 4 and 10, specifically including the following sub-steps:

[0124] Step C1: Filter historical feature point pairs to obtain feature point pairs within the historical coordinate frame and outside the current coordinate frame The specific method is: traverse the historical feature point pair Kpt last The coordinates of the left point [x L ,y L ], if it is located in If the point is outside the box, then the point pair is added to

[0125] Step C2: Obtained from step C1 And the current frame feature point pair Kpt cur Fusion, get fusion feature point pair

[0126] Step C3: Obtained from step C2 According to the overlapping area, calculate the homography matrix {H ij},i=1,...,C1,j=1,...,C2, further, calculate the overlapping area rect convas The homography matrix of each block in the overlapped area is linearly fused with the historical global homography matrix. The homography matrix outside the overlapped area takes the historical global homography matrix (rect convasThe block algorithm may be, but is not limited to, aanap. For speed-up considerations, the present invention makes improvements based on the block algorithm aanap. convas Internally and externally, calculate the corresponding

[0127] For further information, see Fig.12 , the step C3 further comprises the following sub-steps:

[0128] Step C31, if the block midpoint is located in rect convas Outside, the block on the left is the unit matrix, right Hg last ;

[0129] Step C32, if the block midpoint is located in rect convas , then proceed to the following sub-steps (see Fig.13 ):

[0130] Step C321, calculate the block vertex (x, y) and The feature points form a weight matrix W and a coordinate matrix A;

[0131] Sort by weight value, select the first Q data with larger weight value to form W, and the corresponding coordinates to form A matrix;

[0132] Step C322, the homography matrix H1 may be solved by, but is not limited to, weighted SVD;

[0133] Step C323, according to the aanap algorithm idea, calculate the linear combination of Hl_linear, Hl and Hl_linear in the non-overlapping area of ​​Hl, and update the combination result to Hl;

[0134] Step C324, calculate Hl and Hg according to the aanap algorithm last The linear combination of Left

[0135] In this embodiment, refer to Fig.14 , the step D fusion comprises the following sub-steps:

[0136] Step D1: Left Image Y L according to Mapping is performed to obtain a mapping diagram Right picture Y R according to Mapping is performed to obtain a mapping diagram

[0137] Step D2: and The spliced ​​result image is obtained by fusion, and the fusion method can be but is not limited to progressive fusion.

[0138] See also Fig.15 Based on the same inventive concept, in order to realize the above-mentioned real-time infrared binocular image stitching method, the present invention also proposes a real-time infrared binocular image stitching system, which includes a preprocessing module, a stitching information calculation module, an overlapping area inner and outer homography matrix calculation module and a fusion module, wherein the preprocessing module is used to obtain stitching information of continuous N frames of image data, and the N frames of image data are recorded as in, The i-th frame image data obtained by the left and right cameras respectively; the splicing information calculation module is used to calculate the splicing information of the current frame and obtain the coordinate frame information rect of the feature point of the current frame convas And the outer rectangular frame The homography matrix calculation module inside and outside the overlap region is used to calculate the homography matrix {H ij}, i = 1, ..., C1, j = 1, ..., C2, wherein C1 and C2 are the numbers of longitudinal and transverse blocks; the fusion module is used to fuse and obtain a spliced ​​result image.

[0139] The present invention takes into account both target alignment in the overlapping area and inter-frame stabilization outside the overlapping area by means of “blocking in the overlapping area and stabilization outside the overlapping area” and “gradual transition inside and outside the overlapping area is achieved with the help of historical point features and current point features”. It is a fusion of the global homography matrix and the block homography matrix, which gives full play to the speed and stability of the global homography matrix, while at the same time giving full play to the block homography matrix to ensure target alignment in the overlapping area. The use of historical feature points also makes up for the shortcomings of insufficient feature points in certain scenes of infrared images. Finally, it is explained that the present invention is a splicing algorithm that “divides the overlapping area into blocks and stabilizes outside the overlapping area” and “gradual transition inside and outside the overlapping area is achieved with the help of historical point features and current point features”. Within the framework of blocking in the overlapping area and stabilization outside, the use of point features or other features falls within the scope of protection of the present invention.

[0140] It should be noted that, for the above embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification belong to preferred embodiments, and the actions involved are not necessarily required by the present application.

[0141] The above embodiments describe the basic principles and main features of the present invention and the advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the changes and modifications made by those skilled in the art shall be within the scope of protection of the appended claims of the present invention without departing from the spirit and scope of the present invention.

Claims

1. A real-time infrared binocular image stitching method, characterized in that: The steps include: Step A: Pre-stitching, obtaining the stitching information of N consecutive frames of image data, the N frames of image data are recorded as in, The i-th frame image data is obtained by the left and right cameras respectively; the step A includes the following sub-steps: Step A1: extract, match and purify feature points of each frame image; Step A2: purify the feature points of N frames of images to obtain purified feature point pairs Kpt2; Step A3: Calculate the global homography matrix Hg using the feature point pair Kpt2 obtained in step A2 last , historical overlapping areas, historical feature points Kpt last ; Step B: Calculate the current frame stitching information and obtain the coordinate frame information rect of the feature points of the current frame convas And the outer rectangular frame of the current feature point pair in the left image The step B comprises the following sub-steps: Step B1: Get the feature point pair Kpt of the current frame cur ; Step B2: Kpt is obtained from the feature points of the current frame cur Calculate the current global homography matrix Hg cur ; Step B3: Get the coordinate frame information rect of the overlapping area stitching result image convas ; Step B4: Get the current feature point pair Kpt cur In the outer rectangle of the left picture Step C: Calculate the homography matrix {H ij }, i = 1, ..., C1, j = 1, ..., C2, where C1 and C2 are the numbers of vertical and horizontal blocks; the step C includes the following sub-steps: Step C1: Filter historical feature point pairs to obtain feature point pairs within the historical coordinate frame and outside the current coordinate frame Step C2: Obtained from step C1 And the current frame feature point pair Kpt cur Fusion, get fusion feature point pair Step C3: Obtained from step C2 According to the overlapping area, calculate the homography matrix {H ij },i=1,...,C1,j=1,...,C2; calculate the overlapping area rect convas The homography matrix of each block in the overlapped area is linearly fused with the historical global homography matrix. The homography matrix outside the overlapped area takes the historical global homography matrix. convas Internally and externally, calculate the corresponding Step D: Fusion to obtain a spliced ​​result image; Step D includes the following sub-steps: Step D1: Left Image Y L according to Mapping is performed to obtain a mapping diagram Right picture Y R according to Mapping is performed to obtain a mapping diagram Step D2: and The fusion results in a stitched image.

2. A real-time infrared binocular image stitching method as claimed in claim 1, characterized in that: In the step A1, the method for extracting feature points from each frame of image includes orb, the matching method includes brute force matching, and the purification method includes ransac method.

3. A real-time infrared binocular image stitching method as claimed in claim 1, characterized in that: The step A2 comprises the following sub-steps: Step A21: Fusing the purified feature point pair information of N frames; Step A22: Purify the feature point pair obtained in step A21 to obtain a purified feature point pair Kpt2.

4. A real-time infrared binocular image stitching method as claimed in claim 1, characterized in that: The step A3 comprises the following sub-steps: Step A31: Calculate the global homography matrix Hg from the feature point pair Kpt2 obtained in step A2 last ; Step A32: The global homography matrix Hg obtained in step A31 last Get the coordinate frame information of the overlapping area in the left image Step A33: Hg obtained from step A31 last The overlapping area obtained in step A32 is in the coordinate box of the left figure Generate historical feature point pair Kpt last .

5. A real-time infrared binocular image stitching system, used to implement a real-time infrared binocular image stitching method according to any one of claims 1 to 4, characterized in that: It includes a preprocessing module, a splicing information calculation module, a homography matrix calculation module inside and outside the overlapping area, and a fusion module. The preprocessing module is used to obtain the splicing information of N consecutive frames of image data. The N frames of image data are recorded as in, The i-th frame image data obtained by the left and right cameras respectively; the splicing information calculation module is used to calculate the splicing information of the current frame and obtain the coordinate frame information rect of the feature point of the current frame convas And the outer rectangular frame of the current feature point pair in the left image The homography matrix calculation module inside and outside the overlap region is used to calculate the homography matrix {H ij }, i = 1, ..., C1, j = 1, ..., C2, wherein C1 and C2 are the numbers of longitudinal and transverse blocks; the fusion module is used to fuse and obtain a spliced ​​result image.

Images