Fast registration and fusion method and device for heterogeneous images based on optical coaxial structure

Abstract

The present application relates to a kind of fast registration fusion methods of heterogeneous images based on optical coaxial structure, the method includes obtaining image to be processed, image to be processed is the infrared and visible light camera photographed infrared image to be registered and visible light image to be registered;Infrared image to be registered and visible light image to be registered corresponding registered infrared image and registered visible light image are obtained, registered infrared image and registered visible light image are obtained by first pre-processing, first pre-processing is that infrared image to be registered and visible light image to be registered are processed by calibration target image center correction, correction adopts homography matrix algorithm;Registered infrared image and registered visible light image are processed by weighted fusion, to obtain the fused fusion image after fusion.The present application also relates to a kind of fast registration fusion device of heterogeneous images based on optical coaxial structure.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a fast heterogeneous image registration and fusion method and apparatus based on an optical coaxial structure. Background Technology

[0002] Pixel-level image fusion combines the image information of registered images into a single image. This method can compensate for the insufficient information obtained by a single sensor, providing more valuable information for subsequent image target detection and recognition algorithms. Figure 1 As shown in the example, from left to right are the visible light image, the infrared image, and the fused image. Image fusion can enhance the clarity and brightness of nighttime images, while highlighting information with strong thermal radiation.

[0003] Pixel-level heterogeneous image fusion consists of two steps: pixel-level heterogeneous image registration and pixel-level fusion of registered images. Pixel-level heterogeneous image registration involves aligning images of the same target from different sensors pixel by pixel, ensuring that pixels at the same location in the image to be fused correspond to completely consistent positions in actual space.

[0004] Currently, mainstream image fusion methods mainly include traditional methods and deep learning-based methods. Traditional methods include sparse representation-based curve wavelet transform, dual-tree complex wavelet transform, and non-subsampled shear wavelet transform, while deep learning-based methods include Pulse Coupled Neural Network (PCNN).

[0005] like Figure 2 As shown, traditional methods primarily extract feature points using feature point extraction algorithms and then use consistency matching and clustering methods to remove non-corresponding feature points, thereby finding corresponding points in heterogeneous images. The homography transformation matrix H is calculated using these corresponding points. Let the heterogeneous images to be registered be A and B, then:

[0006] A′=AH

[0007] B′=BH

[0008] A' and B' are images after registration, and if the registration accuracy is high enough, A'(x, y) and B'(x, y) represent the same part of the target in space. The traditional pixel-level image registration of heterogeneous images is to extract feature points, remove the wrong matching points after consistency matching, and finally obtain the homography transformation matrix according to the filtered matching feature points to realize the registration of heterogeneous images. However, this method has the following two obvious shortcomings: 1) the conventional feature point extraction algorithms such as SIFT, SURF, ORB and the like have long calculation time and high calculation complexity; and 2) for the fusion of infrared images and visible light images, the feature point extraction algorithm often leads to insufficient effective feature points due to the low spatial resolution of infrared images, resulting in poor consistency matching results and large registration error after registration transformation of the homography matrix.

[0009] For the deep learning type of heterogeneous image registration algorithm, the registration accuracy can be better, but for the supervised deep learning method, a large amount of related data set needs to be obtained, resulting in large amount of work in the early preparation, and for the unsupervised deep learning method, the work amount of the parameter acquisition training process is large. In summary, as a prerequisite for pixel-level image fusion, the current mainstream method cannot realize fast and accurate pixel-level registration, and occupies a large amount of computer resources in the subsequent heterogeneous image fusion process.

[0010] At the same time, the traditional method and the deep learning method based on the correlation coefficient (CC), the structural similarity (SSIM), the mutual information (NMI) and the like reach a relatively high level. However, the main problem of this kind of method is that the parameters need to be manually set, and the second problem is that the calculation time of these algorithms is long. Taking the NSST-based image fusion algorithm with good performance in the traditional algorithm as an example, under the hardware condition of desktop CPU and independent graphics card, the fusion processing frame rate is less than 1 frame / s, which cannot meet the real-time requirement. The real-time performance of the detection algorithm after fusion is also seriously affected. SUMMARY

[0011] The application provides a fast heterogeneous image registration and fusion method and device based on an optical coaxial structure to solve the problem that the traditional image fusion method and the deep learning-based fusion method cannot realize fast and accurate pixel-level registration.

[0012] The technical scheme adopted by the application is as follows:

[0013] In a first aspect, the application discloses a fast heterogeneous image registration and fusion method based on an optical coaxial structure, comprising:

[0014] acquire a to-be-processed image, the to-be-processed image being a to-be-registered infrared image and a to-be-registered visible light image captured by an infrared and visible light camera;

[0015] acquire registered infrared and visible light images corresponding to the to-be-registered infrared and visible light images, the registered infrared and visible light images being acquired through first preprocessing, the first preprocessing being image center correction processing of the to-be-registered infrared and visible light images through a calibration target, the correction adopting a homography matrix algorithm;

[0016] perform weighted fusion processing on the registered infrared and visible light images to obtain a fused image after fusion.

[0017] In an implementable embodiment, acquiring the registered infrared and visible light images corresponding to the to-be-registered infrared and visible light images comprises:

[0018] acquire infrared and visible light images of a calibration target, the infrared and visible light images of the calibration target each having n*n solid circle images, n≥3;

[0019] align the picture centers of the infrared and visible light images of the calibration target;

[0020] perform Hough circle detection on the infrared and visible light images of the calibration target to acquire a center coordinate of each circle and an average center coordinate of all center coordinates;

[0021] determine an included angle between a picture center coordinate of the visible light image and the average center coordinate of the infrared and visible light images of the calibration target;

[0022] determine a rotation matrix according to the included angle;

[0023] fix the visible light image, perform translation and rotation on the infrared image according to the rotation matrix to acquire registered infrared and visible light images with consistent picture centers and angles.

[0024] In an implementable embodiment, after acquiring the registered infrared and visible light images corresponding to the to-be-registered infrared and visible light images, the method further comprises:

[0025] perform scaling processing on the registered infrared image through a scaling matrix.

[0026] In an implementable embodiment, performing weighted fusion processing on the registered infrared and visible light images comprises:

[0027] perform Fourier transform processing on the registered infrared and visible light images;

[0028] performing filtering processing on the registered infrared image and the registered visible light image after Fourier transform processing;

[0029] performing inverse Fourier transform processing on the registered infrared image and the registered visible light image after filtering processing;

[0030] performing weighted fusion processing on the registered infrared image and the registered visible light image after inverse Fourier transform processing.

[0031] In an implementable embodiment, before performing Fourier transform processing on the registered infrared image and the registered visible light image, the method comprises:

[0032] performing enhancement processing on the registered infrared image, the enhancement processing adopting a local gray level histogram equalization algorithm.

[0033] In an implementable embodiment, performing filtering processing on the registered infrared image and the registered visible light image after Fourier transform processing comprises:

[0034] performing high-pass filtering processing and low-pass filtering processing on the registered infrared image and the registered visible light image after Fourier transform processing, respectively.

[0035] In an implementable embodiment, performing weighted fusion processing on the registered infrared image and the registered visible light image after inverse Fourier transform processing comprises:

[0036] performing weighted fusion processing on the registered infrared image and the registered visible light image obtained through low-pass filtering processing to obtain a low-frequency fusion image;

[0037] performing weighted fusion processing on the registered infrared image and the registered visible light image obtained through high-pass filtering processing to obtain a high-frequency fusion image;

[0038] performing weighted fusion processing on the low-frequency fusion image and the high-frequency fusion image to obtain a fused fusion image.

[0039] In a second aspect, the present application further discloses a rapid heterogenous image registration and fusion device based on an optical coaxial structure, comprising:

[0040] an acquisition module configured to acquire to-be-processed images, the to-be-processed images being to-be-registered infrared images and to-be-registered visible light images captured by infrared and visible light cameras;

[0041] The correction module is used for acquiring the registered infrared image and the registered visible light image corresponding to the to-be-registered infrared image and the to-be-registered visible light image, the registered infrared image and the registered visible light image being acquired through a first preprocessing, the first preprocessing being image center correction of the to-be-registered infrared image and the to-be-registered visible light image through a calibration target, and the correction adopting a homography matrix algorithm.

[0042] The fusion module is used for performing weighted fusion processing on the registered infrared image and the registered visible light image to obtain a fused image after fusion.

[0043] In a third aspect, the application further discloses a lens device based on an optical coaxial structure, comprising an infrared lens, a visible light lens and an optical lens.

[0044] The center of the light inlet hole of the lens device, the center of the optical lens and the center of the photosensitive element of the infrared lens are located on the same horizontal line, and the horizontal line serves as a light inlet horizontal line.

[0045] The photosensitive element of the visible light lens is arranged in parallel to the light inlet horizontal line, the center of the photosensitive element of the visible light lens and the center of the optical lens are located on the same straight line, and the straight line serves as a reflection vertical line.

[0046] The reflection vertical line and the light inlet horizontal line are perpendicular to each other and located in the same plane.

[0047] The optical lens is a fused quartz medium film mirror.

[0048] In a fourth aspect, the application further discloses a calibration target for image center correction, which is applied to the above-mentioned fast heterogenous image registration and fusion method based on an optical coaxial structure, and is used for image center correction of a to-be-registered infrared image and a to-be-registered visible light image, and comprises a target plate, target marks and heat-emitting sheets.

[0049] The target plate has n*n target marks on one side, and n is greater than or equal to 3.

[0050] The other side of the target plate has n*n heat-emitting sheets corresponding to the target marks.

[0051] The technical scheme of the application has the following beneficial effects:

[0052] The application discloses a fast heterogeneous image registration fusion method based on an optical coaxial structure, and relates to the technical field of image registration fusion.

[0053] Further, the application provides a lens device based on an optical coaxial structure, which is combined with a calibration target for image center correction, and is matched with a fast pixel-level registration fusion method based on the optical coaxial structure, so that fast and accurate registration of heterogeneous images is realized, and a processing frame rate of more than 30 frames / s can be realized on an embedded platform. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.

[0055] Figure 1 A schematic diagram of visible light images, infrared images and fusion images in the background art;

[0056] Figure 2 A flowchart of a fusion image of a traditional algorithm in the background art;

[0057] Figure 3 A schematic diagram of a lens device based on an optical coaxial structure of the application (1-optical lens; 2-heat dissipation fan; 3-infrared lens; 4-visible light lens);

[0058] Figure 4 Another schematic diagram of a lens device based on an optical coaxial structure of the application (1-optical lens; 2-heat dissipation fan; 3-infrared lens; 4-visible light lens; 5-wiring);

[0059] Figure 5 A front view of a calibration target for image center correction of the application;

[0060] Figure 6 A back view of a heating sheet of a calibration target for image center correction of the application;

[0061] Figure 7 Visible light under the calibration target image schematic diagram of the present application;

[0062] Figure 8 Infrared under the calibration target image schematic diagram of the present application;

[0063] Figure 9 Visible light under the calibration target hough circle detection schematic diagram of the present application;

[0064] Figure 10 Infrared under the calibration target hough circle detection schematic diagram of the present application;

[0065] Figure 11 The schematic diagram of the present application for determining the rotation angle (the angle between the picture center coordinates of the visible light image and the average circle center coordinates of the infrared image and the visible light image of the calibration target) (6- visible light image to be registered; 7- infrared image to be registered; 8- infrared under the calibration target image; 9- visible light under the calibration target image);

[0066] Figure 12 The registered image flow chart of the present application is a kind of fast registration fusion method of heterogeneous image based on optical coaxial structure;

[0067] Figure 13 The flow chart of the present application is a kind of fast registration fusion method of heterogeneous image based on optical coaxial structure;

[0068] Illustration:

[0069] Wherein, 1-optical lens; 2-heat dissipation fan; 3-infrared lens; 4-visible light lens; 5-wiring;

[0070] 6- visible light image to be registered; 7- infrared image to be registered; 8- infrared under the calibration target image; 9- visible light under the calibration target image. DETAILED DESCRIPTION

[0071] The following will be described in detail, its example is shown in the drawings. The following description relates to the drawings, unless otherwise indicated, the same numbers in different figures represent the same or similar elements. The following embodiments described in the embodiment do not represent all embodiments consistent with the present application. Only the example of the system and method consistent with some aspects of the present application detailed in the claims.

[0072] The traditional infrared image and visible light image fusion method is to extract feature points on two images, match the feature points, and calculate a homography matrix. Through the improvement, the registration speed is greatly accelerated. For registration of infrared images and visible light images, the feature point extraction of the infrared image is often inaccurate due to the low spatial resolution, resulting in inaccurate calculation of the homography matrix, and thus a large registration error. The method of the application can avoid the method and achieve high-precision registration.

[0073] In the image fusion link, the application mainly performs pre-calculation of a homographic transformation matrix through the assistance of a designed optical coaxial mechanism and calibration of a double-mode calibration target. The application does not use a pre-link such as multi-scale decomposition or dictionary construction to decompose the image. Instead, the application uses a butterworth filter-based image filtering to separate high-frequency and low-frequency components. In the fusion parameter, a self-defined network fitting layer is used to select parameters, and the number of parameter updates is reduced to 8, which greatly accelerates the fusion speed while ensuring the fusion quality. Details are as follows.

[0074] Referring to Figures 1 to 13 .

[0075] In the first aspect, as shown in Figure 3 and Figure 4 , the application discloses a lens device based on an optical coaxial structure, comprising an infrared lens, a visible light lens, and an optical lens; the center of the light inlet hole of the lens device, the center of the optical lens, and the center of the photosensitive element of the infrared lens are located on the same horizontal line, which serves as a light inlet horizontal line; the photosensitive element of the visible light lens is arranged in parallel with the light inlet horizontal line, and the center of the photosensitive element of the visible light lens and the center of the optical lens are located on the same straight line, which serves as a reflection vertical line; the reflection vertical line and the light inlet horizontal line are perpendicular to each other and located in the same plane; and the optical lens is a fused quartz dielectric film mirror.

[0076] As shown in Figure 3 , a common optical coaxial mechanism is constructed to facilitate the acquisition of center-approximate visible light and infrared images. The optical lens in the figure is a fused quartz dielectric film mirror, which can reflect more than 98% of light with a wavelength of 350nm-950nm, that is, it can almost completely reflect visible light, and 100% of light with a wavelength of 950nm or more, that is, it can completely transmit infrared radiation.

[0077] As shown in Figure 4As shown, the infrared lens and the visible light lens are arranged within a coaxial device. The center of the light-entry aperture, the center of the fused silica dielectric film reflector, and the center of the infrared camera's photosensitive film are all on the same horizontal line, called the light-entry horizontal line. The fused silica dielectric film reflector is placed at a 45° angle. The longitudinal central axis of the infrared camera's photosensitive element is perpendicular to the device base. The visible light camera's photosensitive film is placed parallel to the light-entry horizontal line, and its geometric center is on the same straight line as the light-entry horizontal line and the focal point of the fused silica dielectric film reflector; this straight line is called the reflection vertical line. The reflection vertical line and the light-entry horizontal line are perpendicular and lie in the same plane.

[0078] With this mechanism design, the light reflected from the target is split into visible light and infrared bands after passing through the light entrance aperture, and then captured by the visible light camera and infrared camera respectively. This results in visible light and infrared images with the center of the image aligned. However, due to assembly errors in the installation of the infrared camera, visible light camera, and lens, the image center may not be consistent.

[0079] Secondly, the present invention discloses a calibration target for image center correction, which is applied to the following fast heterogeneous image registration and fusion method based on optical coaxial structure for image center correction of infrared image to be registered and visible light image to be registered. The target includes a target plate, a target, and a heating element; one side of the target plate has n*n targets, n≥3; the other side of the target plate has n*n heating elements corresponding to the targets.

[0080] like Figure 5 As shown, the calibration target for image center correction according to the present invention consists of two parts: a visible light-infrared dual-mode image calibration target. The front side has five rows and five columns of solid black circles, and the back side has 25 heating elements of the same size as the solid black circles, attached to the back of the calibration target in the same positions to form an infrared heating target. The heating elements are as follows... Figure 6 As shown.

[0081] To prevent the heating wires in the heating element from generating high temperatures during operation, which could lead to uneven circular highlight patterns during imaging and affect the subsequent homography matrix calculation, thermally conductive silicone is used to attach the circular heating element to the back of the corresponding circular pattern on the visible light-infrared dual-mode calibration target. The silicone dissipates heat, creating a uniform circular heating area. During attachment, precise alignment with the circular pattern on the front is crucial to prevent errors during calibration and ensure registration accuracy. Once completed, connecting the power supply completes the fabrication of the visible light-infrared dual-mode calibration target. The images of this calibration target formed by the visible light camera and the infrared camera are shown below. Figure 7 and Figure 8 As shown.

[0082] Thirdly, this invention discloses a fast heterogeneous image registration and fusion method based on an optical coaxial structure, comprising:

[0083] S100: acquiring a to-be-processed image, the to-be-processed image being a to-be-registered infrared image and a to-be-registered visible light image captured by an infrared and visible light camera.

[0084] S200: acquiring registered infrared and visible light images corresponding to the to-be-registered infrared and visible light images, the registered infrared and visible light images being acquired through first preprocessing, the first preprocessing being image center correction processing of the to-be-registered infrared and visible light images through a calibration target, and the correction adopting a homography matrix algorithm.

[0085] As shown in Figure 12 , the step S200 specifically includes the following steps:

[0086] S201: acquiring infrared and visible light images of a calibration target, the infrared and visible light images of the calibration target both having n*n solid circle images, n≥3.

[0087] In the embodiment, n is 5. As shown in Figure 7 , it is a visible light calibration target image schematic diagram of the present application, Figure 8 , it is an infrared calibration target image schematic diagram of the present application.

[0088] S202: aligning picture centers of the infrared and visible light images of the calibration target.

[0089] Specifically, due to the existence of an optical coaxial mechanism, the picture centers are basically at consistent positions. Assuming that the resolution of the visible light image is w1xh1, the resolution of the infrared image is w2xh2, the picture center point (w1 / 2, h1 / 2) of the visible light image is aligned with the picture center point (w2 / 2, h2 / 2) of the infrared image through image translation, and the aligned point O is set.

[0090] S203: performing Hough circle detection on the infrared and visible light images of the calibration target to acquire a circle center coordinate of each circle and an average circle center coordinate of all circle center coordinates.

[0091] Specifically, Hough circle detection is performed on the two images, and the circle center positions of all circles are obtained. As shown in Figure 9 , it is a visible light calibration target Hough circle detection schematic diagram of the present application; Figure 10 , it is an infrared calibration target Hough circle detection schematic diagram of the present application.

[0092] As shown in Figure 11 , assuming that the picture center of the target target in the visible light image is , and the picture center of the target target in the infrared image is Using the Hough circle detection result of the above figure, the coordinates of the center of all circles in the visible light image and the infrared image are obtained ij (x ij , y ij ), and the average center coordinates of all circles are obtained Due to the assembly error of the coaxial optical device, the center of the double-mode calibration target in the visible light image and the infrared image is inconsistent, that is There is an included angle between

[0093] S204: Determine the included angle between the picture center coordinates of the visible light image and the average center coordinates of the infrared image and the visible light image of the calibration target.

[0094] Where the rotation angle θ can be calculated by

[0095]

[0096] S205: Determine the rotation matrix according to the included angle.

[0097] Fix the visible light image, and match the visible light image with the infrared image, rotating the infrared picture in the clockwise direction. After the rotation angle is obtained, the rotation matrix H1 can be obtained, as shown in the following formula.

[0098]

[0099] S206: Fix the visible light image, and perform translation and rotation on the infrared image according to the rotation matrix to obtain a registered infrared image and a registered visible light image with consistent picture center and angle.

[0100] Through the above image translation and rotation operation, the infrared and visible light images with completely consistent center and completely consistent imaging picture angle can be obtained.

[0101] S300: Perform scaling processing on the registered infrared image through a scaling matrix.

[0102] Due to the difference between the imaging devices of the visible light camera and the infrared camera, there is a size difference problem in the actual imaging object. The scaling matrix is calculated to perform image scaling:

[0103] The form of the scaling matrix is as shown in the following formula:

[0104]

[0105] Where W is the scaling factor in the image row direction, and H is the scaling factor in the image column direction. At the same time, considering the error in the Hough circle detection of the infrared image, the average scaling scale is calculated to reduce the error. The calculation method is as shown in the following formula.

[0106]

[0107] In the scaling process, considering that the resolution of the visible light image is generally higher than that of the infrared image, the linear interpolation method selected by the present application is used to scale up the infrared image.

[0108] Supposing that the infrared image is B, the transformed infrared image B' is obtained by the following formula.

[0109] B' = H2H1B

[0110] 1) When the infrared image is scaled and rotated, if it exceeds the range of the visible light image, the intersection point of the two can be taken as the vertex of the registration image, and the inscribed rectangle can be taken as the frame of the registration image.

[0111] 2) When the infrared image is rotated and a black edge appears, random pixel or 0 pixel filling can be performed to prevent the influence on subsequent image fusion and target tracking network.

[0112] S400: performing weighted fusion processing on the registered infrared image and the registered visible light image to obtain a fused fusion image.

[0113] Specifically, step S400 includes the following steps.

[0114] S401: performing enhancement processing on the registered infrared image, wherein the enhancement processing adopts a local gray level histogram equalization algorithm.

[0115] In order to improve the distinguishability of the fusion image, the infrared image is first preprocessed. The preprocessing method is to perform local gray level histogram equalization on the infrared image. For different application scenarios, the operation is performed on the part with high or low gray value of the infrared image to enhance the information of the scene with low or high temperature. The present application takes the enhancement of the high temperature point as an example.

[0116] Supposing that the image gray level distribution function is H = h(x), x is the gray value, and H is the number of pixel points corresponding to the gray value. The total gray value of the image is shown in the following formula.

[0117]

[0118] d max is the maximum value of the image gray value, and x is the horizontal axis of the gray histogram. The expression of the total gray value increase function g(x) is shown in the following formula.

[0119] g(x) = H sum (x) = ∫0 x h(t)dt x∈(0, 255)

[0120] Suppose that the required local histogram equalization is achieved by a monotonic nonlinear mapping f: A→B, i.e. applying f to each pixel in the original infrared image A to obtain the equalized image B. In order to enhance the part of the infrared image with higher temperature, i.e. the part with higher gray value. The local histogram equalization transformation region is the interval from the point where the first derivative of the function g(x) is equal to 0 to the maximum value on the right side. That is, the monotonic nonlinear mapping function is as follows:

[0121]

[0122] where x0 satisfies g'(x0)=0. Since it is difficult to obtain the analytical expression of the function g(x) in practice, we obtain x0 by approximation as follows. max x0≈x1-h(x1) in h(x1).

[0123] Suppose that the number of pixel points with a gray value greater than x0 is t, and the gray level depth is 1, then the transformation function f has the following relationship.

[0124]

[0125] The discrete form of the function f is as follows.

[0126]

[0127] Considering the problem of processing pixels, the actual algorithm uses every 5 gray levels as an interval to improve the running speed of the algorithm. Through this step, the infrared image is enhanced.

[0128] S402: Fourier transform processing is performed on the registered infrared image and the registered visible light image.

[0129] After obtaining the enhanced infrared image, Fourier transform is performed on the infrared image and the visible light image. The transformation mode is as follows:

[0130]

[0131] S403: Filtering processing is performed on the registered infrared image and the registered visible light image after Fourier transform processing.

[0132] In step S403, high-pass filtering processing and low-pass filtering processing are respectively performed on the registered infrared image and the registered visible light image after Fourier transform processing.

[0133] After completing the Fourier transform, butterworth filtering is performed on the two images. Butterworth high-pass filtering and butterworth low-pass filtering are respectively performed on the infrared image and the visible light image. The butterworth high-pass filter is as follows:

[0134]

[0135] Butterworth low-pass filter is shown as follows:

[0136]

[0137] wherein D(u,v) is a distance function, n is the order, D0 is the cutoff frequency, and the present application takes n=1 considering the requirement of calculation speed.

[0138] S404: performing inverse Fourier transform processing on the filtered registered infrared image and registered visible light image.

[0139] After the butterworth filtering is completed, inverse Fourier transform is performed, and the inverse Fourier transform is shown as follows:

[0140]

[0141] The visible light image and infrared image after butterworth high-pass filtering and the visible light image and infrared image after butterworth low-pass filtering are obtained, and a total of four images are obtained.

[0142] In the above butterworth filter, a Butterworth low-pass filter with a cutoff frequency of D il , D vl and a Butterworth high-pass filter with a cutoff frequency of D ih and D vh are used respectively.

[0143] S405: performing weighted fusion processing on the registered infrared image and registered visible light image after the inverse Fourier transform processing.

[0144] Step S405 specifically includes:

[0145] S4051: performing weighted fusion processing on the registered infrared image and registered visible light image obtained after low-pass filtering to obtain a low-frequency fusion image; and performing weighted fusion processing on the registered infrared image and registered visible light image obtained after high-pass filtering to obtain a high-frequency fusion image.

[0146] The visible light image and infrared image after butterworth low-frequency filtering are added through weights W v and W i to obtain a low-frequency fusion image; and the visible light image and infrared image after butterworth high-frequency filtering are added through weights W v and Wi The addition operation is performed to obtain a low-frequency fusion image.

[0147] S4052: The low-frequency fusion image and the high-frequency fusion image are subjected to weighted fusion processing to obtain a fused fusion image.

[0148] The low-frequency fusion image and the high-frequency fusion image are subjected to weighted addition operation through weights fl and fh.

[0149] The weighted addition operation in the above steps is shown in the following formula:

[0150] f(x, y) = aA(x, y) + bB(x, y)

[0151] f is the fused image, a and b are fusion weights, and A and B are two images to be fused.

[0152] In summary, there are 8 parameters to be determined in this fusion rule, which are the cut-off frequencies D ih and D vh of the high-frequency filters of the infrared image and the visible light image, the cut-off frequencies D il and D vl of the low-frequency filters of the infrared image and the visible light image, the fusion weights W v and W i of the visible light image and the infrared image, and the fusion weight coefficients fl and fh of the low-frequency fusion image and the high-frequency fusion image.

[0153] Among them, the functions of the above steps 402, 403 and 404 are written using the custom layer of the pytorch framework, and the 8 parameters to be updated are set to torch.nn.Parameter type, indicating that these parameters need to be updated in the training process, i.e. the process of back propagation.

[0154] An unsupervised training method is adopted, and the loss function is a comprehensive fusion of the loss and the structural similarity loss. The loss function loss is shown in the following formula:

[0155]

[0156] loss SSIM = (1-SSIM(F f , F i )) + (1-SSIM(F f , F v ))

[0157] loss = loss SSIM + sloss L2

[0158] F f , Fv and F i To fuse images, visible light images, and infrared images, loss L2 The losses are L2-loss, calculated from the fused image and the visible light image, and from the fused image and the infrared image, respectively, and are used to constrain the overall brightness of the fused result. SSIM Structural similarity loss is used to constrain the correlation of images and preserve image texture, brightness information, and contrast information. The calculation of structural similarity SSIM is shown in the following formula.

[0159]

[0160]

[0161] The optimizer chosen is SGD (Stochastic Gradient Descent). After training for 50 epochs, eight parameters are obtained, which are then used to determine the image fusion parameters.

[0162] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0163] This invention provides a lens device based on an optical coaxial structure. Combined with a calibration target for image center correction, a fast pixel-level, coaxial-based method for rapid heterogeneous image registration and fusion is designed, achieving fast and accurate heterogeneous image registration. On an embedded platform, a processing frame rate exceeding 30 frames per second can be achieved. Based on pixel-level image registration, an adaptive image fusion method based on a bilateral Butterworth filter is designed. This method maintains similarity to or surpasses current traditional algorithms in evaluation metrics such as correlation coefficient (CC), structural similarity (SSIM), and mutual information (NMI), while achieving rapid image fusion. On an embedded platform, a fusion speed exceeding 30 frames per second is achieved, demonstrating high practical value.

[0164] Fourthly, this invention discloses a fast heterogeneous image registration and fusion device based on an optical coaxial structure, comprising:

[0165] The acquisition module is used to acquire the image to be processed, which is an infrared image to be registered and a visible light image to be registered captured by an infrared and a visible light camera;

[0166] The correction module is used to acquire the registered infrared image and the registered visible light image corresponding to the infrared image to be registered and the visible light image to be registered. The registered infrared image and the registered visible light image are acquired through a first preprocessing. The first preprocessing is to perform image center correction on the infrared image to be registered and the visible light image to be registered through a calibration target. The correction adopts the homography matrix algorithm.

[0167] a fusion module configured to perform a weighted fusion process on the registered infrared image and the registered visible light image to obtain a fused image.

[0168] For the specific definition of the fast multi-source image registration and fusion device based on the optical coaxial structure, reference can be made to the definition of the fast multi-source image registration and fusion method based on the optical coaxial structure in the foregoing, which will not be repeated here. In addition, each module in the two-dimensional code fast reading device in the complex background can be realized by software, hardware, and a combination thereof, in whole or in part. Each module described above can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0169] It should be noted that the relationship terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to the process, method, article or device. Without more limitations, the element defined by the statement "including a..." does not exclude the presence of another identical element in the process, method, article or device including the element.

[0170] The above description is merely specific embodiments of the present application, enabling those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0171] It should be understood that the present application is not limited to what has been described and illustrated in the above description and the accompanying drawings, and various modifications and changes can be made without departing from the scope of the present application. The scope of the present application is only limited by the appended claims.

Claims

1. A fast multi-sensor image registration and fusion method based on optical coaxial structure, characterized in that, The method comprises the following steps: acquiring a to-be-processed image, the to-be-processed image being a to-be-registered infrared image and a to-be-registered visible light image captured by an infrared and visible light camera; acquiring registered infrared images and registered visible light images corresponding to the to-be-registered infrared image and the to-be-registered visible light image, the registered infrared images and the registered visible light images being acquired through first preprocessing, the first preprocessing being image center correction processing of the to-be-registered infrared image and the to-be-registered visible light image through a calibration target, the correction adopting a homography matrix algorithm; performing weighted fusion processing on the registered infrared images and the registered visible light images to obtain a fused fusion image; acquiring registered infrared images and registered visible light images corresponding to the to-be-registered infrared image and the to-be-registered visible light image comprises: acquiring infrared images and visible light images of a calibration target, the infrared images and the visible light images of the calibration target both having n*n solid circle images, n≥3; aligning the picture centers of the infrared images and the visible light images of the calibration target; performing Hough circle detection on the infrared images and the visible light images of the calibration target to acquire the center coordinates of each circle and the average center coordinates of all the center coordinates; determining the included angle between the picture center coordinates of the visible light image and the average center coordinates of the infrared images and the visible light images of the calibration target; determining a rotation matrix according to the included angle; fixing the visible light image and performing translation and rotation on the infrared image according to the rotation matrix to acquire registered infrared images and registered visible light images with consistent picture centers and angles; the weighted fusion processing on the registered infrared images and the registered visible light images comprises: performing Fourier transform processing on the registered infrared images and the registered visible light images; the filter processing on the registered infrared images and the registered visible light images after the Fourier transform processing comprises: performing high-pass filter processing and low-pass filter processing on the registered infrared images and the registered visible light images after the Fourier transform processing respectively; performing inverse Fourier transform processing on the registered infrared images and the registered visible light images after the filter processing; the weighted fusion processing on the registered infrared images and the registered visible light images after the inverse Fourier transform processing comprises: performing weighted fusion processing on the registered infrared images and the registered visible light images obtained through the low-pass filter processing to obtain a low-frequency fusion image; performing weighted fusion processing on the registered infrared images and the registered visible light images obtained through the high-pass filter processing to obtain a high-frequency fusion image; performing weighted fusion processing on the low-frequency fusion image and the high-frequency fusion image to obtain a fused fusion image.

2. The fast registration and fusion method of heterogeneous images based on optical coaxial structure according to claim 1, characterized in that, after acquiring the registered infrared images and the registered visible light images corresponding to the to-be-registered infrared image and the to-be-registered visible light image, the method further comprises: performing scaling processing on the registered infrared images through a scaling matrix.

3. The fast registration and fusion method of heterogeneous images based on optical coaxial structure according to claim 1, characterized in that, before performing the Fourier transform processing on the registered infrared images and the registered visible light images, the method comprises: performing enhancement processing on the registered infrared images, the enhancement processing adopting a local gray level histogram equalization algorithm.

4. A fast multi-sensor image registration and fusion device based on optical coaxial structure, characterized in that, ​ An acquisition module is configured to acquire a to-be-processed image, which is a to-be-registered infrared image and a to-be-registered visible light image captured by an infrared and visible light camera; A correction module is configured to acquire registered infrared and visible light images corresponding to the to-be-registered infrared and visible light images, the registered infrared and visible light images being acquired through a first preprocessing, the first preprocessing being image center correction on the to-be-registered infrared and visible light images through a calibration target, the correction adopting a homography matrix algorithm; A fusion module is configured to perform weighted fusion processing on the registered infrared and visible light images to obtain a fused fusion image; The registered infrared and visible light images corresponding to the to-be-registered infrared and visible light images are acquired, including: An infrared image and a visible light image of a calibration target are acquired, the infrared image and the visible light image of the calibration target both having n*n solid circle images, n≥3; The image centers of the infrared image and the visible light image of the calibration target are aligned; Hough circle detection is performed on the infrared image and the visible light image of the calibration target to acquire a center coordinate of each circle and an average center coordinate of all the center coordinates; An included angle between a picture center coordinate of the visible light image and the average center coordinate of the infrared image and the visible light image of the calibration target is determined; A rotation matrix is determined according to the included angle; The visible light image is fixed, and the infrared image is translated and rotated according to the rotation matrix to acquire registered infrared and visible light images with consistent picture centers and angles; The weighted fusion processing on the registered infrared and visible light images includes: The registered infrared and visible light images are subjected to Fourier transform processing; The registered infrared and visible light images subjected to the Fourier transform processing are subjected to filtering processing, including: The registered infrared and visible light images subjected to the Fourier transform processing are subjected to high-pass filtering processing and low-pass filtering processing, respectively; The registered infrared and visible light images subjected to the filtering processing are subjected to inverse Fourier transform processing; The registered infrared and visible light images subjected to the inverse Fourier transform processing are subjected to weighted fusion processing, including: The registered infrared and visible light images obtained through the low-pass filtering processing are subjected to weighted fusion processing to obtain a low-frequency fusion image; The registered infrared and visible light images obtained through the high-pass filtering processing are subjected to weighted fusion processing to obtain a high-frequency fusion image; The low-frequency fusion image and the high-frequency fusion image are subjected to weighted fusion processing to obtain a fused fusion image.

5. A lens device based on optical coaxial structure, applied to the fast heterogeneous image registration fusion method based on optical coaxial structure in any one of claims 1-3, characterized in that, The lens device includes an infrared lens, a visible light lens and an optical lens; Centers of light inlet holes of the lens device, the optical lens and a photosensitive element of the infrared lens are located on a same horizontal line, and the horizontal line serves as a light inlet horizontal line. The photosensitive element of the visible light lens is arranged in parallel with the light inlet horizontal line, the center of the photosensitive element of the visible light lens and the center of the optical lens are located on the same straight line, and the straight line serves as a reflection vertical line; The reflection vertical line and the light inlet horizontal line are perpendicular and located in the same plane; The optical lens is a fused quartz medium film mirror.

6. A calibration target for image center correction, applied to the fast multi-source image registration and fusion method based on optical coaxial structure as claimed in any one of claims 1-3, the image center correction of the to-be-registered infrared image and the to-be-registered visible light image, characterized in that, A target plate, a target and a heating sheet are included; One side of the target plate has n*n targets, n≥3; The other side of the target plate has n*n heating sheets corresponding to the targets.

Images