Infrared and visible light image registration method, system, device, and image processing terminal

Through the combination of YOLOv5 network detection and conditional generation adversarial network conversion, SURF descriptor matching model and RANSAC algorithm, the problem of resolution and color difference in infrared and visible image registration is solved, and high-precision and robust image matching is achieved Quasi-effect.

CN114529593BActive Publication Date: 2024-11-26XIDIAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210029468.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-12
Publication Date
2024-11-26
Estimated Expiration
2042-01-12

AI Technical Summary

Technical Problem

The prior art is difficult to effectively solve the resolution and color differences between infrared and visible images, resulting in low registration accuracy and poor robustness. It is difficult to achieve high-quality image registration in scenarios where the target size difference is large and the background is complex. .

Method used

The YOLOv5 network is used to detect visible image targets, and the conditions are generated and the visible image is converted into pseudo-infrared image. The matching model is constructed based on the SURF descriptor. The error matching points are filtered through distance and slope consistency. Finally, the homography is estimated using the RANSAC algorithm. Transform parameters for registration.

Benefits of technology

It improves the accuracy and robustness of infrared and visible images, and can effectively deal with scenes with large differences in target sizes and complex backgrounds, which significantly improves the accuracy of registration and anti-interference ability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114529593B_ABST
    Figure CN114529593B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of image processing technology, and discloses an infrared and visible light image registration method, system, device, and image processing terminal, including: (1) collecting infrared images and visible light images; (2) constructing a training data set, constructing and training a conditional generative adversarial neural network; (3) using a YOLOv5 network to detect visible light image targets; (4) using a generative adversarial neural network to generate a pseudo infrared image from a visible light image; (5) using a speeded up robust feature (SURF) algorithm to construct a feature descriptor, and using a brute force matching method to construct a matching model; (6) filtering mismatched points by distance and slope consistency; (7) using a random sampling consensus (RANSAC) algorithm to estimate transformation parameters to complete registration. The present invention can complete relatively accurate registration of infrared and visible light images with large target size differences and complex backgrounds, and is a practical solution for infrared and visible light image registration.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to an infrared and visible light image registration method, system, device, and image processing terminal. The method of the present invention can be used for infrared and visible light image registration tasks with large target size differences and complex backgrounds. Background Art

[0002] At present, multimodal images from different imaging devices can provide richer and more comprehensive information than single-modal images, and people have an increasing demand for the integration and utilization of various image information. Image registration refers to the process of determining the transformation parameters between images based on some similarity metrics, so that two or more images of the same scene acquired from different sensors, different perspectives, and different times are transformed to the same coordinate system and optimally matched at the pixel level. Image registration technology is a core technology that provides an operational benchmark for subsequent image processing tasks such as image stitching and image fusion, and is a hot research issue in the field of computer vision.

[0003] As an important multi-sensor image registration method, infrared and visible light image registration plays an important role in computer vision, robotics, power equipment fault detection, remote sensing and military applications. However, due to the huge difference in resolution and color between infrared images and visible light images, it is difficult to achieve the registration of infrared images and visible light images. In terms of resolution, infrared images are mostly within "500x960", which is much lower than the "2160x3840" resolution of visible light images. This leads to serious loss of grayscale details and image blur in infrared images, which is quite different from the clear texture in visible light grayscale images. In addition, due to the different imaging mechanisms of infrared images and visible light images, in the visible light band, the image contrast is determined by reflectivity and shadow; in the infrared band, the contrast is determined by emissivity and temperature. In particular, as a result of temperature difference, the contrast can vary over a wide range, which leads to large differences in color performance in heterogeneous images.

[0004] Aiming at the huge difference between the target information of infrared images and visible light images, as well as the low accuracy and poor robustness of using only SURF feature descriptors, it is hoped to obtain a new registration method that can be used for the registration of infrared and visible light images with large target size differences and complex backgrounds.

[0005] Through the above analysis, the problems and defects of the existing technology are: there is a huge difference between the target information of the existing infrared image and the visible light image, and the accuracy and robustness of using only the SURF feature descriptor are low.

[0006] The difficulty in solving the above problems and defects is: it is difficult to use existing technology to solve the registration of infrared and visible light images with large differences in resolution and color and complex backgrounds, and the registration accuracy and robustness are low. There is currently no better solution.

[0007] The significance of solving the above problems and defects is: the present invention provides a practical solution, improves the accuracy and robustness of infrared and visible light image registration, and effectively solves the problem of infrared and visible light image registration with huge differences in target information and complex background. Summary of the invention

[0008] In view of the problems existing in the prior art, the present invention provides an infrared and visible light image registration method, system, device, and image processing terminal.

[0009] The present invention is implemented as follows: a method for registering infrared and visible light images, wherein the method uses a YOLOv5 network to detect the approximate target area in a visible light image, uses a conditional generative adversarial network to modally convert the visible light image into a pseudo infrared image, constructs a feature description of the infrared image pair based on a SURF descriptor and uses brute force matching to construct a matching model, then uses distance constraints and slope consistency to filter matching points, and finally uses a random sampling consistency RANSAC algorithm to estimate the parameters of the homography transformation and complete the registration of the real infrared image.

[0010] Furthermore, the infrared and visible light image registration method comprises the following steps:

[0011] Step 1: Collect infrared images and visible light images;

[0012] Step 2: Build a training data set and conditional generative adversarial network and train them;

[0013] Step 3: Use the YOLOv5 network to detect targets in visible light images: Use the YOLOv5 network to detect targets in visible light images, and select pedestrians or bicycles based on the label. For pedestrians, the target frame is increased by 1 / 12 of the original height on the top and bottom, and the width is adjusted to 0.8 times the height based on the height; for bicycles, the target frame is increased by 1 / 30 of the original width on the left and right, and the height is adjusted to 1.25 times the width based on the width; for other categories, the original image is directly half the height and the width is 0.8 times the height. The original image is cropped according to the detection frame to obtain the approximate target area; Detecting the approximate area of ​​the visible light target first can effectively solve the problem of large resolution difference between infrared images and visible light images;

[0014] Step 4: Generate a pseudo infrared image using a conditional generative adversarial network. Scale the cropped visible light image to a size of 512 pixels wide and 640 pixels high and input it into the conditional generative adversarial network to obtain the corresponding pseudo infrared image. Remove the watermark from the original real infrared image and scale it to a size of 512 pixels wide and 640 pixels high. Generating a pseudo infrared image from a visible light image through a generative adversarial network can reduce the color difference of the image and reduce the difficulty of heterogeneous image registration.

[0015] Step 5: SURF algorithm constructs descriptors and brute force matching constructs matching models: The accelerated robust feature SURF algorithm is used to extract the positions of key points and construct descriptors for real infrared images and pseudo infrared images respectively. The brute force matching method is used to construct matching models between feature points based on Euclidean distance, and two best matches are selected for each key point.

[0016] Step 6: Filter mismatched points by using distance and slope consistency constraints. Distance and slope consistency constraints can be used to remove obviously incorrect mismatched points, thereby improving the accuracy of subsequent estimated transformation parameters.

[0017] Step 7: The random sampling consistency RANSAC algorithm is used to estimate the transformation parameters to complete the registration: the matching point pairs finally retained by the real infrared image and the pseudo infrared image are used to estimate the parameters of the homography transformation matrix H according to the random sampling consistency algorithm, and the real infrared image is transformed to complete the registration.

[0018] Furthermore, the step 1 of collecting infrared images and visible light images specifically includes: building a ResNet50 network as a feature extraction network of a twin convolutional neural network, the structure of which is: a first convolutional layer, a first BN layer, an activation function layer, a maximum pooling layer, a second convolutional layer, a second BN layer, a third convolutional layer, a third BN layer, a fourth convolutional layer, a fourth BN layer, a fifth convolutional layer, and a fifth BN layer; the number of convolution kernels of the first to fifth convolutional layers is set to 64, 64, 128, 256, and 512, respectively, and the convolution kernel sizes are set to 7, 3, 3, 3, and 3, respectively, the step sizes of the first, second, and third convolutional layers are set to 2, the step sizes of the fourth and fifth convolutional layers are set to 1, and the convolution kernel void rates in the fourth and fifth convolutional layers are set to 2 and 4; the size of the pooling area kernel of the maximum pooling layer is set to 3×3, and the step sizes are set to 2; the first to fifth BN layers all use a batch normalization function, the activation function layer uses a linear rectification function, and the maximum pooling layer uses a regional maximum pooling function.

[0019] Furthermore, the step 2 of constructing a training data set and a conditional generative adversarial network and training the network includes:

[0020] (1) Construct a training data set. The size difference between visible light images and infrared images is large. The visible light images are reduced by 3 times to 1280 pixels in height and 720 pixels in width. Four pairs of matching pixels are manually selected from the infrared and visible light image pairs. The homography transformation matrix H is determined by the matching pixels. The infrared image is transformed to be consistent with the size of the visible light image. The area containing the target is manually selected and cropped into infrared and visible light image pairs whose size is an integer multiple of 4.

[0021] (2) Construct a generator Gnet based on the Resnet network as the feature extraction network of the conditional generative adversarial network. Its structure includes a downsampling convolution module, 9 ResnetBlock modules, an upsampling deconvolution module and a Tanh component. The structure of the downsampling convolution module is as follows: the first padded convolution layer, the first convolution layer, the BN layer, the activation function layer, the second convolution layer, the BN layer, the activation function layer, the third convolution layer, the BN layer, the activation function layer; the parameter of the first padded convolution layer is 3, the number of convolution kernels of the first to third convolution layers is set to 64, 128, and 256 respectively, the convolution kernel size is set to 7, 3, and 3 respectively, the step size is set to 1, 2, and 2 respectively, and the padding is set to 0, 1, and 1 respectively; the structure of the RestnetBlock module is as follows: padded convolution layer, convolution layer, BN layer, activation function layer Layer, DropOut layer, padded convolution layer, convolution layer, BN layer; the padded convolution layer parameter is 1, the number of convolution kernels of the convolution layer is set to 256, the size is 3, the step size is 1, and the padding is 1; the upsampling deconvolution module structure is: the first deconvolution layer, BN layer, activation function layer, the second deconvolution layer, BN layer, activation function layer, the third padded convolution layer, the third convolution layer, activation function layer; the number of convolution kernels of the first and second deconvolution layers are set to 128 and 64 respectively, the convolution kernel size is 3, the step size is 2, and the padding is 1, the third padded convolution layer parameter is 3, the number of convolution kernels of the third convolution layer is 3, the convolution kernel size is 7, the step size is 1, and the padding is 0; all BN layers use batch normalization function, and the activation function layer uses linear rectification function except the last one which uses Tanh function;

[0022] (3) Construct a PatchGan-style Dnet as the discriminator network of the conditional generation network, whose structure includes the first convolution layer, activation function layer, second convolution layer, BN layer, activation function layer, third convolution layer, BN layer, activation function layer, fourth convolution layer, BN layer, activation function layer, and fifth convolution layer in sequence; the number of convolution kernels of the first to fifth convolution layers is set to 64, 128, 256, 512, 1 in sequence, the convolution kernel size is set to 4, the step size is set to 2, 2, 2, 1, 1 in sequence, the padding is set to 1, the BN layer uses the batch normalization function, and the activation function layer is set to the LeakyReLU function;

[0023] (4) Training the conditional generative adversarial network: input the training data set into the conditional generative adversarial network, and use the Adam algorithm to update the weights of the conditional generative adversarial network until the adversarial loss function Loss converges.

[0024] Furthermore, the homography transformation matrix H in (1) is defined as follows:

[0025]

[0026] Where s represents the scale factor, M is the camera intrinsic parameter matrix, and r1, r2, t represent the camera extrinsic parameters.

[0027] Furthermore, the linear rectification function in (2) is defined as follows:

[0028]

[0029] The Tanh function is defined as follows:

[0030]

[0031] The LeakyReLU function is defined as follows:

[0032]

[0033] Where a is a fixed parameter of (1, +∞), and x is the input;

[0034] The loss function Loss in (4) is defined as follows:

[0035]

[0036] The generation loss of Gnet is defined as follows:

[0037] Lg=-log D(G(I vis )-μ||I inf -G(I vis )||;

[0038] The adversarial loss of Dnet is defined as follows:

[0039] L d = -log(lD(I vis , G(I vis ))-log(D(I vis , I inf ));

[0040] Where LcGAN represents the total loss of the conditional generative adversarial network, G represents the generator Gnet, D represents the discriminator Dnet, x represents the input image, z represents the noise vector, and z is replaced by the DropOut layer; μ represents the weight of the second item in Lg, which is set to 100, Iinf represents the real infrared image, and Ivis represents the visible light image.

[0041] Further, the step 6 of filtering mismatched points by distance and slope consistency includes:

[0042] (1) For each key point, if the Euclidean distance of the first best match is less than 0.75 times the Euclidean distance of the second best match, the first best match is retained; otherwise, the two best matches are discarded.

[0043] (2) When the number of correctly matched feature point pairs is large, the slope of the straight line formed by the feature points in the real infrared image and the feature points in the pseudo infrared image is set to be less than 0.1 to filter out the mismatched points;

[0044] Another object of the present invention is to provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the infrared and visible light image registration method.

[0045] Another object of the present invention is to provide an intelligent image processing terminal, which is used to implement the infrared and visible light image registration method.

[0046] Another object of the present invention is to provide an infrared and visible light image registration system for implementing the infrared and visible light image registration method, the infrared and visible light image registration system comprising:

[0047] An image acquisition module, used for acquiring infrared images and visible light images;

[0048] A training data set construction module is used to construct a training data set, build and train a conditional generative adversarial neural network;

[0049] Image object detection module, used to detect visible light image objects using the YOLOv5 network;

[0050] A pseudo infrared image generation module, used to generate a pseudo infrared image from a visible light image using a generative adversarial neural network;

[0051] The matching model building module is used to accelerate the robust feature SURF algorithm to build feature descriptors and use the brute force matching method to build a matching model;

[0052] Mismatch point filtering module, used to filter mismatch points by distance and slope consistency;

[0053] The matching completion module is used to estimate the transformation parameters using the random sampling consistency RANSAC algorithm to complete the alignment.

[0054] Combining all the above technical solutions, the advantages and positive effects of the present invention are as follows: the present invention uses the YOLOv5 network to detect the approximate target area in the visible light image, which can effectively solve the problem of excessive image resolution difference; the conditional generative adversarial network is used to convert the visible light image into a pseudo infrared image, which can greatly reduce the color difference of heterogeneous images and reduce the difficulty of registration; the feature description of the infrared image pair is constructed based on the SURF descriptor and the matching model is constructed by brute force matching; then the matching points are filtered by using distance constraints and slope consistency to improve the registration accuracy; finally, the random sampling consensus (RANSAC) algorithm is used to estimate the parameters of the homography transformation and complete the registration of the real infrared image. It is used to improve the accuracy of the registration of infrared and visible light images with large size differences and complex backgrounds.

[0055] The registration system of the present invention can use the YOLOv5 network to detect the approximate area of ​​the target in the visible light image to solve the problem of large resolution difference between the visible light image and the infrared image, and adopts distance and slope consistency constraints to solve the problem of mismatching, so that the present invention can be applied to infrared and visible light image registration scenarios with large target size difference and complex background, and has the advantages of high precision and good robustness.

[0056] The present invention utilizes a conditional generative adversarial network to perform modal conversion on visible light images to generate pseudo infrared images to solve the problem of large color differences in heterogeneous images, and overcomes the problems of low accuracy and poor anti-interference ability when directly using SURF features for registration. This makes the present invention have the advantages of strong generalization ability and the ability to cope with infrared and visible light image registration scenarios with a lot of interference and complex background. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a flow chart of the infrared and visible light image registration method provided by an embodiment of the present invention.

[0058] Figure 2 is a structural schematic diagram of an infrared and visible light image registration system provided by an embodiment of the present invention;

[0059] Figure 2 In: 1. Image acquisition module; 2. Training data set construction module; 3. Image target detection module; 4. Pseudo infrared image generation module; 5. Matching model construction module; 6. Mismatch point filtering module; 7. Matching completion module.

[0060] Figure 3 It is a flow chart of the implementation of the infrared and visible light image registration method provided by an embodiment of the present invention.

[0061] Figure 4 It is a schematic diagram of the network structure of the generator Gnet constructed by the present invention.

[0062] Figure 5 It is a schematic diagram of the ResnetBlock module structure constructed by the present invention.

[0063] Figure 6 It is a schematic diagram of the network structure of the discriminator Dnet constructed by the present invention.

[0064] Figure 7 This is the result of the registration of infrared and visible light images of bicycles according to the present invention.

[0065] Figure 8 This is the result of the registration of the infrared and visible light images of pedestrians by the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0067] In view of the problems existing in the prior art, the present invention provides an infrared and visible light image registration method, system, device, and image processing terminal. The present invention is described in detail below in conjunction with the accompanying drawings.

[0068] like Figure 1 As shown, the infrared and visible light image registration method provided by the present invention comprises the following steps:

[0069] S101: Collect infrared images and visible light images;

[0070] S102: Build a training data set, build and train a conditional generative adversarial neural network;

[0071] S103: Detect visible light image targets using the YOLOv5 network;

[0072] S104: Generate a pseudo infrared image from the visible light image using a generative adversarial neural network;

[0073] S105: Use the SURF algorithm to construct feature descriptors and use the brute force matching method to build a matching model;

[0074] S106: filtering mismatched points by distance and slope consistency;

[0075] S107: Use the random sampling consensus (RANSAC) algorithm to estimate the transformation parameters to complete the alignment.

[0076] Ordinary technicians in the industry can also use other steps to implement the infrared and visible light image registration method provided by the present invention, such as replacing SURF features with features such as SIFT and ORB, and replacing RANSAC algorithm with the least squares method. Figure 1 The infrared and visible light image registration method provided by the present invention is only a specific embodiment.

[0077] like Figure 2 As shown, the infrared and visible light image registration system provided by the present invention includes:

[0078] Image acquisition module 1, used for acquiring infrared images and visible light images;

[0079] Training data set construction module 2, used to construct a training data set, build and train a conditional generative adversarial neural network;

[0080] Image target detection module 3, used to detect visible light image targets using the YOLOv5 network;

[0081] A pseudo infrared image generation module 4, used for generating a pseudo infrared image from a visible light image by using a generative adversarial neural network;

[0082] The matching model building module 5 is used to build feature descriptors using the accelerated robust feature (SURF) algorithm and to build a matching model using a brute force matching method;

[0083] Mismatched point filtering module 6, used to filter mismatched points by distance and slope consistency;

[0084] The matching completion module 7 is used to use the random sampling consensus (RANSAC) algorithm to estimate the transformation parameters to complete the registration.

[0085] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0086] like Figure 3 As shown, the infrared and visible light image registration method of the present invention comprises the following steps:

[0087] Step 1: Collect infrared images and visible light images: An ordinary mobile phone is equipped with a near-infrared camera to simultaneously capture visible light images and infrared images of the same scene, such as pedestrians or bicycles. The resolution of the visible light image is 2160 pixels wide and 3840 pixels high, and the resolution of the infrared image is 507 pixels wide and 960 pixels high. There are 100 pairs of images for each scene, for a total of 200 pairs of images.

[0088] Step 2: Build a training dataset and conditional generative adversarial network and train it:

[0089] (2a) Construct a training data set. The size difference between visible light images and infrared images is large. The visible light images are reduced by 3 times to 1280 pixels in height and 720 pixels in width. Four pairs of matching pixels are manually selected from the infrared and visible light image pairs. The homography transformation matrix H is determined by the matching pixels. The infrared image is transformed to be consistent with the size of the visible light image. The area containing the target is manually selected and cropped into infrared and visible light image pairs whose size is an integer multiple of 4.

[0090] The homography transformation matrix H is defined as follows:

[0091]

[0092] Where s represents the scale factor, M is the camera intrinsic parameter matrix, and r1, r2, t represent the camera extrinsic parameters.

[0093] like Figure 4 As shown, the Gnet in the conditional generative adversarial network constructed by the present invention is further described.

[0094] Figure 4is a schematic diagram of the structure of Gnet as a generator, which includes a downsampling convolution module, 9 ResnetBlock modules, an upsampling deconvolution module and a Tanh component in sequence. The structure of the downsampling convolution module is: the first padded convolution layer, the first convolution layer, the BN layer, the activation function layer, the second convolution layer, the BN layer, the activation function layer, the third convolution layer, the BN layer, the activation function layer; the parameter of the first padded convolution layer is 3, the number of convolution kernels of the first to third convolution layers is set to 64, 128, 256, respectively, the convolution kernel size is set to 7, 3, 3, the step size is set to 1, 2, 2, and the padding is set to 0, 1, 1; the structure of the upsampling deconvolution module is: the first deconvolution layer, BN layer, activation function layer, the second deconvolution layer, BN layer, activation function layer, the third padded convolution layer, the third convolution layer, activation function layer; the number of convolution kernels in the first and second deconvolution layers are set to 128 and 64 respectively, the convolution kernel size is 3, the step size is 2, and the padding is 1. The parameters of the third padded convolution layer are 3, the number of convolution kernels in the third convolution layer is 3, the convolution kernel size is 7, the step size is 1, and the padding is 0; all BN layers use batch normalization function, and the activation function layer uses linear rectification function except the last one which uses Tanh function.

[0095] The linear rectification function is defined as follows:

[0096]

[0097] The Tanh function is defined as follows:

[0098]

[0099] like Figure 5 As shown, the ResnetBlock module in the Gnet constructed by the present invention is further described.

[0100] Figure 5 This is a schematic diagram of the ResnetBlock module structure. The structure is: padded convolution layer, convolution layer, BN layer, activation function layer, DropOut layer, padded convolution layer, convolution layer, BN layer; the padded convolution layer parameter is 1, the number of convolution kernels in the convolution layer is set to 256, the size is 3, the step size is 1, and the padding is 1.

[0101] like Figure 6 As shown, Dnet in the conditional generative adversarial network constructed by the present invention is further described.

[0102] Figure 6is a schematic diagram of the structure of the discriminator Dnet, which includes the first convolution layer, activation function layer, second convolution layer, BN layer, activation function layer, third convolution layer, BN layer, activation function layer, fourth convolution layer, BN layer, activation function layer, and fifth convolution layer in sequence; the number of convolution kernels of the first to fifth convolution layers are set to 64, 128, 256, 512, 1 in sequence, the convolution kernel size is set to 4, the step size is set to 2, 2, 2, 1, 1 in sequence, the padding is set to 1, the BN layer uses batch normalization function, and the activation function layer is set to LeakyReLU function.

[0103] (2d) Training the conditional generative adversarial network: input the training data set into the conditional generative adversarial network, and use the Adam algorithm to update the weights of the conditional generative adversarial network until the adversarial loss function Loss converges.

[0104] The loss function Loss is defined as follows:

[0105]

[0106] The generation loss of Gnet is defined as follows:

[0107] Lg=-logD(G(I vis ))-μ||I inf -G(I vis )||

[0108] The adversarial loss of Dnet is defined as follows:

[0109] L d = -log(1-D(I vis , G(I vis ))-log(D(I vis , I inf ))

[0110] Where LcGAN represents the total loss of the conditional generative adversarial network, G represents the generator Gnet, D represents the discriminator Dnet, x represents the input image, z represents the noise vector, and z is replaced by the DropOut layer; μ represents the weight of the second item in Lg, which is set to 100, Iinf represents the real infrared image, and Ivis represents the visible light image.

[0111] Step 3: Use the YOLOv5 network to detect visible light image targets:

[0112] The YOLOv5 network is used to detect targets in visible light images. Pedestrians or bicycles are selected based on the label. For pedestrians, the target frame is increased by 1 / 12 of the original height on the top and bottom, and the width is adjusted to 0.8 times the height based on the height. For bicycles, the target frame is increased by 1 / 30 of the original width on the left and right, and the height is adjusted to 1.25 times the width based on the width. For other categories, the original image is directly cropped to obtain the approximate target area based on the detection frame with half the original image height and 0.8 times the width.

[0113] Step 4: Generate a pseudo infrared image using a conditional generative adversarial network:

[0114] The cropped visible light image is scaled to a size of 512 pixels wide and 640 pixels high and input into the conditional generative adversarial network to obtain the corresponding pseudo infrared image. The original real infrared image is removed of the watermark when it was shot and scaled to a size of 512 pixels wide and 640 pixels high.

[0115] Step 5: SURF algorithm constructs descriptors and brute force matching constructs matching models:

[0116] The speeded up robust feature (SURF) algorithm is used to extract the positions of key points and construct descriptors for real infrared images and pseudo infrared images respectively. The brute force matching method is used to build a matching model between feature points based on the Euclidean distance, and two best matches are selected for each key point.

[0117] Step 6: Filter mismatched points based on distance and slope consistency:

[0118] (6a) For each key point, if the Euclidean distance of the first best match is less than 0.75 times the Euclidean distance of the second best match, the first best match is retained, otherwise the two best matches are discarded;

[0119] (6b) When the number of correctly matched feature point pairs is large, the slope of the straight line formed by the feature points in the real infrared image and the feature points in the pseudo infrared image is set to be less than 0.1 to filter out the mismatched points;

[0120] Step 7: The random sampling consensus (RANSAC) algorithm estimates the transformation parameters to complete the registration:

[0121] The matching point pairs finally retained from the real infrared image and the pseudo infrared image are used to estimate the parameters of the homography transformation matrix H according to the random sampling consistency algorithm, and the real infrared image is transformed to complete the registration.

[0122] The technical effects of the present invention are described in detail below in conjunction with simulation experiments.

[0123] 1. Simulation conditions:

[0124] The hardware platform of the simulation experiment of the present invention is: the processor is Intel (R) Core (TM) i7-10700K CPU, the main frequency is 2.9 GHz, the memory is 64 GB, and the graphics card is NVIDIA GeForce RTX3060.

[0125] The software platform of the simulation experiment of the present invention is: Ubuntu20.04 operating system, Pycharm2021 software, python3.7 and Pytorch deep learning framework.

[0126] 2. Simulation content and result analysis:

[0127] When generating the training set and test set in the simulation experiment of the present invention, the pedestrian and bicycle data sets taken on campus were used, with 100 pairs of images in each category, a total of 400 images. In the simulation experiment of the present invention, 50 pairs of images in each category were used as the training data set, and the other 50 pairs were used as the test data set.

[0128] In the simulation experiment, the existing technologies used are:

[0129] The accelerated robust feature method proposed by Herbert Bay, Tinne Tuytelaars et al. in "SURF: Speeded up robust features [C] / / Proceedings of the 9th European conference on Computer Vision-Volume Part I. Springer-Verlag, 2006." is referred to as the SURF algorithm.

[0130] In order to qualitatively evaluate the simulation effect of the present invention, the present invention is used to test the bicycle and pedestrian data sets respectively, and the obtained registration results are as follows: Figure 7 and Figure 8 shown. Figure 7 (a) are the visible light image and the real infrared image detected and cropped by YOLOv5, Figure 7 (b) are the registration results after SURF algorithm and the registration results after the present invention, respectively. Figure 7 (c) are the results of SURF algorithm registration and fusion and the results of the present invention registration and fusion, respectively. Figure 8 Same reason.

[0131] Depend on Figure 7 , Figure 8 It can be concluded that the subjective evaluation effect of the present invention on the registration of infrared and visible light images is better than that based on the SURF algorithm.

[0132] In order to quantitatively evaluate the simulation effect of the present invention, the present invention uses mean absolute error (MAE), peak signal-to-noise ratio (PSNR), normalized mutual information (NMI) and structural similarity (SSIM) as performance evaluation indicators to compare with the prior art. The comparison results are shown in Table 1:

[0133] Table 1 Comparison of evaluation indicators of the present invention and the prior art in simulation experiments

[0134] Methods\Indicators mMAE mPSNR mNMI mSSIM Prior art 56.7484 9.5436 0.1590 0.4692 The present invention 34.0360 13.7719 0.2393 0.6004

[0135] As can be seen from Table 1, in the bicycle and pedestrian test data sets, the present invention is superior to the prior art in all evaluation indicators, proving that the present invention has achieved higher accuracy and stronger robustness in the infrared and visible light image registration task with large target size differences and complex backgrounds, and is a very practical registration method and system.

[0136] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0137] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A method for registering infrared and visible light images. It is characterized in that The infrared and visible light image registration method uses the YOLOv5 network to detect the approximate target area in the visible light image, uses the conditional generative adversarial network to convert the visible light image into a pseudo infrared image, constructs the feature description of the infrared image pair based on the SURF descriptor and uses brute force matching to build a matching model, then uses distance constraints and slope consistency to filter the matching points, and finally uses the random sampling consistency RANSAC algorithm to estimate the parameters of the homography transformation and complete the registration of the real infrared image; The infrared and visible light image registration method comprises the following steps: Step 1: Collect infrared images and visible light images; Step 2: Build a training data set and conditional generative adversarial network and train them; Step 3: Use the YOLOv5 network to detect targets in visible light images: Use the YOLOv5 network to detect targets in visible light images, and select pedestrians or bicycles based on the label. For pedestrians, the target frame is increased by 1 / 12 of the original height on the top and bottom, and the width is adjusted to 0.8 times the height based on the height; for bicycles, the target frame is increased by 1 / 30 of the original width on the left and right, and the height is adjusted to 1.25 times the width based on the width; for other categories, the original image is directly half the height and the width is 0.8 times the height. The original image is cropped according to the detection frame to obtain the approximate area of ​​the target; Step 4: Generate a pseudo infrared image using a conditional generative adversarial network. Scale the cropped visible light image to a size of 512 pixels wide and 640 pixels high and input it into the conditional generative adversarial network to obtain the corresponding pseudo infrared image. Remove the watermark from the original real infrared image and scale it to a size of 512 pixels wide and 640 pixels high. Step 5: SURF algorithm constructs descriptors and brute force matching constructs matching models: The accelerated robust feature SURF algorithm is used to extract the positions of key points and construct descriptors for real infrared images and pseudo infrared images respectively. The brute force matching method is used to construct matching models between feature points based on Euclidean distance, and two best matches are selected for each key point. Step 6: Filter mismatched points based on distance and slope consistency; Step 7: Random sampling consistency RANSAC algorithm is used to estimate the transformation parameters to complete the registration: the matching point pairs finally retained by the real infrared image and the pseudo infrared image are used to estimate the parameters of the homography transformation matrix H according to the random sampling consistency algorithm, and the real infrared image is transformed to complete the registration; The step 6 of filtering mismatched points by distance and slope consistency includes: (1) For each key point, if the Euclidean distance of the first best match is less than 0.75 times the Euclidean distance of the second best match, the first best match is retained; otherwise, the two best matches are discarded. (2) When the number of correctly matched feature point pairs is large, the slope of the straight line formed by the feature points in the real infrared image and the feature points in the pseudo infrared image is set to be less than 0.1 to filter out mismatched points.

2. The infrared and visible light image registration method according to claim 1, It is characterized in that The step 1 of collecting infrared images and visible light images specifically includes: building a ResNet50 network as a feature extraction network of a twin convolutional neural network, wherein the structure is: a first convolutional layer, a first BN layer, an activation function layer, a maximum pooling layer, a second convolutional layer, a second BN layer, a third convolutional layer, a third BN layer, a fourth convolutional layer, a fourth BN layer, a fifth convolutional layer, and a fifth BN layer; the number of convolution kernels of the first to fifth convolutional layers is set to 64, 64, 128, 256, and 512, respectively, and the convolution kernel sizes are set to 7, 3, 3, 3, and 3, respectively; the step sizes of the first, second, and third convolutional layers are set to 2, the step sizes of the fourth and fifth convolutional layers are set to 1, and the convolution kernel void rates in the fourth and fifth convolutional layers are set to 2 and 4; the size of the pooling area kernel of the maximum pooling layer is set to 3×3, and the step sizes are set to 2; the first to fifth BN layers all use a batch normalization function, the activation function layer uses a linear rectification function, and the maximum pooling layer uses a regional maximum pooling function.

3. The infrared and visible light image registration method according to claim 1, It is characterized in that The step 2 of constructing a training data set and a conditional generative adversarial network and training the network includes: (1) Construct a training data set. The size difference between visible light images and infrared images is large. The visible light images are reduced by 3 times to 1280 pixels in height and 720 pixels in width. Four pairs of matching pixels are manually selected from the infrared and visible light image pairs. The homography transformation matrix H is determined by the matching pixels. The infrared image is transformed to be consistent with the size of the visible light image. The area containing the target is manually selected and cropped into infrared and visible light image pairs whose size is an integer multiple of 4. (2) Construct a generator Gnet based on the Resnet network as the feature extraction network of the conditional generative adversarial network. Its structure includes a downsampling convolution module, 9 ResnetBlock modules, an upsampling deconvolution module and a Tanh component. The structure of the downsampling convolution module is as follows: the first padded convolution layer, the first convolution layer, the BN layer, the activation function layer, the second convolution layer, the BN layer, the activation function layer, the third convolution layer, the BN layer, the activation function layer; the parameter of the first padded convolution layer is 3, the number of convolution kernels of the first to third convolution layers is set to 64, 128, and 256 respectively, the convolution kernel size is set to 7, 3, and 3 respectively, the step size is set to 1, 2, and 2 respectively, and the padding is set to 0, 1, and 1 respectively; the structure of the RestnetBlock module is as follows: padded convolution layer, convolution layer, BN layer, activation function layer Layer, DropOut layer, padded convolution layer, convolution layer, BN layer; the padded convolution layer parameter is 1, the number of convolution kernels of the convolution layer is set to 256, the size is 3, the step size is 1, and the padding is 1; the upsampling deconvolution module structure is: the first deconvolution layer, BN layer, activation function layer, the second deconvolution layer, BN layer, activation function layer, the third padded convolution layer, the third convolution layer, activation function layer; the number of convolution kernels of the first and second deconvolution layers are set to 128 and 64 respectively, the convolution kernel size is 3, the step size is 2, and the padding is 1, the third padded convolution layer parameter is 3, the number of convolution kernels of the third convolution layer is 3, the convolution kernel size is 7, the step size is 1, and the padding is 0; all BN layers use batch normalization function, and the activation function layer uses linear rectification function except the last one which uses Tanh function; (3) Construct a PatchGan-style Dnet as the discriminator network of the conditional generation network, whose structure includes the first convolution layer, activation function layer, second convolution layer, BN layer, activation function layer, third convolution layer, BN layer, activation function layer, fourth convolution layer, BN layer, activation function layer, and fifth convolution layer in sequence; the number of convolution kernels of the first to fifth convolution layers is set to 64, 128, 256, 512, 1 in sequence, the convolution kernel size is set to 4, the step size is set to 2, 2, 2, 1, 1 in sequence, the padding is set to 1, the BN layer uses the batch normalization function, and the activation function layer is set to the LeakyReLU function; (4) Training the conditional generative adversarial network: input the training data set into the conditional generative adversarial network, and use the Adam algorithm to update the weights of the conditional generative adversarial network until the adversarial loss function Loss converges.

4. The infrared and visible light image registration method according to claim 3, It is characterized in that The homography transformation matrix H in (1) is defined as follows: Where s represents the scale factor, M is the camera intrinsic parameter matrix, and r1, r2, t represent the camera extrinsic parameters.

5. The infrared and visible light image registration method according to claim 3, It is characterized in that The linear rectification function in (2) is defined as follows: The Tanh function is defined as follows: The LeakyReLU function is defined as follows: Where a is a fixed parameter of (1, +∞), and x is the input; The loss function Loss in (4) is defined as follows: The generation loss of Gnet is defined as follows: Lg=-log D(G(I vis ))-μ||I inf -G(I vis )||; The adversarial loss of Dnet is defined as follows: L d =-log(1-D(I vis ,G(I vis ))-log(D(I vis ,I inf )); Where LcGAN represents the total loss of the conditional generative adversarial network, G represents the generator Gnet, D represents the discriminator Dnet, x represents the input image, z represents the noise vector, and z is replaced by the DropOut layer; μ represents the weight of the second item in Lg, which is set to 100, Iinf represents the real infrared image, and Ivis represents the visible light image.

6. A computer device, It is characterized in that The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the infrared and visible light image registration method according to any one of claims 1 to 5.

7. An intelligent image processing terminal, It is characterized in that The intelligent image processing terminal is used to implement the infrared and visible light image registration method described in any one of claims 1 to 5.

8. An infrared and visible light image registration system for implementing the infrared and visible light image registration method according to any one of claims 1 to 5, It is characterized in that The infrared and visible light image registration system comprises: An image acquisition module, used for acquiring infrared images and visible light images; A training data set construction module is used to construct a training data set, build and train a conditional generative adversarial neural network; Image object detection module, used to detect visible light image objects using the YOLOv5 network; A pseudo infrared image generation module, used to generate a pseudo infrared image from a visible light image using a generative adversarial neural network; The matching model building module is used to accelerate the robust feature SURF algorithm to build feature descriptors and use the brute force matching method to build a matching model; Mismatch point filtering module, used to filter mismatch points by distance and slope consistency; The matching completion module is used to estimate the transformation parameters using the random sampling consistency RANSAC algorithm to complete the alignment.