Visible light-infrared image matching method based on image generation model

Abstract

The invention provides a visible light-infrared image matching method based on an image generation model, and the method is characterized in that the method comprises the following steps: S1, collecting paired visible light image data and far-wave infrared image data through a sensor device; s2, training a visible light-infrared image generation model according to the paired image data; s3, training an infrared image encoder model according to the generation model; s4, training a visible light-infrared image feature point extraction model according to the image generation model and the image encoder model; and S5, according to the model, carrying out image matching on an input visible light-infrared image pair. Compared with the prior art, the method can reduce the significant modal difference between the visible light image and the infrared image, effectively carries out cross-spectrum image matching, and improves the performance of an environment perception system of an automatic driving automobile in a low-illumination (night) environment.

Description

Technical Field

[0001] This invention relates to the fields of intelligent connected vehicles and machine learning, and in particular to a visible light-infrared image matching method based on an image generation model. Background Technology

[0002] Environmental perception systems are crucial for autonomous vehicles, providing rich information about the vehicle's surroundings to guide decision-making and control algorithms in completing autonomous driving tasks. Low-light (nighttime) environmental perception is a very important part of autonomous driving, and most current methods rely on multi-sensor fusion, using sensors such as visible light cameras, LiDAR, and long-wave infrared cameras.

[0003] In practical applications, since the vehicle body structure is not a rigid body, unavoidable vehicle vibrations during driving cause relative pose changes in sensors installed around the vehicle body. In this situation, if the computer still calculates based on the extrinsic parameters of each sensor calibrated at the time of manufacture, the accuracy of various perception tasks will decrease, thereby reducing the performance of the vehicle's environmental perception system. Due to its unique imaging principle, far-wave infrared cameras can image targets such as pedestrians in low-light environments such as at night. Visible-infrared image fusion technology significantly improves the environmental perception performance of autonomous vehicles in low-light environments. However, since pose changes caused by vehicle vibration degrade the performance of this fusion technology, real-time visible-infrared image matching technology is needed to determine the relative pose changes between visible and infrared images for correction. Currently, there are some cross-spectral image matching works, but they are all limited by the huge modal differences between visible and infrared images (especially at night), making it difficult to extract common features for matching. How to fully exploit the common features in visible and infrared images, improve cross-spectral image matching performance, and thus enhance the environmental perception capabilities of autonomous vehicles in low-light environments is an urgent problem to be solved. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a visible light-infrared image matching method based on an image generation model. This method can help reduce the huge modal differences between visible light images and infrared images, extract common features to achieve visible light-infrared image matching, and thus improve the environmental perception capability of autonomous vehicles in low-light environments.

[0005] The objective of this invention can be achieved through the following technical solution: a visible light-infrared image matching method based on an image generation model, comprising the following steps:

[0006] S1. Acquire paired visible light image data and far-wave infrared image data through sensor devices;

[0007] S2. Based on the paired image data, train the visible light-infrared image generation model;

[0008] S3. Train the infrared image encoder based on the image generation model;

[0009] S4. Based on the image generation model and the image encoder, train the visible light-infrared image feature point extraction model;

[0010] S5. Based on the feature point extraction model, perform image matching on the input visible light-infrared image pairs.

[0011] Furthermore, the sensor device in step S1 includes a visible light camera for acquiring visible light image data and a far-wave infrared camera for acquiring far-wave infrared image data.

[0012] Furthermore, the data in step S1 includes three-channel RGB image data output by a visible light camera and single-channel grayscale image data output by a far-wave infrared camera.

[0013] Furthermore, in step S1, the paired data refers to visible light and infrared image pairs aligned in pixel space.

[0014] Further, step S2 includes the following steps:

[0015] S2.1 Construct an image generation model based on generative adversarial networks, consisting of a visible light encoder and decoder connected in series;

[0016] S2.2 Input the visible light image into the image generation model to obtain its output pseudo-infrared image, and use the infrared image for supervision to train the generation model.

[0017] Further, step S3 includes the following steps:

[0018] S3.1 Construct an infrared image encoder;

[0019] S3.2 Input the visible light image into the visible light encoder of the trained image generation model to obtain visible light features;

[0020] S3.3 Input the infrared image into the infrared encoder to obtain infrared features, and use visible light features for supervision to train the infrared encoder.

[0021] Further, step S4 includes the following steps:

[0022] S4.1 Construct a visible light-infrared image feature point extraction model;

[0023] S4.2 Input the visible light image into the visible light encoder of the trained image generation model to obtain visible light features;

[0024] S4.3 Input the infrared image into the trained infrared encoder to obtain infrared features;

[0025] S4.4 Input visible light features and infrared features into the visible light-infrared image feature point extraction model to train the model.

[0026] Further, step S5 includes the following steps:

[0027] S5.1 Input the visible light image into the visible light encoder of the trained image generation model to obtain visible light features;

[0028] S5.2 Input the infrared image into the trained infrared encoder to obtain infrared features;

[0029] S5.3 Input visible light features and infrared features into the visible light-infrared image feature point extraction model to extract feature points of visible light images and infrared images respectively;

[0030] S5.4 Match feature points of visible light and infrared images to obtain visible light-infrared image matching results.

[0031] Compared with the prior art, the present invention has the following advantages:

[0032] 1. This invention trains an image generation model to enable the model to convert visible light to infrared images, and uses its visible light encoder to obtain the visible light features of visible light images.

[0033] 2. This invention trains an infrared encoder to receive infrared images and output infrared features similar to those of visible light.

[0034] 3. This invention reduces the modal differences between visible light and infrared images by processing visible light images and infrared images into visible light features and infrared features respectively, and extracts the common features between the two types of images, laying the foundation for subsequent feature point extraction. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating a visible light-infrared image matching method based on an image generation model in this embodiment.

[0036] Figure 2 for Figure 1 The specific network architecture of the image generation model in the illustrated embodiment;

[0037] Figure 3 for Figure 1The specific network architecture of the feature point extraction model in the illustrated embodiment;

[0038] Figure 4 This is a visual flowchart illustrating a visible light-infrared image matching method based on an image generation model, as described in this embodiment. Detailed Implementation

[0039] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0040] This invention provides a visible light-infrared image matching method based on an image generation model, comprising the following steps:

[0041] S1. Acquire paired visible light image data and far-wave infrared image data through sensor devices;

[0042] The sensor equipment includes a visible light camera and a far-infrared camera. The optical centers of the visible light and far-infrared cameras are aligned, and intrinsic and extrinsic parameters are calibrated using the Zhang Zhengyou method to ensure the accuracy of the three-channel visible light images acquired. v (size 3×H×W) and grayscale single-channel far-wave infrared image I ir (Dimensions are 1×H×W) Aligned in pixel space, where 3 and 1 are the number of image channels, and H and W are the height and width of the image, respectively.

[0043] S2. Based on the paired image data, train the visible light-infrared image generation model; specifically including the following steps:

[0044] S2.1 Construct an image generation model based on generative adversarial networks, consisting of a visible light encoder and decoder connected in series;

[0045] like Figure 2 As shown, the image generation model G consists of a visible light encoder E. vThe encoder consists of two modules: a encoder and a decoder. The encoder comprises four downsampling convolutional modules and three residual convolutional modules. Each downsampling convolutional module downsamples the image features by a factor of 2 and consists of a 3x3 convolution with a stride of 2, a ReLU activation function, and batch normalization. The number of input and output channels for each downsampling convolutional module are (3, 128), (128, 256), (256, 512), and (512, 1024), respectively. Each residual convolutional module consists of a 3x3 convolution with a stride of 1, a ReLU activation function, and batch normalization, and includes residual connections. The number of input and output channels for each residual convolutional module is 1024. The decoder consists of four upsampling convolutional modules. Each upsampling convolutional module upsamples the image features by a factor of 2 and consists of a 3x3 transposed convolution with a stride of 2, a ReLU activation function, and batch normalization. The number of input and output channels of each upsampling convolutional module are (1024, 512), (512, 256), (256, 128), and (128, 1), respectively.

[0046] S2.2, Transform the visible light image I v Input an image generation model G, and obtain its output pseudo-infrared image I. pir (Dimensions are 1×H×W), and infrared image I is used. ir Supervised training is used to generate the model;

[0047] Using the image generation model G, obtain its output pseudo-infrared image I. pir The calculation formula is:

[0048] I pir =G(I v )

[0049] It can also be expressed as:

[0050] I pir =D(E) v (I v ))

[0051] To improve pseudo-infrared image I pir and infrared image I ir In addition to general generative adversarial network (GAN) training methods, the training supervision methods for generative models also include infrared reconstruction loss, which considers the feature similarity between the models. rec and infrared edge reconstruction loss edge The calculation formula is:

[0052] Loss rec =SmoothL1Loss(I ir ,I pir )

[0053] Lossedge =SmoothL1Loss(Sobel(I ir ),Sobel(I pir ))

[0054] The loss calculation method used is SmoothL1 loss, and Sobel() refers to using the Sobel operator to convolve the image to obtain the image edge map.

[0055] S3. Train the infrared image encoder based on the image generation model, specifically including the following steps:

[0056] S3.1 Constructing an infrared image encoder E ir ;

[0057] Infrared image encoder E ir It consists of four downsampling convolutional modules and three residual convolutional modules, and is integrated with the visible light encoder E. v The structures are basically the same, but since infrared images are single-channel grayscale images, the number of channels for the input and output data of each downsampling convolution module of the infrared image encoder are (1, 128), (128, 256), (256, 512), and (512, 1024), respectively.

[0058] S3.2 Input the visible light image into the visible light encoder of the trained image generation model to obtain visible light features;

[0059] After the image generation model is trained, the visible light image I... v Input visible light encoder E v Obtain visible light feature F v (Dimensions are 1024 × (H / 16) × (W / 16)), the calculation formula is:

[0060] F v =E v (I v )

[0061] S3.3 Input the infrared image into the infrared encoder to obtain infrared features, and use visible light features for supervision to train the infrared encoder;

[0062] Infrared image I ir Input infrared encoder E ir Obtain infrared feature F ir (Dimensions are 1024 × (H / 16) × (W / 16)), the calculation formula is:

[0063] F ir =E ir (I ir )

[0064] Using visible light feature F v For infrared feature F ir Supervised training of the infrared encoder is performed, and the loss function is calculated using the following formula:

[0065] Loss feat =SmoothL1Loss(F v ,F ir )

[0066] S4. Based on the image generation model and image encoder, train the visible light-infrared image feature point extraction model; specifically including the following steps:

[0067] S4.1 Construct a visible light-infrared image feature point extraction model;

[0068] like Figure 3 As shown, the visible light-infrared image feature point extraction model M consists of a confidence output head S, a position offset output head P, and a descriptor output head Des.

[0069] S4.2, Transform the visible light image I v The visible light encoder E is input to the trained image generation model G. v Obtain visible light feature E v ;

[0070] S4.3, transfer infrared image I ir Input the trained infrared encoder E ir Obtain infrared feature F ir ;

[0071] S4.4, Visible light characteristics F v and infrared signature F ir Input the visible light-infrared image feature point extraction model M to train the model;

[0072] Visible light feature F v and infrared signature F ir Inputting model M yields the confidence map (Score), position offset map (Pos), and descriptor map (Desc) for both visible and infrared images. The calculation formulas are as follows:

[0073] (Score v Pos v ,Desc v )=(S(I v ),P(I v ),Des(I v ))

[0074] (Score ir Pos ir,Desc ir )=(S(I ir ),P(I ir ),Des(I ir ))

[0075] After obtaining the above results, based on the threshold λ s =0.5 Points with a confidence score greater than the threshold are selected as candidate feature points. The position of the feature points in the pixel coordinate system is corrected using a position offset Pos, resulting in the feature points and their descriptors (Kpt, Desc). The calculation formula is as follows:

[0076] (Kpt v ,Desc v )=M(F v )

[0077] (Kpt ir ,Desc ir )=M(F ir )

[0078] After obtaining the above results, the Homographic Adaption method of Unsuperpoint was used to perform self-supervised training of feature points.

[0079] S5. Based on the feature point extraction model, perform image matching on the input visible light-infrared image pair; specifically including the following steps:

[0080] S5.1, Transform the visible light image I v The visible light encoder E is input to the trained image generation model G. v Obtain visible light feature E v ;

[0081] S5.2, transfer infrared image I ir Input the trained infrared encoder E ir Obtain infrared feature F ir ;

[0082] S5.3, Visible light characteristics F v and infrared signature F ir Input a visible light-infrared image feature point extraction model M to extract feature points (Kpt) from visible light and infrared images respectively. v ,Desc v ), (Kpt ir ,Desc ir );

[0083] S5.4 Match feature points of visible light and infrared images to obtain visible light-infrared image matching results.

[0084] Based on the descriptor matching threshold λ d =0.5, using the bidirectional nearest neighbor method to descriptor (Desc) feature points of visible light and infrared images. v ,Desc ir Matching is performed, and descriptors with a Euclidean distance greater than a threshold λ are filtered out. d Pairs of points that are not nearest neighbors to each other are considered as initial matching pairs (Kpt). v1 ,Kpt ir1 Then, the RANSAC method is used to filter out outlier matching pairs, obtaining the final matching pairs (Kpt). v2 ,Kpt ir2 ).

[0085] The above description of the embodiments is provided to enable those skilled in the art to understand and use the invention. It will be apparent to those skilled in the art that various modifications can be made to these embodiments, and the general principles described herein can be applied to other embodiments without inventive effort. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made by those skilled in the art based on the disclosure of the present invention without departing from the scope of the invention should be within the protection scope of the present invention.

Claims

1. A visible light-infrared image matching method based on an image generation model, characterized in that, Includes the following steps: S1. Acquire paired visible light image data and far-wave infrared image data through sensor devices; S2. Based on the paired image data, train the visible light-infrared image generation model; S3. Train the infrared image encoder based on the image generation model; S4. Based on the image generation model and the image encoder, train the visible light-infrared image feature point extraction model; S5. Based on the feature point extraction model, perform image matching on the input visible light-infrared image pairs.

2. The visible light-infrared image matching method based on an image generation model according to claim 1, characterized in that, The sensor device includes a visible light camera for acquiring visible light image data and a far-wave infrared camera for acquiring far-wave infrared image data.

3. The visible light-infrared image matching method based on an image generation model according to claim 1, characterized in that, The data includes three-channel RGB image data output by a visible light camera and single-channel grayscale image data output by a far-wave infrared camera.

4. The visible light-infrared image matching method based on an image generation model according to claim 1, characterized in that, The paired data refers to visible light and infrared image pairs aligned in pixel space.

5. The visible light-infrared image matching method based on an image generation model according to claim 1, characterized in that, S2 includes the following steps: S2.1 Construct an image generation model based on generative adversarial networks, consisting of a visible light encoder and decoder connected in series; S2.2 Input the visible light image into the image generation model to obtain its output pseudo-infrared image, and use the infrared image for supervision to train the generation model.

6. The visible light-infrared image matching method based on an image generation model according to claim 1, characterized in that, S3 includes the following steps: S3.1 Construct an infrared image encoder; S3.2 Input the visible light image into the visible light encoder of the trained image generation model to obtain visible light features; S3.3 Input the infrared image into the infrared encoder to obtain infrared features, and use visible light features for supervision to train the infrared encoder.

7. The visible light-infrared image matching method based on an image generation model according to claim 1, characterized in that, S4 includes the following steps: S4.1 Construct a visible light-infrared image feature point extraction model; S4.2 Input the visible light image into the visible light encoder of the trained image generation model to obtain visible light features; S4.3 Input the infrared image into the trained infrared encoder to obtain infrared features; S4.4 Input visible light features and infrared features into the visible light-infrared image feature point extraction model to train the model.

8. The visible light-infrared image matching method based on an image generation model according to claim 1, characterized in that, S5 includes the following steps: S5.1 Input the visible light image into the visible light encoder of the trained image generation model to obtain visible light features; S5.2 Input the infrared image into the trained infrared encoder to obtain infrared features; S5.3 Input visible light features and infrared features into the visible light-infrared image feature point extraction model to extract feature points of visible light images and infrared images respectively; S5.4 Match feature points of visible light and infrared images to obtain visible light-infrared image matching results.

Images