Infrared and visible light image registration method and system based on modal conversion

Abstract

The present invention relates to the field of image processing technology, and in particular to a method and system for infrared and visible light image registration based on modal conversion, the method comprising: constructing an image registration model, the model comprising a modal conversion network, a feature extraction network, and a matching and regression network; in a training stage, acquiring an infrared image and a visible light image, and before extracting features from the visible light image, performing a modal conversion operation on the visible light image through a modal conversion network to generate a corresponding pseudo infrared image; inputting the infrared image and the pseudo infrared image into a feature extraction network respectively, performing shallow feature extraction and deep feature extraction in turn, obtaining respective corresponding feature maps, and then inputting the matching and regression network to perform matching and geometric transformation parameter prediction; after meeting a preset training target, obtaining a final image registration model. The network structure of the present invention is simple, and can enhance the feature extraction capability, thereby improving the registration accuracy of infrared and visible light images.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more particularly to a method and system for infrared and visible light image registration based on modal conversion. Background Art

[0002] Visible light images and infrared images each have their own unique advantages and characteristics in the field of imaging. The fusion of infrared and visible light images helps to obtain more substantial information, enhance image recognition, and improve the accuracy and efficiency of target detection and recognition.

[0003] The differences in the characteristics of different sensors and the use environment lead to obvious differences in infrared images and visible light images in terms of bands, imaging conditions, etc., making them inconsistent in geometry. This inconsistency involves multiple aspects such as position, scale, rotation, and deformation, making it difficult to accurately align them on the same epipolar line. This situation will cause parallax and residual distortion, which will in turn affect the image quality and accuracy. It becomes challenging to directly fuse or compare the two. Spectral differences make it difficult to accurately align infrared and visible light images. However, the current neural network has insufficient registration accuracy for images of different modalities. How to improve the registration accuracy of infrared and visible light images has become a technical problem that technicians in this field need to solve urgently. Summary of the invention

[0004] In view of this, the present invention provides an infrared and visible light image registration method and system based on modal conversion, which has a simple network structure and can enhance the feature extraction capability and improve the registration accuracy of infrared and visible light images.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides an infrared and visible light image registration method based on modality conversion, comprising the following steps:

[0007] Constructing an image registration model, wherein the image registration model includes a modality conversion network, a feature extraction network, and a matching and regression network;

[0008] In the training stage, an infrared image I and a visible light image V are obtained, and before feature extraction is performed on the visible light image V, a modal conversion operation is performed on it through the modal conversion network to generate a corresponding pseudo infrared image P;

[0009] The infrared image I and the pseudo infrared image P are respectively input into the feature extraction network, and shallow feature extraction and deep feature extraction are performed in sequence to obtain their respective corresponding feature maps;

[0010] Inputting the feature map of the infrared image I and the feature map of the pseudo infrared image P into the matching and regression network to perform matching and geometric transformation parameter prediction;

[0011] After meeting the preset training objectives, the final image registration model is obtained.

[0012] Furthermore, the modality conversion network uses a Pix2Pix model to convert the visible light image into a pseudo infrared image.

[0013] Furthermore, the feature extraction network includes two branches, which respectively extract features from the infrared image I and the pseudo infrared image P, and each branch includes a shallow feature extraction module and a deep feature extraction module;

[0014] The shallow feature extraction module includes a convolution layer and a pooling layer;

[0015] The deep feature extraction module uses ResNeXt as the basic network and adds an improved BAM attention unit after each residual block;

[0016] The BAM attention unit includes two branches: channel attention and spatial attention. The process of improving it includes:

[0017] Add a max pooling layer on top of the original average pooling layer in the channel attention branch of the BAM attention unit, and add a normal 3×3 convolution and ReLU activation function between the two fully connected layers;

[0018] After separating the channel attention direction from the spatial attention direction, the attention maps of the two branches are merged by element-wise addition, and then a sigmoid function is used to obtain the total attention map.

[0019] A residual connection is added to add the total attention map to the input shallow feature map element-wise to obtain the final feature map.

[0020] Furthermore, the calculation formula of the final feature map is:

[0021]

[0022] Among them, F' represents the final feature map in the input matching and regression network, F represents the shallow feature map of the input, and M(F) represents the total attention map;

[0023] The calculation formula of the total attention map M(F) is:

[0024] M(F)=σ(M C (F)+M s (F)

[0025] Among them, M C (F) represents the output result of the channel attention direction branch, M s(F) represents the output result of the spatial attention direction branch, and σ represents the sigmoid function.

[0026] Furthermore, the matching and regression network first calculates a dense corresponding mapping between the infrared image and the pseudo infrared image through a matching function, and then estimates geometric transformation parameters of the dense corresponding mapping through a regression network.

[0027] Furthermore, the infrared image feature map f is matched by the matching function I To the pseudo infrared image feature map f P The calculation formula for the dense corresponding mapping is:

[0028] C I→P (i,j,k)=C(f I (i k ,j k ),f P (i,j))=f P (i,j) T f I (i k ,j k )

[0029] Among them, (i,j) and (i k ,j k ) represent the registration point pairs in the infrared image feature map f I and pseudo infrared image feature map f P The coordinates in; k represents the feature point (i k ,j k )'s auxiliary index; C I→P represents the infrared image feature map f I and pseudo infrared image feature map f P The dense correspondence mapping between I and f P Multiply them together to get .

[0030] Furthermore, the calculation formula of the geometric transformation parameters of the dense correspondence mapping through the regression network is:

[0031] R:R h′×w′×(h′×w′) →R DoF

[0032] Among them, (h', w') represents the height and width of the dense correspondence map, DoF represents the degree of freedom of the radial transformation model, and R represents the regression network.

[0033] In a second aspect, the present invention provides an infrared and visible light image registration system based on modality conversion, comprising:

[0034] An acquisition module, used for acquiring infrared images and visible light images;

[0035] The registration module is used to register the infrared image and the visible light image using the image registration model trained by the infrared and visible light image registration method based on modal conversion as described above.

[0036] In a third aspect, the present invention provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor; when the processor executes the computer program, the steps of the infrared and visible light image registration method based on modal conversion as described above are implemented.

[0037] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the infrared and visible light image registration method based on modality conversion as described above are implemented.

[0038] It can be seen from the above technical solutions that, compared with the prior art, the present invention has the following beneficial effects:

[0039] Before extracting features from the visible light image and the infrared red line, the present invention first modally converts the visible light image to generate a pseudo infrared image to narrow the modal difference between the infrared and visible light images. In the feature extraction stage, shallow features and deep features are extracted from the infrared image I and the pseudo infrared image in turn to retain more features in the image and improve the registration accuracy of the infrared image and the visible light image.

[0040] In the feature extraction stage, the present invention improves the channel branch based on the original BAM attention, optimizes the feature extraction performance, and effectively enhances the feature extraction capability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0042] Figure 1 A schematic diagram of the structure of the image registration model provided by the present invention;

[0043] Figure 2 A schematic diagram of the structure of the BAMU attention unit provided by the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] like Figure 1 As shown, an embodiment of the present invention discloses a method for infrared and visible light image registration based on modality conversion, comprising the following steps:

[0046] Construct an image registration model, which includes a modality conversion network, a feature extraction network, and a matching and regression network;

[0047] In the training phase, an infrared image I and a visible light image V are obtained. Before feature extraction is performed on the visible light image V, a modal conversion operation is performed on it through a modal conversion network to generate a corresponding pseudo infrared image P.

[0048] The infrared image I and the pseudo infrared image P are respectively input into the feature extraction network, and shallow feature extraction and deep feature extraction are performed in turn to obtain their corresponding feature maps;

[0049] The feature map of the infrared image I and the feature map of the pseudo infrared image P are input into the matching and regression network to perform matching and geometric transformation parameter prediction to obtain the geometric transformation parameters;

[0050] After meeting the preset training objectives, the final image registration model is obtained.

[0051] The core network module in the image registration model is further explained below.

[0052] (1) Modal conversion network.

[0053] The modality conversion network of the present invention adopts the Pix2Pix model to convert the visible light image into a pseudo infrared image.

[0054] The Pix2Pix model consists of two parts: the generator and the discriminator. The generator uses a U-Net architecture, adding jump connections between the image layers in the encoder and decoder stacks. The discriminator uses a convolutional classifier, which tries to identify the authenticity of each image block and classify it as true or false. The discriminator will run over the entire image and average all the results as the final output. If the fake image output by the generator cannot fool the discriminator, then a low similarity score will be obtained; if it fools the discriminator and makes the discriminator make a wrong judgment, then a high similarity score will be obtained. The judgment process of the discriminator is the same.

[0055] Pix2Pix borrows the loss function of the conditional generative adversarial network (cGAN), as shown in formula (1):

[0056]

[0057] Among them, D is the discriminator, G is the generator, x is the input image, y is the corresponding real image, and z is a random variable. For the discriminator, it expects D(x,y) to be as large as possible, that is, the probability that the discriminator can recognize the real image is as large as possible, and D(x,G(x,z)) to be as small as possible, that is, the probability that the generated image is the real image is as small as possible, which is equivalent to 1-D(x,G(x,z)) as large as possible; for the generator, the first term is a constant, and only the second term is expected to be as small as possible.

[0058] In addition, the Pix2Pix model also adds an L1 loss to the generator, which allows the generator to not only deceive the discriminator, that is, to be as realistic as possible, but also to be consistent with the content of the input image. The L1 loss function is added because it produces less blur than the L2 loss function, as shown in formula (2):

[0059]

[0060] The final loss function is shown in formula (3):

[0061]

[0062] (2) Feature extraction network.

[0063] The spectral difference between infrared images and visible light images poses a huge challenge to the registration task, and it is necessary to solve the problem of difficult feature point extraction during the registration process.

[0064] Specifically, the feature extraction network of the present invention includes two branches, which respectively extract features from the infrared image I and the pseudo infrared image P, and each branch includes a shallow feature extraction module and a deep feature extraction module;

[0065] The shallow feature extraction module includes a convolution layer and a pooling layer;

[0066] The deep feature extraction module (BU-Res) uses ResNeXt as the basic network and adds an improved bottleneck attention module BAM attention unit after each residual block. The improved BAM attention unit is called BAMU attention unit. The specific structure is as follows: Figure 2 shown.

[0067] Specifically, the BAM attention unit includes two branches: channel attention and spatial attention. The difference between the BAMU attention unit and the BAM attention unit is that the channel attention branch has been improved. The specific improvement process includes:

[0068] Add a max pooling layer on top of the original average pooling layer in the channel attention branch of the BAM attention unit, and add a normal 3×3 convolution and ReLU activation function between the two fully connected layers;

[0069] After separating the attention of the channel attention direction from the spatial attention direction, the attention maps of the two branches are merged by element addition, and then the total attention map is obtained through a sigmoid function. The calculation formula of the total attention map M(F) is:

[0070] M(F)=σ(M C (F)+M s (F)) (4)

[0071] Among them, M C (F) represents the output result of the channel attention direction branch, M s (F) represents the output result of the spatial attention direction branch, and σ represents the sigmoid function.

[0072] After that, a residual connection is added to add the total attention map to the input shallow feature map element-wise to obtain the final feature map. The calculation formula of the final feature map is:

[0073]

[0074] Among them, F' represents the final feature map in the input matching and regression network, F represents the shallow feature map of the input, and M(F) represents the total attention map;

[0075] The deep feature extraction module (BU-Res) in the feature extraction network proposed in the present invention selects BAMU to be added to the ResNeXt residual structure, which effectively improves the feature extraction capability of the entire image registration model. At the same time, since the process of calculating the attention map is decomposed into two branches, the computational overhead is greatly reduced.

[0076] (3) Matching and regression network.

[0077] The matching and regression network first calculates the dense correspondence mapping between the infrared image and the pseudo infrared image through the matching function, and then estimates the geometric transformation parameters of the dense correspondence mapping through the regression network.

[0078] Matching: The infrared image feature map f is matched by matching function I To the pseudo infrared image feature map f PThe calculation formula for the dense corresponding mapping is:

[0079] C I→P (i,j,k)=C(f I (i k ,j k ),f P (i,j))=f P (i,j) T f I (i k ,j k ) (6)

[0080] Among them, (i,j) and (i k ,j k ) represent the registration point pairs in the infrared image feature map f I and pseudo infrared image feature map f P The coordinates of the feature point (i k ,j k )'s auxiliary index; C I→P Indicates the infrared image feature f I and pseudo infrared image feature f P The dense correspondence mapping between I and f P Multiply them together to get .

[0081] Regression: The calculation formula of the geometric transformation parameters of dense correspondence mapping through the regression network is:

[0082] R:R h′×w′×(h′×w′) →R DoF (7)

[0083] Among them, (h', w') represents the height and width of the dense correspondence map, DoF represents the degree of freedom of the radial transformation model, and R represents the regression network.

[0084] (4) Loss Function

[0085] During the training process, the present invention uses the transformed grid loss as the basic loss function. The parameter value θ of the transformation model between the visible light image and the infrared image predicted by the model is obtained through the regression network R, and the parameter value of the transformation model between the actual visible light image and the infrared image is θ gt , the difference between the two transformed grids is measured by summing the squared distances between the corresponding grid points, and the basic loss function is constructed based on the difference. The calculation formula of the basic loss function is:

[0086]

[0087] Among them, (x i ,y j) represents the pixel at position (i, j); θ represents the predicted affine transformation, θ gt represents the actual affine transformation, τ θ and Respectively represent the parameters θ and θ gt The transformation operation under gt ) represents θ and θ gt The basis loss between ; N represents the number of grid points.

[0088] In general, the image registration model proposed in the present invention uses both infrared image I and visible light image V as input, wherein the visible light image V is first modally converted using the Pix2Pix model before feature extraction, and the corresponding pseudo infrared image P is generated as a substitute input for the input visible light image V, which is equivalent to using infrared image I and pseudo infrared image P of similar modality for registration. After the input image enters the feature extraction step, it is first subjected to a layer of convolution and pooling for shallow extraction, and then input into BU-Res for further feature extraction. The BAMU attention proposed in the present invention is added to the original ResNeXt residual structure, which effectively improves the model's feature extraction capability for complex images. Feature map matching is then performed to generate a corresponding mapping map, and then a regression operation is performed to predict the transformation parameters. The present invention has shown through experiments on the FLIR data set that the PCK index is greatly improved compared to the current network MTIVRNet, and visually presents a superior registration effect.

[0089] In one embodiment, the present invention further provides an infrared and visible light image registration system based on modality conversion, comprising:

[0090] An acquisition module, used for acquiring infrared images and visible light images;

[0091] The registration module is used to register the infrared image and the visible light image using the image registration model trained by the above-mentioned infrared and visible light image registration method based on modal conversion.

[0092] In other embodiments, the present invention also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor; when the processor executes the computer program, the steps of the above-mentioned infrared and visible light image registration method based on modal conversion are implemented.

[0093] In addition, the present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the infrared and visible light image registration method based on modal conversion as described above are implemented.

[0094] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0095] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for infrared and visible light image registration based on modality conversion, characterized in that: The following steps are involved: Constructing an image registration model, wherein the image registration model includes a modality conversion network, a feature extraction network, and a matching and regression network; In the training stage, an infrared image I and a visible light image V are obtained, and before feature extraction is performed on the visible light image V, a modal conversion operation is performed on it through the modal conversion network to generate a corresponding pseudo infrared image P; The infrared image I and the pseudo infrared image P are respectively input into the feature extraction network, and shallow feature extraction and deep feature extraction are performed in sequence to obtain their respective corresponding feature maps; Inputting the feature map of the infrared image I and the feature map of the pseudo infrared image P into the matching and regression network to perform matching and geometric transformation parameter prediction; After meeting the preset training objectives, the final image registration model is obtained; The feature extraction network includes two branches, which respectively extract features from the infrared image I and the pseudo infrared image P, and each branch includes a shallow feature extraction module and a deep feature extraction module; The shallow feature extraction module includes a convolution layer and a pooling layer; The deep feature extraction module uses ResNeXt as the basic network and adds an improved BAM attention unit after each residual block; The BAM attention unit includes two branches: channel attention and spatial attention. The process of improving it includes: Add a max pooling layer on top of the original average pooling layer in the channel attention branch of the BAM attention unit, and add a normal 3×3 convolution and ReLU activation function between the two fully connected layers; After separating the channel attention direction from the spatial attention direction, the attention maps of the two branches are merged by element-wise addition, and then a sigmoid function is used to obtain the total attention map. A residual connection is added to add the total attention map to the input shallow feature map element-wise to obtain the final feature map.

2. The infrared and visible light image registration method based on modality conversion according to claim 1 is characterized in that: The modality conversion network uses the Pix2Pix model to convert the visible light image into a pseudo infrared image.

3. The infrared and visible light image registration method based on modality conversion according to claim 1, characterized in that: The calculation formula for the final feature map is: Among them, F' represents the final feature map in the input matching and regression network, F represents the shallow feature map of the input, and M(F) represents the total attention map; The calculation formula of the total attention map M(F) is: M(F)=σ(M C (F)+M s (F)) Among them, M C (F) represents the output result of the channel attention direction branch, M s (F) represents the output result of the spatial attention direction branch, and σ represents the sigmoid function.

4. The infrared and visible light image registration method based on modality conversion according to claim 1, characterized in that: The matching and regression network first calculates the dense corresponding mapping between the infrared image and the pseudo infrared image through a matching function, and then estimates the geometric transformation parameters of the dense corresponding mapping through a regression network.

5. The infrared and visible light image registration method based on modality conversion according to claim 4 is characterized in that: The infrared image feature map f is matched by the matching function I To the pseudo infrared image feature map f P The calculation formula for the dense corresponding mapping is: C I→P (i,j,k)=C(f I (i k ,j k ),f P (i,j))=f P (i,j) T f I (i k ,j k ) Among them, (i,j) and (i k ,j k ) represent the registration point pairs in the infrared image feature map f I And the pseudo infrared image feature map f P The coordinates in k represent the feature point (i k ,j k )'s auxiliary index; C I→P Represents the infrared image feature map f I And the pseudo infrared image feature map f P The dense correspondence mapping between I and f P Multiply them together to get .

6. The infrared and visible light image registration method based on modality conversion according to claim 4 is characterized in that: The calculation formula of the geometric transformation parameters of dense correspondence mapping through the regression network is: R:R h'×w'×(h'×w') →R DoF Among them, (h', w') represents the height and width of the dense correspondence map, DoF represents the degree of freedom of the radial transformation model, and R represents the regression network.

7. An infrared and visible light image registration system based on modal conversion, characterized in that: include: An acquisition module, used for acquiring infrared images and visible light images; A registration module is used to register the infrared image and the visible light image using the image registration model trained by the infrared and visible light image registration method based on modal conversion as described in any one of claims 1 to 6.

8. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor; characterized in that when the processor executes the computer program, the steps of the infrared and visible light image registration method based on modal conversion as described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the infrared and visible light image registration method based on modality conversion as described in any one of claims 1 to 6 are implemented.

Images