Infrared and weak visible light image registration method and system based on color edge enhancement

Abstract

The present invention relates to the field of image processing technology, and in particular to an infrared and weak visible light image registration method and system based on color edge enhancement, the method comprising: building an image registration model, the image registration model comprising two branches, the first branch performing feature extraction and registration processing on an infrared image and a weak visible light image, the second branch performing feature extraction and registration processing on an infrared image and an enhanced weak visible light image, and combining the registration results of the two branches to obtain a more accurate infrared and weak visible light image registration effect; when extracting features of the infrared image and the weak visible light image of the two branches, additionally integrating edge features extracted by a method based on an attention mechanism, so that the edge features of the infrared image are fully utilized; by performing color enhancement processing on the weak visible light image, the comprehensiveness of the feature expression of the weak visible light image is improved, and the alignment accuracy between the infrared image and the visible light image is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more particularly to an infrared and weak visible light image registration method and system based on color edge enhancement. Background Art

[0002] Visible light images and infrared images each have their own unique advantages and characteristics in the field of imaging. Visible light images cover a wavelength range from 380 nanometers to 780 nanometers, and are intuitive, clear, and colorful. They are suitable for daily photography and video shooting, but are easily affected by changes in lighting. Weak visible light images are images whose features are not significant due to the influence of lighting. In contrast, infrared images cover a wavelength range from 3 microns to 14 microns, and use the thermal radiation information of objects to generate images. They are not affected by lighting and distance, and are suitable for night monitoring and hidden object detection. Therefore, the fusion of infrared and visible light images helps to obtain more substantial information, enhance image recognition, and improve the accuracy and efficiency of target detection and recognition. By rationally fusing visible light images and infrared images, more comprehensive information acquisition can be achieved and its application areas can be expanded.

[0003] However, the characteristics of different sensors and differences in the use environment lead to obvious differences between infrared images and visible light images in terms of bands, imaging conditions, etc., making them inconsistent in geometry. This inconsistency involves multiple aspects such as position, scale, rotation, and deformation, making it difficult to accurately align them on the same epipolar line. This situation will cause parallax and residual distortion, which will in turn affect the image quality and accuracy. It is not easy to achieve multimodal image registration. Direct fusion or comparison of the two becomes challenging. Spectral differences make it difficult to accurately align infrared and visible light images. However, the current neural network has insufficient registration accuracy for images of different modalities. How to improve the registration accuracy of infrared and visible light images has become a technical problem that technicians in this field need to solve urgently. Summary of the invention

[0004] In view of this, the present invention provides an infrared and weak visible light image registration method based on color edge enhancement, which improves the alignment accuracy between infrared images and visible light images.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides an infrared and weak visible light image registration method based on color edge enhancement, comprising the following steps:

[0007] Building an image registration model, wherein the image registration model includes two branches, namely a first branch and a second branch;

[0008] In the training stage, an infrared image I and a weak visible light image V are obtained, and the weak visible light image V is internally enhanced to generate a corresponding enhanced image A;

[0009] The first branch includes two feature extraction networks, two edge detection networks and a matching and regression network. The two feature extraction networks on the branch extract content features of the infrared image I and the weak visible light image V respectively, and the two edge detection networks extract edge features of the infrared image I and the weak visible light image V respectively. Then, the content features and edge features of the infrared image I and the weak visible light image V are fused to obtain their respective fused features.

[0010] The second branch includes a feature extraction network and a matching and regression network, wherein the feature extraction network on the branch extracts content features of the enhanced image A, and fuses the extracted content features with edge features of the weak visible light image V in the first branch to obtain fused features of the enhanced image A;

[0011] Input the fusion features of the infrared image I and the fusion features of the weak visible light image V into the matching and regression network in the first branch for matching and geometric transformation parameter prediction; input the fusion features of the infrared image I and the fusion image of the enhanced image A into the matching and regression network in the second branch for matching and geometric transformation parameter prediction;

[0012] The two branches are trained in parallel and weights are shared. After the preset goals are met, the final image registration model is obtained.

[0013] Furthermore, the internal enhancement process of the weak visible light image is only performed in the training stage. In each round of training, the weak visible light image V is internally enhanced by random color jittering to generate different enhanced images. In the testing and inference stages, the first branch is used to align the infrared image and the weak visible light image.

[0014] Furthermore, the edge detection network on the first branch is based on the RCF model and introduces the CBAM attention mechanism.

[0015] Furthermore, the feature extraction networks on the first branch and the second branch are composed of CBAM attention units, ELU activation functions, group convolutions and residual connections.

[0016] Furthermore, the matching and regression network on the first branch uses the fusion features of the weak visible light image and the fusion features of the infrared image as reference images and floating images in both directions to fuse the parameters of the two-way registration; the matching and regression network on the second branch uses the fusion features of the infrared image and the fusion features of the enhanced image corresponding to the weak visible light image as reference objects and floating objects in both directions to fuse the parameters of the two-way registration.

[0017] Furthermore, the weak visible light image in the first branch and the enhanced image in the second branch are collectively referred to as visible light images. The matching and regression networks on the first branch and the second branch first calculate the dense correspondence mapping between the infrared image features and the visible light image features through a matching function, and then estimate the geometric transformation parameters of the dense correspondence mapping through a regression network.

[0018] Furthermore, the dense correspondence mapping calculation method from infrared image features to visible light image features is:

[0019] C I→V (i,j,k)=C(f I (i k ,j k ),f V (i,j))=f V (i,j) T f I (i k ,j k )

[0020] Among them, C I→V Represents the infrared image feature map f I And the visible light image feature map f V The dense correspondence mapping between I and f V Multiply them to get (i,j) and (i k ,j k ) represent the matching point pairs P and P' in the visible light image feature map f V And infrared image feature map f I The coordinates in k represent the feature point (i k ,j k )'s auxiliary index;

[0021] The dense correspondence mapping calculation method from visible light image features to infrared image features is:

[0022] C V→I (i,j,k)=C(f V (i k ,j k ),f I (i,j))=f I (i,j) T f V (i k ,j k )

[0023] Among them, C V→I Represents the visible light image feature map f V And infrared image feature map f IThe dense correspondence mapping between V and f I Multiply them together to get .

[0024] Furthermore, the geometric transformation parameters of the regression network estimating the dense correspondence mapping are expressed as:

[0025] R:R h′×w′×(h′×w′) →R DoF

[0026] Among them, (h', w') represents the height and width of the dense correspondence map, DoF represents the degree of freedom of the affine transformation model, and R represents the regression network.

[0027] Furthermore, the process of constructing the loss function of the image registration model includes:

[0028] A point grid is constructed in the image space, and the difference between the two grids is measured by summing the square distances between the corresponding grid points. The basic loss function is constructed based on the difference. The calculation formula of the basic loss function is:

[0029]

[0030] Among them, θ represents the predicted affine transformation, θ gt represents the actual affine transformation, l(θ,θ gt ) represents θ and θ gt The basic loss between ; N represents the number of grid points, (x i ,y j ) represents the pixel at position (i, j); τ θ and Respectively represented by θ and θ gt Parameterized transformation operations;

[0031] On the basis of the basic loss function, an additional branch for calculating the reverse transformation is added to construct a bidirectional loss function. The calculation formula is:

[0032] L org = l(θ I→V ,θ gt I→V )+l(θ V→I ,θ gt V→I )

[0033] Among them, θ I→V represents the predicted affine transformation from infrared image I to visible light image V, θ gt I→V represents the affine transformation from the actual infrared image I to the visible light image V, θ V→I represents the predicted affine transformation from the visible light image V to the infrared image I, θgt V→I Represents the affine transformation from the actual visible light image V to the infrared image I

[0034] On the basis of the bidirectional loss function, considering the enhanced image, two losses L are added aug and L id , respectively:

[0035] L aug = l(θ I→A ,θ gt I→V )+l(θ A→I ,θ gt I→V )

[0036] L id = l(θ I→V ,θ I→A )+l(θ V→I ,θ A→I )

[0037] Among them, θ I→A represents the affine transformation from the predicted infrared image I to the enhanced image A, θ A→I Represents the affine transformation from the actual enhanced image A to the infrared image I;

[0038] The final constructed loss function is expressed as:

[0039] Loss = α·L org +β·L aug +γ·L id

[0040] Among them, α, β, and γ are the balance parameters of each loss function.

[0041] In a second aspect, the present invention provides an infrared and weak visible light image registration system based on color edge enhancement, comprising:

[0042] An acquisition module, used for acquiring infrared images and weak visible light images to be registered;

[0043] The registration module is used to register the infrared image and the weak visible light image using the image registration model trained according to the above-mentioned infrared and weak visible light image registration method based on color edge enhancement.

[0044] It can be seen from the above technical solutions that, compared with the prior art, the present invention has the following beneficial effects:

[0045] The present invention performs color enhancement on weak visible light images through random color jittering to obtain weak visible light images of different colors, and forms a training pair with the corresponding infrared image to form a stream of the network, which is parallel to the training pair composed of the original weak visible light image and the infrared image to form a dual-stream structure. The two streams are trained simultaneously to improve the comprehensiveness of the feature expression of the initial weak visible light image.

[0046] The feature extraction network of the present invention adds CBAM attention units and adopts ELU activation function, which can enhance the network's feature extraction capability for complex images. At the same time, considering the importance of edge features in images, the edge detection network of the present invention adds CBAM attention, enhances edge feature extraction capability, better emphasizes the edge information of the image, and helps capture key feature information that is easily overlooked in the image during the feature extraction stage. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0048] Figure 1 A schematic diagram of the structure of the image registration model provided by the present invention;

[0049] Figure 2 A schematic diagram of an original weak visible light image and a corresponding enhanced image provided by the present invention;

[0050] Figure 3 A comparison chart of the effects of the edge detection network provided by the present invention and other edge detection algorithms;

[0051] Figure 4 A schematic diagram of the structure of the feature extraction network provided by the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] like Figure 1 As shown, an embodiment of the present invention discloses a method for registering infrared and weak visible light images based on color edge enhancement, comprising the following steps:

[0054] Building an image registration model, the image registration model includes two branches, namely a first branch and a second branch;

[0055] In the training stage, an infrared image I and a weak visible light image V are obtained, and the weak visible light image V is internally enhanced to generate a corresponding enhanced image A;

[0056] The first branch includes two feature extraction networks, two edge detection networks and a matching and regression network. The two feature extraction networks on this branch extract the content features of the infrared image I and the weak visible light image V respectively to obtain F i-ce and F v-ce , the two edge detection networks extract the edge features of the infrared image I and the weak visible light image V respectively, and obtain F i-ed and F v-ed Then, the content features and edge features of the infrared image I and the weak visible light image V are fused to obtain their respective fusion features F i and F v ;

[0057] The second branch includes a feature extraction network and a matching and regression network. The feature extraction network on this branch extracts the content feature F of the enhanced image A. a-ce , extract the content feature F a-ce and the edge feature F of the weak visible light image V in the first branch v-ed Fusion is performed to obtain the fusion feature F of the enhanced image A a ;

[0058] The fusion feature F of the infrared image I i The fusion feature F of the weak visible light image V v Input the matching and regression network in the first branch to perform matching and geometric transformation parameter prediction; the fusion feature F of the infrared image I i The fused image F with the enhanced image A a Input the matching and regression network in the second branch to perform matching and geometric transformation parameter prediction;

[0059] The two branches are trained in parallel and weights are shared. After the preset goals are met, the final image registration model is obtained.

[0060] In general, the input infrared image I, the weak visible light image V and the enhanced image A corresponding to the weak visible light image are used to perform color enhancement, edge detection, feature extraction, matching and regression, so that the image registration model can finally learn the affine transformation parameters required for image registration. It can be divided into four stages:

[0061] The first stage is internal enhancement. The weak visible light image is enhanced by random color jitter and other methods. After enhancement, the infrared image I, the weak visible light image V and the enhanced image A corresponding to the weak visible light image are simultaneously input into the feature extraction network for edge detection and feature extraction respectively.

[0062] The second stage is edge detection.

[0063] The third stage is feature extraction. The feature information extracted by the feature extraction network and the edge information detected by the edge detection network are fused using a 3x3 ordinary convolution to obtain the corresponding fused information. The enhanced image A is randomly generated in each epoch, so the edge features are not extracted separately, but are fused with the edge features of the low-light image V during each epoch training. The fused information of I and V constitutes one branch of the network, and the fused information of A constitutes another branch, forming a dual-stream structure with weight sharing between the two branches.

[0064] The fourth stage is matching and regression. In the matching stage, matrix multiplication is performed on the feature map obtained after feature extraction to calculate the similarity, and the final transformation parameters are obtained through the regression network.

[0065] The following is a further explanation of the core modules involved in the image registration model (hereinafter referred to as CEREG).

[0066] (1) Color enhancement module

[0067] In the process of image registration, it is difficult to extract enough common features for subsequent prediction. In view of this, the present invention uses random color jittering to perform color enhancement on weak visible light images.

[0068] Before feature extraction, CEREG will perform random color jitter on the weak visible light image in each training epoch. By changing the color of each pixel in the image, the comprehensiveness of the weak visible light image expression in the dataset is expanded. During the training process, the original weak visible light image and infrared image constitute one stream, and the enhanced image and infrared image constitute another stream. The two streams are trained in parallel and share weights. Different enhanced images are generated by random color jittering, which makes the CEREG network highly robust to color. The internal enhancement process of weak visible light images is only performed in the training phase. In each round of training, the weak visible light image V is internally enhanced by random color jittering to generate different enhanced images. In the testing and inference phases, only a single-stream architecture without an internal enhancement process, that is, the first branch, is needed to align the infrared image and the weak visible light image. This can improve computational efficiency. The specific color enhancement effect is as follows: Figure 2 shown.

[0069] (2) Edge Detection Network

[0070] Edges are obvious areas of change in an image. Through edge detection, key features such as the outline, shape, and structure of an object can be extracted. These advantages make edge detection an important technical tool in image processing and computer vision. Therefore, the edge detection method is introduced into the registration network CEREG proposed in this invention to enhance the network's ability to extract common features between infrared images and corresponding weak visible light images.

[0071] Traditional edge detection operators (such as the Canny operator) are susceptible to noise interference, and deep learning methods (such as rich convolution features RCF) are also limited in effect. The context-aware tracking strategy CATS is integrated into RCF to obtain the CATS-RCF model. Although the CATS-RCF model can effectively improve the edge detection performance, there is still a noise problem. Therefore, the present invention introduces the convolutional block attention module CBAM into the CATS-RCF model, named CCR edge detection network, to obtain more accurate image edge features. The specific steps of CCR edge detection are as follows:

[0072] I CCR_Edge =E CCR (I in ) (1)

[0073] Among them, E CCR (·) and I CCR_Edge They represent the edge detection operation of the CCR edge detection network and the extracted edge feature map respectively.

[0074] In order to intuitively reflect the advantages of the CCR edge detection network proposed in this invention, the results of edge detection of images are compared using the traditional edge detection method Canny, the edge detection methods RCF, CATS-RCF and CCR based on deep learning. The comparison results are shown in Figure 2. Figure 3 As shown, it can be seen that the edge information extracted by the traditional edge detection method Canny is very poor, while the edge information extracted by the deep learning method is very rich. In the deep learning method, the edge information extracted by RCF is sufficient but there is more noise, and there is a problem of blurred edges; the CATS-RCF method reduces the noise, but the edge information is also reduced a lot, resulting in some missing details; the present invention improves on the basis of CATS-RCF, adds the CBAM attention mechanism to obtain the final CCR edge detection network, and the image edge information obtained by the CCR edge detection network is sufficient, and there is no problem of a lot of noise interference.

[0075] (3) Feature extraction network

[0076] Since the spectra of infrared images and visible light images are very different, the effect of registering them using existing registration methods is not ideal. Therefore, the present invention improves the deep learning-based method as a feature extraction method to alleviate the problem of unsatisfactory registration effect.

[0077] The ResNeXt network introduces group convolution based on the residual idea of ​​ResNet and shows good performance in visual tasks. Therefore, the present invention selects ResNeXt as the basic network and takes the first three residual modules of 3, 4, and 23, which are called CE-Res modules. The specific structure is as follows: Figure 4 As shown in the figure, the module consists of a convolutional block attention module CBAM, an ELU activation function, a group convolution, and a residual connection. The input of the CE-Res module is a 256-d feature Fin, which is processed by 32 branches to obtain Fc1,...,Fc32, which are then connected to Fin and then to obtain Fout through the ELU activation function. The processing of the branches is 1×1 convolution, ELU activation function, 3×3 convolution, ELU activation function, 1×1 convolution, and CBAM.

[0078] The CE-Res module avoids the gradient vanishing and exploding problems that may occur in deep neural network training through residual connection; the use of group convolution can better capture the complementarity and diversity between different features; the CBAM attention unit is inserted into the latter part of each residual block structure to adaptively learn the channel and spatial relationship in the image to enhance the feature representation ability of the convolutional neural network; the ReLU activation function in the original residual block is changed to the ELU activation function to enhance the network feature extraction capability while speeding up the network training.

[0079] The principle of CBAM attention is to automatically select and adjust the features that the model focuses on by learning attention weights. It is based on two key modules: the channel module and the spatial module. Through the combination of these two attention modules, CBAM can simultaneously focus on the feature relationship of the channel and spatial dimensions, thereby improving the expressiveness of the features. Therefore, in the feature extraction stage of CEREG, the CE-Res module of the present invention additionally integrates the CBAM attention unit, which can improve the perception and expression capabilities of the model.

[0080] The overall structure of the CBAM attention unit includes channel attention and spatial attention. The channel attention module aims to capture the relationship between channels in the input feature map to determine the importance of different channels. Specifically, the channel attention weights are obtained by performing global average pooling and two fully connected layers. Then, these weights are applied to the original feature map to perform feature weighting operations on the channel dimension.

[0081] The spatial attention module is used to capture the relationship between different spatial positions in the input feature map to determine the importance of each position. To achieve this goal, the maximum pooling and average pooling operations are used to extract spatial attention information from the input feature map. This information is then applied to the original feature map to perform feature weighting operations on the spatial dimension. The implementation process of the CBAM attention module is as follows:

[0082]

[0083]

[0084] Among them, F represents the input feature map, F' represents the feature map extracted by channel attention and the feature map after fusion of F, Denotes element-level multiplication. During the multiplication process, a broadcast mechanism is used to perform dimension transformation and matching. In this process, the channel attention value is broadcast along the spatial dimension while the spatial attention value is also broadcast along the channel dimension. F” denotes the final output.

[0085] M c (F) and M s (F) is obtained by the following formula:

[0086]

[0087] Among them, formula (4) provides a representation of the calculation process of channel attention. First, the input features are respectively average pooled and max pooled, and then the generated feature descriptors are Input to the shared perceptron network to generate the channel attention map M C ∈R C×1×1 Finally, the feature vectors are added by element-wise summation and processed using an activation function to obtain the final output. Formula (5) provides a representation of the computational process of spatial attention. First, the input features are sequentially passed through two pooling operations, and then concatenated and convolved through simple convolutions to generate a two-dimensional spatial attention map. Where σ represents the Sigmoid function, and the weight W0∈R C / r×C , weight W1∈R C×C / r .

[0088] In the field of deep learning, activation functions are a crucial component of neural networks. They are responsible for introducing nonlinear properties, allowing neural networks to adapt to complex data patterns and tasks. ELU (Exponential Linear Unit) is a commonly used activation function. It is an improved activation function for the rectified linear unit (ReLU) and is widely used in neural networks. The definition of the ELU activation function is as follows:

[0089]

[0090] Among them, x is the input value, and α is a predefined positive number, which indicates the slope when the input is negative. Compared with the ReLU activation function, the nonlinear part of ELU is an exponential function, which remains smooth and continuous in the negative region without stagnant gradients. Therefore, compared with other common activation functions (such as Sigmoid, ReLU, etc.), the ELU activation function has some unique characteristics:

[0091] ① Smoothness: The ELU function is smooth and continuous in the entire input domain. This is because it uses an exponential function to approximate the negative input part, avoiding the gradient mutation problem of the ReLU function.

[0092] ② Non-zero gradient: In the negative input area, the gradient of the ELU function is not zero. This is the biggest difference from the ReLU function. It can avoid the situation where neurons "die" or the gradient disappears.

[0093] ③ Approximate zero mean: When α is set to a small positive number, the output mean of the ELU function is close to zero, which is beneficial for some optimization techniques (such as batch normalization) and model stability.

[0094] ④ Robustness: The ELU function has good adaptability to Gaussian offset and noise of the data, which makes it more advantageous in processing complex and noisy data.

[0095] The above characteristics bring many advantages to the ELU activation function during the training process: by introducing an exponential function on negative inputs, the problem of zero gradient of the ReLU function in the negative area is solved; due to the non-zero gradient characteristics of the ELU activation function, the neural network is easier to converge using the ELU function, which improves the training speed; by improving the gradient problem and providing approximate zero mean characteristics, the ELU activation function can improve the generalization ability and overall performance of the model.

[0096] The ELU activation function is of great significance in the application of neural networks. By maintaining the characteristics of smooth and non-zero gradients, the ELU function can effectively solve the problems of the ReLU function and improve the stability, convergence speed and generalization ability of the neural network. Therefore, the ELU activation function is widely used in deep learning models, providing advantages for model training and performance.

[0097] (4) Matching and regression network

[0098] After the input image passes through the feature extraction network, the weak visible light image in the first branch and the enhanced image in the second branch are collectively referred to as visible light images. The matching and regression networks on the first branch and the second branch first calculate the dense correspondence mapping between the infrared image features and the visible light image features through the matching function, and then estimate the geometric transformation parameters of the dense correspondence mapping through the regression network.

[0099] Most current registration methods only consider registration in one direction. For infrared and visible light image registration, the infrared image is generally registered with the visible light image, that is, one-way learning (I->V): the visible light image is used as the reference image, and the infrared image is used as the floating image (geometric transformation is performed to move it closer to the visible light image).

[0100] The present invention adds an additional branch to estimate the symmetric transformation corresponding to the opposite direction. This two-way learning method can help reduce the differences between the estimated geometric transformation parameters between infrared images and visible light images and between visible light images and infrared images, that is, two-way learning (I->V and V->I): on the one hand, the visible light image is used as the reference image and the infrared image is used as the floating image; on the other hand, the infrared image is used as the reference image and the visible light image is used as the floating image.

[0101] By processing the registration in these two directions simultaneously, more consistent and accurate results can be achieved. CEREG calculates the similarity between them by matrix multiplication, and the similarity increases with the increase of cosine value.

[0102] Specifically, in the present invention, the matching and regression network on the first branch uses the fusion features of the weak visible light image and the fusion features of the infrared image as the reference image and the floating image in both directions, and fuses the parameters of the two-way registration; the matching and regression network on the second branch uses the fusion features of the infrared image and the fusion features of the enhanced image corresponding to the weak visible light image as the reference image and the floating image in both directions, and fuses the parameters of the two-way registration.

[0103] That is to say, the matching and regression network on the first branch uses the fusion features of the weak visible light image as the reference image and the fusion features of the infrared image as the floating image to register the two; the matching and regression network on the second branch uses the fusion features of the infrared image as the reference object and the fusion features of the enhanced image corresponding to the weak visible light image as the floating object to register the two;

[0104] Alternatively: the matching and regression network on the first branch uses the fusion features of the infrared image as the reference image and the fusion features of the weak visible light image as the floating image to align the two; the matching and regression network on the second branch uses the fusion features of the enhanced image corresponding to the weak visible light image as the reference object and the fusion features of the infrared image as the floating object to align the two.

[0105] 1) Matching

[0106] The matching function C is used as a method for calculating the dense correspondence mapping between two feature maps, where the dense correspondence mapping from infrared image features to visible light image features is calculated as:

[0107] C I→V (i,j,k)=C(f I (i k ,j k ),f V (i,j))=f V (i,j) T f I (i k ,j k ) (7)

[0108] Among them, C I→V Represents the infrared image feature map f I And the visible light image feature map f V The dense correspondence mapping between I and f V Multiply them to get (i,j) and (i k ,j k ) represent the matching point pairs P and P' in the visible light image feature map f V And infrared image feature map f I The coordinates in; k represents the feature point (i k ,j k )'s auxiliary index;

[0109] The dense correspondence mapping calculation method from visible light image features to infrared image features is:

[0110] C V→I (i,j,k)=C(f V (i k ,j k ),f I (i,j))=f I (i,j) T f V (i k ,j k ) (8)

[0111] Among them, C V→I Represents the visible light image feature map f V And infrared image feature map f I The dense correspondence mapping between V and f I Multiply them together to get .

[0112] In this process, the similarity between points is represented by the feature point position (i, j). After obtaining the similarity feature map, it is normalized to generate the corresponding mapping map. Here, CEREG uses ReLU+L2Norm as the normalization method.

[0113] 2) Regression of transformation parameters

[0114] The regression step is used to predict the transformation parameters. When the dense correspondence map passes through the regression network R, the network R directly estimates the geometric transformation parameters as follows:

[0115] R:R h′×w′×(h′×w′) →R DoF (9)

[0116] Among them, (h', w') represents the height and width of the dense correspondence map, DoF represents the degree of freedom of the affine transformation model, and R represents the regression network.

[0117] (5) Construction of loss function, including:

[0118] During the training process, the present invention uses the transformed grid loss as the basic loss function. A point grid is constructed in the image space, and the neural network is used to predict the value τ θ and tags Use parameters θ and θ respectively gt Transform it, measure the difference between the two transformed grids by summing the square distances between the corresponding grid points, and construct the basic loss function based on the difference. The calculation formula of the basic loss function is:

[0119]

[0120] Among them, θ represents the predicted affine transformation, θ gt represents the actual affine transformation, l(θ,θ gt ) represents θ and θ gt The basic loss between them; N represents the number of grid points, τ θ and Respectively represented by θ and θ gt Parameterized transformation operation; (x i ,y j ) represents the pixel at position (i, j);

[0121] In order to achieve bidirectional learning, on the basis of the basic loss function, an additional branch for calculating the reverse transformation is added to construct a bidirectional loss function. The calculation formula is:

[0122]

[0123] Among them, θ I→Vrepresents the predicted affine transformation from infrared image I to visible light image V, θ gt I→V represents the affine transformation from the actual infrared image I to the visible light image V, θ V→I represents the predicted affine transformation from the visible light image V to the infrared image I, θ gt V→I Represents the affine transformation from the actual visible light image V to the infrared image I;

[0124] In order to regularize the training, on the basis of the bidirectional loss function, consider the enhanced image and add two losses L aug and L id , respectively:

[0125] L aug = l(θ I→A ,θ gt I→V )+l(θ A→I ,θ gt I→V ) (12)

[0126] L id = l(θ I→V ,θ I→A )+l(θ V→I ,θ A→I ) (13)

[0127] Among them, θ I→A represents the affine transformation from the predicted infrared image I to the enhanced image A, θ A→I Represents the affine transformation from the actual enhanced image A to the infrared image I;

[0128] Since the geometric relationship between the two images is equivalent to the original image pair, the enhanced image pair also shares labels. The identity term in formula (13) induces training to ensure that the prediction values ​​from the original pair and the enhanced pair are equal. The final constructed loss function is expressed as:

[0129] Loss = α·L org +β·L aug +γ·L id (14)

[0130] Among them, α, β, and γ are the balance parameters of each loss function. In the experiment, these parameters are set to 0.5, 0.3, and 0.2 respectively.

[0131] In general, in the preprocessing stage, the present invention performs color enhancement on weak visible light images by random color jittering to obtain weak visible light images of different colors, and forms a training pair with the corresponding infrared image to form a stream of the network, which is parallel to the training pair composed of the original weak visible light image and the infrared image to form a dual-stream structure. The two streams are trained simultaneously to improve the comprehensiveness of the feature expression of the initial weak visible light image. In the feature extraction stage, the present invention proposes a CE-Res feature extraction network, which is mainly composed of CBAM attention and ELU activation function to enhance the network's feature extraction ability for complex images. Considering the importance of edge features in images, the present invention proposes a CCR edge detection network. This module adds CBAM attention before CATS-RCF to enhance the edge feature extraction capability and better emphasize the edge information of the image. Adding CCR in the feature extraction stage helps CEREG capture key feature information that is easily overlooked in the image. The present invention has been experimentally demonstrated on the LLVIP dataset that when α=0.1, the probability of correct keypoints (PCK) is increased by 5.62% compared to the current optimal network TEIVRNet, while visually presenting a superior registration effect. This indicates that CEREG has achieved significant results in improving the accuracy of image registration, and effectively fused the failed infrared and weak visible light images.

[0132] In other embodiments, the present invention provides an infrared and weak visible light image registration system based on color edge enhancement, comprising:

[0133] An acquisition module, used for acquiring infrared images and weak visible light images to be registered;

[0134] The registration module is used to register the infrared image and the weak visible light image using the image registration model trained according to the above-mentioned infrared and weak visible light image registration method based on color edge enhancement.

[0135] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0136] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for infrared and weak visible light image registration based on color edge enhancement, characterized in that: The following steps are involved: Building an image registration model, wherein the image registration model includes two branches, namely a first branch and a second branch; In the training stage, an infrared image I and a weak visible light image V are obtained, and the internal color of the weak visible light image V is enhanced to generate a corresponding enhanced image A; The first branch includes two feature extraction networks, two edge detection networks and a matching and regression network. The two feature extraction networks on the branch extract content features of the infrared image I and the weak visible light image V respectively, and the two edge detection networks extract edge features of the infrared image I and the weak visible light image V respectively. Then, the content features and edge features of the infrared image I and the weak visible light image V are fused to obtain their respective fused features. The second branch includes a feature extraction network and a matching and regression network. The feature extraction network on the branch extracts content features of the enhanced image A, and fuses the extracted content features with edge features of the weak visible light image V in the first branch to obtain fused features of the enhanced image A. Input the fusion features of the infrared image I and the fusion features of the weak visible light image V into the matching and regression network in the first branch to perform matching and geometric transformation parameter prediction; The fused features of the infrared image I and the fused image of the enhanced image A are input into the matching and regression network in the second branch for matching and geometric transformation parameter prediction; The two branches are trained in parallel and weights are shared. After the preset goals are met, the final image registration model is obtained.

2. The infrared and weak visible light image registration method based on color edge enhancement according to claim 1 is characterized in that: The internal enhancement process of weak visible light images is only performed in the training stage. In each round of training, the weak visible light image V is internally enhanced by random color jittering to generate different enhanced images. In the testing and inference stages, the first branch is used to align the infrared image and the weak visible light image.

3. The infrared and weak visible light image registration method based on color edge enhancement according to claim 1, characterized in that: The edge detection network on the first branch is based on the rich convolutional feature RCF model, integrates the context-aware tracking strategy CATS into RCF, and introduces the convolutional block attention module CBAM attention mechanism.

4. The infrared and weak visible light image registration method based on color edge enhancement according to claim 1, characterized in that: The feature extraction networks on the first branch and the second branch are composed of CBAM attention units, ELU activation functions, group convolutions and residual connections.

5. The infrared and weak visible light image registration method based on color edge enhancement according to claim 1, characterized in that: The matching and regression network on the first branch uses the fusion features of the weak visible light image and the fusion features of the infrared image as the reference image and the floating image in both directions to fuse the parameters of the two-way registration; the matching and regression network on the second branch uses the fusion features of the infrared image and the fusion features of the enhanced image corresponding to the weak visible light image as the reference image and the floating image in both directions to fuse the parameters of the two-way registration.

6. The infrared and weak visible light image registration method based on color edge enhancement according to claim 5, characterized in that: The weak visible light image in the first branch and the enhanced image in the second branch are collectively referred to as visible light images. The matching and regression networks on the first branch and the second branch first calculate the dense corresponding mapping between the infrared image features and the visible light image features through a matching function, and then estimate the geometric transformation parameters of the dense corresponding mapping through a regression network.

7. The infrared and weak visible light image registration method based on color edge enhancement according to claim 6, characterized in that: The dense correspondence mapping from infrared image features to visible light image features is calculated as follows: C I→V (i,j,k)=C(f I (i k ,j k ),f V (i,j))=f V (i,j) T f I (i k ,j k ) Among them, C I→V Represents the infrared image feature map f I And the visible light image feature map f V The dense correspondence mapping between I and f V Multiply them to get (i,j) and (i k ,j k ) represent the matching point pairs P and P' in the visible light image feature map f V And infrared image feature map f I The coordinates in k represent the feature point (i k ,j k )'s auxiliary index; The dense correspondence mapping calculation method from visible light image features to infrared image features is: C V→I (i,j,k)=C(f V (i k ,j k ),f I (i,j))=f I (i,j) T f V (i k ,j k ) Among them, C V→I Represents the visible light image feature map f V And infrared image feature map f I The dense correspondence mapping between V and f I Multiply them together to get .

8. The infrared and weak visible light image registration method based on color edge enhancement according to claim 6, characterized in that: The geometric transformation parameters of the regression network estimating the dense correspondence mapping are expressed as: R:R h′×w′×(h′×w′) →R DoF Among them, (h', w') represents the height and width of the dense correspondence map, DoF represents the degree of freedom of the affine transformation model, and R represents the regression network.

9. The infrared and weak visible light image registration method based on color edge enhancement according to claim 1, characterized in that: The process of constructing the loss function of the image registration model includes: A point grid is constructed in the image space, and the difference between the two grids is measured by summing the square distances between the corresponding grid points. The basic loss function is constructed based on the difference. The calculation formula of the basic loss function is: Among them, θ represents the predicted affine transformation, θ gt represents the actual affine transformation, l(θ,θ gt ) represents θ and θ gt The basic loss between ; N represents the number of grid points, (x i ,y j ) represents the pixel at position (i, j); τ θ and Respectively represented by θ and θ gt Parameterized transformation operations; On the basis of the basic loss function, an additional branch for calculating the reverse transformation is added to construct a bidirectional loss function. The calculation formula is: L org =l(θ I→V ,i gt I→V )+l(θ V→I ,i gt V→I ) Among them, θ I→V represents the predicted affine transformation from infrared image I to visible light image V, θ gt I→V represents the affine transformation from the actual infrared image I to the visible light image V, θ V→I represents the predicted affine transformation from the visible light image V to the infrared image I, θ gt V→I Represents the affine transformation from the actual visible light image V to the infrared image I; On the basis of the bidirectional loss function, considering the enhanced image, two losses L are added aug and L id , respectively: L aug =l(θ I→A ,i gt I→V )+l(θ A→I ,i gt I→V ) L id =l(θ I→V ,i I→A )+l(θ V→I ,i A→I ) Among them, θ I→A represents the affine transformation from the predicted infrared image I to the enhanced image A, θ A→I Represents the affine transformation from the actual enhanced image A to the infrared image I; The final constructed loss function is expressed as: Loss=α·L org +β·L aug +γ·L id Among them, α, β, and γ are the balance parameters of each loss function.

10. An infrared and weak visible light image registration system based on color edge enhancement, characterized in that: include: An acquisition module, used for acquiring infrared images and weak visible light images to be registered; A registration module is used to register the infrared image and the weak visible light image using the image registration model trained by the infrared and weak visible light image registration method based on color edge enhancement as described in any one of claims 1 to 9.

Images