Infrared and weak visible light image registration method and system based on color edge enhancement
By building an image registration model, using feature extraction and edge detection networks, combined with CBAM attention mechanism and ELU activation function, the problem of insufficient alignment accuracy between infrared images and visible light images is solved, and a more efficient image registration effect is achieved.
Patent Information
- Application Number
- CN202411131169.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-17
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-17
AI Technical Summary
Due to differences in bands and imaging conditions, infrared images and visible light images are difficult to accurately align in geometric shapes, resulting in parallax and residual distortion, affecting image quality and accuracy. The existing neural networks lack the accuracy of multimodal images registration for multimodal images.
Using infrared and weak visible light image registration method based on color edge enhancement, the image registration model is built, and feature extraction network, edge detection network, and matching and regression network are used to carry out parallel training and weight sharing, combining CBAM attention mechanism and ELU Activate the function to improve feature extraction and edge detection capabilities, and achieve accurate registration of infrared images and visible light images.
The alignment accuracy between infrared images and visible light images is improved, the accuracy and robustness of image registration is enhanced, and the difference in geometric transformation parameters is reduced through two-way learning and loss function optimization, and the effect of image fusion is improved.
Smart Images

Figure CN119068002A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and more particularly to a method and system for registering infrared and weak visible light images based on color edge enhancement. Background Art
[0002] Visible light images and infrared images each have their own unique advantages and characteristics in the imaging field. Visible light images, covering a wavelength range from 380 nanometers to 780 nanometers, offer intuitive clarity and rich colors, making them suitable for everyday photography and videography. However, they are easily affected by changes in lighting. Weak visible light images are images whose features are blurred due to the effects of lighting. In contrast, infrared images, covering a wavelength range from 3 microns to 14 microns, generate images using the thermal radiation information of objects. They are unaffected by lighting and distance, making them suitable for nighttime monitoring and hidden object detection. Therefore, fusing infrared and visible light images helps obtain more comprehensive information, enhance image recognition, and improve the accuracy and efficiency of target detection and recognition. By rationally fusing visible light and infrared images, more comprehensive information can be acquired, expanding their application areas.
[0003] However, differences in the characteristics of different sensors and their usage environments lead to significant differences between infrared and visible light images in terms of wavelength bands, imaging conditions, etc., making them inconsistent in geometry. This inconsistency involves multiple aspects such as position, scale, rotation, and deformation, making it difficult to accurately align them on the same epipolar line. This situation will cause parallax and residual distortion, which in turn affects image quality and accuracy. It is not easy to achieve multimodal image registration. Directly fusing or comparing the two becomes challenging. Spectral differences make it difficult to accurately align infrared and visible light images. However, the current neural network has insufficient registration accuracy for images of different modalities. How to improve the registration accuracy of infrared and visible light images has become a technical problem that technicians in this field urgently need to solve. Summary of the Invention
[0004] In view of this, the present invention provides an infrared and weak visible light image registration method based on color edge enhancement, which improves the alignment accuracy between infrared images and visible light images.
[0005] In order to achieve the above object, the present invention adopts the following technical solutions:
[0006] In a first aspect, the present invention provides a method for registering infrared and weak visible light images based on color edge enhancement, comprising the following steps:
[0007] Building an image registration model, wherein the image registration model includes two branches, namely a first branch and a second branch;
[0008] In the training phase, an infrared image I and a weak visible light image V are obtained, and the weak visible light image V is internally enhanced to generate the corresponding enhanced image A;
[0009] The first branch includes two feature extraction networks, two edge detection networks, and a matching and regression network. The two feature extraction networks in this branch respectively extract content features of the infrared image I and the weak visible light image V, and the two edge detection networks respectively extract edge features of the infrared image I and the weak visible light image V. The content features and edge features of the infrared image I and the weak visible light image V are then fused to obtain their respective fused features.
[0010] The second branch includes a feature extraction network and a matching and regression network. The feature extraction network on the branch extracts content features of the enhanced image A, and fuses the extracted content features with the edge features of the weak visible light image V in the first branch to obtain fused features of the enhanced image A.
[0011] Input the fused features of the infrared image I and the fused features of the weak visible light image V into the matching and regression network in the first branch for matching and geometric transformation parameter prediction; input the fused features of the infrared image I and the fused image of the enhanced image A into the matching and regression network in the second branch for matching and geometric transformation parameter prediction;
[0012] The two branches are trained in parallel and weights are shared. After the preset goals are met, the final image registration model is obtained.
[0013] Furthermore, the internal enhancement process of the weak visible light image is only performed in the training phase. In each round of training, the weak visible light image V is internally enhanced by random color jittering to produce different enhanced images. In the testing and inference phases, the first branch is used to align the infrared image and the weak visible light image.
[0014] Furthermore, the edge detection network on the first branch is based on the RCF model and introduces the CBAM attention mechanism.
[0015] Furthermore, the feature extraction networks on the first branch and the second branch are composed of CBAM attention units, ELU activation functions, group convolutions and residual connections.
[0016] Furthermore, the matching and regression network on the first branch uses the fusion features of the weak visible light image and the fusion features of the infrared image as the reference image and the floating image in both directions to fuse the parameters of the bidirectional registration; the matching and regression network on the second branch uses the fusion features of the infrared image and the fusion features of the enhanced image corresponding to the weak visible light image as the reference object and the floating object in both directions to fuse the parameters of the bidirectional registration.
[0017] Furthermore, the weak visible light image in the first branch and the enhanced image in the second branch are collectively referred to as visible light images. The matching and regression networks on the first branch and the second branch first calculate the dense correspondence mapping between the infrared image features and the visible light image features through a matching function, and then estimate the geometric transformation parameters of the dense correspondence mapping through the regression network.
[0018] Furthermore, the dense correspondence mapping from infrared image features to visible light image features is calculated as follows:
[0019] C I→V (i,j,k)=C(f I (i k ,j k ),f V (i,j))=f V (i,j) T f I (i k ,j k )
[0020] Among them, C I→V Represents the infrared image feature map f I and visible light image feature map f V The dense correspondence mapping between I and f V Multiply them together to get (i,j) and (i k ,j k ) represent the matching point pairs P and P' in the visible light image feature map f V and infrared image feature map f I The coordinates in k represent the feature point (i k ,j k )'s auxiliary index;
[0021] The dense correspondence mapping from visible light image features to infrared image features is calculated as follows:
[0022] C V→I (i,j,k)=C(f V (i k ,j k ),f I (i,j))=f I (i,j) T f V (i k ,j k )
[0023] Among them, C V→I Represents the visible light image feature map f V and infrared image feature map f IThe dense correspondence mapping between V and f I Multiply them together to get .
[0024] Furthermore, the regression network estimates the geometric transformation parameters of the dense correspondence mapping as:
[0025] R:R h′×w′×(h′×w′) →R DoF
[0026] Among them, (h', w') represents the height and width of the dense correspondence map, DoF represents the degree of freedom of the affine transformation model, and R represents the regression network.
[0027] Furthermore, the process of constructing the loss function of the image registration model includes:
[0028] A point grid is constructed in the image space. The difference between the two grids is measured by summing the squared distances between the corresponding grid points. The basic loss function is constructed based on this difference. The calculation formula of the basic loss function is:
[0029]
[0030] Among them, θ represents the predicted affine transformation, θ gt represents the actual affine transformation, l(θ,θ gt ) represents θ and θ gt The basis loss between; N represents the number of grid points, (x i ,y j ) represents the pixel at position (i, j); τ θ and Represented by θ and θ gt Parameterized transformation operations;
[0031] On the basis of the basic loss function, an additional branch of the reverse transformation is added to construct a bidirectional loss function. The calculation formula is:
[0032] L org =l(θ I→V ,θ gt I→V )+l(θ V→I ,θ gt V→I )
[0033] Among them, θ I→V represents the predicted affine transformation from infrared image I to visible light image V, θ gt I→V represents the affine transformation from the actual infrared image I to the visible light image V, θ V→I represents the predicted affine transformation from the visible light image V to the infrared image I, θgt V→I Represents the affine transformation from the actual visible light image V to the infrared image I
[0034] On the basis of the bidirectional loss function, considering the enhanced image, two losses L are added aug and L id , respectively:
[0035] L aug =l(θ I→A ,θ gt I→V )+l(θ A→I ,θ gt I→V )
[0036] L id =l(θ I→V ,θ I→A )+l(θ V→I ,θ A→I )
[0037] Among them, θ I→A represents the affine transformation from the predicted infrared image I to the enhanced image A, θ A→I Represents the affine transformation from the actual enhanced image A to the infrared image I;
[0038] The final constructed loss function is expressed as:
[0039] Loss = α·L org +β·L aug +γ·L id
[0040] Among them, α, β, and γ are the balance parameters of each loss function.
[0041] In a second aspect, the present invention provides an infrared and weak visible light image registration system based on color edge enhancement, comprising:
[0042] An acquisition module, used for acquiring infrared images and weak visible light images to be registered;
[0043] The registration module is used to register the infrared image and the weak visible light image using the image registration model trained according to the above-mentioned infrared and weak visible light image registration method based on color edge enhancement.
[0044] It can be seen from the above technical solutions that compared with the prior art, the present invention has the following beneficial effects:
[0045] The present invention performs color enhancement on weak visible light images through random color dithering to obtain weak visible light images of different colors, and forms a training pair with the corresponding infrared image to form a stream of the network. In parallel with the training pair composed of the original weak visible light image and the infrared image, a dual-stream structure is formed. The two streams are trained simultaneously to improve the comprehensiveness of the feature expression of the initial weak visible light image.
[0046] The feature extraction network of the present invention adds a CBAM attention unit and adopts the ELU activation function to enhance the network's feature extraction capabilities for complex images. Furthermore, considering the importance of edge features in images, the edge detection network of the present invention adds CBAM attention to enhance edge feature extraction capabilities, better emphasizing edge information in images and helping to capture key features that are easily overlooked in the image during the feature extraction stage. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0048] Figure 1 A schematic diagram of the structure of the image registration model provided by the present invention;
[0049] Figure 2 Schematic diagram of the original weak visible light image and the corresponding enhanced image provided by the present invention;
[0050] Figure 3 A comparison chart of the effects of the edge detection network provided by the present invention and other edge detection algorithms;
[0051] Figure 4 This is a schematic diagram of the structure of the feature extraction network provided by the present invention. DETAILED DESCRIPTION
[0052] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0053] like Figure 1 As shown, an embodiment of the present invention discloses a method for registering infrared and weak visible light images based on color edge enhancement, comprising the following steps:
[0054] Build an image registration model, which includes two branches, namely the first branch and the second branch;
[0055] In the training phase, an infrared image I and a weak visible light image V are obtained, and the weak visible light image V is internally enhanced to generate the corresponding enhanced image A;
[0056] The first branch includes two feature extraction networks, two edge detection networks and a matching and regression network. The two feature extraction networks on this branch extract the content features of the infrared image I and the weak visible light image V respectively to obtain F i-ce and F v-ce , the two edge detection networks extract the edge features of the infrared image I and the weak visible light image V respectively, and obtain F i-ed and F v-ed , then fuse the content features and edge features of the infrared image I and the weak visible light image V to obtain their respective fusion features F i and F v ;
[0057] The second branch contains a feature extraction network and a matching and regression network. The feature extraction network on this branch extracts the content features F of the enhanced image A. a-ce , extract the content feature F a-ce and the edge feature F of the weak visible light image V in the first branch v-ed Fusion is performed to obtain the fusion feature F of the enhanced image A a ;
[0058] The fusion feature F of the infrared image I i Fusion feature F with weak visible light image V v Input the matching and regression network in the first branch to perform matching and geometric transformation parameter prediction; the fusion feature F of the infrared image I i The fused image F with the enhanced image A a Input the matching and regression network in the second branch to perform matching and geometric transformation parameter prediction;
[0059] The two branches are trained in parallel and weights are shared. After the preset goals are met, the final image registration model is obtained.
[0060] In general, the input infrared image I, the weak visible light image V and the enhanced image A corresponding to the weak visible light image are used. Through color enhancement, edge detection, feature extraction, matching and regression, the image registration model finally learns the affine transformation parameters required for image registration. Specifically, it can be divided into four stages:
[0061] The first stage is internal enhancement. The weak visible light image is enhanced using methods such as random color dithering. After enhancement, the infrared image I, the weak visible light image V, and the enhanced image A corresponding to the weak visible light image are simultaneously input into the feature extraction network for edge detection and feature extraction, respectively.
[0062] The second stage is edge detection.
[0063] The third stage is feature extraction. The feature information extracted by the feature extraction network is fused with the edge information detected by the edge detection network using a 3x3 normal convolution to obtain the corresponding fused information. The enhanced image A is randomly generated at each epoch, so edge features are not extracted separately. Instead, they are fused with the edge features of the low-light image V during each training epoch. The fused information of I and V forms one branch of the network, and the fused information of A forms another branch, forming a dual-stream structure with weight sharing between the two branches.
[0064] The fourth stage is matching and regression. In the matching stage, matrix multiplication is performed on the feature map obtained after feature extraction to calculate the similarity, and the final transformation parameters are obtained through the regression network.
[0065] The following is a further explanation of the core modules involved in the image registration model (hereinafter referred to as CEREG).
[0066] (1) Color enhancement module
[0067] During the image registration process, it is difficult to extract enough common features for subsequent prediction. To address this, the present invention uses random color jittering to perform color enhancement on weak visible light images.
[0068] Before feature extraction, CEREG will perform random color dithering on the weak visible light image in each training epoch. By changing the color of each pixel in the image, the comprehensiveness of the weak visible light image expression in the dataset is expanded. During the training process, the original weak visible light image and infrared image constitute one stream, and the enhanced image and infrared image constitute another stream. The two streams are trained in parallel and share weights. Different enhanced images are generated by random color dithering, which makes the CEREG network highly robust to color. The internal enhancement process of weak visible light images is only performed in the training phase. In each round of training, the weak visible light image V is internally enhanced by random color dithering to generate different enhanced images; in the testing and inference phase, only a single-stream architecture without an internal enhancement process, that is, the first branch, is needed to align the infrared image and the weak visible light image. This can improve computational efficiency. The specific color enhancement effect is as follows: Figure 2 shown.
[0069] (2) Edge Detection Network
[0070] Edges are distinct regions of change in an image. Edge detection can be used to extract key features such as an object's outline, shape, and structure. These advantages make edge detection a crucial tool in image processing and computer vision. Therefore, we incorporate edge detection into the proposed registration network, CEREG, to enhance the network's ability to extract shared features between infrared images and their corresponding weak visible light images.
[0071] Traditional edge detection operators (such as the Canny operator) are susceptible to noise interference, and the effects of deep learning methods (such as rich convolution features RCF) are also limited. The context-aware tracking strategy CATS is integrated into RCF to obtain the CATS-RCF model. Although the CATS-RCF model can effectively improve the edge detection performance, the noise problem still exists. Therefore, the present invention introduces the convolutional block attention module CBAM into the CATS-RCF model, named CCR edge detection network, which aims to obtain more accurate image edge features. The specific steps of CCR edge detection are as follows:
[0072] I CCR_Edge =E CCR (I in ) (1)
[0073] Among them, E CCR (·) and I CCR_Edge They represent the edge detection operation of the CCR edge detection network and the extracted edge feature map respectively.
[0074] In order to intuitively demonstrate the advantages of the CCR edge detection network proposed in this paper, the results of edge detection of images are compared using the traditional edge detection method Canny, the edge detection method RCF based on deep learning, CATS-RCF and CCR. The comparison results are shown in the figure. Figure 3 As shown in the figure, the traditional edge detection method Canny extracts very poor edge information, while the deep learning method extracts very rich edge information. Among the deep learning methods, RCF extracts sufficient edge information but has a lot of noise, resulting in blurred edges. The CATS-RCF method reduces noise, but also significantly reduces edge information, resulting in some missing details. The present invention improves on CATS-RCF and adds the CBAM attention mechanism to obtain the final CCR edge detection network. The image edge information obtained by the CCR edge detection network is sufficient and does not suffer from the problem of large amounts of noise interference.
[0075] (3) Feature extraction network
[0076] Due to the huge spectral difference between infrared images and visible light images, the effect of registering them using existing registration methods is not ideal. Therefore, the present invention improves the deep learning-based method as a feature extraction method to alleviate the problem of unsatisfactory registration effect.
[0077] The ResNeXt network introduces group convolution based on the residual idea of ResNet and shows good performance in visual tasks. Therefore, this paper selects ResNeXt as the basic network and takes the first three residual modules of 3, 4, and 23, which are called CE-Res modules. The specific structure is as follows: Figure 4 As shown in Figure 1, this module consists of a convolutional block attention module (CBAM), an ELU activation function, group convolution, and a residual connection. The input to the CE-Res module is a 256-d feature Fin. Fin is processed through 32 branches to obtain Fc1, ..., Fc32. Fc1, ..., Fc32 are concatenated and then connected to Fin. The resulting Fout is obtained through the ELU activation function. The branch processing sequence is 1×1 convolution, ELU activation function, 3×3 convolution, ELU activation function, 1×1 convolution, and CBAM.
[0078] The CE-Res module avoids the gradient vanishing and exploding problems that may occur in deep neural network training through residual connections; the use of group convolution can better capture the complementarity and diversity between different features; the CBAM attention unit is inserted at the end of each residual block structure to adaptively learn the channel and spatial relationships in the image to enhance the feature representation ability of the convolutional neural network; the ReLU activation function in the original residual block is changed to the ELU activation function, which enhances the network feature extraction capability while speeding up network training.
[0079] The principle of CBAM attention is to automatically select and adjust the features the model focuses on by learning attention weights. It is based on two key modules: the channel module and the spatial module. By combining these two attention modules, CBAM can simultaneously focus on feature relationships in both channel and spatial dimensions, thereby improving the expressiveness of features. Therefore, during the feature extraction stage of CEREG, the CE-Res module of the present invention additionally integrates the CBAM attention unit, which can improve the model's perception and expressive capabilities.
[0080] The overall structure of the CBAM attention unit comprises channel attention and spatial attention. The channel attention module aims to capture the relationship between channels in the input feature map to determine the importance of each channel. Specifically, channel attention weights are obtained by performing global average pooling and two fully connected layers. These weights are then applied to the original feature map to perform feature weighting on the channel dimension.
[0081] The spatial attention module is used to capture the relationship between different spatial locations in the input feature map to determine the importance of each location. To achieve this goal, maximum pooling and average pooling operations are used to extract spatial attention information from the input feature map. This information is then applied to the original feature map to perform feature weighting operations on the spatial dimension. The implementation process of the CBAM attention module is as follows:
[0082]
[0083]
[0084] Among them, F represents the input feature map, F' represents the feature map extracted by channel attention and the feature map after fusion of F, Denotes element-wise multiplication. During the multiplication, a broadcasting mechanism is used to transform and match dimensions. During this process, the channel attention values are broadcast along the spatial dimension, while the spatial attention values are also broadcast along the channel dimension. F" denotes the final output.
[0085] M c (F) and M s (F) is obtained from the following formula:
[0086]
[0087] Among them, formula (4) provides the calculation process of channel attention. First, the input features are average pooled and max pooled respectively, and then the generated feature descriptors are Input to the shared perceptron network to generate the channel attention map M C ∈R C×1×1 Finally, the feature vectors are added together by element-wise summation and processed using the activation function to obtain the final output. Formula (5) provides a representation of the computational process of spatial attention. First, the input features are sequentially passed through two pooling operations, and then concatenated and convolved by simple convolution to generate a two-dimensional spatial attention map. Where σ represents the Sigmoid function, and the weight W0∈R C / r×C , weight W1∈R C×C / r .
[0088] In the field of deep learning, activation functions are a crucial component of neural networks. They are responsible for introducing nonlinear properties, allowing neural networks to adapt to complex data patterns and tasks. ELU (Exponential Linear Unit) is a commonly used activation function. It is an improved activation function for the rectified linear unit (ReLU) and is widely used in neural networks. The definition of the ELU activation function is as follows:
[0089]
[0090] Where x is the input value and α is a predefined positive number representing the slope for negative inputs. Compared to the ReLU activation function, the nonlinear part of the ELU is an exponential function, which remains smooth and continuous in the negative region without stagnant gradients. Therefore, compared to other common activation functions (such as Sigmoid, ReLU, etc.), the ELU activation function has some unique characteristics:
[0091] ① Smoothness: The ELU function is smooth and continuous in the entire input domain. This is because it uses an exponential function to approximate the negative input part, avoiding the gradient mutation problem of the ReLU function.
[0092] ② Non-zero gradient: In the negative input area, the gradient of the ELU function is not zero. This is the biggest difference from the ReLU function. It can avoid the situation where neurons "die" or the gradient disappears.
[0093] ③ Approximate zero mean: When α is set to a small positive number, the output mean of the ELU function is close to zero, which is beneficial for some optimization techniques (such as batch normalization) and model stability.
[0094] ④ Robustness: The ELU function has good adaptability to Gaussian offset and noise of the data, which makes it more advantageous in processing complex and noisy data.
[0095] The above characteristics bring many advantages to the ELU activation function during training: by introducing an exponential function on negative inputs, the problem of the ReLU function's zero gradient in the negative region is solved; due to the non-zero gradient characteristic of the ELU activation function, the neural network is easier to converge using the ELU function, which improves the training speed; by improving the gradient problem and providing an approximate zero mean characteristic, the ELU activation function can improve the generalization ability and overall performance of the model.
[0096] The ELU activation function is of great significance in neural network applications. By maintaining smooth, non-zero gradients, the ELU function effectively addresses the issues of the ReLU function, improving the stability, convergence speed, and generalization capabilities of neural networks. Consequently, the ELU activation function is widely used in deep learning models, providing advantages for both training and performance.
[0097] (4) Matching and regression network
[0098] After the input image passes through the feature extraction network, the weak visible light image in the first branch and the enhanced image in the second branch are collectively referred to as the visible light image. The matching and regression networks on the first and second branches first calculate the dense correspondence mapping between the infrared image features and the visible light image features through the matching function, and then estimate the geometric transformation parameters of the dense correspondence mapping through the regression network.
[0099] Most current registration methods only consider registration in one direction. For infrared and visible light image registration, the infrared image is generally registered with the visible light image, that is, one-way learning (I->V): the visible light image is used as the reference image, and the infrared image is used as the floating image (geometric transformation is performed to move it closer to the visible light image).
[0100] The present invention adds an additional branch to estimate the symmetric transformation corresponding to the opposite direction. This bidirectional learning method can help reduce the differences between the estimated geometric transformation parameters between infrared images and visible light images and between visible light images and infrared images, namely bidirectional learning (I->V and V->I): on the one hand, the visible light image is used as the reference image and the infrared image is used as the floating image; on the other hand, the infrared image is used as the reference image and the visible light image is used as the floating image.
[0101] By processing the registration in these two directions simultaneously, more consistent and accurate results can be achieved. CEREG calculates the similarity between them by matrix multiplication, and the similarity increases with the increase of cosine value.
[0102] Specifically, in the present invention, the matching and regression network on the first branch uses the fusion features of the weak visible light image and the fusion features of the infrared image as the reference image and the floating image in both directions to fuse the parameters of the two-way registration; the matching and regression network on the second branch uses the fusion features of the infrared image and the fusion features of the enhanced image corresponding to the weak visible light image as the reference image and the floating image in both directions to fuse the parameters of the two-way registration.
[0103] That is, the matching and regression network on the first branch uses the fused features of the weak visible light image as the reference image and the fused features of the infrared image as the floating image to register the two. The matching and regression network on the second branch uses the fused features of the infrared image as the reference object and the fused features of the enhanced image corresponding to the weak visible light image as the floating object to register the two.
[0104] Alternatively: the matching and regression network on the first branch uses the fused features of the infrared image as the reference image and the fused features of the weak visible light image as the floating image to align the two; the matching and regression network on the second branch uses the fused features of the enhanced image corresponding to the weak visible light image as the reference object and the fused features of the infrared image as the floating object to align the two.
[0105] 1) Matching
[0106] The matching function C is used as a method to calculate the dense correspondence mapping between the two feature maps, where the dense correspondence mapping from the infrared image features to the visible light image features is calculated as follows:
[0107] C I→V (i,j,k)=C(f I (i k ,j k ),f V (i,j))=f V (i,j) T f I (i k ,j k ) (7)
[0108] Among them, C I→V Represents the infrared image feature map f I and visible light image feature map f V The dense correspondence mapping between I and f V Multiply them together to get (i,j) and (i k ,j k ) represent the matching point pairs P and P' in the visible light image feature map f V and infrared image feature map f I The coordinates in; k represents the feature point (i k ,j k )'s auxiliary index;
[0109] The dense correspondence mapping from visible light image features to infrared image features is calculated as follows:
[0110] C V→I (i,j,k)=C(f V (i k ,j k ),f I (i,j))=f I (i,j) T f V (i k ,j k ) (8)
[0111] Among them, C V→I Represents the visible light image feature map f V and infrared image feature map f I The dense correspondence mapping between V and f I Multiply them together to get .
[0112] In this process, the similarity between points is represented by the feature point position (i, j). After obtaining the similarity feature map, it is normalized to generate the corresponding mapping. Here, CEREG uses ReLU+L2Norm as the normalization method.
[0113] 2) Regression of transformation parameters
[0114] The regression step is used to predict the transformation parameters. When the dense correspondence map passes through the regression network R, the network R directly estimates the geometric transformation parameters as follows:
[0115] R:R h′×w′×(h′×w′) →R DoF (9)
[0116] Among them, (h', w') represents the height and width of the dense correspondence map, DoF represents the degree of freedom of the affine transformation model, and R represents the regression network.
[0117] (5) Construction of loss function, including:
[0118] During the training process, the present invention uses the transformed grid loss as the basic loss function. A point grid is constructed in the image space, and the neural network prediction value τ is used. θ and tags Use parameters θ and θ respectively gt Transform it, measure the difference between the two transformed grids by summing the squared distances between the corresponding grid points, and construct a basic loss function based on the difference. The calculation formula of the basic loss function is:
[0119]
[0120] Among them, θ represents the predicted affine transformation, θ gt represents the actual affine transformation, l(θ,θ gt ) represents θ and θ gt The basis loss between them; N represents the number of grid points, τ θ and Represented by θ and θ gt Parameterized transformation operation; (x i ,y j ) represents the pixel at position (i, j);
[0121] In order to achieve bidirectional learning, on the basis of the basic loss function, an additional branch of the calculation of the reverse transformation is added to construct a bidirectional loss function. The calculation formula is:
[0122]
[0123] Among them, θ I→Vrepresents the predicted affine transformation from infrared image I to visible light image V, θ gt I→V represents the affine transformation from the actual infrared image I to the visible light image V, θ V→I represents the predicted affine transformation from the visible light image V to the infrared image I, θ gt V→I Represents the affine transformation from the actual visible light image V to the infrared image I;
[0124] In order to regularize the training, on the basis of the bidirectional loss function, considering the enhanced image, two losses L are added aug and L id , respectively:
[0125] L aug =l(θ I→A ,θ gt I→V )+l(θ A→I ,θ gt I→V ) (12)
[0126] L id =l(θ I→V ,θ I→A )+l(θ V→I ,θ A→I ) (13)
[0127] Among them, θ I→A represents the affine transformation from the predicted infrared image I to the enhanced image A, θ A→I Represents the affine transformation from the actual enhanced image A to the infrared image I;
[0128] Since the geometric relationship between the two images is equivalent to the original image pair, the enhanced image pair also shares the label. The identity term in formula (13) induces training to ensure that the prediction values from the original pair and the enhanced pair are equal. The final constructed loss function is expressed as:
[0129] Loss = α·L org +β·L aug +γ·L id (14)
[0130] Among them, α, β, and γ are the balance parameters of each loss function. In the experiment, these parameters are set to 0.5, 0.3, and 0.2 respectively.
[0131] In general, in the preprocessing stage, the present invention performs color enhancement on weak visible light images through random color dithering to obtain weak visible light images of different colors, and forms a training pair with the corresponding infrared image to form a stream of the network, which is parallel to the training pair composed of the original weak visible light image and the infrared image to form a dual-stream structure. The two streams are trained simultaneously to improve the comprehensiveness of the feature expression of the initial weak visible light image. In the feature extraction stage, the present invention proposes a CE-Res feature extraction network, which is mainly composed of CBAM attention and ELU activation function to enhance the network's feature extraction capability for complex images. Considering the importance of edge features in images, the present invention proposes a CCR edge detection network. This module adds CBAM attention before CATS-RCF to enhance edge feature extraction capabilities and better emphasize the edge information of the image. Adding CCR in the feature extraction stage helps CEREG capture key feature information that is easily overlooked in the image. Experiments on the LLVIP dataset show that, when α = 0.1, the probability of correct keypoints (PCK) is improved by 5.62% compared to the current optimal network TEIVRNet, while visually demonstrating superior registration results. This demonstrates that CEREG has achieved significant results in improving image registration accuracy and effectively integrating failed infrared and weak visible light images.
[0132] In other embodiments, the present invention provides an infrared and weak visible light image registration system based on color edge enhancement, comprising:
[0133] An acquisition module, used for acquiring infrared images and weak visible light images to be registered;
[0134] The registration module is used to register the infrared image and the weak visible light image using the image registration model trained according to the above-mentioned infrared and weak visible light image registration method based on color edge enhancement.
[0135] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.
[0136] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for infrared and weak visible light image registration based on color edge enhancement, characterized in that: The following steps are involved: Building an image registration model, wherein the image registration model includes two branches, namely a first branch and a second branch; In the training stage, an infrared image I and a weak visible light image V are obtained, and the internal color of the weak visible light image V is enhanced to generate a corresponding enhanced image A; The first branch includes two feature extraction networks, two edge detection networks and a matching and regression network. The two feature extraction networks on the branch extract content features of the infrared image I and the weak visible light image V respectively, and the two edge detection networks extract edge features of the infrared image I and the weak visible light image V respectively. Then, the content features and edge features of the infrared image I and the weak visible light image V are fused to obtain their respective fused features. The second branch includes a feature extraction network and a matching and regression network. The feature extraction network on the branch extracts content features of the enhanced image A, and fuses the extracted content features with edge features of the weak visible light image V in the first branch to obtain fused features of the enhanced image A. Input the fusion features of the infrared image I and the fusion features of the weak visible light image V into the matching and regression network in the first branch to perform matching and geometric transformation parameter prediction; The fused features of the infrared image I and the fused image of the enhanced image A are input into the matching and regression network in the second branch for matching and geometric transformation parameter prediction; The two branches are trained in parallel and weights are shared to obtain the final image registration model after the preset goals are met.
2. The infrared and weak visible light image registration method based on color edge enhancement according to claim 1 is characterized in that: The internal enhancement process of weak visible light images is only performed in the training stage. In each round of training, the weak visible light image V is internally enhanced by random color jittering to generate different enhanced images. In the testing and inference stages, the first branch is used to align the infrared image and the weak visible light image.
3. The infrared and weak visible light image registration method based on color edge enhancement according to claim 1, characterized in that: The edge detection network on the first branch is based on the rich convolutional feature RCF model, integrates the context-aware tracking strategy CATS into RCF, and introduces the convolutional block attention module CBAM attention mechanism.
4. The infrared and weak visible light image registration method based on color edge enhancement according to claim 1, characterized in that: The feature extraction networks on the first branch and the second branch are composed of CBAM attention units, ELU activation functions, group convolutions and residual connections.
5. The infrared and weak visible light image registration method based on color edge enhancement according to claim 1, characterized in that: The matching and regression network on the first branch uses the fusion features of the weak visible light image and the fusion features of the infrared image as the reference image and the floating image in both directions to fuse the parameters of the two-way registration; the matching and regression network on the second branch uses the fusion features of the infrared image and the fusion features of the enhanced image corresponding to the weak visible light image as the reference image and the floating image in both directions to fuse the parameters of the two-way registration.
6. The infrared and weak visible light image registration method based on color edge enhancement according to claim 5, characterized in that: The weak visible light image in the first branch and the enhanced image in the second branch are collectively referred to as visible light images. The matching and regression networks on the first branch and the second branch first calculate the dense corresponding mapping between the infrared image features and the visible light image features through a matching function, and then estimate the geometric transformation parameters of the dense corresponding mapping through a regression network.
7. The infrared and weak visible light image registration method based on color edge enhancement according to claim 6, characterized in that: The dense correspondence mapping calculation method from infrared image features to visible light image features is: C I→V (i,j,k)=C(f I (i k ,j k ),f V (i,j))=f V (i,j) T f I (i k ,j k ) Among them, C I→V Represents the infrared image feature map f I And the visible light image feature map f V The dense correspondence mapping between I and f V Multiply them to get (i,j) and (i k ,j k ) represent the matching point pairs P and P' in the visible light image feature map f V And infrared image feature map f I The coordinates in k represent the feature point (i k ,j k )'s auxiliary index; The dense correspondence mapping calculation method from visible light image features to infrared image features is: C V→I (i,j,k)=C(f V (i k ,j k ),f I (i,j))=f I (i,j) T f V (i k ,j k ) Among them, C V→I Represents the visible light image feature map f V And infrared image feature map f I The dense correspondence mapping between V and f I Multiply them together to get .
8. The infrared and weak visible light image registration method based on color edge enhancement according to claim 6, characterized in that: The geometric transformation parameters of the regression network estimating the dense correspondence mapping are expressed as: R:R h′×w′×(h′×w′) →R DoF Among them, (h', w') represents the height and width of the dense correspondence map, DoF represents the degree of freedom of the affine transformation model, and R represents the regression network.
9. The infrared and weak visible light image registration method based on color edge enhancement according to claim 1, characterized in that: The process of constructing the loss function of the image registration model includes: A point grid is constructed in the image space, and the difference between the two grids is measured by summing the square distances between the corresponding grid points. The basic loss function is constructed based on the difference. The calculation formula of the basic loss function is: Among them, θ represents the predicted affine transformation, θ gt represents the actual affine transformation, l(θ,θ gt ) represents θ and θ gt The basic loss between ; N represents the number of grid points, (x i ,y j ) represents the pixel at position (i, j); τ θ and Respectively represented by θ and θ gt Parameterized transformation operations; On the basis of the basic loss function, an additional branch for calculating the reverse transformation is added to construct a bidirectional loss function. The calculation formula is: L org =l(θ I→V ,i gt I→V )+l(θ V→I ,i gt V→I ) Among them, θ I→V represents the predicted affine transformation from infrared image I to visible light image V, θ gt I→V represents the affine transformation from the actual infrared image I to the visible light image V, θ V→I represents the predicted affine transformation from the visible light image V to the infrared image I, θ gt V→I Represents the affine transformation from the actual visible light image V to the infrared image I; On the basis of the bidirectional loss function, considering the enhanced image, two losses L are added aug and L id , respectively: L aug =l(θ I→A ,i gt I→V )+l(θ A→I ,i gt I→V ) L id =l(θ I→V ,i I→A )+l(θ V→I ,i A→I ) Among them, θ I→A represents the affine transformation from the predicted infrared image I to the enhanced image A, θ A→I Represents the affine transformation from the actual enhanced image A to the infrared image I; The final constructed loss function is expressed as: Loss=α·L org +β·L aug +γ·L id Among them, α, β, and γ are the balance parameters of each loss function.
10. An infrared and weak visible light image registration system based on color edge enhancement, characterized in that: include: An acquisition module, used for acquiring infrared images and weak visible light images to be registered; A registration module is used to register the infrared image and the weak visible light image using the image registration model trained by the infrared and weak visible light image registration method based on color edge enhancement as described in any one of claims 1 to 9.
Citation Information
Patent Citations
Multi-target vehicle trajectory extraction method based on pixel-level image fusion
CN115457080A
Feature fusion and dense connection based infrared plane target detection method
WO2020102988A1
Cited By
Lake aquatic vegetation remote sensing classification method fusing image spatial spectrum characteristics
CN122156968A
