Infrared image and visible light image registration method based on adaptive graph reasoning

Abstract

The invention relates to the technical field of image processing, in particular to an infrared image and visible light image registration method based on adaptive graph reasoning, and the method comprises the steps: obtaining an infrared image and a visible light image, taking the visible light image as a reference standard, carrying out the multi-scale feature extraction and feature enhancement of the infrared image and the visible light image, and obtaining an infrared image and a visible light image; obtaining a multi-scale infrared enhancement feature and a multi-scale visible light enhancement feature; sequentially performing dimension reduction, splicing and fusion on the multi-scale infrared enhancement features and the multi-scale visible light enhancement features, and performing residual fusion based on an adaptive feature enhancement mechanism to obtain refined infrared fusion features and refined visible light fusion features; performing node embedding based on the refined infrared fusion features and the refined visible light fusion features, performing enhancement by using context node reasoning and an adaptive graph convolutional network, and finally performing calculation to obtain a deformation field; and re-sampling the infrared image to be registered by using the calculated deformation field to obtain a registered infrared image.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a registration method for infrared images and visible light images based on adaptive graph reasoning. Background Art

[0002] Image registration is a fundamental task in computer vision technology, aiming to precisely align images from different sources spatially to achieve information fusion and enhancement. As an important branch of this field, infrared and visible light image registration, due to their critical role in multimodal perception, has been widely used in scenarios such as detection and reconnaissance, video surveillance, medical image processing, and autonomous driving. Infrared images primarily capture the thermal radiation characteristics of objects, while visible light images retain rich texture and structural information. The two differ significantly in imaging mechanisms and image content, making high-precision and robust registration a core research issue.

[0003] Currently proposed registration methods for infrared and visible light images are mainly divided into feature-based registration methods and deep learning-based registration methods. Feature-based registration methods extract structural features such as edges and corners, and use descriptors and matching strategies to complete image alignment. However, when infrared images have fewer details or there are non-rigid changes, the matching accuracy is often less than ideal. In recent years, deep learning-based registration methods have gradually become a research hotspot. A typical method is to use convolutional neural networks to extract multi-scale semantic features and construct a registration network to learn the spatial transformation relationship between images. To improve registration accuracy, technicians usually introduce a multimodal feature fusion mechanism, combining the thermal information of infrared images and the texture structure of visible light images to achieve precise positioning and alignment of the target area.

[0004] Deep learning-based registration methods first utilize a pretrained two-stream neural network to extract features from both infrared and visible light images, extracting discriminative deep semantic features from both modalities. The infrared and visible light feature maps are then fed into a fusion module, which interacts and integrates the multimodal features through an attention mechanism or channel weighting to extract key feature information for alignment. During the prediction phase of the registration transformation, the network feeds the feature matching results into a transformation estimation subnetwork to predict the geometric transformation parameters between the images (such as an affine transformation or optical flow field). Finally, the infrared and visible light images are spatially remapped using the geometric transformation to produce the registered output. During model training, a contrastive loss is often used to measure the consistency of the registered images, thereby optimizing the model's network parameters to ensure that the registration results maintain visual structural consistency and spatial alignment accuracy.

[0005] However, the inventors of this application discovered that infrared images reflect an object's thermal radiation information, while visible light images reflect its light reflection information. These images differ significantly in geometric features such as edges and textures, leading to mismatches during subsequent feature matching. Due to differences in the imaging mechanisms of infrared and visible light sensors, infrared and visible light images often appear differently in areas with significant texture information, such as license plates and road signs. This modality difference not only limits the algorithm's adaptability in complex and changing scenarios but also introduces errors into the feature matching process, resulting in the inability of currently proposed deep learning-based registration methods to meet high-precision registration requirements. Summary of the Invention

[0006] In view of this, an embodiment of the present application proposes a registration method for infrared images and visible light images based on adaptive graph reasoning, which effectively alleviates the impact of modal differences and can achieve higher precision and stronger robust image registration in complex scenes. It has good application prospects and practical value.

[0007] In the first aspect, an embodiment of the present application proposes a registration method for infrared images and visible light images based on adaptive graph reasoning, which is implemented based on a registration model composed of a bidirectional multi-scale feature enhancement module, a cross-modal adaptive aggregation module, a graph adaptive reasoning flow estimation module and an output module. The method includes the following steps: obtaining an infrared image and a visible light image to be registered, taking the visible light image as a reference benchmark, using the bidirectional multi-scale feature enhancement module to perform multi-scale feature extraction and feature enhancement on the infrared image and the visible light image to obtain multi-scale infrared enhancement features and multi-scale visible light enhancement features; using the cross-modal adaptive aggregation module to perform multi-scale infrared enhancement and multi-scale visible light enhancement on the infrared image and the visible light image. The features are sequentially dimensionally reduced, concatenated, and fused, and then residual fusion is performed based on the adaptive feature enhancement mechanism to obtain refined infrared fusion features and refined visible light fusion features. The graph adaptive inference flow estimation module is used to perform node embedding based on the refined infrared fusion features and refined visible light fusion features to obtain infrared image nodes and visible light image nodes. The infrared image nodes and visible light image nodes are then enhanced using context node inference and an adaptive graph convolutional network. Finally, the deformation field is calculated based on the infrared enhanced image nodes and the visible light enhanced image nodes. The calculated deformation field is used to resample the infrared image to be registered through the output module to obtain the infrared image registered with the visible light image.

[0008] This application proposes an infrared and visible light image registration method based on adaptive graph reasoning. This method utilizes a registration model consisting of a bidirectional multi-scale feature enhancement module, a cross-modal adaptive aggregation module, a graph-adaptive reasoning flow estimation module, and an output module to achieve infrared and visible light image registration. By introducing these modules, the performance of infrared and visible light image registration is significantly improved. First, the bidirectional multi-scale feature enhancement module utilizes a bidirectional guidance mechanism to effectively fuse global structural information with local detail information at different scales, effectively alleviating the problem of local semantic loss caused by scale variation and improving feature representation capabilities. Second, the graph-adaptive reasoning flow estimation module fully preserves the spatial structural relationships of the images by embedding image features into a graph structure and performing reasoning in the graph space, effectively reducing the interference caused by modal differences on feature matching. Finally, the cross-modal adaptive aggregation module extracts shared consistent features between the infrared and visible light images through a weighted fusion mechanism. This enhances feature consistency while maintaining the original feature differences of the images, further alleviating the impact of modal differences. In summary, this method can achieve higher-precision and more robust image registration effects in complex scenes, and has good application prospects and practical value.

[0009] Optionally, let the infrared image be , the visible light image is , , , and Represents the height and width of infrared image and visible light image respectively; The bidirectional multi-scale feature enhancement module first extracts the feature vectors through two backbones that do not share weights. and Multi-scale features of and multi-scale visible light features , ; Among them, each backbone consists of four convolutional layers, instance regularization and LeakReLU activation function. and The dimensions of , and The dimensions of , and The dimensions of ; Using 1×1 convolution and normalization operations, the uniform 、 、 and The number of channels is expanded to and The same 128, we get 、 、 and ; Based on the idea of ​​bidirectional multi-scale guidance, learnable weights are used to 、 、 、 、 and Perform multi-scale adaptive guidance fusion to obtain multi-scale infrared enhancement features 、 、 and multi-scale visible infrared enhancement features 、 、 ; Will and Adjust the number of channels back to 32 and and The number of channels is adjusted back to 64 to obtain the final multi-scale infrared enhanced features and multi-scale visible infrared enhancement features , .

[0010] Optionally, 、 、 、 、 、 It is expressed by the formula: ; ; ; ; ; ; ; ; in, represents depthwise separable convolution, represents the upsampling operation, represents the downsampling operation, A preset very small positive number to avoid the denominator being 0. 、 、 、 、 、 、 、 and All are infrared adaptive weights, 、 、 、 、 、 、 、 and Both are visible light adaptive weights, and are initially 1.

[0011] Optionally, the cross-modal adaptive aggregation module first and Perform feature dimensionality reduction to halve the feature size and halve the number of channels to obtain infrared dimensionality reduction features and visible light dimensionality reduction features ; and It is expressed by the formula: ; ; in, Represents 3×3 convolution; Through the following formula, the channel dimension and Perform splicing and use 3×3 convolution to further extract joint features from the spliced ​​features; ; ; in, represents channel dimension splicing, Represents the features after splicing, Represents joint features; Will According to the number of channels, it is divided into two parts. As the representation of infrared images in the joint feature space, the second half As a representation of the visible light image in the joint feature space; The adaptive feature enhancement mechanism and Sigmoid activation function are used to learn the importance weight of each channel. Based on the importance weight, element-by-element multiplication is used to enhance the feature expression ability, and residual fusion is performed by element-by-element addition to finally obtain the refined infrared fusion feature. and refinement of visible light fusion features ; and It is expressed by the formula: ; ; ; ; ; ; in, and represents two different 3×3 convolutions, and represents the importance weight of the infrared channel, and represents the importance weight of the visible light channel, represents element-wise multiplication, represents element-by-element addition, Represents the Sigmoid activation function.

[0012] Optionally, the graph adaptive reasoning flow estimation module is composed of a node embedding unit, a graph adapter, a graph convolutional network, an adaptive graph convolutional network and a deformation field calculation unit; the node embedding unit is used to perform node embedding based on refined infrared fusion features and refined visible light fusion features to obtain infrared image nodes and visible light image nodes; the graph adapter is used to dynamically model the relationship between the infrared modality and the visible light modality based on the infrared image nodes and the visible light image nodes, and guide the infrared image nodes to generate an adjacency matrix through the visible light image nodes; the graph convolutional network is used to perform context node reasoning on the visible light image nodes to achieve enhancement, thereby obtaining and outputting visible light output features; the adaptive graph convolutional network is used to enhance the infrared image nodes based on the adjacency matrix, thereby obtaining and outputting infrared output features; the deformation field calculation unit is used to splice the infrared output features and the visible light output features, and then calculate the deformation field through a convolution operation.

[0013] Optionally, performing node embedding based on the refined infrared fusion features and the refined visible light fusion features to obtain infrared image nodes and visible light image nodes includes: Refined infrared fusion features and refinement of visible light fusion features Adjust the channel dimension and use convolution operation to and The number of channels is adjusted to ,get and ; Create the projection matrix , , Represents the first pixels and the first The weight relationship between nodes; Based on the projection matrix and Perform weighted summation to establish a global graph representation of the image and obtain infrared image nodes and Visible Light Image Node ; and It is expressed by the formula: ; ; in, represents L2 regularization.

[0014] Optionally, based on the infrared image nodes and the visible light image nodes, the relationship between the infrared modality and the visible light modality is dynamically modeled, and the infrared image nodes are guided by the visible light image nodes to generate an adjacency matrix, including: Based on visible light image node , through convolution and Softmax operations, the context weight is obtained , It is expressed by the formula: ; in, represents a multilayer perceptron, Represents the Softmax function; use For infrared image nodes Weighted to get a new representation , based on Constructing the adjacency matrix , , It consists of two layers of MLP. The first layer consists of a linear function and a ReLU activation function, and the second layer uses an adaptive kernel to achieve interaction. The graph convolutional network performs context node reasoning on visible light image nodes to achieve enhancement, thereby obtaining and outputting visible light output features, which is achieved through the following formula: ; ; in, Represents a graph convolutional network; The adaptive graph convolutional network enhances the infrared image nodes based on the adjacency matrix to obtain and output infrared output features, which is achieved through the following formula: ; in, Represents an adaptive graph convolutional network.

[0015] Optionally, the infrared output features and the visible light output features are concatenated and then the deformation field is calculated through a convolution operation, which is implemented by the following formula: ; in, is the infrared output characteristic, is the visible light output characteristic, is the calculated deformation field.

[0016] In a second aspect, an embodiment of the present application proposes an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute a registration method for infrared images and visible light images based on adaptive graph reasoning as described in the first aspect above.

[0017] In a third aspect, an embodiment of the present application proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement a registration method for infrared images and visible light images based on adaptive graph reasoning as described in the first aspect above.

[0018] It can be understood that the beneficial effects of the second to third aspects mentioned above can be found in the relevant description of the first aspect mentioned above. In order to keep the text concise and avoid repetition, they will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related technologies, the following is a brief introduction to the drawings required for use in the embodiments of the present application or the description of the related technologies. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 is a flowchart of a registration method for infrared images and visible light images based on adaptive graph reasoning provided in one embodiment of the present application; Figure 2 is a schematic structural diagram of a registration model provided in one embodiment of the present application; Figure 3 is a schematic structural diagram of a bidirectional multi-scale feature enhancement module provided in one embodiment of the present application; Figure 4 is a schematic diagram of a bidirectional guidance mechanism provided in one embodiment of the present application; Figure 5 is a schematic diagram of the structure of a graph adaptive reasoning flow estimation module provided in one embodiment of the present application; Figure 6 is a schematic diagram of the working principle of a graph adapter provided in one embodiment of the present application; Figure 7 is a comparative schematic diagram of registered infrared images provided in one embodiment of the present application; Figure 8 It is a structural diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below with reference to the accompanying drawings. In the various embodiments of the present application, many technical details are proposed to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can be implemented. The division of the following embodiments is only for the convenience of description and should not constitute any limitation on the specific implementation of the present application. The various embodiments can be combined with each other and referenced to each other under the premise of no contradiction.

[0022] An embodiment of the present application proposes a registration method for infrared images and visible light images based on adaptive graph reasoning. The registration model is implemented based on a bidirectional multi-scale feature enhancement module, a cross-modal adaptive aggregation module, a graph adaptive reasoning flow estimation module, and an output module, and is applied to an electronic device, wherein the electronic device can be a terminal or a server. In this embodiment and the following embodiments, the electronic device is described using a server as an example. The following is a detailed description of the implementation details of the registration method for infrared images and visible light images based on adaptive graph reasoning proposed in this embodiment. The following content is only the implementation details provided for the convenience of understanding and is not necessary for the implementation of this solution.

[0023] The specific process of the infrared image and visible light image registration method based on adaptive graph reasoning proposed in this embodiment can be as follows: Figure 1 As shown in the figure, the specific structure of the registration model is as follows Figure 2 As shown, the method includes: Step 11: Obtain the infrared image and visible light image to be registered, use the visible light image as a reference, and use the bidirectional multi-scale feature enhancement module to perform multi-scale feature extraction and feature enhancement on the infrared image and the visible light image to obtain multi-scale infrared enhancement features and multi-scale visible light enhancement features.

[0024] In practice, the first step in registering infrared and visible light images is to obtain the infrared and visible light images to be registered. The visible light image serves as a reference, referred to as the reference visible light image. The goal of registration is to align the infrared image to the visible light image, so the infrared image can be referred to as a floating infrared image. Both the infrared and visible light images are input into the registration model's bidirectional multi-scale feature enhancement module, which extracts and enhances multi-scale features from the infrared and visible light images, generating multi-scale infrared enhancement features and multi-scale visible light enhancement features.

[0025] In one example, when training the registration model, mainstream public MSRS and RoadScene datasets can be selected as training sample sets and test sample sets. These two datasets include combinations of paired visible light images and infrared images of multiple scenes.

[0026] In an example, suppose the infrared image is , the visible light image is , , , and Represents the height and width of infrared image and visible light image respectively.

[0027] In one example, the specific structure of the bidirectional multi-scale feature enhancement module can be as follows: Figure 3 As shown in the figure, the bidirectional multi-scale feature enhancement module first extracts the feature vectors through two backbones (feature extraction skeletons) whose weights are not shared. and Multi-scale features of and multi-scale visible light features , Among them, each backbone consists of four convolutional layers, instance regularization and LeakReLU activation function. and The dimensions of , and The dimensions of , and The dimensions of .

[0028] Next, use 1×1 convolution and normalization operations to unify 、 、 and The number of channels is expanded to and The same 128, we get 、 、 and .

[0029] 、 、 and It is expressed by the formula: ; ; ; ; in, represents a 1×1 convolution operation, Represents a normalization operation.

[0030] Then, based on the idea of ​​bidirectional multi-scale guidance, the global structure and local details of multi-scale features are fully explored. Since features of different scales contribute differently to the final flow estimation, learnable weights can be used to guide the fusion of features of different scales, that is, learnable weights can be used to 、 、 、 、 and Perform multi-scale adaptive guidance fusion to obtain multi-scale infrared enhancement features 、 、 and multi-scale visible infrared enhancement features 、 、 The principle of this process can be described as follows Figure 4 shown.

[0031] 、 、 、 、 、 It can be expressed by the formula: ; ; ; ; ; ; ; ; in, represents depthwise separable convolution, represents the upsampling operation, represents the downsampling operation, A preset very small positive number, usually set to , used to avoid the situation where the denominator is 0, 、 、 、 、 、 、 、 and All are infrared adaptive weights, 、 、 、 、 、 、 、 and Both are visible light adaptive weights, and are initially 1.

[0032] Finally, and The number of channels is adjusted back to 32, and The number of channels is adjusted back to 64 to obtain the final multi-scale infrared enhanced features and multi-scale visible infrared enhancement features , .

[0033] 、 、 and It is expressed by the formula: ; ; ; ; in, Indicates a convolution operation with an output channel of 32. Indicates a convolution operation with an output channel of 64. Represents a normalization operation.

[0034] It is understandable that and The dimensions of , and The dimensions of , and The dimensions of .

[0035] In step 12, a cross-modal adaptive aggregation module is used to sequentially reduce the dimensionality, concatenate, and fuse the multi-scale infrared enhancement features and the multi-scale visible light enhancement features, and then residual fusion is performed based on the adaptive feature enhancement mechanism to obtain refined infrared fusion features and refined visible light fusion features.

[0036] In the specific implementation, the multi-scale infrared enhancement features and multi-scale visible light enhancement features output by the bidirectional multi-scale feature enhancement module will enter the cross-modal adaptive aggregation module. The cross-modal adaptive aggregation module needs to reduce the dimension, splice and fuse the multi-scale infrared enhancement features and the multi-scale visible light enhancement features in turn, and then perform residual fusion based on the adaptive feature enhancement mechanism to obtain refined infrared fusion features and refined visible light fusion features.

[0037] In one example, the cross-modal adaptive aggregation module first and Perform feature dimensionality reduction to halve the feature size and halve the number of channels to obtain infrared dimensionality reduction features and visible light dimensionality reduction features ; and It is expressed by the formula: ; ; in, Represents a 3×3 convolution with a stride of 2.

[0038] In order to make full use of the complementary information of infrared images and visible light images, the cross-modal adaptive aggregation module will perform and The concatenation is performed and 3×3 convolution is used to further extract joint features from the concatenated features. This process can be achieved by the following formula: ; ; in, represents channel dimension splicing, Represents the features after splicing, Represents joint features.

[0039] In order to further extract complementary information, the cross-modal adaptive aggregation module needs to According to the number of channels, it is divided into two parts. As the representation of infrared images in the joint feature space, the second half As the representation of visible light images in the joint feature space.

[0040] Since the feature representation of infrared images and visible light images is different, only splicing and convolution cannot fully utilize their respective important information. Therefore, the cross-modal adaptive aggregation module also needs to use the adaptive feature enhancement mechanism and Sigmoid activation function to learn the importance weight of each channel to enhance the robustness of the registration. Then, based on the obtained importance weight, element-by-element multiplication is used to enhance the feature expression ability, and residual fusion is performed by element-by-element addition to finally obtain the refined infrared fusion feature. and refinement of visible light fusion features .

[0041] and It is expressed by the formula: ; ; ; ; ; ; in, and represents two different 3×3 convolutions, and represents the importance weight of the infrared channel, and represents the importance weight of the visible light channel, represents element-wise multiplication, represents element-by-element addition, Represents the Sigmoid activation function.

[0042] In step 13, the graph adaptive reasoning flow estimation module is used to perform node embedding based on the refined infrared fusion features and the refined visible light fusion features to obtain infrared image nodes and visible light image nodes. The infrared image nodes and visible light image nodes are then enhanced using context node reasoning and the adaptive graph convolutional network. Finally, the deformation field is calculated based on the infrared enhanced image nodes and the visible light enhanced image nodes.

[0043] In the specific implementation, the refined infrared fusion features and refined visible light fusion features output by the cross-modal adaptive aggregation module will enter the graph adaptive reasoning flow estimation module. The graph adaptive reasoning flow estimation module needs to perform node embedding based on the refined infrared fusion features and refined visible light fusion features to obtain infrared image nodes and visible light image nodes, and then use context node reasoning and adaptive graph convolutional network to enhance the infrared image nodes and visible light image nodes. Finally, the deformation field is calculated based on the infrared enhanced image nodes and the visible light enhanced image nodes.

[0044] It is understandable that the presence of the cross-modal adaptive aggregation module effectively alleviates the modal differences between infrared and visible light images. To further reduce the modal differences, the feature space is converted to a graph space. Node embedding is a core component of graph structure learning. This technology maps input features to a graph space, enabling nodes to express global relationships and local structures well. In the infrared and visible light image registration task, projecting the infrared and visible light image features from the conventional feature space into the graph space and generating infrared image nodes and visible light image nodes, respectively, can effectively reduce the impact of modal differences.

[0045] In an example, the specific structure of the graph adaptive reasoning flow estimation module can be as follows: Figure 5 As shown, the graph adaptive reasoning flow estimation module is specifically composed of a node embedding unit, a graph adapter, a graph convolutional network, an adaptive graph convolutional network, and a deformation field calculation unit. The node embedding unit is used to perform node embedding based on the refined infrared fusion features and the refined visible light fusion features to obtain infrared image nodes and visible light image nodes. The graph adapter is used to dynamically model the relationship between the infrared modality and the visible light modality based on the infrared image nodes and the visible light image nodes, and guide the infrared image nodes to generate an adjacency matrix through the visible light image nodes. The graph convolutional network is used to perform contextual node reasoning on the visible light image nodes to achieve enhancement, thereby obtaining and outputting visible light output features. The adaptive graph convolutional network is used to enhance the infrared image nodes based on the adjacency matrix, thereby obtaining and outputting infrared output features. The deformation field calculation unit is used to splice the infrared output features and the visible light output features, and then calculate the deformation field through a convolution operation.

[0046] In one example, when the node embedding unit performs node embedding again, it refines the infrared fusion feature and refinement of visible light fusion features Adjust the channel dimension and use convolution operation to and The number of channels from Adjust to ,get and .in, represents a convolution operation with output channel number The intrinsic dimension of the node feature in the projection space is independent of the spatial resolution of the input feature, and aims to represent a high-dimensional representation of each node.

[0047] Next, the projection matrix is established. represents the weight relationship between the th pixel in the grid space and the th node in the graph space. The construction of the weight relationship is the core step of feature projection, which determines the allocation proportion of the input feature in the node representation. Specifically, the construction of the projection matrix combines the similarity measure of the feature and the node to achieve efficient mapping of the input feature to the node representation.

[0048] After the projection matrix is established, the weighted sum of and is performed based on the projection matrix to establish the global graph representation of the image, and the infrared image node and the visible light image node are obtained.

[0049] and are represented by the formula as follows: wherein, represents L2 regularization.

[0050] In one example, the infrared image node and the visible light image node can represent the features of the infrared image and the features of the visible light image. However, since the infrared image node and the visible light image node only encode the corresponding relationship between the nodes in the image, the representation difference between the infrared image node and the visible light image node due to the modal difference is ignored, which will hinder the effective information propagation in the graph reasoning process. Therefore, a graph adapter is designed to dynamically model the relationship between the features of the infrared image and the features of the visible light image. The working principle of the graph adapter can be shown as follows: Figure 6 The graph adapter is based on the visible light image node , and obtains the context weight by convolution and Softmax operation. The formula is represented as follows: wherein,​​​​​​​ denotes a multi-layer perceptron, denotes a Softmax function.

[0051] Next, the weighted infrared image nodes are used to obtain new representations , and an adjacency matrix is constructed based on , , , consists of two layers of MLP, the first layer consists of linear functions and ReLU activation functions, and the second layer uses adaptive kernels to realize interaction, thereby learning and adapting the relationship between infrared features and visible light features.

[0052] In one example, the graph convolution network performs context node reasoning on the visible light image nodes to realize enhancement, thereby obtaining and outputting visible light output features, which is realized by the following formula: ; ; wherein, denotes the graph convolution network.

[0053] In one example, the adaptive graph convolution network performs enhancement on the infrared image nodes based on the adjacency matrix, thereby obtaining and outputting infrared output features, which is realized by the following formula: ; wherein, denotes the adaptive graph convolution network.

[0054] In one example, the deformation field calculation unit concatenates the infrared output features and the visible light output features, and then calculates the deformation field through convolution operation, which can be realized by the following formula: ; wherein, is the infrared output feature, is the visible light output feature, is the calculated deformation field.

[0055] Step 14, through the output module, using the calculated deformation field to resample the infrared image to be registered, obtaining the infrared image registered with the visible light image.

[0056] In specific implementation, after the output module receives the deformation field output by the graph adaptive reasoning flow estimation module, the deformation field can be used to resample the infrared image to be registered, obtaining the infrared image registered with the visible light image.

[0057] In an example, if the infrared image is used as a reference and the visible light image is used as a float, then the deformation field output by the graph adaptive inference flow estimation module is the deformation field from the visible light image to the infrared image. The output module needs to use the deformation field to resample the visible light image to be registered to obtain the visible light image after registration with the infrared image.

[0058] In one example, Figure 7 The comparison of registration results of different registration methods is shown. The registration method for infrared images and visible light images based on adaptive graph reasoning proposed in this embodiment has the best registration effect.

[0059] This embodiment proposes an infrared image and visible light image registration method based on adaptive graph reasoning. This method constructs and utilizes a registration model consisting of a bidirectional multi-scale feature enhancement module, a cross-modal adaptive aggregation module, a graph-adaptive reasoning flow estimation module, and an output module to achieve infrared and visible light image registration. By introducing these modules, the performance of infrared and visible light image registration in the cross-modal registration task is significantly improved. First, the bidirectional multi-scale feature enhancement module utilizes a bidirectional guidance mechanism to effectively fuse global structural information with local detail information at different scales, effectively alleviating the problem of local semantic loss caused by scale variations and improving feature representation capabilities. Second, the graph-adaptive reasoning flow estimation module fully preserves the spatial structural relationships of the images by embedding image features into a graph structure and performing reasoning in the graph space, effectively reducing the interference caused by modal differences on feature matching. Finally, the cross-modal adaptive aggregation module extracts shared consistent features between the infrared and visible light images through a weighted fusion mechanism. This enhances feature consistency while maintaining the original feature differences of the images, further alleviating the impact of modal differences. In summary, this method can achieve higher-precision and more robust image registration effects in complex scenes, and has good application prospects and practical value.

[0060] The steps of the various methods above are divided for clarity of description only. During implementation, they can be combined into a single step, or some steps can be broken down into multiple steps. As long as they contain the same logical relationships, they are all within the scope of protection of this application. Adding minor modifications or introducing minor design changes to the algorithm or process, but not changing the core design of the algorithm or process, are also within the scope of protection of this application.

[0061] Another embodiment of the present application provides an electronic device, the specific structure of which is as follows: Figure 8As shown, it includes: at least one processor 21; and a memory 22 communicatively connected to the at least one processor 21; wherein the memory 22 stores instructions that can be executed by the at least one processor 21, and the instructions are executed by the at least one processor 21 to enable the at least one processor 21 to execute a registration method for infrared images and visible light images based on adaptive graph reasoning as described in the above method embodiment.

[0062] The memory and processor are connected using a bus, which includes any number of interconnected buses and bridges. The bus connects various circuits of one or more processors and the memory. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. These are all well known in the art and therefore will not be described further in this article. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over a wireless medium via an antenna. Furthermore, the antenna receives data and transmits it to the processor.

[0063] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. The memory can be used to store relevant data used by the processor when performing operations.

[0064] Another embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement a registration method for infrared images and visible light images based on adaptive graph reasoning as described in the above method embodiment.

[0065] That is, those skilled in the art will understand that all or part of the steps in the above-described method embodiments can be accomplished through hardware associated with program instructions. The program is stored in a storage medium and includes a number of instructions for causing a device (such as a single-chip microcomputer, chip, etc.) or processor to execute all or part of the steps of the method for registering infrared and visible light images based on adaptive graph reasoning described in the method embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory, a random access memory, a magnetic disk, or an optical disk.

[0066] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application. In actual applications, various modifications may be made to the embodiments in form and detail without departing from the spirit and scope of the present application. Those skilled in the art will appreciate that improvements and modifications may be made without departing from the principles of the present application, and such improvements and modifications are also considered to be within the scope of protection of the present application.

Claims

1. A registration method for infrared and visible light images based on adaptive graph reasoning, implemented based on a registration model consisting of a bidirectional multi-scale feature enhancement module, a cross-modal adaptive aggregation module, a graph adaptive reasoning flow estimation module, and an output module, characterized by: The method comprises: Obtain the infrared image and visible light image to be registered, use the visible light image as a reference, and use the bidirectional multi-scale feature enhancement module to perform multi-scale feature extraction and feature enhancement on the infrared image and visible light image to obtain multi-scale infrared enhancement features and multi-scale visible light enhancement features; Using a cross-modal adaptive aggregation module, the multi-scale infrared enhancement features and multi-scale visible light enhancement features are sequentially reduced in dimension, concatenated, and fused. Then, residual fusion is performed based on the adaptive feature enhancement mechanism to obtain refined infrared fusion features and refined visible light fusion features. Using the graph adaptive reasoning flow estimation module, node embedding is performed based on the refined infrared fusion features and the refined visible light fusion features to obtain infrared image nodes and visible light image nodes. Contextual node reasoning and an adaptive graph convolutional network are then used to enhance the infrared image nodes and visible light image nodes. Finally, the deformation field is calculated based on the infrared enhanced image nodes and the visible light enhanced image nodes. Through the output module, the calculated deformation field is used to resample the infrared image to be registered, and the infrared image registered with the visible light image is obtained.

2. The infrared image and visible light image registration method based on adaptive graph reasoning according to claim 1, characterized in that: Assume that the infrared image is , the visible light image is , , , and Represents the height and width of infrared image and visible light image respectively; The bidirectional multi-scale feature enhancement module first extracts the feature vectors through two weight-unshared backbones. and Multi-scale features of and multi-scale visible light features , ; Among them, each backbone consists of four convolutional layers, instance regularization and LeakReLU activation function. and The dimensions of , and The dimensions of , and The dimensions of ; Using 1×1 convolution and normalization operations, the uniform 、 、 and The number of channels is expanded to and The same 128, we get 、 、 and ; Based on the idea of ​​bidirectional multi-scale guidance, learnable weights are used to 、 、 、 、 and Perform multi-scale adaptive guidance fusion to obtain multi-scale infrared enhancement features 、 、 and multi-scale visible infrared enhancement features 、 、 ; Will and Adjust the number of channels back to 32 and and The number of channels is adjusted back to 64 to obtain the final multi-scale infrared enhanced features and multi-scale visible infrared enhancement features , .

3. The infrared image and visible light image registration method based on adaptive graph reasoning according to claim 2, characterized in that: 、 、 、 、 、 It is expressed by the formula: ； ； ； ； ； ； ； ； in, represents depthwise separable convolution, represents the upsampling operation, represents the downsampling operation, A preset very small positive number to avoid the denominator being 0. 、 、 、 、 、 、 、 and All are infrared adaptive weights, 、 、 、 、 、 、 、 and Both are visible light adaptive weights, and are initially 1.

4. The method for registering infrared images and visible light images based on adaptive graph reasoning according to claim 2, characterized in that: The cross-modal adaptive aggregation module first and Perform feature dimensionality reduction to halve the feature size and halve the number of channels to obtain infrared dimensionality reduction features and visible light dimensionality reduction features ; and It is expressed by the formula: ； ； in, Represents 3×3 convolution; By the following formula, we can get and Perform splicing and use 3×3 convolution to further extract joint features from the spliced ​​features; ； ； in, represents channel dimension splicing, Represents the features after splicing, Represents joint features; Will According to the number of channels, it is divided into two parts. As the representation of infrared images in the joint feature space, the second half As a representation of the visible light image in the joint feature space; The adaptive feature enhancement mechanism and Sigmoid activation function are used to learn the importance weight of each channel. Based on the importance weight, element-by-element multiplication is used to enhance the feature expression ability, and residual fusion is performed by element-by-element addition to finally obtain the refined infrared fusion feature. and refinement of visible light fusion features ; and It is expressed by the formula: ； ； ； ； ； ； in, and represents two different 3×3 convolutions, and represents the importance weight of the infrared channel, and represents the importance weight of the visible light channel, represents element-wise multiplication, represents element-by-element addition, Represents the Sigmoid activation function.

5. The method for registering infrared images and visible light images based on adaptive graph reasoning according to claim 4, characterized in that: The graph adaptive reasoning flow estimation module consists of a node embedding unit, a graph adapter, a graph convolutional network, an adaptive graph convolutional network, and a deformation field calculation unit; The node embedding unit is used to perform node embedding based on the refined infrared fusion features and the refined visible light fusion features to obtain infrared image nodes and visible light image nodes; The graph adapter is used to dynamically model the relationship between the infrared modality and the visible light modality based on the infrared image node and the visible light image node, and guide the infrared image node to generate an adjacency matrix through the visible light image node; The graph convolutional network is used to perform contextual node reasoning on visible light image nodes to achieve enhancement, thereby obtaining and outputting visible light output features; The adaptive graph convolutional network is used to enhance the infrared image nodes based on the adjacency matrix to obtain and output infrared output features; The deformation field calculation unit is used to splice the infrared output features and the visible light output features, and then calculate the deformation field through a convolution operation.

6. The method for registering infrared images and visible light images based on adaptive graph reasoning according to claim 5, characterized in that: Node embedding is performed based on the refined infrared fusion features and the refined visible light fusion features to obtain infrared image nodes and visible light image nodes, including: Refined infrared fusion features and refinement of visible light fusion features Adjust the channel dimension and use convolution operation to and The number of channels is adjusted to ,get and ; Create the projection matrix , , represents the first pixels and the first The weight relationship between nodes; Based on the projection matrix and Perform weighted summation to establish a global graph representation of the image and obtain infrared image nodes and Visible Light Image Node ; and It is expressed by the formula: ； ； in, represents L2 regularization.

7. The method for registering infrared images and visible light images based on adaptive graph reasoning according to claim 6, characterized in that: Based on infrared image nodes and visible light image nodes, the relationship between the infrared modality and the visible light modality is dynamically modeled. The visible light image node guides the infrared image node to generate an adjacency matrix, including: Based on visible light image node , through convolution and Softmax operations, the context weight is obtained , It is expressed by the formula: ； in, represents a multilayer perceptron, Represents the Softmax function; use For infrared image nodes Weighted to get a new representation , based on Constructing the adjacency matrix , , It consists of two layers of MLP. The first layer consists of a linear function and a ReLU activation function, and the second layer uses an adaptive kernel to achieve interaction. The graph convolutional network performs context node reasoning on visible light image nodes to achieve enhancement, thereby obtaining and outputting visible light output features, which is achieved through the following formula: ； ； in, Represents a graph convolutional network; The adaptive graph convolutional network enhances the infrared image nodes based on the adjacency matrix to obtain and output infrared output features, which is achieved through the following formula: ； in, Represents an adaptive graph convolutional network.

8. The method for registering infrared images and visible light images based on adaptive graph reasoning according to claim 7, characterized in that: The infrared output features and visible light output features are spliced ​​together, and then the deformation field is calculated through convolution operation, which is achieved by the following formula: ； in, is the infrared output characteristic, is the visible light output characteristic, is the calculated deformation field.

9. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; In which, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a registration method for infrared images and visible light images based on adaptive graph reasoning as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it can implement the infrared image and visible light image registration method based on adaptive graph reasoning as described in any one of claims 1 to 8.

Images