Infrared-visible light image registration method and system

Abstract

The present disclosure provides an infrared-visible light image registration method and system, comprising: acquiring an infrared image and a visible light image; extracting feature points from the acquired infrared image and visible light image using a self-supervised learning network, and describing the feature points; matching feature points in the infrared image and the visible light image using a deep graph convolutional network; obtaining transformation parameters from the infrared image to the visible light image using a progressive consistent sampling algorithm based on the pixel coordinates of the matched feature point pairs in the two images; transforming the coordinates of the infrared image into the visible light image coordinate system based on the transformation parameters, and registering the infrared-visible light images; through the method of the present disclosure, more accurate registration of infrared images and visible light images can be achieved compared with traditional methods; and accurate registration results can be obtained for different images taken at different angles.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular, relates to an infrared-visible light image registration method and system. Background Art

[0002] Since the infrared and visible light lenses are located in different positions and have different focal lengths, image resolutions, and distortion parameters, there are inevitably deformation problems such as offset and scaling between the two modal images collected by the same infrared thermal imager. To fuse and analyze infrared images with visible light images, it is first necessary to accurately align the images of the two modalities; the alignment process generally includes corner point extraction, feature calculation, and feature matching.

[0003] The inventors of the present disclosure have found that although traditional corner point extraction and feature calculation methods such as scale-invariant feature transform (SIFT) have good scale and rotation invariance and are also robust to changes in viewing angles in single-modal images, they are often not robust enough to changes in different modalities such as infrared and visible light; the traditional kd-tree nearest neighbor query algorithm (Best Bin First, BBF) feature matching method processes each feature point in isolation during matching, while ignoring the contextual information of neighboring feature points, which is also prone to matching errors. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes an infrared-visible light image registration method and system; the present invention adopts the SuperPoint feature point extraction and descriptor extraction method based on deep self-supervised learning, and adopts the SuperGlue matching method based on deep graph convolution, and finally adopts the progressive consistent sampling (PROSAC) method of asymptotic sampling consistency to estimate the transformation parameters between infrared and visible light images, thereby realizing the registration and fusion of infrared images and visible light images; through the method in the present invention, more accurate registration of infrared images and visible light images can be achieved compared with traditional methods; and accurate registration results can be obtained for different images taken at different angles.

[0005] In a first aspect, the present disclosure provides an infrared-visible light image registration method, comprising:

[0006] Acquire infrared images and visible light images;

[0007] Using the self-supervised learning network, feature points of the acquired infrared image and visible light image are extracted and described;

[0008] Use deep graph convolutional networks to match feature points in infrared images and visible light images;

[0009] According to the pixel coordinates of the matched feature point pairs in the two images, the transformation parameters from the infrared image to the visible light image are obtained by using the progressive consistent sampling algorithm;

[0010] According to the transformation parameters, the coordinates of the infrared image are transformed into the visible light image coordinate system, and the infrared-visible light images are registered.

[0011] Furthermore, the learning process of the self-supervised learning network is:

[0012] Construct a set of synthetic images containing triangles, quadrilaterals, cubes, and checkerboard shapes to obtain corner point information;

[0013] Perform random homography transformation on the image and add noise to obtain the position of feature points in the transformed image;

[0014] The transformed image and its corner point information are used to supervise the self-supervised learning network, and a self-supervised learning network with the ability to extract feature points of infrared images and visible light images is obtained.

[0015] Furthermore, the deep graph convolutional network includes an attention graph convolutional network and an optimized matching layer;

[0016] The input of the attention graph convolutional network is a set of feature point positions and description vectors of a pair of images, and outputs a feature descriptor after spatial information aggregation; the input of the optimization matching layer is the feature descriptor output by the attention graph convolutional network, and outputs a matching result.

[0017] Furthermore, in the attention graph convolutional network, first, the position of the feature point is dimensionally upgraded, and the feature point position and description vector after dimension upgrading are added to the feature descriptor vector to obtain the initial representation of each feature point; then, a multivariate graph is constructed; the vertices of the multivariate graph are all the feature points in the two images, and the edges include intra-image edges and cross-image edges; the intra-image edges connect the feature point pairs in a single image, and the cross-image edges connect the feature point pairs from the two images;

[0018] After constructing the multivariate graph, the features of all vertices in the constructed multivariate graph are aggregated and updated.

[0019] Furthermore, the optimization matching layer calculates a similarity matrix based on the updated features, and each unit in the matrix represents the similarity of the features in the two images;

[0020] When a feature point in one image does not have a matching feature point in another image, the matrix is ​​expanded, and a new row and column are added to the expanded matrix to describe the situation where the feature point does not have a matching feature point.

[0021] Furthermore, after obtaining the matching feature point pairs in the infrared image and the visible light image, the transformation parameters between the images are calculated based on the pixel coordinates of the feature point pairs; wherein, quality evaluation is performed on all matching point pairs to obtain Q values, and the points are arranged in descending order according to the Q values. Random sampling is performed when the results are arranged in descending order according to the Q values ​​in each iteration to perform model hypothesis and verification.

[0022] Furthermore, according to the transformation parameters, the coordinates of the infrared image are transformed into the coordinate system of the visible light image, and the average value of the pixels at the corresponding positions is taken to obtain the final fused image.

[0023] In a second aspect, the present disclosure further provides an infrared-visible light image registration system, including an image acquisition module, a feature point extraction module, a feature point matching module, a transformation parameter calculation module and an image registration module;

[0024] The image acquisition module is configured to: acquire infrared images and visible light images;

[0025] The feature point extraction module is configured to: extract feature points from the acquired infrared image and visible light image using a self-supervised learning network, and describe the feature points;

[0026] The feature point matching module is configured to: match feature points in the infrared image and the visible light image using a deep graph convolutional network;

[0027] The transformation parameter calculation module is configured to: obtain the transformation parameters from the infrared image to the visible light image using a progressive consistent sampling algorithm according to the pixel coordinates of the matched feature point pairs in the two images;

[0028] The image registration module is configured to: transform the coordinates of the infrared image into a visible light image coordinate system according to the transformation parameters, and perform infrared-visible light image registration.

[0029] In a third aspect, the present disclosure further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the infrared-visible light image registration method described in the first aspect are implemented.

[0030] In a fourth aspect, the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the infrared-visible light image registration described in the first aspect.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. In the present invention, based on the use of a self-supervised learning network to extract feature points, a deep graph convolutional network to match feature points, and a progressively consistent sampling algorithm to obtain transformation parameters, the infrared-visible light image registration is achieved, and the registration result is more accurate, achieving a more effective matching of infrared images and visible light images.

[0033] 2. Compared with traditional image registration methods, the present invention can, on the one hand, extract more reliable feature points and descriptors through SuperPoint based on deep self-supervision, and on the other hand, effectively utilize the feature point position information and context information based on the SuperGlue feature matching method based on the deep graph convolutional network to achieve more effective matching; on this basis, the asymptotic sampling consistency method adopted can obtain more accurate transformation parameter estimation results, and accurate registration results can be obtained for images taken at different angles. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings in the specification that constitute a part of this embodiment are used to provide a further understanding of this embodiment. The schematic embodiments of this embodiment and their descriptions are used to explain this embodiment and do not constitute improper limitations on this embodiment.

[0035] Figure 1 This is a flow chart of Embodiment 1 of the present disclosure;

[0036] Figure 2 A schematic diagram of a SuperPoint feature extraction and description network according to Embodiment 2 of the present disclosure;

[0037] Figure 3 This is a rendering of the infrared image of Example 2 of the present disclosure after SuperPoint feature point extraction;

[0038] Figure 4 This is a rendering of the visible light image of Example 2 of the present disclosure after SuperPoint feature point extraction;

[0039] Figure 5 A schematic diagram of a SuperGlue feature matching network according to Embodiment 2 of the present disclosure;

[0040] Figure 6 This is a rendering of the infrared image and visible light image of Example 2 of the present disclosure after SuperGlue feature point matching;

[0041] Figure 7 This is the final fused image effect diagram of Example 2 of the present disclosure. DETAILED DESCRIPTION

[0042] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0043] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0044] Currently, most infrared thermal imagers can collect infrared images and visible light images at the same time; infrared images record the radiation information of objects, which can directly reflect the temperature of substation equipment and the environment, but it is difficult to distinguish equipment components with similar temperatures; visible light images record the reflection information of objects, which can clearly present the appearance and outline of the equipment; therefore, fusing and analyzing infrared images with visible light images can effectively utilize the advantages of the two modes, thereby improving the accuracy of equipment positioning and defect diagnosis. The following is an introduction to the method for fusing infrared images and visible light images involved in the present disclosure through embodiments.

[0045] Embodiment 1:

[0046] like Figure 1 As shown, this embodiment provides an infrared-visible light image registration method, including:

[0047] Acquire infrared images and visible light images;

[0048] Using the self-supervised learning network, feature points of the acquired infrared image and visible light image are extracted and described;

[0049] Use deep graph convolutional networks to match feature points in infrared images and visible light images;

[0050] According to the pixel coordinates of the matched feature point pairs in the two images, the transformation parameters from the infrared image to the visible light image are obtained by using the progressive consistent sampling algorithm;

[0051] According to the transformation parameters, the coordinates of the infrared image are transformed into the visible light image coordinate system, and the infrared-visible light images are registered.

[0052] In this embodiment, the learning process of the self-supervised learning network is:

[0053] Construct a set of synthetic images containing triangles, quadrilaterals, cubes, and checkerboard shapes to obtain corner information;

[0054] Perform random homography transformation on the image and add noise to obtain the position of feature points in the transformed image;

[0055] The transformed image and its corner point information are used to supervise the self-supervised learning network, and a self-supervised learning network with the ability to extract feature points of infrared images and visible light images is obtained.

[0056] In this embodiment, the deep graph convolutional network includes an attention graph convolutional network and an optimized matching layer; the input of the attention graph convolutional network is a set of feature point positions and description vectors of a pair of images, and outputs a feature descriptor after spatial information aggregation; the input of the optimized matching layer is the feature descriptor output by the attention graph convolutional network, and outputs a matching result.

[0057] In this embodiment, in the attention graph convolution network, first, the position of the feature point is dimensionally upgraded, and the feature point position and description vector after dimension upgrading are added to the feature descriptor vector to obtain the initial representation of each feature point; then, a multivariate graph is constructed; the vertices of the multivariate graph are all the feature points in the two images, and the edges include intra-image edges and cross-image edges; the intra-image edges connect the feature point pairs in a single image, and the cross-image edges connect the feature point pairs from the two images;

[0058] After constructing the multivariate graph, the features of all vertices in the constructed multivariate graph are aggregated and updated.

[0059] In this embodiment, the optimization matching layer calculates a similarity matrix based on the updated features, and each unit in the matrix represents the similarity of the features in the two images;

[0060] When a feature point in one image does not have a matching feature point in another image, the matrix is ​​expanded, and a new row and column are added to the expanded matrix to describe the situation where the feature point does not have a matching feature point.

[0061] In this embodiment, after obtaining the matching feature point pairs in the infrared image and the visible light image, the transformation parameters between the images are calculated based on the pixel coordinates of the feature point pairs; wherein, quality evaluation is performed on all matching point pairs to obtain Q values, and they are arranged in descending order according to the Q values. Random sampling is performed when the results are arranged in descending order according to the Q values ​​in each iteration to perform model hypothesis and verification.

[0062] In this embodiment, the coordinates of the infrared image are transformed into the coordinate system of the visible light image according to the transformation parameters, and the average value is taken according to the pixels at the corresponding positions to obtain the final fused image.

[0063] Embodiment 2:

[0064] This embodiment provides an infrared-visible light image registration method. The infrared-visible light image registration method is described in detail by applying the method to an image of a substation. The specific scheme is as follows:

[0065] First, infrared images and visible light images of the substation equipment are acquired. Specifically, existing technologies are used for image acquisition.

[0066] In this embodiment, the SuperPoint method of deep self-supervised learning is used to extract sparse feature points from the acquired infrared image and visible light image, and describe the feature points;

[0067] In view of the lack of robustness of artificially designed point feature extraction and description methods such as SIFT and SURF (Speeded Up Robust Features) to modal changes, in this embodiment, the SuperPoint method based on deep learning is used to extract and describe feature points of two different modal images, infrared and visible light.

[0068] like Figure 2 As shown, the SuperPoint feature extraction and description network in this embodiment is composed of an encoder and two decoders; the encoder is a convolutional network similar to VGGNet, comprising multiple convolutional layers and pooling layers, for encoding an input image of size W×H; the two decoders are used for feature point extraction and feature point description, respectively; wherein the feature point extraction decoder is composed of a convolutional layer, a Softmax layer, and a Reshape layer, and outputs a W×H×1 image, in which the value of each pixel represents the probability that the pixel is a feature point; the feature description decoder is composed of a convolutional layer, a bilinear interpolation layer, and an L2 norm normalization layer, and finally outputs a W×H×D feature map, in which each pixel corresponds to a D-dimensional feature vector.

[0069] Since the true value of the feature point is difficult to be manually labeled, a self-supervised learning strategy that does not require the true value is designed in this embodiment; specifically, first, a synthetic image set containing simple shapes such as triangles, quadrilaterals, cubes, and checkerboards is constructed, and these images have clear corner point information; then, a variety of homography transformations are randomly performed on the images and noise is added; since these homography transformations are generated by simulation, the positions of the feature points in the transformed images can be accurately obtained; finally, various simulated images and their corner point information are used to supervise the feature extraction network and learn to obtain the SuperPoint network; in this embodiment, the results of extracting feature points are as follows Figure 3 and 4 As shown:

[0070] In this embodiment, the SuperGlue method of the deep graph convolutional network is used to match the sparse feature points in the infrared image and the visible light image. In view of the fact that most existing feature matching methods match each feature point independently and are prone to errors, in this embodiment, the SuperGlue method based on the deep graph neural network is used to utilize the spatial geometric relationship between the feature points to improve the accuracy of feature matching.

[0071] The structure based on the deep graph neural network used in this embodiment is as follows Figure 5 As shown, it is mainly composed of two modules: an attention graph convolutional network and an optimization matching layer; wherein the input of the attention graph convolutional network is a set of feature point positions and description vectors of a pair of images, and outputs a feature descriptor after spatial information aggregation; the input of the optimization matching layer is the feature descriptor output by the attention graph convolutional network, and outputs the matching result.

[0072] Assume that the infrared image and the visible light image are image A and image B respectively, and the total number of feature points of image A and image B is M and N respectively; the position and description vector of the feature points in image A are recorded as: and The feature point positions and description vectors in image B are recorded as: and

[0073] In the attention graph convolutional network, first, the position of the feature point is passed through a feature point encoder composed of a multi-layer perceptron (MLP) to increase the dimension, and then the high-dimensional vector is added to the feature descriptor vector to obtain the initial representation of each feature point; that is:

[0074] (0) x i =d i +MLP enc (p i ) (1)

[0075] Then, a multivariate graph is constructed; the vertices of the graph are all the feature points in the two images, and the edges include intra-image edges and cross-image edges; among them, intra-image edges connect feature point pairs in a single image, while cross-image edges connect feature point pairs from two images; after the graph is constructed, the message passing mechanism is used to aggregate and update the features of all vertices in the graph; the update method is as follows:

[0076]

[0077] Among them, [·||·] represents the connection, m ε→i is the information transmitted from all other nodes in the graph to node i through the self-attention mechanism. After updating through information aggregation, the description vector of each feature point is obtained:

[0078]

[0079] Thus, each feature vector gathers the spatial and descriptive information of all feature points in its image and all feature points in another image; where W· (L) is the weight of the deep graph convolutional network after L rounds of iterations, and b is the bias parameter.

[0080] The optimized matching layer calculates an M×N similarity matrix S based on the updated features. Each cell (i, j) in the matrix represents the feature in image A. and the features in image B The similarity is:

[0081]

[0082] Due to occlusion or different field of view, the feature points in one image may not have matching feature points in another image; for this reason, the matrix S is expanded to a (M+1)×(N+1) matrix The newly added row and column are used to describe the situation where there is no match for the feature points, namely:

[0083]

[0084] Then, the problem of feature point matching is transformed into an optimal transportation problem, which can be solved using the Sinkhorn algorithm. Since the Sinkhorn algorithm is differentiable, it can be implemented with one network layer.

[0085] In this embodiment, the SuperGlue module is used to obtain the feature point matching results as follows: Figure 6 shown.

[0086] In this embodiment, the transformation parameters from the infrared image to the visible light image are estimated using the asymptotic sampling consistency PROSAC algorithm according to the pixel coordinates of the matched feature point pairs in the two images.

[0087] Specifically, after obtaining the matching feature point pairs in the infrared image and the visible light image, in this embodiment, the transformation parameters between the images are estimated according to the pixel coordinates of the feature point pairs; since the offset between the infrared lens and the visible light lens is very small relative to the distance of the photographed object, they can be approximated as a common optical center, and the infrared image is transformed into the coordinate system of the visible light image using the homography transformation matrix H; in the homogeneous coordinate system, the pixel coordinate transformation in the two images can be expressed as the following relationship:

[0088]

[0089] Among them, h in the homography transformation matrix H33 = 1. Therefore, the parameter freedom of the transformation matrix is ​​8, and it can be estimated using four or more pairs of feature points.

[0090] Due to the existence of matching noise or even wrong matching outliers, the least square method or RANSAC method is often used for parameter estimation; however, in RANSAC, each pair of feature points is treated equally, and samples are randomly selected from the entire set of feature point pairs, which has problems such as randomness of the estimation results and slow convergence speed. Therefore, in this embodiment, the PROSAC algorithm is used to estimate the transformation parameters.

[0091] In this embodiment, the PROSAC algorithm designs a semi-random method to perform quality evaluation on all matching point pairs to obtain Q values, and then arrange them in descending order according to the Q values. In each iteration, random sampling is performed preferentially among high-quality point pairs to perform model assumptions and verifications, thereby reducing algorithm complexity, improving efficiency, and avoiding the situation where convergence cannot be guaranteed faced by the RANSAC random algorithm.

[0092] In this embodiment, the coordinates of the infrared image are transformed into the visible light image coordinate system according to the transformation parameters to achieve registration.

[0093] Specifically, after calculating the transformation model, the coordinates of the infrared image are transformed into the coordinate system of the visible light image, and then the average value of the pixels at the corresponding positions is taken to obtain the final fused image, such as Figure 7 shown.

[0094] In view of the differences between infrared and visible light modalities, this embodiment adopts SuperPoint feature point extraction and descriptor calculation based on deep self-supervision, SuperGlue feature matching method based on deep graph convolutional network and progressive consistent sampling algorithm to achieve the matching of infrared images and visible light images. It can achieve more accurate registration of infrared images and visible light images compared with traditional methods, and accurate registration results can be obtained for images of different substation equipment taken at different angles.

[0095] Embodiment 3:

[0096] This embodiment provides an infrared-visible light image registration system, including an image acquisition module, a feature point extraction module, a feature point matching module, a transformation parameter calculation module and an image registration module;

[0097] The image acquisition module is configured to: acquire infrared images and visible light images;

[0098] The feature point extraction module is configured to: extract feature points from the acquired infrared image and visible light image using a self-supervised learning network, and describe the feature points;

[0099] The feature point matching module is configured to: match feature points in the infrared image and the visible light image using a deep graph convolutional network;

[0100] The transformation parameter calculation module is configured to: obtain the transformation parameters from the infrared image to the visible light image using a progressive consistent sampling algorithm according to the pixel coordinates of the matched feature point pairs in the two images;

[0101] The image registration module is configured to: transform the coordinates of the infrared image into a visible light image coordinate system according to the transformation parameters, and perform infrared-visible light image registration.

[0102] Embodiment: 4:

[0103] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the infrared-visible light image registration method described in Embodiment 1 are implemented.

[0104] Embodiment 5:

[0105] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the infrared-visible light image registration method described in Embodiment 1 are implemented.

[0106] The above description is only a preferred embodiment of the present embodiment and is not intended to limit the present embodiment. For those skilled in the art, the present embodiment may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present embodiment shall be included in the protection scope of the present embodiment.

Claims

1. An infrared-visible light image registration method, characterized in that: include: Acquire infrared images and visible light images; Using the self-supervised learning network, feature points of the acquired infrared image and visible light image are extracted and described; Use deep graph convolutional networks to match feature points in infrared images and visible light images; According to the pixel coordinates of the matched feature point pairs in the two images, the transformation parameters from the infrared image to the visible light image are obtained by using the progressive consistent sampling algorithm; According to the transformation parameters, the coordinates of the infrared image are transformed into the visible light image coordinate system, and the infrared-visible light image is registered; The deep graph convolutional network includes an attention graph convolutional network and an optimized matching layer; The input of the attention graph convolutional network is a set of feature point positions and description vectors of a pair of images, and outputs a feature descriptor after spatial information aggregation; the input of the optimization matching layer is the feature descriptor output by the attention graph convolutional network, and outputs a matching result; In the attention graph convolutional network, first, the position of the feature point is upgraded, and the upgraded feature point position and description vector are added to the feature descriptor vector to obtain the initial representation of each feature point; Then, a multivariate graph is constructed; the vertices of the multivariate graph are all feature points in the two images, and the edges include intra-image edges and cross-image edges; the intra-image edges connect feature point pairs in a single image, and the cross-image edges connect feature point pairs from two images; After constructing the multivariate graph, the features of all vertices in the constructed multivariate graph are aggregated and updated; The optimization matching layer calculates a similarity matrix based on the updated features, and each unit in the matrix represents the similarity of the features in the two images; When a feature point in one image does not have a matching feature point in another image, the matrix is ​​expanded, and a new row and column are added to the expanded matrix to describe the situation where the feature point does not have a matching feature point.

2. The infrared-visible light image registration method according to claim 1, characterized in that: The learning process of the self-supervised learning network is: Construct a set of synthetic images containing triangles, quadrilaterals, cubes, and checkerboard shapes to obtain corner information; Perform random homography transformation on the image and add noise to obtain the position of feature points in the transformed image; The transformed image and its corner point information are used to supervise the self-supervised learning network, and a self-supervised learning network with the ability to extract feature points of infrared images and visible light images is obtained.

3. The infrared-visible light image registration method according to claim 1, characterized in that: After obtaining the matching feature point pairs in the infrared image and the visible light image, the transformation parameters between the images are calculated according to the pixel coordinates of the feature point pairs. Among them, the quality of all matching point pairs is evaluated to obtain the Q value, and the points are arranged in descending order according to the Q value. Each iteration, random sampling is performed when the results are arranged in descending order according to the Q value to perform model hypothesis and verification.

4. The infrared-visible light image registration method according to claim 1, characterized in that: According to the transformation parameters, the coordinates of the infrared image are transformed into the coordinate system of the visible light image, and the average value of the pixels at the corresponding positions is taken to obtain the final fused image.

5. An infrared-visible light image registration system, characterized in that: It includes an image acquisition module, a feature point extraction module, a feature point matching module, a transformation parameter calculation module and an image registration module; The image acquisition module is configured to: acquire infrared images and visible light images; The feature point extraction module is configured to: extract feature points from the acquired infrared image and visible light image using a self-supervised learning network, and describe the feature points; The feature point matching module is configured to: match feature points in the infrared image and the visible light image using a deep graph convolutional network; The transformation parameter calculation module is configured to: obtain the transformation parameters from the infrared image to the visible light image using a progressive consistent sampling algorithm according to the pixel coordinates of the matched feature point pairs in the two images; The image registration module is configured to: transform the coordinates of the infrared image into a visible light image coordinate system according to the transformation parameters, and perform infrared-visible light image registration; The deep graph convolutional network includes an attention graph convolutional network and an optimized matching layer; The input of the attention graph convolutional network is a set of feature point positions and description vectors of a pair of images, and outputs a feature descriptor after spatial information aggregation; the input of the optimization matching layer is the feature descriptor output by the attention graph convolutional network, and outputs a matching result; In the attention graph convolutional network, first, the position of the feature point is upgraded, and the upgraded feature point position and description vector are added to the feature descriptor vector to obtain the initial representation of each feature point; Then, a multivariate graph is constructed; the vertices of the multivariate graph are all feature points in the two images, and the edges include intra-image edges and cross-image edges; the intra-image edges connect feature point pairs in a single image, and the cross-image edges connect feature point pairs from two images; After constructing the multivariate graph, the features of all vertices in the constructed multivariate graph are aggregated and updated; The optimization matching layer calculates a similarity matrix based on the updated features, and each unit in the matrix represents the similarity of the features in the two images; When a feature point in one image does not have a matching feature point in another image, the matrix is ​​expanded, and a new row and column are added to the expanded matrix to describe the situation where the feature point does not have a matching feature point.

6. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the infrared-visible light image registration method as claimed in any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the steps in the infrared-visible light image registration method as described in any one of claims 1 to 4 are implemented.

Images