Dual light image registration method, apparatus and device, and storage medium

Abstract

Embodiments of the present application provide a dual-light image registration method, device and equipment, and a storage medium, the dual-light image registration method comprising: acquiring infrared images and visible light images of the same group collected for a target scene; converting the infrared images into visible light images to obtain corresponding visible light conversion images; performing feature detection and matching on the visible light images and the visible light conversion images to determine a target homography matrix; and performing coordinate mapping on the visible light conversion images through the target homography matrix to obtain infrared registration images registered with the visible light images.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a dual-light image registration method, device and equipment, and a computer readable storage medium. BACKGROUND

[0002] Infrared thermal imaging and visible light imaging are two most important detection means, which are widely used in military equipment and civilian occasions. Due to the different imaging mechanisms and application scenarios of infrared images and visible light images, they have natural complementarity. Visible light images are formed by the reflection of the target, have high spatial resolution, can provide rich and clear appearance information, contain a large amount of information, are conducive to the cognition of the scene and the target by the human eye, and are better for establishing a discrimination model. The infrared detector receives the thermal radiation intensity information of the object in the field of view to form an infrared image, and the working principle of the infrared imaging sensor also determines that it is less affected by changes in visible light, fog and other bad weather, but the imaging quality is slightly worse than that of visible light. Registration and fusion of the two can effectively enrich the image texture details and significantly improve the image information richness.

[0003] However, the wavebands of infrared and visible light are not the same, and the differences between infrared images and visible light images are large. The core of registration of different modal images is to extract matching features of the spatial corresponding positions of different modal images, and the known various traditional feature extraction methods or deep feature extraction methods cannot better solve the problem that the image features of the spatial corresponding positions of infrared images and visible light images are difficult to match, thereby leading to low registration accuracy of the existing infrared image and visible light image registration algorithm, and inevitable flicker noise in the fused image. In addition, by designing a complex mechanical structure, the rotation and translation of 6 degrees of freedom in the three-dimensional space of infrared and visible light images are adjusted to realize the registration of infrared images and visible light images. This not only leads to difficult adjustment in the production process and is difficult to be engineered, but also can only adjust the rotation and translation of 6 degrees of freedom in the three-dimensional space by the mechanical structure adjustment device, and cannot solve the problem of inconsistency of the image features caused by different focal lengths and different fields of view of infrared images and visible light images.

[0004] Therefore, how to effectively solve the registration problem of infrared images and visible light images is a problem to be solved in the field. SUMMARY

[0005] To solve the existing technical problems, the present application provides a dual-light image registration method, device and equipment, and a computer readable storage medium, which can effectively match the image features of the spatial corresponding positions of infrared images and visible light images.

[0006] To achieve the above purpose, the technical scheme of the embodiments of the present application is as follows:

[0007] In a first aspect, embodiments of the present application provide a dual-light image registration method, comprising:

[0008] obtaining an infrared image and a visible light image of the same group collected for a target scene;

[0009] converting the infrared image into a visible light image to obtain a corresponding visible light converted image;

[0010] performing feature detection and matching on the visible light image and the visible light converted image to determine a target homography matrix;

[0011] performing coordinate mapping on the visible light converted image through the target homography matrix to obtain an infrared registration image registered with the visible light image.

[0012] In a second aspect, embodiments of the present application provide a dual-light image registration device, comprising:

[0013] an obtaining module configured to obtain an infrared image and a visible light image of the same group collected for a target scene;

[0014] a converting module configured to convert the infrared image into a visible light image to obtain a corresponding visible light converted image;

[0015] a matching module configured to perform feature detection and matching on the visible light image and the visible light converted image to determine a target homography matrix;

[0016] a registration module configured to perform coordinate mapping on the visible light converted image through the target homography matrix to obtain an infrared registration image registered with the visible light image.

[0017] In a third aspect, embodiments of the present application provide an infrared thermal imaging device, comprising a processor, a memory connected to the processor, and a computer program stored in the memory and executable by the processor, wherein the computer program is executed by the processor to implement the dual-light image registration method according to any of the embodiments of the present application.

[0018] In a fourth aspect, embodiments of the present application provide a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the dual-light image registration method according to any of the embodiments of the present application.

[0019] In the above embodiment, by converting the infrared image into a visible light image, feature detection and matching are performed based on the converted visible light image and the original visible light image in the same group to obtain a target homography matrix, and the target homography matrix is used to perform coordinate mapping on the visible light image converted from the infrared image to obtain an infrared registration image that can be registered with the original visible light image. In the above embodiment, the corresponding visible light image is obtained by converting the infrared image, and the image features in the visible light image retain the spatial correlation of the image features in the original infrared image. Feature detection and matching are performed based on the visible light image and the original visible light image to extract matched feature points to calculate the target homography matrix, so that image registration can be performed in the visible light domain, the differences between images in different domains are reduced, and the matched image features of the spatial corresponding positions of the infrared image and the visible light image can be more easily and accurately detected, thereby effectively improving the registration accuracy.

[0020] In the above embodiment, the dual-light image registration device, the infrared thermal imaging equipment, and the computer readable storage medium belong to the same concept as the corresponding dual-light image registration method embodiment, and thus have the same technical effects as the corresponding dual-light image registration method embodiment, which will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 An application scenario diagram of the dual-light image registration method in an embodiment;

[0022] Figure 2 A flowchart of the dual-light image registration method in an embodiment;

[0023] Figure 3 An architecture diagram of the modal conversion network in an embodiment;

[0024] Figure 4 A principle diagram of calculating a spatial correlation map in an embodiment;

[0025] Figure 5 A visible light image in an example;

[0026] Figure 6 A visible light image in an example; Figure 5 An infrared image in an example that is in the same group as the visible light image;

[0027] Figure 7 A visible light image in an example; Figure 6 A visible light conversion image obtained by converting the infrared image shown in FIG. 8A through the modal conversion network;

[0028] Figure 8 A feature matching result diagram of the visible light conversion image shown in FIG. 8B and the visible light image shown in FIG. 8A; Figure 7 Figure 5 ​​

[0029] Figure 9 A flowchart of a two-light image registration method in an optional specific example;

[0030] Figure 10 for Figure 6 The infrared configuration image shown is the result of infrared image transformation using the target homography matrix.

[0031] Figure 11 for Figure 10 The infrared registration image shown is Figure 5 The fused image of the visible light images shown:

[0032] Figure 12 This is a schematic diagram of an infrared registration device in one embodiment. Detailed Implementation

[0033] The technical solution of this application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] In the following description, the phrase "some embodiments" refers to a subset of all possible embodiments. It should be noted that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0036] In the following description, the terms "first, second, and third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, and third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0037] Please see Figure 1An optional application scenario of the dual-light image registration method provided by the embodiment of the present application is shown in the figure, wherein the infrared thermal imaging device 11 includes a processor 12, a memory 13 connected with the processor 12, and a shooting module 14. The shooting module 14 includes an infrared shooting assembly 141 and a visible light shooting assembly 142, which are fixed in the same structure and have parallel optical axes. When data is collected, the infrared shooting assembly 141 and the visible light shooting assembly 142 are clock-synchronized, and the infrared image and the visible light image of the same scene are collected at the same time. The infrared image and the visible light image of the same scene collected at the same time are regarded as the same group. The infrared thermal imaging device 11 collects the infrared image and the visible light image of the same group through the shooting module 14 and sends them to the processor 12. The memory 13 stores a computer program for implementing the dual-light image registration method provided by the embodiment of the present application. The processor 12 executes the computer program, converts the infrared image into a visible light conversion image, calculates a target homography matrix based on the feature detection and matching of the visible light conversion image and the original visible light image, and performs registration on the infrared image and the visible light image based on the target homography matrix. The infrared thermal imaging device 11 can be a shooting module 14 integrated with infrared image and visible light image shooting functions and various intelligent terminals with storage and processing functions, such as handheld observation instruments, various sighting devices, security monitoring devices, vehicle-mounted / airborne devices, etc.

[0038] Please refer to Figure 2 The dual-light image registration method provided by the embodiment of the present application can be applied to the infrared thermal imaging device in the application scenario shown in the figure. The dual-light image registration method includes the following steps: Figure 1 The dual-light image registration method includes the following steps:

[0039] S101, acquiring the same group of infrared image and visible light image collected for a target scene.

[0040] The same group of infrared image and visible light image refers to the infrared image and the visible light image of the same target scene collected at the same time. The target scene can be any scene for which the infrared thermal imaging device collects images. The infrared thermal imaging device can include a shooting module, which includes an infrared shooting assembly and a visible light shooting assembly. The infrared thermal imaging device acquires the infrared image and the visible light image of the target scene in real time through the shooting module. In other optional embodiments, the infrared thermal imaging device does not include a shooting module, and the infrared thermal imaging device acquires the same group of infrared image and visible light image sent by other intelligent devices with image shooting functions. Here, the other intelligent devices can be mobile phone terminals, cloud terminals, etc.

[0041] S103, convert the infrared image into a visible light image to obtain a corresponding visible light conversion image.

[0042] For the infrared image and the visible light image in the same group, the infrared image is converted into a visible light image. In the embodiments of the present application, in order to facilitate distinction and description, the visible light image obtained by converting the infrared image is referred to as a visible light conversion image. Wherein, the conversion of the infrared image can be realized by selecting some known algorithms, such as constructing a residual (ResNet) and a dense (DenseNet) generator two-branch network model, inputting the infrared image, and respectively generating a visible light conversion image through the residual and dense generator two-branch network, and fusing the conversion images of each generator branch network to obtain the visible light conversion image through the linear interpolation algorithm based on image segmentation.

[0043] S104, performing feature detection and matching on the visible light image and the visible light conversion image to determine a target homography matrix.

[0044] The homography matrix refers to the projection relationship between different domains of images, which can be a projection matrix from one plane to another, and can be used to describe the position mapping relationship between the object in the world coordinate system and the pixel coordinate system. The homography matrix is established according to the relationship of the feature point pairs in the visible light image and the visible light conversion image. The feature point pair refers to two feature points in the visible light image and the visible light conversion image respectively representing the same spatial corresponding position. Taking the homography matrix H as an example, the coordinates of the feature point pair are (x', y', 1) and (x, y, 1), and the homography matrix H is:

[0045]

[0046] The conversion relationship of the feature point pair is:

[0047]

[0048] According to the conversion relationship of the feature point pair, the feature point pair is converted into a polynomial equation group, and the coordinates of the feature points obtained based on image detection and matching are substituted into the polynomial equation group to determine the homography matrix.

[0049] Wherein, for the polynomial equation system with 8 degrees of freedom, the number of feature point pairs required for solving one homography matrix is 4 pairs. Through image detection and matching, the number of feature point pairs obtained can far exceed 4 pairs, and the better one can be selected from the multiple homography matrices solved as the target homography matrix. It can be understood that in some specific applications, the solved homography matrix can also be directly regarded as the optimal target homography matrix. After the infrared image is converted into the visible light conversion image, the image features of the visible light conversion image still retain the spatial correlation relationship of the image features in the original infrared image, and the feature detection and matching based on the converted visible light conversion image and the visible light image reduces the difference between the image features of the images in different domains, so that the matching image features at the spatial corresponding positions of the infrared image and the visible light image can be more easily and accurately detected. Wherein, the feature detection and matching can be realized by selecting some known algorithms, such as the Harris-based corner detection and the Flann-based matching method.

[0050] S105, mapping the coordinates of the visible light conversion image through the target homography matrix to obtain an infrared registration image registered with the visible light image.

[0051] The target homography matrix is used to map the coordinates of the visible light conversion image to the same coordinate system registered with the visible light image, that is, the infrared image is converted to the same coordinate system registered with the visible light image to obtain an infrared registration image registered with the visible light image. The infrared registration image can be used as the basis for image fusion.

[0052] In the above embodiment, the infrared image is converted into a visible light image, the feature detection and matching are performed based on the converted visible light conversion image and the original visible light image in the same group to obtain a target homography matrix, the coordinates of the infrared image converted to obtain the visible light conversion image are mapped through the target homography matrix to obtain an infrared registration image that can be registered with the original visible light image. Wherein, the corresponding visible light conversion image is obtained by converting the infrared image, the image features in the visible light conversion image retain the spatial correlation relationship of the image features in the original infrared image, the feature detection and matching are performed based on the visible light conversion image and the original visible light image to extract the matching feature points to calculate the target homography matrix, so that the image registration can be performed in the visible light domain, the difference between the image features of the images in different domains is reduced, and the matching image features at the spatial corresponding positions of the infrared image and the visible light image can be more easily and accurately detected, thereby effectively improving the registration accuracy.

[0053] In some embodiments, S103, converting the infrared image into a visible light image to obtain a corresponding visible light converted image, comprises:

[0054] The modal conversion network constructed based on the generative adversarial network converts the infrared image to obtain a corresponding visible light converted image;

[0055] The generative adversarial network includes a generative model, and a first discriminative model and a second discriminative model connected to the generative model, and the first discriminative model and the second discriminative model respectively take an infrared image of an original scale and an infrared image sampled at a set sampling rate as input.

[0056] The generative adversarial network (GAN, Generative Adversarial Networks) is a kind of deep learning model, which is a kind of unsupervised learning neural network. The generative adversarial network includes a generative model (Generative Model) and a discriminative model (Discriminative Model), which learns through mutual game to generate an output that can fit the input. The basic architecture of the modal conversion network adopts the generative adversarial network, the generative model converts the infrared image into a visible light image through learning, and the discriminative model identifies whether the generated visible light image is real enough. In the present embodiment, please refer to Figure 3 , the generative adversarial network includes two parallel discriminative models, the first discriminative model and the second discriminative model respectively take an infrared image of an original scale and an infrared image sampled at a set rate as input, form a multi-scale discriminative network to identify whether the visible light image converted by the generative model is real, and the set rate can be 2 times down sampling.

[0057] In the above embodiments, the modal conversion network is constructed based on the generative adversarial network, the discriminative model in the generative adversarial network adopts a discriminative model that takes an infrared image sampled at different rates as input to form a multi-scale discriminative network, in this way, the generative model will provide clearer texture details, and when the up sampling process is based on the main edge of the image, the distortion probability can be reduced.

[0058] In some embodiments, before the modal conversion network constructed based on the generative adversarial network converts the infrared image to obtain a corresponding visible light converted image, it includes:

[0059] An initial modal conversion network is constructed based on the generative adversarial network, and the initial modal conversion network is trained; in an iteration process, the generative model converts an infrared sample image into a visible light fitting image, and the first discriminative model and the second discriminative model judge whether the visible light fitting image is real;

[0060] The modal conversion network is trained by iteration until the loss function in the network converges.

[0061] The initial modal conversion network is constructed based on a generative adversarial network. During the training process, the generative model is used to convert an infrared sample image to generate a corresponding visible light fitting image as observation data by giving certain implicit information, the discriminative model is used to judge whether the visible light fitting image is true, the loss functions of the generative model and the discriminative model in the generative adversarial network are made to converge through a large number of iterative training, the first discriminative model and the second discriminative model predict that the visible light fitting image converted by the generative model is true, so as to judge the conversion quality of the visible light fitting image converted by the generative model, and the trained modal conversion network is obtained.

[0062] In the above embodiment, the modal conversion network is obtained by training, the corresponding visible light conversion image is obtained by converting the infrared image through the modal conversion network, and the visible light conversion image generated by the generative model is discriminated by using the multi-scale discriminative network, so that the image conversion result is more accurate and real.

[0063] In some embodiments, the first discriminative model and the second discriminative model respectively adopt a U-net network structure, and the loss function of the first discriminative model and the second discriminative model can be shown in the following formula one:

[0064]

[0065] X t is an infrared image; X f is a visible light conversion image; D = σ(C), C is an MxN output matrix, and each element in the output matrix represents the confidence of the corresponding pixel being true.

[0066] The U-net network structure is an encoding-decoding structure, the compression channel is an encoder for extracting features of the image layer by layer, and the expansion channel is a decoder for restoring the position information of the image. The U-Net network structure can be divided into three parts: a main feature extraction part, a strengthened feature extraction part, and a prediction part. The main feature extraction part can be a stack of convolution and maximum pooling, the features are extracted through multiple convolution layers, and the maximum pooling reduces the dimension of the feature extraction result and removes redundancy. The strengthened feature extraction part performs up-sampling on the effective feature layer obtained by the main feature extraction part and performs feature fusion to obtain the final effective feature layer which fuses all features. The prediction part classifies each feature using the effective feature layer. The first discriminative model and the second discriminative model adopt the same U-net network structure, the first discriminative model takes the infrared image at the original scale as input, and in the loss function of the first discriminative model, X tis the original ratio of the infrared image; the second discriminant model takes the 2 times down-sampled infrared image as input, as in the above formula one, the loss function of the second discriminant model is X t is the 2 times down-sampled infrared image, the output of each discriminant model is an MxN output matrix, each element in the output matrix represents the possibility of the corresponding pixel in the visible light conversion image being true, the outputs of the two discriminant models are normalized using the sigmoid function, and the binary cross-entropy loss is used to calculate the total loss of the multi-scale discriminant network composed of the two discriminant models.

[0067] Optionally, the total loss of the multi-scale discriminant network can be as shown in the following formula two:

[0068]

[0069] Wherein, λ1 and λ2 can be adjusted according to actual application requirements, for the case that the confidence of the output results of the two discriminant models is the same, λ1 and λ2 can be set to 0.5.

[0070] In the above embodiment, the multi-scale discriminant network is composed of two U-net network structures, each hidden layer in the U-Net network structure can have more feature dimensions, thereby facilitating the model to learn more diverse and comprehensive features, and more accurate output results can be obtained by using different levels of features through superposition combination.

[0071] In some embodiments, the loss function of the generation model can be as follows:

[0072] L S = d(S x , S y ); (formula three)

[0073] Wherein, S x is the first set of spatial correlation maps corresponding to the number of sampling channels of the feature extraction of the infrared sample image by the generation model, S y is the second set of spatial correlation maps corresponding to the number of sampling channels of the feature extraction of the visible light fitting image by the generation model, and d(·) is the distance between each spatial correlation map in the first set of spatial correlation maps and each spatial correlation map in the second set of spatial correlation maps.

[0074] The generation model takes the original infrared image before and after conversion and the corresponding visible light conversion image as two images in different domains, and uses the spatial correlation loss between the images in different domains to set the loss function. Please refer to Figure 4 , assuming that the original infrared image x and the corresponding visible light conversion image y are given, the image features f x and fy , and the self-similarity of both is calculated in the form of feature maps, referred to as spatial correlation maps.

[0075] Taking the infrared image x as an example, the calculation of the spatial correlation map is shown in the following Formula Five:

[0076]

[0077] wherein, represents the features of the query point x i with K sampling channels, and the K sampling channels represent feature extraction of the infrared image x by dividing it into K image blocks, and the query point x i may be a key point in the image block corresponding to each sampling channel. represents the corresponding features of N P feature points in the feature extraction results output by the K sampling channels, represents the calculation of the feature space correlation between the query point and the features of other points in the feature extraction results output by the K sampling channels. In this way, by calculating the self-similarity in the form of feature maps, the spatial correlation between feature points is captured, and the structure of the entire image is represented as a first spatial correlation map set S x , as shown in the following Formula Six:

[0078]

[0079] wherein, N s represents the number of sampling channels.

[0080] According to the above principle, the second spatial correlation map set S y of the visible light converted image y is calculated. x Using the first spatial correlation map set S y and the second spatial correlation map set S y , the spatial correlation map with structural similarity between the infrared image x and the corresponding visible light converted image y is compared, and the distance d(·) between each spatial correlation map in the first spatial correlation map set and each spatial correlation map in the second spatial correlation map set is calculated as the loss function of the generation model, wherein the distance d(·) can be the calculation of the Euclidean distance L1, L1=||S x -S y ||, or the calculation of the cosine distance L2, L2=||1-cos(S x , S yThe modality conversion network includes the loss function of the generative model. By training the modality conversion network until the loss function in the network converges, that is, the loss function of the generative model converges, the image features of the visible light converted image obtained by the generative model after training can not only encode the illumination, texture and other attribute information in the original infrared image, but also retain the spatial correlation of each feature point in the original infrared image.

[0081] In the above embodiments, the generation model can introduce a feature extraction network that calculates the spatial correlation between image features of images in different domains, so that the visible light converted image after infrared image conversion retains the spatial positional relationship of feature points of the original infrared image.

[0082] Optionally, the loss function of the generative model can be as shown in Formula 4 below:

[0083] L C =d(Z) x Z y (Formula 4)

[0084] Among them, Z x Z represents the first transformation block feature set corresponding to the number of sampling channels used by the generation model to extract features from the infrared sample image. y Let d(·) be the second conversion block feature set corresponding to the number of sampling channels for feature extraction of the visible light fitted image by the generation model, and let d(·) be the cross-correlation between each conversion block feature in the first conversion block feature set and each conversion block image in the second conversion block feature set.

[0085] Generative models can incorporate attention mechanisms, calculating attention matrices between multiple sampling channels to optimize the cross-correlation calculation of spatial locations between images in different domains. Assuming a given original infrared image x and a converted visible light image y, the structural features of the entire image can be represented as a set of converted block features Z calculated based on the attention mechanism, which can be expressed as Z = [Z1, ..., Zn]. n Taking an infrared image x as an example, the input infrared image x is divided into n sampling channels (patch) for feature extraction, and each patch is encoded as a tag. All labels are fed into a series of transformer blocks, each of which uses self-attention to mix the labels. Then, the similarity is calculated using the current key feature point (Query) and all other feature points (Key). The similarity is then normalized to calculate the attention matrix. The features of each transition block under self-attention are obtained by summing the products of the attention matrix and the corresponding weights (Value), as shown in Formula 7-10 below:

[0086]

[0087] Q = W Q X + b Q ; (Equation Eight)

[0088] K = W K X + b K ; (Equation Nine)

[0089] V = W V X + b V ; (Equation Ten)

[0090] wherein Q, K, V are linear transformations, W Q , W K , W B are linear transformation parameters corresponding to Q, K, V respectively, is a scale factor, Z n represents the conversion block features obtained by calculation; the structure of the entire image is represented as a first conversion block feature set Z x formed by multiple conversion block features.

[0091] According to the above principle, a second conversion block feature set Z y of the visible light conversion image y is calculated, and by given a case of marking x j in a domain, the corresponding spatial position y j in the conversion domain is found to calculate the distance d(·) between each conversion block feature map in the first conversion block feature set and each conversion block feature map in the second conversion block feature set as the loss function of the generation model, and the cross-correlation of x j and y j is calculated. The modal conversion network includes the loss function of the generation model, and by training the modal conversion network until the loss function in the network converges, that is, the loss function of the generation model converges, so that the image features of the visible light conversion image converted from the infrared image by the trained generation model not only encode the illumination, texture and other attribute information in the original infrared image, but also retain the spatial correlation of each feature point in the original infrared image, and by introducing the attention mechanism to calculate the cross-correlation of different domains, the spatial position matching information in the patch can be found faster, which can improve the conversion efficiency and accuracy and save the computing resources.

[0092] In the above embodiments, the generation model can introduce a feature extraction network that uses an attention mechanism to calculate the feature similarity of image features between images in different domains, so that the visible light conversion image converted from the infrared image retains the spatial position relationship of the feature points of the original infrared image, and the network running time can be reduced and the computing resources can be saved.

[0093] It should be noted that in some embodiments, the generation model can be configured to include both the above-mentioned two feature extraction networks at the same time, and the loss function of the modal conversion network can be as shown in the following Formula XI:

[0094]

[0095] μ1 and μ2 are hyperparameters, respectively. Please refer to Figure 5 and Figure 6 represent the visible light images and infrared images in the same group, respectively, as shown in Figure 7 , is a visible light converted image generated after the infrared image is converted by the modal conversion network. Compared with the conversion results of the currently known algorithms, the conversion accuracy of the modal conversion network in the embodiments of the present application is improved.

[0096] In some embodiments, the feature detection and matching of the visible light image and the visible light converted image, and the determination of the target homography matrix, include:

[0097] detecting the features of the visible light image and the visible light converted image by the SuperGlue network to extract the positions and visual feature descriptors of the feature points, and coupling the positions and the visual feature descriptors corresponding to each of the feature points to form the descriptors of each of the feature points;

[0098] matching the feature points between the visible light image and the visible light converted image, and determining the matching feature points according to the matching of the descriptors of each of the feature points;

[0099] determining the initial homography matrix according to the matching feature points;

[0100] adaptively optimizing the initial homography matrix to determine the target homography matrix.

[0101] wherein, the descriptor refers to a data structure that describes a feature, and the dimension of a descriptor can be multi-dimensional. For the feature position p i and the visual feature descriptor d i of the i-th key point input to the SuperGlue network, the position of the key point is embedded into a high-dimensional vector by coupling through a multi-layer perceptron (MLP), which can be as shown in the following Formula XII:

[0102]

[0103] so that the initial representation of each key point i is It combines its visual appearance and location. For two images, such as visible light image A and visible light converted image B, each image has a set of keypoint locations p and associated visual descriptors d, and these (p, d) are collectively referred to as local features. Given the initial local features, a matching descriptor f can be computed through communication between the features. i ∈R D By combining the attention aggregation mechanism with neural network training, the final matching descriptor can be obtained as shown in Formula Thirteen below:

[0104]

[0105] After L rounds of iterative optimization using self-attention and cross-attention on a visible light image A, we obtain the enhanced feature f of the i-th feature point in the visible light image A. i A This feature f i A Used for subsequent matching to find matching feature points from the visible light converted image B. See also Figure 8 The above refers to the feature matching results obtained by using the SuperGlue network to perform feature detection and matching on visible light images and visible light converted images in this embodiment of the application. Based on a set number of matched feature point pairs in the obtained feature matching results, an initial homography matrix is ​​determined, and the initial homography matrix is ​​adaptively optimized to determine the final optimal target homography matrix.

[0106] Optionally, before determining the initial homography matrix based on the matching feature points, the method further includes: filtering the matching feature points using the DEGENSAC algorithm to remove unmatched feature points. The DEGENSAC algorithm can better eliminate outliers and mismatches, thereby improving the convergence speed of model training and the accuracy of estimation results.

[0107] In the above embodiments, the visible light converted image obtained by converting the infrared image through the modality conversion network retains the spatial correlation of the image features in the original infrared image. The SuperGlue network extracts the position of feature points and visual feature descriptors. The visual feature descriptors contain the spatial correlation features of the features, which can improve the efficiency and accuracy of feature extraction and matching.

[0108] In some embodiments, the adaptive optimization of the initial homography matrix to determine the target homography matrix includes:

[0109] The initial homography matrix is ​​adaptively optimized based on the trained adaptive transformation network to determine the target homography matrix;

[0110] The loss function of the adaptive conversion network is determined by the distance between the visible light image and the target visible light conversion image and the cross-correlation between the visible light image and the target visible light conversion image.

[0111] The loss function of the adaptive conversion network is determined by the distance between the images before and after the homography matrix conversion and the cross-correlation between the images before and after the homography matrix conversion. The optimal homography matrix for image registration can be found by training the adaptive conversion network. In the two images of image registration, taking the image to be registered img' and the target image img as an example, the pixel alignment by the homography matrix H can be shown in the following formula fourteen:

[0112] img' = H(img); (Formula fourteen)

[0113] In the homogeneous coordinate system, it can be shown in the following formula fifteen:

[0114]

[0115] (m, n) represents the feature position of the image to be registered img', and (m', n') represents the feature position of the target image img. In the homography matrix, h 33 = 1, therefore, the degree of freedom of the homography matrix is 8, that is, only 4 pairs of feature points are needed to estimate the homography matrix.

[0116] In this embodiment, given the original visible light image v and the corresponding visible light conversion image v' after modal conversion of the original infrared image, after feature detection and matching by the SuperGlue network, the initial homography matrix H is output to the adaptive conversion network, and the following formula sixteen shows the loss function for constraint by joint training:

[0117] Loss = L1(v, v") + L SC (v, v"); (Formula sixteen)

[0118] The optimal target homography matrix H' is output by the adaptive conversion network, and the target visible light conversion image v" is obtained by homographic transformation of the visible light conversion image v' by the target homography matrix H', v" = H'( v '). L1(.) represents the Euclidean distance between the visible light image and the target visible light conversion image. L sc (·) represents the spatial correlation loss between the visible light image and the target visible light conversion image. By iteratively training the adaptive conversion network, when the loss function converges, the optimal target homography matrix is considered to be solved.

[0119] In the above embodiment, after feature detection and matching of the visible light image and the visible light converted image to determine the homography matrix, the image registration accuracy can be improved by establishing an adaptive conversion network taking the distance and cross correlation between the visible light image and the target visible light converted image as a loss function to adaptively optimize the homography matrix.

[0120] Optionally, before the adaptive optimization of the initial homography matrix based on the trained adaptive conversion network to determine the target homography matrix, the method further comprises:

[0121] An initial adaptive conversion network is constructed based on a self-attention transformer network model.

[0122] The initial homography matrix is input into the initial adaptive conversion network, and the modal conversion network and the initial adaptive conversion network are jointly trained until the loss function of the adaptive conversion network converges, and the trained adaptive conversion network is obtained.

[0123] The initial adaptive conversion network can be a self-attention transformer network model obtained by pre-training a known data set. By jointly training the modal conversion network and the adaptive conversion network, the trained adaptive conversion network can be obtained after training. The adaptive conversion network constructed by the transformer network model has the advantage of parallel computing compared with the general recurrent neural network, and can produce more interpretable results combined with the self-attention mechanism. By attention adjustment, it can learn to perform different tasks, and it is convenient to quickly and accurately adapt to obtain the optimal target homography matrix.

[0124] For a more comprehensive understanding of the dual light image registration method provided by the embodiments of the present application, the flow of the dual light image registration method will be described below with a specific example: Figure 9

[0125] S11, acquiring a same group of infrared image and visible light image; as shown in Figure 5 and Figure 6 .

[0126] S12, converting the infrared image into a visible light image by a modal conversion network, called a visible light converted image; as shown in Figure 7 .

[0127] ​Wherein, the basic framework of the modal conversion network adopts a generative adversarial network, the generative adversarial network includes a generative model and two discriminative models D1 and D2, the discriminative model D1 takes the infrared image of the original scale as input, the discriminative model D2 takes the infrared image of 2 times down-sampling as input, the discriminative models D1 and D2 form a multi-scale discriminator network, and the loss function is set as shown in the previous formula two. The generative model introduces a feature extraction network for calculating the spatial correlation of image features between images in different domains, and the loss function is set as shown in the previous formula three. A feature extraction network using attention mechanism can also be introduced to calculate the feature similarity of image features between images in different domains, and the loss function is set as shown in the previous formula four.

[0128] S13, feature points are extracted and matched in the visible light conversion image and the visible light image through the feature detection and matching network, a feature matching result containing a plurality of feature point pairs is obtained, and an initial homography matrix is output according to the feature matching result;

[0129] Wherein, the key point position and the descriptor are extracted based on the SuperGlue network, the feature point descriptor can be shown in the previous formula twelve, and the matching descriptor is calculated through the communication between features as shown in the previous formula thirteen; and the image feature matching result is shown in the previous formula Figure 8 .

[0130] S14, the initial homography matrix is adaptively optimized through the adaptive conversion network to determine an optimal target homography matrix;

[0131] Wherein, the loss function of the adaptive conversion network is set as shown in the previous formula sixteen.

[0132] S15, the visible light conversion image is position-mapped through the target homography matrix to obtain an infrared registration image; as Figure 10 shown.

[0133] Image registration can be used to improve the image fusion effect, as Figure 11 shown, the original visible light image and the infrared registration image are fused to obtain a visible light and infrared fusion image; from the fusion image, it can be seen that the registration area has been aligned and has not produced displacement, ghosting and other phenomena, and has high fusion precision.

[0134] Please refer to Figure 12In another aspect of the present application, a dual-light image registration device is provided, which can be implemented in an exemplary embodiment by an infrared handheld sighting device. The dual-light image registration device comprises: an acquisition module 131 configured to acquire a same set of infrared images and visible light images collected for a target scene; a conversion module 132 configured to convert the infrared images into visible light images to obtain corresponding visible light converted images; a matching module 133 configured to perform feature detection and matching on the visible light images and the visible light converted images to determine a target homography matrix; and a registration module 134 configured to perform coordinate mapping on the visible light converted images by using the target homography matrix to obtain infrared registration images registered with the visible light images.

[0135] Optionally, the conversion module 132 is specifically configured to convert the infrared images into corresponding visible light converted images based on a modal conversion network constructed based on a generative adversarial network; wherein the generative adversarial network comprises a generative model and first and second discriminative models connected to the generative model, and the first and second discriminative models take infrared images at an original scale and infrared images sampled at a set sampling rate as inputs, respectively.

[0136] Optionally, the conversion module 132 is specifically configured to construct an initial modal conversion network based on a generative adversarial network, and train the initial modal conversion network; in an iteration process, the generative model converts infrared sample images into visible light fitting images, and the first and second discriminative models judge whether the visible light fitting images are true; and the network is trained iteratively until a loss function in the network converges, to obtain the trained modal conversion network.

[0137] Optionally, the first and second discriminative models adopt a U-net network structure, and a loss function of the first and second discriminative models is as follows:

[0138]

[0139] X t is an infrared image; X f is a visible light converted image; D = σ(C), C is an MxN output matrix, and each element in the output matrix represents a confidence degree that a corresponding pixel is true.

[0140] Optionally, a loss function of the generative model is as follows:

[0141] L S = d(S x , S y );

[0142] wherein, Sx a first set of spatial correlation maps corresponding to a number of sampling channels for feature extraction of the infrared sample image by the generation model, S y a second set of spatial correlation maps corresponding to a number of sampling channels for feature extraction of the visible light fitting image by the generation model, d(·) is a distance between each spatial correlation map in the first set of spatial correlation maps and each spatial correlation map in the second set of spatial correlation maps; or,

[0143] The loss function of the generation model is as follows:

[0144] L C = d(Z x , Z y );

[0145] wherein Z x is a first set of converted block features corresponding to a number of sampling channels for feature extraction of the infrared sample image by the generation model, Z y is a second set of converted block features corresponding to a number of sampling channels for feature extraction of the visible light fitting image by the generation model, d(·) is a cross-correlation between each converted block feature in the first set of converted block features and each converted block image in the second set of converted block features.

[0146] Optionally, the matching module 133 is configured to perform feature detection on the visible light image and the visible light converted image by a SuperGlue network to extract positions and visual feature descriptors of feature points, couple the positions and the visual feature descriptors corresponding to each of the feature points to form a descriptor of each of the feature points, match the feature points between the visible light image and the visible light converted image, determine matching feature points according to matching of the descriptors of the feature points, and determine an initial homography matrix according to the matching feature points, and perform adaptive optimization on the initial homography matrix to determine a target homography matrix.

[0147] Optionally, the matching module 133 is configured to perform adaptive optimization on the initial homography matrix based on a trained adaptive conversion network to determine a target homography matrix, wherein a loss function of the adaptive conversion network is determined by a distance between the visible light image and a target visible light converted image and a cross-correlation between the visible light image and the target visible light converted image.

[0148] Optionally, the matching module 133 is further configured to construct an initial adaptive conversion network based on a self-attention transformer network model; input the initial homography matrix into the initial adaptive conversion network; and perform joint training on the modality conversion network and the initial adaptive conversion network until a loss function of the adaptive conversion network converges, so as to obtain the trained adaptive conversion network.

[0149] It should be noted that, in the process of registering the same group of infrared images and visible light images by using the dual-light image registration device provided in the above embodiments, only the division of the above program modules is exemplified, and in actual application, the above processing can be completed by different program modules according to needs, that is, the internal structure of the device can be divided into different program modules to complete all or part of the above-described method steps. In addition, the dual-light image registration device and the dual-light image registration method provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0150] In another aspect, the present application also provides an infrared thermal imaging device, which is an optional hardware structure schematic diagram of the infrared thermal imaging device provided in the embodiments of the present application. The infrared thermal imaging device includes a processor and a memory connected to the processor. The memory is used to store various types of data to support the operation of the image processing device, and stores a computer program for implementing the dual-light image registration method provided in any embodiment of the present application. When the computer program is executed by the processor, the steps of the dual-light image registration method provided in any embodiment of the present application are implemented, and the same technical effects can be achieved. To avoid repetition, details will not be repeated here.

[0151] Optionally, the infrared thermal imaging device further includes a shooting module and a display module connected to the processor. The shooting module includes an infrared shooting assembly and a visible light shooting assembly arranged in parallel along an optical axis. The infrared shooting assembly and the visible light shooting assembly are clock-synchronized, and collect infrared images and visible light images of a target scene and send them to the processor.

[0152] The processor performs image fusion based on the registered infrared registration image and the visible light image to obtain a dual-light fusion image. The display module is configured to display the dual-light fusion image.

[0153] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to realize each process of the above-mentioned double-light image registration method embodiment and achieve the same technical effects. To avoid repetition, details are not described herein. The computer readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0154] It should be noted that in this document, the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article or device that includes a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0155] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products, which are stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and include a plurality of instructions for making a terminal (which can be an infrared image device, a mobile phone, a computer, a server or a network device, etc.) execute the method described in each embodiment of the present application.

[0156] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A two-light image registration method, characterized in that, include: Acquire a set of infrared and visible light images for the target scene; The infrared image is converted into a visible light image to obtain a corresponding visible light converted image; the visible light converted image is the visible light image obtained by converting the infrared image. Feature detection and matching are performed on the visible light image and the visible light converted image to determine the target homography matrix; The visible light converted image is mapped to coordinates using the target homography matrix to obtain an infrared registered image that is registered with the visible light image.

2. The dual-light image registration method as described in claim 1, characterized in that, The step of converting the infrared image into a visible light image to obtain the corresponding visible light converted image includes: The infrared image is converted using a mode conversion network built based on a generative adversarial network to obtain the corresponding visible light converted image; The generative adversarial network includes a generative model and a first discriminant model and a second discriminant model connected to the generative model. The first discriminant model and the second discriminant model take infrared images at the original scale and infrared images sampled at a set sampling rate as inputs, respectively.

3. The dual-light image registration method as described in claim 2, characterized in that, Before the mode conversion network based on generative adversarial network converts the infrared image to obtain the corresponding visible light converted image, the process includes: An initial mode conversion network is constructed based on a generative adversarial network, and the initial mode conversion network is trained. In one iteration, the generative model converts the infrared sample image into a visible light fitted image, and the first discriminant model and the second discriminant model determine whether the visible light fitted image is real. The trained mode conversion network is obtained by iterative training until the loss function in the network converges.

4. The dual-light image registration method as described in claim 3, characterized in that, The first discriminant model and the second discriminant model both adopt the U-net network structure, and the loss functions of the first discriminant model and the second discriminant model are as follows: X t Infrared image; X f For the visible light converted image; D = σ(C), C is an M×N output matrix, each element in the output matrix represents the confidence level of the corresponding pixel as true, and σ() is the sigmoid function; Indicates X t Expected value Indicates X f Expected value; X represents t and X f The result output by the discriminant model.

5. The dual-light image registration method as described in claim 3, characterized in that, The loss function of the generative model is as follows: L S =d(S x ,S y ); Among them, S x S is the set of first spatial correlation maps corresponding to the number of sampling channels for feature extraction of infrared sample images by the generative model. y The second set of spatial correlation maps corresponds to the number of sampling channels used by the generative model to extract features from the visible light fitted image, where d(·) is the distance between each spatial correlation map in the first set of spatial correlation maps and each spatial correlation map in the second set of spatial correlation maps; or, The loss function of the generative model is as follows: L C =d(Z x ,Z y ); Among them, Z x Z represents the first transformation block feature set corresponding to the number of sampling channels used by the generation model to extract features from the infrared sample image. y Let d(·) be the second conversion block feature set corresponding to the number of sampling channels for feature extraction of the visible light fitted image by the generation model, and let d(·) be the cross-correlation between each conversion block feature in the first conversion block feature set and each conversion block image in the second conversion block feature set.

6. The dual-light image registration method as described in claim 2, characterized in that, The step of performing feature detection and matching on the visible light image and the visible light converted image to determine the target homography matrix includes: The visible light image and the visible light converted image are subjected to feature detection by the SuperGlue network to extract the position of feature points and visual feature descriptors. The position and visual feature descriptor corresponding to each feature point are coupled to form the descriptor of each feature point. The feature points between the visible light image and the visible light converted image are matched, and the matching feature points are determined based on the matching of the descriptors of each feature point; Determine the initial homography matrix based on the matched feature points; The initial homography matrix is ​​adaptively optimized to determine the target homography matrix.

7. The dual-light image registration method as described in claim 6, characterized in that, The adaptive optimization of the initial homography matrix to determine the target homography matrix includes: The initial homography matrix is ​​adaptively optimized based on the trained adaptive transformation network to determine the target homography matrix; The loss function of the adaptive conversion network is determined by the distance between the visible light image and the target visible light converted image, and the cross-correlation between the visible light image and the target visible light converted image.

8. The dual-light image registration method as described in claim 7, characterized in that, Before determining the target homography matrix by adaptively optimizing the initial homography matrix based on the trained adaptive transformation network, the process further includes: The initial adaptive transformation network is constructed based on the self-attention transformer network model; The initial homography matrix is ​​input into the initial adaptive transformation network, and the mode transformation network and the initial adaptive transformation network are jointly trained until the loss function of the adaptive transformation network converges, thus obtaining the trained adaptive transformation network.

9. A dual-light image registration device, characterized in that, include: The acquisition module is used to acquire a set of infrared and visible light images of the same target scene. A conversion module is used to convert the infrared image into a visible light image to obtain a corresponding visible light converted image; the visible light converted image is the visible light image obtained by converting the infrared image. The matching module is used to perform feature detection and matching on the visible light image and the visible light converted image to determine the target homography matrix; The registration module is used to perform coordinate mapping on the visible light converted image through the target homography matrix to obtain an infrared registered image registered with the visible light image.

10. An infrared thermal imaging device, characterized in that, The method includes a processor, a memory connected to the processor, and a computer program stored in the memory and executable by the processor, wherein the computer program, when executed by the processor, implements the dual-light image registration method as described in any one of claims 1 to 8.

11. The infrared thermal imaging device as described in claim 10, characterized in that, It also includes a shooting module and a display module connected to the processor. The shooting module includes an infrared shooting component and a visible light shooting component arranged with their optical axes parallel. The infrared shooting component and the visible light shooting component are clock-synchronized and capture infrared and visible light images of the target scene and send them to the processor. The processor performs image fusion based on the registered infrared image and the visible light image to obtain a dual-light fused image. The display module is used to display the dual-light fusion image.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the dual-light image registration method as described in any one of claims 1 to 8.

Images