Multi-modal image registration method and fusion method

Abstract

The invention discloses a multi-modal image registration method and fusion method, a multi-modal image comprises a visible light image and at least two spectral images, and the method comprises the following steps: respectively obtaining gradient maps of the at least two spectral images, and fusing the gradient maps of the at least two spectral images to obtain a fused gradient map; detecting an extreme point of each spectral image of the at least two spectral images to obtain a first feature point, and generating a first descriptor of the first feature point according to the fusion gradient map; detecting an extreme point of the visible light image to obtain a second feature point, and generating a second descriptor of the second feature point according to the visible light image; and registering the first feature point and the second feature point according to the first descriptor of the first feature point and the second descriptor of the second feature point so as to register the visible light image and the at least two spectral images. According to the invention, the stability of the feature points is enhanced through the cross-band gradient fusion of the multispectral image, and the accuracy and robustness of the registration of the multimodal image can be improved.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to a multimodal image registration and fusion method. Background Technology

[0002] Multimodal image registration is a crucial step in remote sensing image processing, directly impacting the accuracy of subsequent information fusion and analysis. However, current cross-modal, multi-view registration methods face numerous challenges. In remote sensing image processing, visible light and multispectral imagery each have their advantages. Visible light imagery offers high spatial resolution and provides rich texture and shape information, but its spectral information is weak, making it difficult to distinguish similar ground features. Multispectral imagery, on the other hand, boasts even higher spectral resolution, effectively identifying the spectral characteristics of ground features, particularly excelling in vegetation and water body monitoring, but its spatial resolution is lower, resulting in less detailed images. Therefore, combining visible light and multispectral imagery can balance high spatial resolution with rich spectral information, thereby improving the accuracy and reliability of remote sensing analysis.

[0003] However, due to the significant differences between visible light images and multispectral images in terms of resolution, illumination, and noise, traditional feature matching methods are not robust enough in cross-modal registration, especially when the feature descriptors are significantly different, the matching effect drops significantly. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal image registration and fusion method that can improve the accuracy and robustness of multimodal image registration.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A multimodal image registration method, wherein the multimodal image includes a visible light image and at least two spectral images, wherein the visible light image is an image generated by acquiring visible light from a target, and the at least two spectral images correspond to at least two wavelengths, and any one of the spectral images is an image generated by acquiring light of the corresponding wavelength from the target;

[0007] The multimodal image registration method includes:

[0008] The gradient maps of the at least two spectral images are obtained separately, and the gradient maps of the at least two spectral images are fused to obtain a fused gradient map;

[0009] For each of the at least two spectral images, extreme points are detected in the spectral image to obtain a first feature point, and a first descriptor of the first feature point is generated based on the fused gradient map.

[0010] Extreme points are detected in the visible light image to obtain second feature points, and a second descriptor for the second feature points is generated based on the visible light image.

[0011] Based on the first descriptor of the first feature point and the second descriptor of the second feature point, the first feature point and the second feature point are registered to register the visible light image and the at least two spectral images.

[0012] Optionally, fusing the gradient maps of the at least two spectral images to obtain a fused gradient map includes:

[0013] The preset products of corresponding pixels in the at least two spectral images are summed to obtain the value of the corresponding pixel in the fused gradient map. The preset product of corresponding pixels in any spectral image is the product of the gradient value of the corresponding pixel in any spectral image and the weight coefficient of the corresponding pixel.

[0014] Optionally, for each of the at least two spectral images, detecting extreme points in the current spectral image to obtain the first feature point includes:

[0015] For any one of the at least two spectral images, obtain a multi-scale image of the spectral image and detect extreme points in the multi-scale image of the spectral image. From the extreme points of the at least two spectral images, select the extreme points that consistently meet the preset conditions in the at least two spectral images as the first feature point.

[0016] Optionally, selecting the first feature point from the extreme points of the at least two spectral images that consistently meet a preset condition includes:

[0017] If the same extreme point is detected in at least two spectral images, or in a predetermined number of spectral images, then the extreme point is taken as the first feature point.

[0018] Optionally, before generating the first descriptor of the first feature point based on the fused gradient map, the method further includes:

[0019] For any one of the at least two spectral images, obtain a multi-scale image of the spectral image and perform difference on any two adjacent layers in the multi-scale image of the spectral image to obtain a scale space difference image and calculate the gradient of the scale space difference image.

[0020] For the first feature point, the contrast of the first feature point is obtained based on the response value of the first feature point in the scale space difference image of the spectral image and the gradient vector of the first feature point. If the contrast of the first feature point meets the preset requirements, the first feature point is removed.

[0021] Optionally, before generating the first descriptor of the first feature point based on the fused gradient map, the method further includes:

[0022] For any one of the at least two spectral images, obtain a multi-scale image of the spectral image and perform difference on any two adjacent layers in the multi-scale image of the spectral image to obtain a scale space difference image.

[0023] For the first feature point, according to the preset matrix of the first feature point in the scale space difference image of the spectral image, if it is determined that the first feature point is located in the edge region of the spectral image, the first feature point is removed. The preset matrix is ​​formed by the second-order partial derivatives and mixed partial derivatives of the first feature point in the x-direction and y-direction of the scale space difference image of the spectral image.

[0024] Optionally, generating a first descriptor for the first feature point based on the fused gradient map includes:

[0025] For the gradient map of any one of the at least two spectral images, in the gradient map of this spectral image, the neighborhood of the first feature point is divided into multiple sub-regions, and multiple directional gradient histograms are obtained for each sub-region. Based on the multiple directional gradient histograms of the multiple sub-regions, the descriptor of the first feature point in this spectral image is obtained.

[0026] The first feature point is obtained by stitching together the descriptors of the first feature point in the at least two spectral images.

[0027] Optionally, the registration of the first feature point and the second feature point includes:

[0028] An affine transformation matrix is ​​obtained based on the first feature point and the second feature point, and the elements of the affine transformation matrix are encoded into vectors to obtain an affine matrix encoding. The affine transformation matrix describes the positional transformation relationship between the first feature point in the spectral image and the second feature point corresponding to the first feature point in the visible light image.

[0029] A position code is generated based on the position of the first feature point / second feature point using sine and cosine functions;

[0030] The position encoding of the first feature point / second feature point and the affine matrix encoding are concatenated to generate the position awareness of the first feature point / second feature point, so as to register the first feature point and the second feature point based on the position awareness of the first feature point / second feature point.

[0031] Optionally, the registration of the first feature point and the second feature point includes:

[0032] The first feature point and the second feature point are input into a preset network. The preset network is used to output the position residual of the first feature point and the relative displacement of the second feature point relative to the first feature point, so as to register the first feature point and the second feature point according to the position residual of the first feature point and the position of the second feature point after adjustment according to the relative displacement.

[0033] The preset network includes multiple self-attention layers and a feedforward network. Each self-attention layer learns the relationship between the first feature point and the second feature point. The feedforward network is used to further process the output information of the self-attention layer. The preset network defines a query vector Query(Q), a key vector Key(K), and a value vector Value(V). The query vector Query(Q) is generated based on the first feature point, and the key vector Key(K) and the value vector Value(V) are generated based on the second feature point.

[0034] Optionally, the registration of the first feature point and the second feature point includes:

[0035] The first feature point and the second feature point are input into a preset model, and the preset model is used to output the registration result of the first feature point and the second feature point;

[0036] The loss during training the preset model includes displacement residual loss, which represents the difference between the displacement residual predicted by the preset model and the true label, and / or the loss during training the preset model includes epipolar geometric loss, which constrains the geometric relationship between the first feature point and the second feature point.

[0037] A multimodal image fusion method, wherein the multimodal image includes a visible light image and at least two spectral images, wherein the visible light image is an image generated by acquiring visible light from a target, and the at least two spectral images correspond to at least two wavelengths, and any one of the spectral images is an image generated by acquiring light of the corresponding wavelength from the target;

[0038] The multimodal image fusion method includes:

[0039] The visible light image and the at least two spectral images are registered using any of the above-described multimodal image registration methods, and the registered visible light image and the at least two spectral images are then fused.

[0040] The visible light image and the at least two spectral images are fused using a conditional generative adversarial network. The conditional generative adversarial network includes a generator and a discriminator. The generator's input is the fusion result of the spectral image, the deformation field, and the affine transformation matrix. The generator's output is the generated image. The deformation field is the result of deformation processing on the visible light image. The affine transformation matrix describes the positional transformation relationship between the first feature point in the spectral image and the second feature point in the visible light image corresponding to the first feature point.

[0041] Optionally, the loss during training of the conditional generative adversarial network includes a spectral loss, which describes the difference between the spectral feature vector of the generated image and the spectral feature vector of the real image.

[0042] Or / and, the loss during training of the conditional generative adversarial network includes spatial gradient loss, which describes the difference between the gradient of the generated image and the gradient of the real image.

[0043] As can be seen from the above technical solution, the multimodal image registration and fusion method of the present invention extracts feature points from each spectral image of the multispectral image and generates descriptors of the feature points based on the gradient map of each spectral image. Furthermore, it registers the feature points of the multispectral image and the feature points of the visible light image based on the descriptors of the feature points of the multispectral image and the feature points of the visible light image, thereby registering the multispectral image and the visible light image. By enhancing the stability of the feature points through gradient fusion across bands of the multispectral image, it can improve the accuracy and robustness of the registration. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 A flowchart illustrating a multimodal image registration method provided in one embodiment;

[0046] Figure 2 This is a flowchart illustrating the process of obtaining a second descriptor from a spectral image in a multimodal image registration method provided in one embodiment. Detailed Implementation

[0047] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0048] This embodiment provides a multimodal image registration method. The multimodal image includes a visible light image and at least two spectral images. The visible light image is an image generated by acquiring visible light from a target. The at least two spectral images correspond to at least two wavelengths, and any one of the spectral images is an image generated by acquiring light of the corresponding wavelength from the target.

[0049] The target refers to the object to be imaged. Visible light from the target is acquired to generate a visible light image. Any band of light from the target is acquired to generate a spectral image corresponding to that band. For example, the b-th band of light from the target is acquired to generate a spectral image of the b-th band, where b∈[1,n], and n represents a total of n spectral images, where n is a positive integer greater than 1.

[0050] For reference Figure 1 , Figure 1 A flowchart of a multimodal image registration method provided in one embodiment is shown below. Figure 1 As shown, the multimodal image registration method includes the following steps:

[0051] S11: Calculate the gradients of the at least two spectral images respectively, and fuse the gradient maps of the at least two spectral images to obtain a fused gradient map.

[0052] For any one of the at least two spectral images, the gradient of the current spectral image is calculated to obtain a gradient map of the current spectral image. The gradient maps of the at least two spectral images are then fused to obtain a fused gradient map.

[0053] S12: For each of the at least two spectral images, detect extreme points in the spectral image to obtain a first feature point, and generate a first descriptor for the first feature point based on the fused gradient map.

[0054] An image's extreme point refers to the point in the image where the pixel value is the highest or lowest within the neighborhood of that pixel.

[0055] A first feature point is obtained based on the extreme points detected in the spectral image. A first descriptor for the first feature point is further generated based on the fused gradient map. The first descriptor describes the characteristics of the first feature point in the at least two spectral images.

[0056] S13: Detect extreme points in the visible light image, obtain second feature points, and generate a second descriptor for the second feature points based on the visible light image.

[0057] A second feature point is obtained based on the extreme points detected in the visible light image. A second descriptor for the second feature point is then generated based on the visible light image, describing the characteristics of the second feature point in the visible light image.

[0058] S14: Based on the first descriptor of the first feature point and the second descriptor of the second feature point, register the first feature point and the second feature point to register the visible light image and the at least two spectral images.

[0059] Based on the first descriptor of the first feature point and the second descriptor of the second feature point, the first feature point and the second feature point that can correspond to the target are determined, thereby registering the first feature point in the spectral image and the second feature point in the visible light image, and then registering the visible light image and the at least two spectral images.

[0060] In this embodiment of the multimodal image registration method, feature points are extracted from each spectral image of the multispectral image and descriptors of the feature points are generated based on the gradient maps of each spectral image. Furthermore, the feature points of the multispectral image and the feature points of the visible light image are registered based on the descriptors of the feature points of the multispectral image and the feature points of the visible light image, thereby registering the multispectral image and the visible light image. The stability of the feature points is enhanced by gradient fusion across bands of the multispectral image, which can improve the accuracy and robustness of the registration.

[0061] In some embodiments, fusing the gradient maps of the at least two spectral images to obtain a fused gradient map includes: summing a preset product of corresponding pixels from the at least two spectral images to obtain the value of the corresponding pixel in the fused gradient map, wherein the preset product of corresponding pixels from any spectral image is the product of the gradient value of the corresponding pixel in any spectral image and the weight coefficient of the corresponding pixel. The weight coefficient of the corresponding pixel in any spectral image is determined by the correlation between the pixel value of any spectral image and the pixel values ​​of other spectral images.

[0062] In some implementations, the weighting coefficient of a corresponding pixel in any given spectral image is the average of the correlation coefficients between the pixel values ​​of the given spectral image and the pixel values ​​of other spectral images. The correlation coefficient between the pixel values ​​of the given spectral image and the pixel values ​​of other spectral images is the ratio of the product of the covariance of the pixel values ​​of the given spectral image and the standard deviation of the pixel values ​​of the given spectral image and the standard deviation of the pixel values ​​of other spectral images. The correlation coefficients between the pixel values ​​of the given spectral image and the pixel values ​​of other spectral images are obtained, and then the correlation coefficients between the given spectral image and all other spectral images are averaged.

[0063] For example, the fused gradient map can be represented as:

[0064] ;

[0065] in, I represents the value of pixel (x,y) in the fused gradient map. b (x,y) represents the pixel value of pixel (x,y) in the spectral image of band b. This represents the gradient value of pixel (x,y) in the spectral image of band b, also known as the gradient magnitude. b This represents the weighting coefficient, determined by the spectral correlation between bands. 'n' indicates that there are a total of n spectral images, where n is a positive integer greater than 1.

[0066] The weighting coefficients can be obtained using the following formula:

[0067] ;

[0068] Among them, cov(I b ,I i ) represents the covariance between the spectral image of band b and the spectral image of band i, σ b σ represents the standard deviation of the pixel values ​​in the spectral image of band b. i This represents the standard deviation of the pixel values ​​in the spectral image of the i-th band.

[0069] For example, refer to Figure 2 , Figure 2 This is a flowchart illustrating the process of obtaining a second descriptor from spectral images in a multimodal image registration method according to one embodiment. In some implementations, acquiring the at least two spectral images includes: noise quantization of any one of the at least two spectral images to obtain the signal-to-noise ratio (SNR) of any spectral image, wherein the SNR of any spectral image is the ratio of the mean of multiple measured signal values ​​of any spectral image to the standard deviation of the multiple measured signal values. Exemplarily, it can be expressed as:

[0070] ;

[0071] in, This represents the average of multiple measurements of the signal value in the spectral image of band b. This represents the standard deviation of the signal values ​​measured multiple times for the spectral image of band b. Since each band is independently dimmed in practical applications to ensure image quality, the signal-to-noise ratio (SNR) value can be used directly thereafter.

[0072] In this embodiment, the noise in each band is quantified to evaluate the noise-to-signal ratio (i.e., signal-to-noise ratio). By setting appropriate gain and integration time, the response of each band reaches 80% of the full-well, and 100 sets of integrating sphere data can be continuously captured to calculate the signal-to-noise ratio (SNR) of each band.

[0073] In some embodiments, for each of the at least two spectral images, detecting extreme points in the spectral image includes: obtaining a multi-scale image of the spectral image for any one of the at least two spectral images, and detecting extreme points in the multi-scale image of the spectral image. Obtaining the multi-scale image of the spectral image includes: processing the spectral image using a Gaussian kernel to obtain the multi-scale image of the spectral image. The scale of the Gaussian kernel is obtained based on the product of a reference scale and a preset adjustment coefficient. The preset adjustment coefficient is obtained based on the ratio of the maximum signal-to-noise ratio to the signal-to-noise ratio of the spectral image, where the maximum signal-to-noise ratio is the highest among the at least two spectral images.

[0074] In this embodiment, the scale of the Gaussian kernel used for spectral image processing is dynamically adjusted based on the signal-to-noise ratio (SNR) of the spectral image. This dynamic adjustment of the Gaussian kernel scale suppresses the influence of noise. A smaller Gaussian kernel scale is used when processing spectral images in bands with high SNR, while a larger Gaussian kernel scale is used when processing spectral images in bands with low SNR, to better handle noise. The Gaussian kernel scale can also be considered as the filtering scale of the Gaussian kernel.

[0075] For example, it can be represented as:

[0076] ;

[0077] in, σ represents the scale of the Gaussian kernel used in the spectral image processing of band b. base Represents the baseline scale, max(SNR) 1:n ) represents the maximum signal-to-noise ratio in an n-band spectral image, and ϵ represents a constant, a small constant to prevent division by zero errors, for example, ϵ=10. -5 .

[0078] In some embodiments, for each of the at least two spectral images, detecting extreme points in the spectral image to obtain a first feature point includes: obtaining a multi-scale image of the spectral image for any of the at least two spectral images and detecting extreme points in the multi-scale image; selecting, from the extreme points of the at least two spectral images, extreme points that consistently meet preset conditions as the first feature point. Obtaining a multi-scale image of any spectral image and then detecting extreme points in the multi-scale image improves the accuracy of obtaining extreme points from the spectral image. The extreme points detected in the spectral image are potential first feature points. In some embodiments, a multi-scale image can be generated by constructing a scale-space pyramid from the spectral image.

[0079] For example, a scale-space pyramid is constructed for the spectral image, and a multi-scale image L is generated for the spectral image of band b. b (x,y,σ) is represented as:

[0080] ;

[0081] Among them, I b (x,y) represents the spectral image of band b, G(x,y,σ) represents the Gaussian kernel function, σ represents the Gaussian kernel scale, and the Gaussian kernel scale σ increases exponentially: s is a constant, and k represents the total number of layers constructed.

[0082] In some implementations, detecting extrema in a multi-scale image of the spectral image includes: differencing any two adjacent layers in the multi-scale image of the spectral image to obtain a scale-space difference image, and then detecting extrema in the scale-space difference image. For example, in an implementation where a multi-scale image is obtained by processing the spectral image using a Gaussian kernel function, differencing any two adjacent Gaussian layers in the multi-scale image of the spectral image is performed to form a scale-space difference image (Difference of Gaussian, DoG) pyramid, and then extrema detection is performed.

[0083] In some implementations, detecting extreme points in a multi-scale image of the spectral image includes: for any pixel in the multi-scale image of the spectral image, if the pixel value of the pixel is greater than the maximum pixel value in its neighborhood or less than the minimum pixel value in its neighborhood, then the pixel is determined as an extreme point. For example, in a three-dimensional scale space (x, y, σ), the intensity values ​​of each pixel and its neighboring pixels are compared; if the following conditions are met: or If it is, then it is determined to be a candidate extreme point, DoG b(x,y,σ) represents the value of pixel (x,y) in a three-dimensional scale space, maxDoG b minDoG represents the maximum value in the neighborhood of a pixel (x,y) in a three-dimensional scale space. b denoted as the minimum value in the neighborhood of pixel (x,y) in three-dimensional scale space, and b represents the b-th band.

[0084] In some implementations, selecting an extreme point from the at least two spectral images that consistently meets a preset condition as a first feature point includes: if the same extreme point is detected in at least two spectral images in a number of spectral images greater than or equal to a preset number, then that extreme point is selected as the first feature point.

[0085] In this embodiment, verifying the consistency of extreme points across multiple spectral bands involves cross-band consistency verification. Feature points that meet the criteria are retained, while inconsistent feature points are removed. This ensures that the feature point can be found in spectral images across multiple bands, thereby improving the cross-modal matching capability for feature points acquired from multispectral images.

[0086] For example, it can be represented as:

[0087] .

[0088] Where Ⅱ(⋅) represents the indicator function, k represents the preset number, also known as the minimum number of uniform bands, and B represents the spectral image of B bands.

[0089] In some implementations, before generating the first descriptor of the first feature point based on the fused gradient map, the method further includes: for the first feature point, performing a second-order Taylor expansion on the grayscale information or gradient information surrounding the location of the first feature point, and calculating the position correction value of the first feature point. This further refines the positioning of the first feature point, thereby improving the accuracy and robustness of registration. The aim is to ensure that the extracted feature points are stable and effective, so that subsequent registration algorithms can work smoothly.

[0090] In some implementations, before generating the first descriptor of the first feature point based on the fused gradient map, the method further includes: obtaining a multi-scale image of the spectral image for any one of the at least two spectral images, and performing difference operations on any two adjacent layers in the multi-scale image of the spectral image to obtain a scale-space difference image, and calculating the gradient of the scale-space difference image; for the first feature point, obtaining the contrast of the first feature point based on the response value of the first feature point in the scale-space difference image of the spectral image and the gradient vector of the first feature point; if the contrast of the first feature point meets a preset requirement, then the first feature point is removed. This process of removing first feature points whose contrast meets the preset requirement improves the accuracy and robustness of registration. The aim is to ensure that the extracted feature points are stable and effective so that subsequent registration algorithms can work smoothly.

[0091] Meeting the preset requirement for the contrast of the first feature point can be determined by judging that the contrast of the first feature point is too low, considering that the first feature point does not have obvious features and is not suitable as a stable key point, and therefore it is removed. In some embodiments, meeting the preset requirement for the contrast of the first feature point can include the contrast of the first feature point being less than a first preset threshold.

[0092] For example, the contrast ratio is calculated according to the following formula:

[0093] ;

[0094] Where D(x) represents the contrast of point (x), D represents the response value of point (x) in the scale-space difference image, and ∂D represents the gradient vector of point (x) in the scale-space difference image. This represents the position correction value for point (x). This value reflects the brightness difference between this point and its surrounding pixels, i.e., local contrast. If |D(x)| is less than a first preset threshold, the contrast of this point is considered too low. The first preset threshold is, for example, 0.03. This process removes low-contrast points to improve the accuracy and robustness of registration, ensuring that the extracted feature points are stable and effective so that subsequent registration algorithms can function smoothly.

[0095] In some embodiments, before generating the first descriptor of the first feature point based on the fused gradient map, the method further includes: obtaining a multi-scale image of the current spectral image from any of the at least two spectral images, and performing difference operations on any two adjacent layers in the multi-scale image of the current spectral image to obtain a scale-space difference image; for the first feature point, based on a preset matrix of the first feature point in the scale-space difference image of the current spectral image, if it is determined that the first feature point is located in the edge region of the current spectral image, then the first feature point is removed. The preset matrix is ​​formed by the second-order partial derivatives and mixed partial derivatives of the first feature point in the x-direction and y-direction of the scale-space difference image of the current spectral image. In this embodiment, removing edge response points improves the accuracy and robustness of registration, aiming to ensure that the extracted feature points are stable and effective so that subsequent registration algorithms can work smoothly.

[0096] In some implementations, for a first feature point, the ratio of the square of the trace of a preset matrix of the first feature point to the determinant of the preset matrix is ​​obtained. If the ratio is greater than a second preset threshold, the first feature point is discarded. If the ratio of the first feature point is greater than the second preset threshold, the first feature point is considered to be located in the edge region of the spectral image and is discarded. For example, the preset matrix is ​​a Hessian matrix, represented as:

[0097] ;

[0098] Among them, D xx D yy Let D represent the second-order partial derivatives of the scale-space difference image in the x and y directions, respectively. xy This represents the mixed partial derivative of the scale-space difference image.

[0099] like: If r = 10, then the first feature point is rejected.

[0100] In some implementations, generating a first descriptor for the first feature point based on the fused gradient map includes:

[0101] S121: For the gradient map of any one of the at least two spectral images, in the gradient map of this spectral image, the neighborhood of the first feature point is divided into multiple sub-regions, and multiple directional gradient histograms of each sub-region are obtained. Based on the multiple directional gradient histograms of the multiple sub-regions, the descriptor of the first feature point in this spectral image is obtained.

[0102] S122: The first feature point is stitched together with the descriptors of the at least two spectral images to obtain the first descriptor of the first feature point.

[0103] For any spectral image, in the gradient map of this spectral image, the neighborhood of the first feature point is divided into multiple sub-regions. For each sub-region, multiple directional gradient histograms of this sub-region are obtained. Further, based on the multiple directional gradient histograms of the multiple sub-regions of the first feature point, a descriptor for the first feature point in this spectral image is obtained.

[0104] The multiple directional gradient histograms correspond to multiple different directions. In some embodiments, obtaining the multiple different directions includes: obtaining the directional histogram of the fused gradient map, dividing the directional histogram into multiple intervals, taking the direction with the largest gradient magnitude in any interval as the main direction of that interval, and determining the multiple different directions based on the main directions of the multiple intervals.

[0105] For example, for the gradient fusion graph G fusion The calculated direction histogram is represented as follows:

[0106] ;

[0107] in, This represents the gradient of the gradient fusion map in the x-direction, and represents the rate of change of pixel (x,y) in the x-direction. This represents the gradient of the gradient fusion map in the y-direction, and the rate of change of the pixel (x, y) in the y-direction. arctan is the arctangent function, used to calculate the angle given the gradient direction.

[0108] Divide the orientation histogram into 'a' intervals, take the direction with the largest peak as the main direction, and take all directions with a gradient magnitude greater than 80% of the main direction as secondary directions. For example, a=8.

[0109] In some implementations, obtaining the descriptor of the first feature point in this spectral image based on the multiple directional gradient histograms of the multiple sub-regions includes: the descriptor of the first feature point in this spectral image is a vector with elements being the weighted sum of the gradient magnitudes of the multiple sub-regions of the first feature point and the weighted sum of the angles of the same gradient direction of the multiple sub-regions of the first feature point. For example, the neighborhood of the feature point is rotated to the main direction, dividing the neighborhood of the first feature point into m×m sub-regions. An 8-directional gradient histogram is calculated for each sub-region, and the descriptor of the first feature point in the spectral image of the b-band is represented as:

[0110] ;

[0111] Where, d b w represents the descriptor of the spectral image in band b. b Represents the weighting coefficient, ∇I b This represents the gradient value of the spectral image in band b, also known as the gradient magnitude. ,…, The directions of each interval of the calculated direction histogram are represented sequentially.

[0112] Finally, the descriptors of the spectral images from all bands are stitched together and normalized to obtain the final multispectral descriptor. The high-dimensionality of spectral images increases the complexity of feature extraction, making it difficult for existing methods to effectively utilize their multi-band information for accurate matching. In this method, an adaptive feature extraction strategy is used to dynamically adjust the Gaussian filtering scale according to the signal-to-noise ratio of the spectral images in each band, and the stability of feature points is enhanced through a cross-band gradient fusion strategy.

[0113] In some implementations, the second descriptor of the second feature point is generated based on the visible light image. The feature point and its principal direction can be calculated using the scale-invariant feature transform (SIFT) method to generate the feature point descriptor. Finally, it is normalized to a unit vector to ensure that its scale is not affected.

[0114] In some implementations, detecting extrema in visible light images includes: obtaining multi-scale images of the visible light image and detecting extrema in the multi-scale images. A scale-space pyramid can be constructed from the visible light image to obtain multi-scale images. Gaussian blurring can be applied to the input visible light image, and a scale-space pyramid can be constructed, i.e., generating blurred versions of the image at multiple scales to capture structural information at different scales. In some implementations, the difference between any two adjacent layers in the multi-scale image of the visible light image is performed to obtain a scale-space difference image, and extrema are detected in the scale-space difference image. Two adjacent layers in the multi-scale image can also be considered as Gaussian images at adjacent scales; a difference Gaussian image is constructed by subtracting the two layers for subsequent extrema detection.

[0115] In some implementations, detecting extreme points in a multi-scale image of a visible light image includes: for any pixel in a multi-scale image of a visible light image, performing extreme value detection in the neighborhood of the pixel in space (x, y) and scale direction (σ); if the pixel has a local maximum or local minimum in the neighborhood of its own scale and adjacent scales, it is considered a potential feature point.

[0116] In some implementations, a second-order Taylor expansion is performed on the positions of the extreme points in the visible light image to calculate their precise locations. The contrast ratio is then obtained based on these precise locations. If an extreme point is determined to be a low-contrast point, it is discarded. Alternatively, an extreme point can be considered a low-contrast point if its contrast ratio is less than a first preset threshold.

[0117] In some implementations, a second-order Taylor expansion is performed on the positions of the extrema of the visible light image to calculate their precise locations. Based on these precise locations, it is determined whether the extrema are located in an edge region; if so, they are discarded. An extrema located in an edge region is considered an edge point, determined by whether the principal curvature ratio of the Hessian matrix exceeds a set threshold. For details, please refer to the implementation method described above for determining whether the first feature point is located in an edge region of the spectral image.

[0118] In some implementations, generating a second descriptor for a second feature point based on a visible light image includes: obtaining the principal direction of the second feature point from the visible light image; calculating the magnitude and direction of the image gradient within the neighborhood of the second feature point; constructing a histogram based on the directions; and taking the direction corresponding to the maximum value of the direction histogram after weighting the gradient magnitude as the principal direction of the second feature point. If the magnitude of other directions exceeds 80% of the principal direction, multiple principal directions are assigned to the second feature point, which can improve the robustness and rotation invariance of matching. For example, when constructing the direction histogram, 360° is typically divided into 36 intervals.

[0119] In some implementations, generating a second descriptor for a second feature point based on a visible light image includes: rotating and normalizing the neighborhood of the second feature point based on its principal direction, dividing it into multiple sub-regions, obtaining a gradient direction histogram for each sub-region, and obtaining a second descriptor for the second feature point based on the gradient direction histograms of the multiple sub-regions. For example, the neighborhood of the second feature point is divided into 4×4 sub-regions, for a total of 16 sub-regions. Gradient direction histograms are calculated for each sub-region in 8 directions, resulting in a 16×8=128-dimensional descriptive vector.

[0120] In some implementations, the second descriptor of the second feature point is normalized using the L2 norm to resist the influence of illumination changes on the gradient magnitude, thereby ensuring that the second descriptor has scale and brightness invariance. The final second descriptor is a unit vector, which is used in the subsequent feature matching process.

[0121] In this embodiment of the multimodal image registration method, for visible light images, the SIFI algorithm is used to extract feature points and obtain visible light descriptors. For multispectral images: ① A dynamic Gaussian kernel is used to overcome the limitations of fixed-scale filtering. This method employs a dynamic Gaussian kernel adjustment mechanism based on the signal-to-noise ratio (SNR) quantization value. Specifically, the SNR of each band in the multispectral image is quantized, and the Gaussian filter kernel scale is dynamically adjusted according to the SNR. A smaller Gaussian kernel is used when the SNR is high to retain more details; a larger Gaussian kernel is used when the SNR is low to smooth noise and improve the stability of feature points. ② A multispectral gradient fusion strategy is adopted to improve the consistency of cross-band features. Traditional methods calculate gradients in a single band, making it difficult to fuse multi-band information and affecting cross-modal matching capabilities. This method generates a stable gradient map by weighted fusion of gradient information from each band. This strategy effectively combines information from different bands, ensuring that features from different bands maintain consistency during the fusion process, thereby enhancing the robustness of feature points. ③ A scale-space pyramid is constructed using a dynamic Gaussian kernel for extremum detection. Images are smoothed using Gaussian kernels of different scales to construct a scale space; extremum point detection is then performed within this scale space to extract potential feature points. ④ To address the issue that traditional extremum detection is only performed in a single band and is susceptible to noise interference, leading to unstable feature point matching, this method employs multi-band extremum response consistency verification. Only feature points that respond in multiple bands are retained, selecting globally representative feature points and improving the cross-modal matching capability of multispectral image feature points. ⑤ Key point localization and selection. Registration accuracy and robustness are improved through sub-pixel level localization correction, removal of low-contrast points, and elimination of edge response points. ⑥ The multispectral image descriptor generation process is optimized. An orientation histogram is calculated on the gradient fusion map, and feature point descriptors are constructed based on this, replacing traditional single-band descriptors and avoiding the accuracy degradation caused by information loss in single-band descriptors. Descriptors from different bands are normalized and then stitched together to generate the final multispectral descriptor, effectively combining multi-band information and enhancing the global consistency of the descriptor.

[0122] In some implementations, registration of the first feature point and the second feature point includes: using the bidirectional nearest neighbor distance ratio (NNDR) to filter and match the first feature point and the second feature point, thus performing geometric coarse registration to remove inaccurate matches.

[0123] In some implementations, registration of the first feature point and the second feature point includes:

[0124] S141: Obtain an affine transformation matrix based on the first feature point and the second feature point, and encode the elements of the affine transformation matrix into vectors to obtain an affine matrix encoding. The affine transformation matrix describes the positional transformation relationship between the first feature point in the spectral image and the second feature point corresponding to the first feature point in the visible light image.

[0125] S142: Generate a position code based on the position of the first feature point / second feature point using a sine function and a cosine function;

[0126] S143: The position encoding of the first feature point / second feature point and the affine matrix encoding are concatenated to generate the position awareness of the first feature point / second feature point, so as to register the first feature point and the second feature point according to the position awareness of the first feature point / second feature point.

[0127] The elements of the affine transformation matrix are the affine transformation parameters. These elements are encoded as vectors to provide additional spatial information. In this embodiment, the positions of the first feature point and the second feature point are encoded to help the model understand the spatial location information of the feature points and effectively combine it with the affine transformation parameters.

[0128] In some implementations, the Random Sample Consensus (RANSAC) algorithm can be used to estimate the optimal affine transformation matrix, minimizing the point pair error to solve for the affine transformation matrix M.

[0129] In some implementations, obtaining the affine transformation matrix based on the first and second feature points includes: First, based on the set of matching point pairs obtained in the feature extraction and matching stages, several sets of matching points are randomly selected. For each set of matching points, an affine transformation model is established to map the point coordinates in the source image to the target image; for example, each set of matching points contains 3 pairs of points. Since the affine transformation has 6 degrees of freedom, a minimum of 3 pairs of points is sufficient for calculation. Subsequently, the affine transformation parameters are calculated using these point pairs. The affine transformation parameters include a linear transformation part and a translation part, which can describe the rotation, scaling, shearing, and translation relationships between images. The calculated affine transformation parameters are applied to all original matching points, and the number of points with an error below a preset threshold under this transformation is counted; these are considered inliers. Through multiple rounds of iterative sampling and evaluation, the affine transformation model that results in the largest number of inliers is finally selected as the optimal affine transformation matrix. Finally, the affine transformation matrix M is refitted using the set of inliers corresponding to this model. The least squares method is generally used for fitting.

[0130] In some implementations, the parameters of the affine transformation matrix M obtained in the coarse registration stage are encoded into a conditional vector c using a multilayer perceptron (MLP). M For example, it is represented as:

[0131] ;

[0132] Where vec(M) represents the vectorized representation of the affine transformation matrix, typically arranged into a one-dimensional vector for network processing. c Indicates the dimension of the encoding.

[0133] For example, position encoding generates a standard position code from the spatial coordinates (x, y) of a feature point using sine and cosine functions to help the model understand the coordinate information.

[0134] ;

[0135] Where (x,y) represents the position coordinates of the feature point, PE view (x,y) represents the positional encoding of the feature point (x,y), d represents the encoding dimension, i represents the index (position) of the encoding dimension, that is, the i-th dimension of the vector, i∈[0,d / 2−1], and ⊕ represents the vector concatenation operation. This encoding method transforms the spatial position of the feature point into a vector, allowing the Transformer model to understand the spatial positional information of the feature point.

[0136] Combining the above encoded information, the positional encoding and affine matrix encoding are concatenated to generate the final position-aware token. This can be represented as:

[0137] .

[0138] In this way, based on the positional awareness of feature points, spatial location information and affine transformation matrices can be effectively used for feature point registration. In this embodiment, a model can be established to obtain affine matrix encoding, obtain feature point position encoding, and register feature points. The model can be called a Transformer model. This embodiment can be considered as the fine registration stage. The role of the fine registration stage is to capture the interrelationships between feature points in multimodal images (visible light images and multispectral images) through a self-attention mechanism, thereby aligning the feature points in a refined manner.

[0139] In some implementations, registering the first feature point and the second feature point includes: generating a token for the first feature point / second feature point based on the location perception of the first feature point / second feature point, a first descriptor of the first feature point, or a second descriptor of the second feature point, and registering the first feature point and the second feature point based on the token of the first feature point and the token of the second feature point.

[0140] In this embodiment, the first feature point / second feature point is processed using tokenization. The token for the first feature point / second feature point consists of three parts, represented as follows:

[0141] ;

[0142] (x, y) represents the position coordinates of the feature point, which are two-dimensional coordinates.

[0143] The second descriptor representing the feature points, i.e. the descriptor obtained from the visible light image, is represented as a k-dimensional vector and can capture the local texture information of the feature points. Specifically, it can be a scale-invariant feature transformation descriptor.

[0144] This represents an m×m image patch (i.e., local image information) surrounding a feature point in a spectral image, containing detailed information about the area around the feature point. This represents an m×m image block (i.e., local image information) surrounding a feature point in a visible light image, containing detailed information about the area around the feature point.

[0145] The tokens of the first feature point and the tokens of the second feature point constitute the input tokens of the first feature point and the second feature point into the Transformer model.

[0146] In some implementations, registration of the first feature point and the second feature point includes: inputting the first feature point and the second feature point into a preset network, wherein the preset network is used to output the position residual of the first feature point and the relative displacement of the second feature point relative to the first feature point, so as to register the first feature point and the second feature point according to the position residual of the first feature point and the position of the second feature point adjusted according to the relative displacement; wherein the preset network includes multiple self-attention layers and a feedforward network, each self-attention layer learns the relationship between the first feature point and the second feature point, and the feedforward network is used to further process the output information of the self-attention layer; wherein the preset network defines a query vector Query(Q), a key vector Key(K), and a value vector Value(V), wherein the query vector Query(Q) is generated based on the first feature point, and the key vector Key(K) and the value vector Value(V) are generated based on the second feature point.

[0147] For example, the query vector Query(Q) is defined as generated based on the first feature point, denoted as q. i The key vector Key (K) and value vector Value (V) are generated based on the second feature point, and are represented as k respectively. j and v j。 is represented as:

[0148] ;

[0149] , , ;

[0150] Where Q represents the query vector of the first feature point, Token MS Token represents the token of the first feature point, and K represents the key vector of the second feature point. RGB The token represents the second feature point, V represents the value vector of the second feature point, and W represents the second feature point. Q W K W V These are the weight matrices used for mapping, d k It is the dimension of the key vector K.

[0151] Feedforward networks transform input vectors into probability distributions, represented as:

[0152] ;

[0153] Where, x i x j All correspond The function of this expression is to: convert a set of real numbers {x1, x2, ..., x...} n The value x is mapped to a set of positive numbers whose sum is 1, which can be understood as a probability distribution. i The larger, the corresponding The larger the value, the higher the degree of "attention" the item receives after normalization. In this embodiment, multiple self-attention layers and feedforward networks are stacked to form a deep structure of the preset network. Each layer learns the relationship between feature points through a self-attention mechanism, and then further processes the information through a feedforward network. The preset network outputs the displacement residual Δp of the feature points. i This refers to the predicted precise location of the feature points. It is represented as:

[0154] ;

[0155] Where, Δp i v represents the displacement residual at the first feature point. j =[Δx j ,Δy j The image represents the relative displacement of the second feature point. These displacement residuals are used to update the feature point positions, thereby achieving finer registration. The preset network can be a Transformer model, and the improved self-attention layer can be a deformable cross-attention layer.

[0156] In some implementations, registering the first feature point and the second feature point includes: inputting the first feature point and the second feature point into a preset model, wherein the preset model is used to output the registration result of the first feature point and the second feature point.

[0157] In some implementations, the loss during training of the preset model includes displacement residual loss, which represents the difference between the displacement residual predicted by the preset model and the true label.

[0158] The displacement residual loss is calculated by identifying the difference between the displacement residual predicted by the pre-defined model and the actual label. For example, it can be represented as:

[0159] ;

[0160] Among them, L residual Δp represents the displacement residual loss. i It is the displacement residual predicted by the preset model, p i ground truth This represents the actual displacement.

[0161] In some implementations, the loss during training of the preset model includes epipolar geometric loss, which constrains the geometric relationship between the first and second feature points. By constraining the geometric relationship of the feature points through epipolar geometric loss, the registration result is ensured to conform to the geometric constraints of the image.

[0162] ;

[0163] Among them, L epipolar Let F represent the epipolar geometric loss, p and q represent the corresponding feature points, M represent the set of feature points, and ϵ represent a constant, which can be a small regularization constant.

[0164] The total loss is a weighted sum of the displacement residual loss and the epipolar geometric loss, expressed as:

[0165] ;

[0166] λ represents the trade-off coefficient between the two parts of the loss function.

[0167] The Adam optimization algorithm can be used to minimize the loss function. Each training iteration uses multiple samples for batch computation, and the model is trained via backpropagation to reduce loss.

[0168] In multimodal image registration, the non-rigid deformation caused by multi-view imaging makes it difficult for traditional affine transformation models to accurately model the data. While pure deep learning solutions have strong feature extraction capabilities, they rely on a large amount of labeled data and lack geometric interpretability, making it difficult to guarantee the reliability of the matching. In this method, a hybrid registration network combining traditional geometric coarse registration and Transformer fine registration is used in the multimodal image registration stage. Furthermore, to improve the robustness of registration under multi-view differences, a cross-attention mechanism is designed and epipolar geometric loss is introduced.

[0169] This embodiment of the multimodal image registration method addresses the issues of multi-viewpoint and non-rigid deformation by employing a coarse-fine two-stage registration strategy to improve registration robustness and accuracy.

[0170] Phase 1: Coarse Geometric Registration. First, SIFT feature points are used for matching, and RANSAC is used to optimize the affine transformation. Through coarse registration, images of different modalities are aligned to an initial affine transformation matrix, laying the foundation for subsequent fine registration.

[0171] Phase Two: Fine-tuning. ① To optimize feature representation and address the issue of insufficient correlation between cross-modal features, this method improves the format of the input token. Based on the results of coarse registration, feature point coordinates, descriptors, and local image patches are concatenated into fused features, enhancing the correlation between feature points in multimodal images and providing a more comprehensive feature representation for subsequent registration processes. ② A geometrically aware positional encoding combining affine matrix parameters is adopted to overcome the problem of traditional positional encoding being insensitive to non-rigid deformations. The token is embedded in a high-dimensional space through linear mapping and fused with affine matrix parameters to generate a spatial positional encoding that is aware of geometric changes. This design enables the model to more accurately understand and process geometric transformations during registration, especially when facing non-rigid deformations, maintaining better registration accuracy. ③ A cross-attention mechanism is adopted to address the problem of insufficient cross-modal information coupling. Using multispectral feature points as queries and visible light feature points as keys / values, the network automatically learns the global and local spatial relationships between multispectral and visible light image feature points during training and ensures accurate alignment between different modalities by calculating cross-modal attention weights. ④ To address the issue of poor geometric consistency of registration points under multi-view imaging, this method constructs an epipolar constraint loss function. This forces the matching points to satisfy multi-view projection consistency, improving the geometric rationality of cross-modal registration and making the registration results more accurate.

[0172] This embodiment also provides a multimodal image fusion method. The multimodal image includes a visible light image and at least two spectral images. The visible light image is an image generated by acquiring visible light from a target. The at least two spectral images correspond to at least two wavelengths, and any one of the spectral images is an image generated by acquiring light of the corresponding wavelength from the target.

[0173] The multimodal image fusion method includes:

[0174] The visible light image and the at least two spectral images are registered using the multimodal image registration method described in any of the above embodiments, and the registered visible light image and the at least two spectral images are then fused.

[0175] This embodiment of the multimodal image fusion method extracts feature points from each spectral image of a multispectral image and generates descriptors for the feature points based on the gradient maps of each spectral image. Furthermore, it registers the feature points of the multispectral image and the visible light image based on the descriptors of the feature points of the multispectral image and the visible light image, thereby registering and fusing the multispectral image and the visible light image. By performing gradient fusion across bands of the multispectral image, the stability of the feature points is enhanced, which can improve the accuracy and robustness of multimodal image registration and fusion.

[0176] In some implementations, a conditional generative adversarial network is used to fuse the visible light image and the at least two spectral images. The conditional generative adversarial network includes a generator and a discriminator. The generator's input is the fusion result of the spectral image, the deformation field, and the affine transformation matrix. The generator's output is a generated image. The deformation field is the result of deforming the visible light image. The affine transformation matrix describes the positional transformation relationship between the first feature point in the spectral image and the second feature point in the visible light image corresponding to the first feature point.

[0177] A Conditional Generative Adversarial Network (cGAN) is constructed. During image upsampling, the cGAN guides the upsampling process to improve the resolution and quality of the generated image. This process involves a generator G and a discriminator D. adv The adversarial training process generates high-resolution images while ensuring that the generated images are as close as possible to real high-resolution images in both space and spectrum. Through adversarial training, the generator and discriminator compete against each other. The generator learns how to produce more realistic high-resolution images, while the discriminator learns how to distinguish between real and generated images.

[0178] Spectral image I LR MS Deformation field and affine transformation matrix M affineThese three parts, when combined, constitute the input to the generator. For visible light image I... RGB Deformation processing is performed to better capture image details during upsampling, achieving deformation-field-guided upsampling. In some implementations, this is achieved through a deformable convolution module D and a deformation field W. deform The visible light image is deformed to obtain the deformation field. The deformable convolutional module is represented by D, and the deformation field is represented by W. deform For example, the input to the generator can be represented as:

[0179] ;

[0180] Where ⊕ represents the concatenation operation, ⊗ represents the deformation convolution operation, and D represents the deformable convolution module, which is responsible for passing the deformation field W. deform The visible light image is processed. With this input format, the generator G can generate images with higher resolution, more accurate spatial positioning, and more realistic textures.

[0181] Discriminator D adv Used to determine the generated image G(I) LR Is it close to the real image? HR Discriminator D adv The goal is to maximize the discriminant value of the real image and minimize the discriminant value of the generated image.

[0182] In some implementations, the loss function of the discriminator is expressed as:

[0183] ;

[0184] Among them, L adv The loss of the discriminator is represented by E[·], which can be called the adversarial loss. E[·] represents the expectation operation, which is to sum the average loss over all samples. It is the expected value of the loss function and represents the overall performance of the network during training. adv (I HR ) is the discriminator's output of the true high-resolution image, D adv (G(I LR The output of the discriminator to the generated image is shown in the image. To enhance the stability of training, spectral normalization is used to constrain the weights of the discriminator, thereby avoiding mode collapse during training.

[0185] In some implementations, the loss during training of the conditional generative adversarial network includes a spectral loss, which describes the difference between the spectral feature vectors of the generated image and the spectral feature vectors of the real image. This imposes a spectral fidelity constraint on the conditional generative adversarial network to ensure that the generated high-resolution image is as spectrally realistic as possible.

[0186] In some implementations, spectral loss includes spectral angle loss, which describes the angle between the spectral feature vectors of the generated image and the real image. Spectral angle loss measures the spectral difference between the generated and real images. The spectral angle is represented by calculating the angle between two spectral vectors; the smaller the angle, the more similar the spectral information of the two images. Spectral angle loss is expressed as:

[0187] ;

[0188] Among them, L spectral ϕ(I) represents the spectral angular loss. HR ) indicates high-resolution generated image I HR The spectral eigenvector, ϕ(I GT ) represents true high-resolution image I GT The spectral eigenvectors are given by , where arccos is the inverse cosine function used to calculate the angle between two vectors. The more similar the spectral eigenvectors are, the smaller the angle, and vice versa. · represents the vector dot product, and ||| represents the L2 norm of the vector.

[0189] In some implementations, the loss during training of the conditional generative adversarial network includes a spatial gradient loss, which describes the difference between the gradient of the generated image and the gradient of the ground truth image. By calculating the gradient difference between the generated and ground truth images to measure their spatial structural differences, the spatial gradient loss is used to constrain the spatial details of the generated image to maintain consistency with the ground truth image. For example, the spatial gradient loss is expressed as:

[0190] ;

[0191] Where: L gradent Represents the spatial gradient loss, ∇I HR (b) Indicates the generated image I HR The gradient in the b-th band, ∇I GT (b) represents the gradient of the real image in band b; || ||1 represents the L1 norm, used to calculate the absolute value of the gradient difference.

[0192] In some implementations, the loss during training of the conditional generative adversarial network includes adversarial loss, spectral loss, and spatial gradient loss, which can be a weighted sum of the adversarial loss, spectral loss, and spatial gradient loss to obtain the total loss. For example, it can be expressed as:

[0193] ;

[0194] Where: λ1 represents the weighting coefficient of the adversarial loss, λ2 represents the weighting coefficient of the spectral loss, and λ3 represents the weighting coefficient of the spatial gradient loss. The weighting coefficients λ1, λ2, and λ3 are used to balance the influence of different loss terms.

[0195] Through adversarial training and loss function optimization, generator G is able to generate high-resolution images while maintaining spectral consistency and the realism of spatial structure.

[0196] In this embodiment, the multimodal image fusion method generates a high-resolution image by using deformation field-guided super-resolution reconstruction after registration, and combines multi-loss joint optimization to ensure the consistency of multi-view and multi-spectral information.

[0197] ① Deformation field-guided upsampling was employed to resolve the issue of inconsistent resolution in the fusion results. A conditional generative adversarial network (cGAN) was used to process low-resolution multispectral and visible light images through deformation field processing, generating a high-resolution image. The generator's input included not only the low-resolution image but also deformation field weights and an affine transformation matrix, ensuring that the spatial structure and spectral information of the generated image were consistent with the real image.

[0198] ② To overcome the problem of spectral distortion during the fusion process, this method designs spectral fidelity constraints to avoid distortion of spectral information after fusion. During image generation, spectral angle loss and spatial gradient loss are introduced to maintain the spectral consistency and spatial structure accuracy of the generated image. Spectral angle loss ensures the fidelity of the generated image in terms of spectral features, avoiding errors caused by the loss of spectral information during reconstruction; while spatial gradient loss helps to preserve the fidelity of spatial details in the generated image, especially in multi-view scenarios, maintaining the edge and texture details of the image.

[0199] The multimodal image registration and fusion method in this embodiment has the following advantages:

[0200] (1) Improved the accuracy of multi-view image registration.

[0201] In existing technologies, traditional registration methods (such as feature-point-based matching methods) are prone to matching errors when faced with images from different viewpoints due to differences in viewpoint, illumination variations, and noise. While deep learning-based registration methods can perform end-to-end learning, they often lack geometric constraints on viewpoint transformations and struggle to handle the complexities brought about by these changes. This invention combines the advantages of traditional geometric registration with deep learning. Specifically, it first performs initial alignment through coarse geometric registration, followed by fine registration using Transformer. Transformer's cross-attention mechanism and epipolar geometric loss effectively capture the global spatial relationships between images from multiple viewpoints, further optimizing the registration results. Therefore, this invention can accurately align images from different viewpoints, avoiding mismatches and registration errors in traditional methods, thereby significantly improving the accuracy of multi-view registration.

[0202] (2) Enhanced spectral and spatial consistency between multispectral and visible light images.

[0203] Existing methods often struggle to ensure spectral consistency between multispectral and visible light images, especially during high-resolution reconstruction. This invention, through deformation field-guided upsampling and spectral fidelity constraints, generates high-resolution images that maintain spectral and spatial consistency, avoiding errors caused by spectral information loss or spatial distortion during reconstruction. This approach not only improves image quality but also ensures more natural and realistic fusion of multimodal images.

[0204] (3) Enhanced the registration accuracy and stability of multispectral images.

[0205] In existing technologies, due to differences in spectral and radiometric characteristics across different bands, traditional registration methods are prone to mismatches or registration failures when processing multispectral images. This invention, however, enhances the consistency of feature points in multispectral images through adaptive Gaussian filtering and cross-band gradient weighted fusion strategies. This results in higher stability of feature points across different bands, reducing the risk of mismatches and improving the registration accuracy and stability of multispectral images.

[0206] The above provides a detailed description of the multimodal image registration and fusion method provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of this invention.

Claims

1. A multimodal image registration method, characterized in that, The multimodal imagery includes a visible light image and at least two spectral images. The visible light image is an image generated by acquiring visible light from the target. The at least two spectral images correspond to at least two wavelengths, and any one of the spectral images is an image generated by acquiring light of the corresponding wavelength from the target. The multimodal image registration method includes: The gradient maps of the at least two spectral images are obtained separately, and the gradient maps of the at least two spectral images are fused to obtain a fused gradient map; For each of the at least two spectral images, extreme points are detected in the spectral image to obtain a first feature point, and a first descriptor of the first feature point is generated based on the fused gradient map. Extreme points are detected in the visible light image to obtain second feature points, and a second descriptor for the second feature points is generated based on the visible light image. Based on the first descriptor of the first feature point and the second descriptor of the second feature point, the first feature point and the second feature point are registered to register the visible light image and the at least two spectral images.

2. The multimodal image registration method according to claim 1, characterized in that, The gradient maps of the at least two spectral images are fused to obtain a fused gradient map, which includes: The preset products of corresponding pixels in the at least two spectral images are summed to obtain the value of the corresponding pixel in the fused gradient map. The preset product of corresponding pixels in any spectral image is the product of the gradient value of the corresponding pixel in any spectral image and the weight coefficient of the corresponding pixel.

3. The multimodal image registration method according to claim 1, characterized in that, For each of the at least two spectral images, detecting extreme points in the spectral image to obtain the first feature point includes: For any one of the at least two spectral images, obtain a multi-scale image of the spectral image and detect extreme points in the multi-scale image of the spectral image. From the extreme points of the at least two spectral images, select the extreme points that consistently meet the preset conditions in the at least two spectral images as the first feature point.

4. The multimodal image registration method according to claim 3, characterized in that, Selecting the first feature point from the extreme points of the at least two spectral images that consistently meet a preset condition includes: If the same extreme point is detected in at least two spectral images, or in a predetermined number of spectral images, then the extreme point is taken as the first feature point.

5. The multimodal image registration method according to claim 1, characterized in that, Before generating the first descriptor of the first feature point based on the fused gradient map, the method further includes: For any one of the at least two spectral images, obtain a multi-scale image of the spectral image and perform difference on any two adjacent layers in the multi-scale image of the spectral image to obtain a scale space difference image and calculate the gradient of the scale space difference image. For the first feature point, the contrast of the first feature point is obtained based on the response value of the first feature point in the scale space difference image of the spectral image and the gradient vector of the first feature point. If the contrast of the first feature point meets the preset requirements, the first feature point is removed.

6. The multimodal image registration method according to claim 1, characterized in that, Before generating the first descriptor of the first feature point based on the fused gradient map, the method further includes: For any one of the at least two spectral images, obtain a multi-scale image of the spectral image and perform difference on any two adjacent layers in the multi-scale image of the spectral image to obtain a scale space difference image. For the first feature point, according to the preset matrix of the first feature point in the scale space difference image of the spectral image, if it is determined that the first feature point is located in the edge region of the spectral image, the first feature point is removed. The preset matrix is ​​formed by the second-order partial derivatives and mixed partial derivatives of the first feature point in the x-direction and y-direction of the scale space difference image of the spectral image.

7. The multimodal image registration method according to claim 1, characterized in that, Generating a first descriptor for the first feature point based on the fused gradient map includes: For the gradient map of any one of the at least two spectral images, in the gradient map of this spectral image, the neighborhood of the first feature point is divided into multiple sub-regions, and multiple directional gradient histograms are obtained for each sub-region. Based on the multiple directional gradient histograms of the multiple sub-regions, the descriptor of the first feature point in this spectral image is obtained. The first feature point is obtained by stitching together the descriptors of the first feature point in the at least two spectral images.

8. The multimodal image registration method according to claim 1, characterized in that, Registration of the first feature point and the second feature point includes: An affine transformation matrix is ​​obtained based on the first feature point and the second feature point, and the elements of the affine transformation matrix are encoded into vectors to obtain an affine matrix encoding. The affine transformation matrix describes the positional transformation relationship between the first feature point in the spectral image and the second feature point corresponding to the first feature point in the visible light image. A position code is generated based on the position of the first feature point / second feature point using sine and cosine functions; The position encoding of the first feature point / second feature point and the affine matrix encoding are concatenated to generate the position awareness of the first feature point / second feature point, so as to register the first feature point and the second feature point based on the position awareness of the first feature point / second feature point.

9. The multimodal image registration method according to claim 1, characterized in that, Registration of the first feature point and the second feature point includes: The first feature point and the second feature point are input into a preset network. The preset network is used to output the position residual of the first feature point and the relative displacement of the second feature point relative to the first feature point, so as to register the first feature point and the second feature point according to the position residual of the first feature point and the position of the second feature point after adjustment according to the relative displacement. The preset network includes multiple self-attention layers and a feedforward network. Each self-attention layer learns the relationship between the first feature point and the second feature point. The feedforward network is used to further process the output information of the self-attention layer. The preset network defines a query vector Query, a key vector Key, and a value vector Value. The query vector Query is generated based on the first feature point, and the key vector Key and the value vector Value are generated based on the second feature point.

10. The multimodal image registration method according to claim 1, characterized in that, Registration of the first feature point and the second feature point includes: The first feature point and the second feature point are input into a preset model, and the preset model is used to output the registration result of the first feature point and the second feature point; The loss during training the preset model includes displacement residual loss, which represents the difference between the displacement residual predicted by the preset model and the true label, and / or the loss during training the preset model includes epipolar geometric loss, which constrains the geometric relationship between the first feature point and the second feature point.

11. A multimodal image fusion method, characterized in that, The multimodal imagery includes a visible light image and at least two spectral images. The visible light image is an image generated by acquiring visible light from the target. The at least two spectral images correspond to at least two wavelengths, and any one of the spectral images is an image generated by acquiring light of the corresponding wavelength from the target. The multimodal image fusion method includes: The visible light image and the at least two spectral images are registered using the multimodal image registration method according to any one of claims 1 to 10, and the registered visible light image and the at least two spectral images are then fused. The visible light image and the at least two spectral images are fused using a conditional generative adversarial network. The conditional generative adversarial network includes a generator and a discriminator. The generator's input is the fusion result of the spectral image, the deformation field, and the affine transformation matrix. The generator's output is the generated image. The deformation field is the result of deformation processing on the visible light image. The affine transformation matrix describes the positional transformation relationship between the first feature point in the spectral image and the second feature point in the visible light image corresponding to the first feature point.

12. The multimodal image fusion method according to claim 11, characterized in that, The loss during training of the conditional generative adversarial network includes a spectral loss, which describes the difference between the spectral feature vector of the generated image and the spectral feature vector of the real image. Or / and, the loss during training of the conditional generative adversarial network includes spatial gradient loss, which describes the difference between the gradient of the generated image and the gradient of the real image.

Images