Multi-band image registration fusion method and device

Abstract

The embodiment of the invention discloses a registration and fusion method and device for multiband images, which are used for realizing efficient registration and fusion of the multiband images. Comprising the steps that a first image group and a second image group to be registered are acquired, each image group comprises at least one image arranged based on a time sequence, the imaging wavebands of the images belonging to the same image group are the same, and at least part of the imaging wavebands of the first image group and the second image group do not coincide; performing registration on the first image group and the second image group based on the spatial dimension to obtain the first image group and the second image group which are spatially aligned; performing registration on the first image group and the second image group which are spatially aligned based on the time dimension to obtain at least one image pair which comprises a first image and a second image; and inputting the image pair into an image fusion model of adjacent attention transformation, and performing feature fusion on the associated information and the complementary information between the first image and the second image through the image fusion model to obtain a fused image of the image pair.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method and device for registration and fusion of multi-band images. Background Art

[0002] In the field of image processing, for the same image acquisition area, the images collected by different devices are often different. For example, different devices collect images in different bands, and images of different imaging bands such as visible light images and infrared images can be obtained. In addition, there are also differences in the acquisition angles and image timestamps of different devices, which makes the temporal and spatial characteristics of images collected by different devices different, resulting in poor multi-band image fusion effects.

[0003] How to achieve efficient registration and fusion of multi-band images is the technical problem to be solved by this application. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method and device for registration and fusion of multi-band images, so as to realize efficient registration and fusion of multi-band images.

[0005] In a first aspect, a multi-band image registration and fusion method is provided, comprising: Acquire a first image group to be registered and a second image group to be registered, wherein any image group contains at least one image arranged based on time sequence, the images belonging to the same image group have the same imaging band, and there is at least a partial non-overlap between the imaging band of the first image group and the imaging band of the second image group; Performing registration on the first image group and the second image group based on a spatial dimension to obtain a first image group and a second image group that are spatially aligned; Performing registration on the spatially aligned first image group and the second image group based on the time dimension to obtain at least one image pair, wherein any of the image pairs includes a first image and a second image aligned based on the spatial dimension and the time dimension, the first image belongs to the first image group, and the second image belongs to the second image group; The at least one image pair is input into an image fusion model to perform feature fusion on the association information and complementary information between the first image and the second image through the image fusion model to obtain a fused image of the image pair, and the image fusion model is a neighboring attention transformation model.

[0006] In a second aspect, a multi-band image registration and fusion device is provided, comprising: an acquisition module, acquiring a first image group to be registered and a second image group to be registered, wherein any image group contains at least one image arranged based on a time sequence, the images belonging to the same image group have the same imaging band, and there is at least a partial non-overlap between the imaging band of the first image group and the imaging band of the second image group; A spatial registration module, performing registration on the first image group and the second image group based on a spatial dimension to obtain a first image group and a second image group that are spatially aligned; a time registration module, performing registration on the spatially aligned first image group and the second image group based on the time dimension to obtain at least one image pair, wherein any of the image pairs includes the first image and the second image aligned based on the spatial dimension and the time dimension, the first image belongs to the first image group, and the second image belongs to the second image group; A fusion module inputs the at least one image pair into an image fusion model to perform feature fusion on the association information and complementary information between the first image and the second image through the image fusion model to obtain a fused image of the image pair, wherein the image fusion model is a neighboring attention transformation model.

[0007] According to a third aspect, an electronic device is provided. The electronic device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the method according to the first aspect are implemented.

[0008] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method according to the first aspect are implemented.

[0009] In a fifth aspect, a computer program product is provided, the computer program product comprising a non-transitory computer-readable storage medium storing a computer program, the computer program being operable to cause a computer to execute part or all of the steps of the method of the first aspect.

[0010] In an embodiment of the present application, first, a first image group to be registered and a second image group to be registered are obtained, wherein any image group contains at least one image arranged based on time sequence, the images belonging to the same image group have the same imaging band, and there is at least partial non-overlap between the imaging band of the first image group and the imaging band of the second image group. Among them, images of different imaging bands can show different image details from different bands. Then, the first image group and the second image group are registered based on the spatial dimension to obtain the first image group and the second image group that are spatially aligned. Next, the first image group and the second image group that are spatially aligned are registered based on the time dimension to obtain at least one pair of image pairs, wherein any image pair includes the first image and the second image that are aligned based on the spatial dimension and the time dimension, the first image belongs to the first image group, and the second image belongs to the second image group. Among them, performing time alignment after spatial alignment can effectively improve the effectiveness of image alignment. At least one image pair is input into an image fusion model to perform feature fusion on the association information and complementary information between the first image and the second image through the image fusion model to obtain a fused image of the image pair, and the image fusion model is a neighboring attention transformation model. The neighboring attention transformation model combines the local invariance of neighboring attention and the large receptive field of the transformation model. Through the image fusion model, it can effectively fuse the spatiotemporally aligned image pairs and optimize the image fusion effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1a This is one of the flow charts of a method for registration and fusion of multi-band images according to an embodiment of the present application; Figure 1b It is a schematic diagram of window attention and neighborhood attention of a multi-band image registration and fusion method according to an embodiment of the present application; Figure 2a This is the second flow chart of a multi-band image registration and fusion method according to an embodiment of the present application; Figure 2b It is a schematic diagram of registering a visible light image and an infrared light image in a multi-band image registration and fusion method according to an embodiment of the present application; Figure 2c It is a schematic diagram of modality conversion of a multi-band image registration and fusion method according to an embodiment of the present application; Figure 3a This is a third flow chart of a multi-band image registration and fusion method according to an embodiment of the present application; Figure 3b It is a schematic diagram of the result of masking and dilation operations on the first image of the multi-band image registration and fusion method according to an embodiment of the present application; Figure 4a This is a fourth flow chart of a multi-band image registration and fusion method according to an embodiment of the present application; Figure 4b This is a schematic diagram of forward-backward difference calculation of the Lucas-Kanade optical flow method of a multi-band image registration and fusion method in one embodiment of the present application; Figure 5a This is a fifth flow chart of a multi-band image registration and fusion method according to an embodiment of the present application; Figure 5b It is a feature fusion schematic diagram of a multi-band image registration and fusion method according to an embodiment of the present application; Figure 6a This is the sixth flow chart of a multi-band image registration and fusion method according to an embodiment of the present application; Figure 6b It is a structural schematic diagram of an image fusion model of a multi-band image registration and fusion method according to an embodiment of the present application; Figure 7 It is a structural schematic diagram of a multi-band image registration and fusion device according to an embodiment of the present application. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application. The numbering of the drawings in this application is only used to distinguish the various steps in the scheme, and is not used to limit the execution order of the various steps. The specific execution order is subject to the description in the specification.

[0013] In the field of image acquisition, image acquisition can be achieved through different bands. For example, visible light imaging is the most common imaging method. The resulting image can provide the observer with a clear and intuitive description of the scene. Visible light images conform to the visual characteristics of the human eye, but the imaging effect is poor in dark light, strong light, overexposure, underexposure, rain, fog and other conditions. Infrared images detect the thermal radiation of objects and are sensitive to temperature. They have good thermal target detection capabilities, strong anti-interference capabilities, and can work in harsh environments. However, the images formed by infrared sensors are generally blurred, the resolution is not high, and compared with visible light images, the texture of the imaged target is weaker, resulting in poor human eye perception and machine perception.

[0014] Considering the imaging characteristics and complementary properties of infrared and visible light images, by using the collaborative imaging and fusion of the two, it is possible to obtain fused images with prominent targets and rich background information in dark light, strong light, overexposure, underexposure, rain, fog, etc., thereby improving the accuracy of subsequent target detection, recognition, and tracking. However, there are many shortcomings in using multiple sensors to describe an actual scene. For example, compared with the single-modal sensor method, the data of multiple sensors is huge and redundant. In order to solve the problems caused by the joint use of multiple sensors and retain their advantages, people have proposed image fusion technology.

[0015] Image fusion technology aims to fuse the effective information in different modal images of the same scene obtained by multiple sensors to generate a fused image. Ideally, the fused image can contain complementary effective information from each modality, discard redundant information, accurately and comprehensively describe the actual scene, and effectively alleviate the problem of increased data volume. The fused image can also improve the accuracy of subsequent perception tasks. Therefore, this technology has been widely used in practical engineering fields such as forest fire prevention, agricultural monitoring, traffic monitoring and aerial reconnaissance.

[0016] However, to fuse visible light images with infrared images, it is required that the visible light images and infrared images are aligned. In the currently applicable dual-spectrum (visible light and infrared dual-spectrum) cameras, visible light images and infrared images often come from different sensors. Among them, different sensors are located in different positions, and it is difficult to ensure that the angles, positions, and focus of the images taken by different sensors are consistent, which often leads to the images being captured not being aligned in the spatial dimension.

[0017] In the time dimension, although the visible light and infrared images can be synchronized in time, this synchronization will also accumulate errors. In addition, network instability will also lead to time asynchrony. The visible light and infrared images received by the back-end algorithm are usually not aligned in time.

[0018] The misalignment between the spatial dimension and the temporal dimension often results in the fused image being unable to accurately express the features of the same scene at the same time, resulting in poor fused image effects.

[0019] In order to solve the problems existing in the related art, the present application embodiment provides a multi-band image registration and fusion method, such as Figure 1a As shown, including: S11: Acquire a first image group to be registered and a second image group to be registered, wherein any image group contains at least one image arranged based on time sequence, the images belonging to the same image group have the same imaging band, and there is at least partial non-overlap between the imaging band of the first image group and the imaging band of the second image group.

[0020] The first image group and the second image group may be acquired by the same image device or by different devices, and each image group contains at least one image. In practical applications, the first image group and the second image group may be images acquired from the same area during the same period of time. The imaging band of the first image group and the imaging band of the second imaging group at least partially do not overlap, for example, the imaging band of the first image group is the visible light band, and the imaging band of the second image group is the infrared band.

[0021] The images included in any of the above image groups are arranged based on time sequence, and the time intervals between multiple images can be uniform or non-uniform. In practical applications, the image group can be, for example, multiple image frames in a video.

[0022] S12: Perform registration on the first image group and the second image group based on a spatial dimension to obtain a first image group and a second image group that are spatially aligned.

[0023] In the solution provided in the embodiment of the present application, the images of the first image group and the second image group are registered. Specifically, the registration is performed based on the features contained in the images, by extracting the significantly distinguishable features of the same name between the image pairs, and using the correspondence between these features of the same name to perform feature matching to determine the paired images.

[0024] In this step, the first image group and the second image group are first registered from the spatial dimension to ensure that the images present the content of the same area. In practical applications, the image may contain moving objects, which may affect the accuracy of spatial registration. Based on this, in this step, the registration can be performed based on the fixed area in the image. For example, in a road scene, a vehicle may be driving on the road, and in the step of spatial registration, the registration can be performed based on the unchanged environmental area in the road. For example, spatial registration is performed based on the area where road markers such as green belts and driving lanes are located, which can effectively improve the accuracy of spatial registration.

[0025] S13: Perform registration on the spatially aligned first image group and the second image group based on the time dimension to obtain at least one image pair, wherein any image pair includes a first image and a second image aligned based on the spatial dimension and the time dimension, the first image belongs to the first image group, and the second image belongs to the second image group.

[0026] In actual application scenarios, visible light and infrared images can be acquired through commercial dual-lens dual-spectrum cameras. Different lenses acquire images of different modalities. Although different lenses are time-synchronized, the problem of time asynchrony is still inevitable. In this regard, although time-asynchronous images can be registered, the position of the same object in the visible light and infrared images will be different after registration, thus affecting the subsequent image fusion effect. Specifically, the visible light features and infrared features of an object will appear in different positions, that is, an object will appear twice in the fused image.

[0027] To achieve image registration that is not aligned in time and space, it is first necessary to align the space of images taken by different lenses. Only when the spatial positions of different images are aligned first, the image registration in time is meaningful. Otherwise, the spaces of different images are not aligned, and there is no reference for time alignment. By performing spatial alignment first and then temporal alignment, the matching degree of the aligned image pairs can be effectively improved, thereby improving the effect of subsequent image fusion.

[0028] S14: Inputting the at least one image pair into an image fusion model to perform feature fusion on the association information and complementary information between the first image and the second image through the image fusion model to obtain a fused image of the image pair, wherein the image fusion model is a neighboring attention transformation model.

[0029] In this step, at least one image pair is input into the image fusion model, so that the features of the image pair are fused through the model, so that the output fused image can simultaneously contain the features of the two paired images. Specifically, the model can perform feature extraction and feature reconstruction on the input image, and these two processes can also be called encoding and decoding.

[0030] In the feature fusion stage, the required feature fusion can be achieved based on the feature fusion of Convolutional Neural Networks (CNN), the feature fusion based on self-attention, and the feature fusion based on cross-attention. Among them, due to the large difference in feature distribution between infrared images and visible light images, if the feature fusion based on CNN or self-attention is used to directly fuse the visible light and infrared image features through CNN or transformer, it will often lead to information loss and information confusion. The feature fusion based on cross-attention can usually only focus on the relevant information of the image features of different modalities, and does not focus on the irrelevant features (such as complementary features) of the image features of different modalities.

[0031] In terms of model structure, the CNN structure can only obtain a local receptive field. In this embodiment, the Transformer model is applied, which has a longer range of receptive fields. However, the Transformer model does not have local invariance. In this regard, in the solution provided in the embodiment of the present application, the Transformer model is combined with neighboring attention, so that the image fusion model has both a longer range of receptive fields and local invariance.

[0032] Figure 1b A schematic diagram of window attention and neighborhood attention is shown. In the implementation of information extraction, window attention can be used to extract information over a large range, wherein attention is performed on all feature points in the window and feature points in a fixed window. In the solution provided in the embodiment of the present application, a neighborhood-invariant attention mechanism is used to pay attention to feature points in the nearest neighboring area of ​​any point, thereby ensuring local invariance.

[0033] In the transformer structure, the feedforward neural network structure plays the role of channel information mixing, but the feedforward neural network structure of the basic neighborhood attention mechanism does not have the ability to mix local information. In this regard, the image fusion model of the embodiment of the present application has a multi-scale feedforward neural network structure with local information mixing. Specifically, the model divides the input features into two parts, and then inputs the two features into different convolution modules for processing. Subsequently, the features processed by the convolution module are superimposed, and then convolution is used for information mixing.

[0034] Through the solution provided in the embodiment of the present application, the image fusion model performs feature extraction and reconstruction based on the transformer of the nearest neighbor attention, which has both a large receptive field and local invariance. Among them, the complete attention mechanism is adopted, so that when the features are fused, both relevant information and complementary information are paid attention to, thereby obtaining a better fusion effect.

[0035] In an embodiment of the present application, the first image group and the second image group have different imaging bands, and images of different imaging bands can show different image details from different bands. Then, spatial dimension alignment and temporal dimension alignment are performed on the first image group and the second image group to improve the effectiveness of image alignment and obtain at least one image pair. At least one image pair is input into an image fusion model to perform feature fusion on the correlation information and complementary information between the first image and the second image through the image fusion model to obtain a fused image of the image pair. The image fusion model is a neighboring attention transformation model. The neighboring attention transformation model has both the local invariance of neighboring attention and the large receptive field of the transformation model. Through the image fusion model, the image pairs aligned in time and space can be effectively fused to optimize the effect of image fusion.

[0036] Based on the solution provided in the above embodiment, optionally, Figure 2a As shown, in the above step S12, the first image group and the second image group are registered based on the spatial dimension to obtain the spatially aligned first image group and the second image group, including: S21: Performing a modality conversion process on the images in the first image group based on the imaging style of the second image group.

[0037] Among them, due to the large modality difference between visible light images and infrared images, there are fewer image matching points, which makes it difficult to align them later, affecting the accuracy of image matching. In this regard, in the solution provided in the embodiment of the present application, a modality conversion process is performed on the images in the first image group, so that the images in the converted first image group can present the same imaging style as the second image group. In addition, the converted first image group and the second image group have a higher similarity, narrowing the modality difference between the visible light image and the infrared image, thereby improving the effectiveness of image registration.

[0038] S22: performing registration on the first image group and the second image group after the modality conversion processing based on the spatial dimension to obtain the first image group and the second image group that are spatially aligned.

[0039] Figure 2b A schematic diagram of registering a visible light image with an infrared light image is shown. The upper left image is a first image of visible light, the upper right image is a second image of infrared light, and the matching feature points of direct registration are shown as hollow circles connected by line segments. The lower left image is obtained by modal conversion of the upper left first image based on the imaging style of the second image. Registering the first image after modal conversion with the second image shown in the lower right can significantly improve the similarity between the images, increase the number of matching feature points, and thus improve the accuracy of spatial registration.

[0040] Figure 2cA schematic diagram of modal conversion is shown. In this example, a first image of visible light is input into a modal conversion neural network to obtain a first image of modal conversion output by the model. The modal conversion neural network can be a pre-trained neural network, which can be trained using image sample pairs including visible light images and infrared images.

[0041] The imaging styles of the modally converted first image and the second image are closer, so that the number of matching points is greater, which is conducive to achieving spatial dimension alignment between images based on matching feature points.

[0042] In this example, performing the modal conversion can determine the matching feature points between the first image and the second image, and based on the correspondence between these feature points, the first image and the second image are spatially registered, thereby obtaining a spatially aligned first image group and a second image group. It should be noted that the above-mentioned first image after modal conversion is only used to determine the matching feature points between the first image and the second image, and the subsequently obtained spatially aligned first image group contains the first image before the modal conversion, that is, the visible light image.

[0043] Through the solution provided in the embodiment of the present application, the similarity between the first image and the second image is determined by modal conversion, and then a larger number of matching feature points are determined, thereby improving the effectiveness of spatial registration and optimizing the quality of spatial alignment between the first image group and the second image group.

[0044] Based on the solution provided in the above embodiment, optionally, Figure 3a As shown, in the above step S13, the spatially aligned first image group and the second image group are registered based on the time dimension to obtain at least one pair of images, including: S31: performing background separation processing on each image in the spatially aligned first image group and each image in the second image group respectively to obtain a first background area corresponding to the image.

[0045] In the embodiment of the present application, the spatially aligned first image group and the second image group are further time aligned. In this step, background separation processing is performed on each image in the first image group and the second image group to determine the first background area corresponding to each image.

[0046] In practical applications, the image may contain moving people, vehicles, etc., and also contain non-moving environmental backgrounds such as roads. For two images taken at the same time, there are also differences in the presentation of moving objects in the two images. In this step, the first background area is extracted from each image through background separation processing. On the one hand, the extracted first background area is affected by the moving object. As the object moves, the first background area obtained by the background separation processing will also change accordingly, that is, the first background area can reflect the characteristics of the foreground moving object to a certain extent. On the other hand, the background contains objects that are not easy to move. Temporal alignment based on the first background area can help improve the efficiency of temporal alignment and avoid the problem of difficulty in temporal alignment due to the change of the moving object in a short time.

[0047] S32: performing mask dilation processing on the foreground of the image based on the first background area to obtain a second background area corresponding to the image.

[0048] In this step, based on the first background area obtained in the above step, the foreground of each image is subjected to mask dilation processing to obtain the second background area corresponding to each image. The mask dilation processing can expand the foreground area in the image and reduce the features presented by the moving object in the image, so that the second background area can reduce the details of the foreground object while retaining a certain degree of influence of the moving object on the background, which is conducive to improving the efficiency of time matching.

[0049] S33: Perform registration on the second background region of the image in the first image group and the second background region of the image in the second image group that are spatially aligned based on the time dimension to obtain at least one image pair.

[0050] In this step, the first image group or the second image group can be selected as the reference image group, an image to be registered can be determined from the reference image, and registration can be performed on each image in the other image group based on the image to be registered, and the images with the highest similarity can be determined as image pairs.

[0051] The image pair includes a first image and a second image, and the second background region of the second image has a high similarity with the second background region of the first image. The background region is used to achieve similarity comparison between images, so as to determine the closest first image and second image to form an image pair.

[0052] The solution provided in the embodiment of the present application can extract appropriate features from the image through background separation and foreground mask expansion. On the one hand, it retains a certain degree of foreground object features, and on the other hand, it avoids the foreground object edge details from excessively affecting the registration efficiency, thereby improving the efficiency of image time alignment.

[0053] Specifically, in order to avoid the negative impact of object movement on image registration, this solution uses a mask to filter out moving objects in the image. The moving objects need to be filtered out first. To filter out the moving objects, the moving objects can be masked. Figure 3b Schematic diagram showing the result of masking and dilation operation on the first image. For example, the Gaussian mixture model background separation algorithm (MOG2 algorithm) can be used to separate the first image ( Figure 3b Left) performs background extraction and obtains the mask image ( Figure 3b ), the mask image can be obtained by inverting the foreground mask. Then, to avoid insufficient background extraction, a 3*3 dilation morphological operation is performed on the first image based on the first background area to obtain a mask image after the dilation operation ( Figure 3b In comparison, the mask image after the dilation operation removes the detailed features of the foreground edge in the image, reducing the negative impact of the moving object edge on the image temporal registration.

[0054] Based on the solution provided in the above embodiment, optionally, Figure 4a As shown, in the above step S13, the spatially aligned first image group and the second image group are registered based on the time dimension to obtain at least one pair of images, including: S41: Determine a first image to be temporally registered from a group of spatially aligned first images.

[0055] In this step, the first image to be temporally registered is determined from the spatially aligned first image group. Optionally, first images that have not yet been constructed into an image pair are selected in sequence along the time sequence in the first image group, so that the most similar second image is selected from the second image group in a subsequent step to form an image pair. The spatially aligned first image group and the second image group can be pre-stored in a cache so as to be called when performing temporal registration.

[0056] S42: Calculate similarity between the first image to be temporally registered and each second image in the group of spatially aligned second images based on an optical flow method.

[0057] In this step, the optical flow method is used to perform temporal registration. For example, assume that the first image group after spatial alignment includes images taken continuously within 10 seconds. Then, the middle first image (for example, the image at the 5th second) can be taken out in time sequence as the first image to be registered. Subsequently, each second image is taken out from the second image group, and the forward-backward difference is calculated based on the optical flow method to determine the difference between each second image and the first image to be registered.

[0058] Figure 4bThe figure shows a schematic diagram of the forward-backward difference calculation of the Lucas-Kanade optical flow method, where d is the forward-backward error, which is determined based on the difference between the forward trajectory and the backward trajectory.

[0059] S43: constructing the second image with the highest similarity and the first image to be temporally registered into an image pair.

[0060] Based on the difference calculated in the above steps, it can be directly determined that the second image with the smallest difference matches the first image to be registered, and then the matching first image and second image are stored in the cache.

[0061] Optionally, after determining the second image with the smallest difference with the first image, it is further determined whether the difference is less than a preset difference, and if it is less than the preset difference, the second image with the smallest difference and the first image are constructed as an image pair. If the difference is greater than or equal to the preset difference, it is determined that there is no second image in the second image group that matches the first image to be registered, and other first images in the first image group are selected to continue to try temporal registration.

[0062] In an application scenario, if the first image group and the second image group are continuous images taken evenly in the same period of time, the images of the first image group and the second image group can be paired sequentially in chronological order based on the 5th second image pair determined through the above steps to efficiently achieve time alignment of the first image group and the second image group.

[0063] Based on the solution provided in the above embodiment, optionally, the image fusion model includes a cross-attention CA module and a spatially embedded single-head channel cross-attention SSCA module.

[0064] Among them, CA refers to cross-attention, and SSCA refers to Spatial-embedded single-headchannel-cross attention.

[0065] Among them, the CA module is used to fuse the associated information in the input features, and the SSCA module is used to fuse the complementary information in the input features.

[0066] Among them, Figure 5a As shown, in the above step S14, the at least one image pair is input into the image fusion model, so as to perform feature fusion on the association information and complementary information between the first image and the second image through the image fusion model to obtain a fused image of the image pair, including: S51: Input the at least one image pair into the image fusion model to perform feature fusion on the association information between the first image and the second image through the CA module, and to perform feature fusion on the complementary information between the first image and the second image through the SSCA module, so as to obtain a fused image of the image pair.

[0067] In this embodiment, the images are input into the image fusion model in pairs, the image fusion model performs feature extraction on the input images, and at least part of the extracted features are input into the CA module, and at least part of the extracted features are input into the SSCA module. The features input into the CA module and the features input into the SSCA module may be different. The associated features of the two input images are fused through the CA module, and the complementary features of the two input images are fused through the SSCA module, and then the features processed by the CA module and the SSCA module are further superimposed and fused through the convolution module to obtain a fused image that contains both the associated feature fusion results of the image pair and the complementary feature fusion results.

[0068] Feature fusion is an important step in image fusion. Unlike the previous use of CNN and self-attention for feature fusion, which easily leads to information loss and confusion, the use of cross-attention (CA) for feature fusion only focuses on the relevant information of different modal images and ignores complementary information. The solution provided in the embodiment of the present application adopts a complete attention mechanism, so that when fusing features, it pays attention to both relevant information and complementary information, thereby obtaining a better fusion effect.

[0069] like Figure 5b As shown in the figure, the upper left (a) shows the feature fusion based on CNN, the A features and B features are merged through MERGE, and the fused features are obtained through the convolution module.

[0070] The upper right (b) shows the feature fusion based on the self-attention mechanism. First, the A features and B features are merged through MERGE, and then the self-attention SA (self-attention) feature fusion is performed based on Q (query), K (key) and V (value) to obtain the fused features. Among them, Q represents the query, which can determine the part that the model needs to pay attention to. K represents the key value, which indicates the importance of each part. V represents the value, which is represented by the numerical value of each part of the input and can be used for subsequent weighted summation.

[0071] The lower left (c) shows the feature fusion based on the cross-attention mechanism. The attention-based Q, K, and V are fused through the CA module, and then merged through MERGE to obtain the fused features.

[0072] The lower right corner (d) shows the feature fusion of the complete cross-attention mechanism provided in an embodiment of the present application. Based on Q, K and V of the attention mechanism, the features are split into the SSCA module and the CA module, and then the feature processing results output by the SSCA module and the CA module are merged through MERGE to obtain the fused features.

[0073] Based on the solution provided in the above embodiment, the optional Figure 6a As shown, the image fusion model includes multiple complete cross-attention CCA modules, wherein any CCA module includes a CA module and a SSCA module; Wherein, in the above step S51, the at least one image pair is input into the image fusion model, so as to perform feature fusion on the association information between the first image and the second image through the CA module, and perform feature fusion on the complementary information between the first image and the second image through the SSCA module, so as to obtain a fused image of the image pair, including: S61: Perform multi-dimensional feature segmentation on the first image and the second image to obtain multiple sub-dimensional features corresponding to the images.

[0074] In this step, feature extraction and splitting are performed on the first image and the second image to obtain multiple sub-dimensional features corresponding to the first image and multiple sub-dimensional features corresponding to the second image. Optionally, the input features are divided into two parts and input into the CA module and the SSCA module respectively.

[0075] S62: Randomly input multiple sub-dimensional features corresponding to the image into multiple CCA modules, and perform feature fusion on the feature information output by the multiple CCA modules through the convolution module.

[0076] In the image fusion model, multiple complete cross-attention CCA modules are included. Figure 6b The model structure is shown in the figure. The above CCA module can be connected to the CCA Block position shown in the figure. Among them, (a) shows the overall architecture of the model. (b) shows the structure of the neighboring attention conversion module, (c) shows the structure of the complete cross-attention CCA module, and (d) shows the complete cross-attention structure in the CCA module.

[0077] The image fusion model used in the embodiment of the present application is based on an end-to-end network of an iNAT (improve neighborhood attention) transformer structure and a CCA module. The transformer of a traditional neural network is usually stacked in large numbers, for example, each block is 4-12. The network of this patent contains fewer blocks and is a lightweight neural network.

[0078] Optionally, in the structure of the image fusion model provided in the embodiment of the present application, CCA modules are set at both the higher and lower layers of the model to perform feature fusion. Among them, high-level feature fusion can ensure fusion at the semantic layer, and low-level fusion can make up for information loss during feature reconstruction, so that the final fused image has more detailed information.

[0079] Through the solution provided in the embodiment of the present application, the relevant features and complementary features of the image pair are fused based on the CCA module, and then merged through the convolution module, so that the fused image has both the fused relevant features and complementary features.

[0080] Considering that related image registration technologies often only consider the spatial misalignment of visible light and infrared images, without considering that images are also misaligned in time in actual application scenarios, the solution provided in the embodiment of the present application uses background extraction, morphological operations, modal conversion, forward and backward difference calculation based on optical flow, and feature matching algorithms to achieve registration of visible light and infrared images in time and space.

[0081] In the process of feature extraction and reconstruction, related technologies often cannot simultaneously have a large receptive field and local invariance. The embodiment of the present application applies an optimized nearest neighbor attention-based transformer for feature extraction and reconstruction, which has both a large receptive field and local invariance.

[0082] In the process of feature fusion, if fusion is performed directly, it often leads to information loss and information confusion, or the problem of focusing on the relevant information of different modal images and ignoring the complementary information when fusion is performed based on the cross-attention mechanism. The embodiment of the present application uses a complete attention mechanism, which focuses on both relevant information and complementary information in the process of feature fusion, and can effectively improve the image fusion effect.

[0083] The solution provided by the embodiment of the present application can realize image registration in time and space and image fusion with complete attention, and can effectively solve the fusion problem of visible light and infrared images that are not aligned in time and space. By extracting the background of the image and avoiding the negative impact of moving objects on image registration, the robustness of image registration based on the time dimension can be effectively improved. In the process of mask expansion, multiple expansion operations can be performed until the result of the mask expansion eliminates the amplitude effect of foreground edge details on image registration, thereby improving the robustness of image registration to moving objects and background extraction. In addition, in the process of time registration, the visible light and infrared images that are not aligned in time can be effectively registered by calculating the forward-backward difference of the LK optical flow.

[0084] During the image registration process, the solution provided in the embodiment of the present application first converts the visible light image modality, then spatially aligns the converted visible light image and infrared image, and finally performs background extraction, multiple expansions, and forward-backward difference calculation of the LK optical flow on the spatially aligned image, which can effectively align images that are not aligned in time and space at the same time.

[0085] In terms of image fusion, the embodiment of the present application applies a transformer with neighborhood attention of a multi-scale feedforward neural network. The model has both local invariance and large-scale information extraction capabilities, effectively improving the ability to extract image features and fuse visible light and infrared images. The SSCA mechanism is applied in the model, which can pay attention to and fuse the complementary information of visible light and infrared images, and improve the ability to fuse visible light and infrared images. The model combines the cross-attention mechanism with the SSCA mechanism, and can simultaneously fuse the relevant information and complementary information of visible light and infrared images, and improve the ability to fuse visible light and infrared images. The model has many advantages such as lightweight, fast speed, and good image fusion effect.

[0086] In order to solve the problems existing in the related art, the present application embodiment also provides a multi-band image registration and fusion device 70, such as Figure 7 As shown, including: An acquisition module 71 is configured to acquire a first image group to be registered and a second image group to be registered, wherein any image group includes at least one image arranged based on a time sequence, the images belonging to the same image group have the same imaging band, and there is at least a partial non-overlap between the imaging band of the first image group and the imaging band of the second image group; A spatial registration module 72 performs registration on the first image group and the second image group based on a spatial dimension to obtain a spatially aligned first image group and a second image group; a time registration module 73, performing registration on the spatially aligned first image group and the second image group based on the time dimension to obtain at least one image pair, wherein any of the image pairs includes the first image and the second image aligned based on the spatial dimension and the time dimension, the first image belongs to the first image group, and the second image belongs to the second image group; The fusion module 74 inputs the at least one image pair into an image fusion model to perform feature fusion on the association information and complementary information between the first image and the second image through the image fusion model to obtain a fused image of the image pair, and the image fusion model is a neighboring attention transformation model.

[0087] Through the device provided by the embodiment of the present application, first, a first image group to be registered and a second image group to be registered are obtained, wherein any image group contains at least one image arranged based on time sequence, the imaging bands of the images belonging to the same image group are the same, and there is at least partial non-overlap between the imaging bands of the first image group and the imaging bands of the second image group. Among them, images of different imaging bands can show different image details from different bands. Then, the first image group and the second image group are registered based on the spatial dimension to obtain the first image group and the second image group that are spatially aligned. Next, the first image group and the second image group that are spatially aligned are registered based on the time dimension to obtain at least one pair of image pairs, wherein any image pair includes the first image and the second image that are aligned based on the spatial dimension and the time dimension, the first image belongs to the first image group, and the second image belongs to the second image group. Among them, performing time alignment after spatial alignment can effectively improve the effectiveness of image alignment. Input at least one image pair into the image fusion model to perform feature fusion on the association information and complementary information between the first image and the second image through the image fusion model to obtain a fused image of the image pair, and the image fusion model is a neighboring attention transformation model. The neighboring attention transformation model combines the local invariance of neighboring attention and the large receptive field of the transformation model. Through the image fusion model, it can effectively fuse the spatiotemporally aligned image pairs and optimize the image fusion effect.

[0088] Among them, the above modules in the device provided by the embodiment of the present application can also implement the method steps provided by the above method embodiment. Alternatively, the device provided by the embodiment of the present application can also include other modules in addition to the above modules to implement the method steps provided by the above method embodiment. And the device provided by the embodiment of the present application can achieve the technical effects that can be achieved by the above method embodiment.

[0089] Preferably, an embodiment of the present application also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, each process of the above-mentioned multi-band image registration and fusion method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0090] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the above-mentioned multi-band image registration and fusion method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0091] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. The computer program can be operated to enable a computer to execute part or all of the steps of the above-mentioned multi-band image registration and fusion method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0092] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0093] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate a device for implementing the functions specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0094] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0095] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0096] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0097] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0098] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0099] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0100] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0101] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A multi-band image registration and fusion method, characterized in that: include: Acquire a first image group to be registered and a second image group to be registered, wherein any image group contains at least one image arranged based on time sequence, the images belonging to the same image group have the same imaging band, and there is at least a partial non-overlap between the imaging band of the first image group and the imaging band of the second image group; Performing registration on the first image group and the second image group based on a spatial dimension to obtain a first image group and a second image group that are spatially aligned; Performing registration on the spatially aligned first image group and the second image group based on the time dimension to obtain at least one image pair, wherein any of the image pairs includes a first image and a second image aligned based on the spatial dimension and the time dimension, the first image belongs to the first image group, and the second image belongs to the second image group; The at least one image pair is input into an image fusion model to perform feature fusion on the association information and complementary information between the first image and the second image through the image fusion model to obtain a fused image of the image pair, and the image fusion model is a neighboring attention transformation model.

2. The method according to claim 1, characterized in that Performing registration on the first image group and the second image group based on a spatial dimension to obtain a spatially aligned first image group and a second image group includes: performing a modality conversion process on the images in the first image group based on the imaging style of the second image group; The first image group and the second image group after the modality conversion processing are registered based on the spatial dimension to obtain the first image group and the second image group that are spatially aligned.

3. The method according to claim 1, characterized in that Performing registration on the spatially aligned first image group and the second image group based on the time dimension to obtain at least one image pair includes: Performing background separation processing on each image in the spatially aligned first image group and each image in the second image group respectively to obtain a first background area corresponding to the image; Based on the first background area, a mask dilation process is performed on the foreground of the image to obtain a second background area corresponding to the image; Based on the time dimension, the second background region of the image in the first image group and the second background region of the image in the second image group that are spatially aligned are registered to obtain at least one image pair.

4. The method according to claim 1, characterized in that Performing registration on the spatially aligned first image group and the second image group based on the time dimension to obtain at least one image pair includes: determining a first image to be temporally registered from the set of spatially aligned first images; Performing similarity calculation based on an optical flow method on the first image to be temporally registered and each second image in a group of second images that are spatially aligned; The second image with the highest similarity and the first image to be temporally registered are constructed as an image pair.

5. The method according to any one of claims 1 to 4, characterized in that The image fusion model includes a cross-attention CA module and a spatially embedded single-head channel cross-attention SSCA module; The step of inputting the at least one image pair into an image fusion model to perform feature fusion on the association information and complementary information between the first image and the second image through the image fusion model to obtain a fused image of the image pair includes: The at least one image pair is input into the image fusion model to perform feature fusion on the association information between the first image and the second image through the CA module, and to perform feature fusion on the complementary information between the first image and the second image through the SSCA module, so as to obtain a fused image of the image pair.

6. The method according to claim 5, characterized in that The image fusion model includes multiple complete cross-attention CCA modules, wherein any CCA module includes a CA module and a SSCA module; The step of inputting the at least one image pair into the image fusion model, performing feature fusion on the association information between the first image and the second image through the CA module, and performing feature fusion on the complementary information between the first image and the second image through the SSCA module, to obtain a fused image of the image pair includes: Performing multi-dimensional feature segmentation on the first image and the second image to obtain a plurality of sub-dimensional features corresponding to the images; The multiple sub-dimensional features corresponding to the image are randomly input into the multiple CCA modules, and the feature information output by the multiple CCA modules is subjected to feature fusion through the convolution module.

7. A multi-band image registration and fusion device, characterized in that: include: an acquisition module, acquiring a first image group to be registered and a second image group to be registered, wherein any image group contains at least one image arranged based on a time sequence, the images belonging to the same image group have the same imaging band, and there is at least a partial non-overlap between the imaging band of the first image group and the imaging band of the second image group; A spatial registration module, performing registration on the first image group and the second image group based on a spatial dimension to obtain a first image group and a second image group that are spatially aligned; a time registration module, performing registration on the spatially aligned first image group and the second image group based on the time dimension to obtain at least one image pair, wherein any of the image pairs includes the first image and the second image aligned based on the spatial dimension and the time dimension, the first image belongs to the first image group, and the second image belongs to the second image group; A fusion module inputs the at least one image pair into an image fusion model to perform feature fusion on the association information and complementary information between the first image and the second image through the image fusion model to obtain a fused image of the image pair, wherein the image fusion model is a neighboring attention transformation model.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the method according to any one of claims 1 to 6 when executed by the processor.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a non-transitory computer-readable storage medium storing a computer program, the computer program being operable to cause a computer to perform the steps of the method according to any one of claims 1 to 6.

Images