Method and apparatus for registration fusion of multi-band images

Abstract

The embodiment of the application discloses a multi-band image registration fusion method and device, which is used for efficient registration and fusion of multi-band images. The method comprises the following steps: obtaining a first image group and a second image group to be registered, the image group containing at least one image arranged based on time sequence, the images belonging to the same image group having the same imaging band, and the imaging bands of the first image group and the second image group having at least partial non-overlapping; performing registration on the first image group and the second image group based on a spatial dimension to obtain a spatially aligned first image group and a spatially aligned second image group; performing registration on the spatially aligned first image group and the spatially aligned second image group based on a time dimension to obtain at least one image pair, the image pair comprising a first image and a second image; inputting the image pair into a neighboring attention transformation image fusion model, and performing feature fusion on the associated information and complementary information between the first image and the second image by the image fusion model to obtain a fusion image of the image pair.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular to a multi-band image registration fusion method and device. BACKGROUND

[0002] In the field of image processing, images collected by different devices for the same image collection area are often different. For example, different devices collect images at different bands, and different imaging band images such as visible light images and infrared images can be obtained. In addition, there are differences in the collection angle, image timestamp, etc. of different devices, so that the time characteristics and spatial characteristics of images collected by different devices are different, thereby resulting in poor multi-band image fusion effect.

[0003] How to efficiently register and fuse multi-band images is a technical problem to be solved by the present application. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a multi-band image registration fusion method and device to efficiently register and fuse multi-band images.

[0005] In a first aspect, a multi-band image registration fusion method is provided, comprising:

[0006] Obtaining a first image group to be registered and a second image group to be registered, wherein any image group contains at least one image arranged based on time sequence, the images belonging to the same image group have the same imaging band, and there is at least partial non-coincidence between the imaging band of the first image group and the imaging band of the second image group;

[0007] Performing registration on the first image group and the second image group based on spatial dimension to obtain spatially aligned first image group and second image group;

[0008] Performing registration on the spatially aligned first image group and second image group based on time dimension to obtain at least one image pair, wherein any image pair includes a first image and a second image aligned based on spatial dimension and time dimension, the first image belongs to the first image group, and the second image belongs to the second image group;

[0009] Inputting the at least one image pair into an image fusion model to perform feature fusion on the associated information and complementary information between the first image and the second image by the image fusion model to obtain a fusion image of the image pair, the image fusion model being a neighbor attention transformation model.

[0010] In a second aspect, a multi-band image registration fusion device is provided, comprising:

[0011] an acquisition module, configured to acquire a first image group to be registered and a second image group to be registered, wherein any of the image groups comprises at least one image arranged based on a time sequence, and images belonging to a same image group are of a same imaging band, and there is at least partial non-overlapping between an imaging band of the first image group and an imaging band of the second image group;

[0012] a spatial registration module, configured to perform registration on the first image group and the second image group based on a spatial dimension, to obtain a spatially aligned first image group and a spatially aligned second image group;

[0013] a temporal registration module, configured to perform registration on the spatially aligned first image group and the spatially aligned second image group based on a time dimension, to obtain at least one image pair, wherein any of the image pairs comprises a first image and a second image aligned based on the spatial dimension and the time dimension, the first image belonging to the first image group, and the second image belonging to the second image group;

[0014] a fusion module, configured to input the at least one image pair into an image fusion model, to perform feature fusion on associated information and complementary information between the first image and the second image by the image fusion model, to obtain a fused image of the image pair, and the image fusion model is a neighbor attention transformation model.

[0015] In a third aspect, an electronic device is provided, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and when the computer program is executed by the processor, the steps of the method according to the first aspect are implemented.

[0016] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the method according to the first aspect are implemented.

[0017] In a fifth aspect, a computer program product is provided, which includes a non-transitory computer readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform some or all of the steps of the method according to the first aspect.

[0018] In the embodiment of the present application, first, a first image group to be registered and a second image group to be registered are acquired, wherein any image group contains at least one image arranged based on time sequence, the images belonging to the same image group have the same imaging waveband, and there is at least partial non-overlapping between the imaging waveband of the first image group and the imaging waveband of the second image group. Wherein the images of different imaging wavebands can show different image details from different wavebands. Then, registration is performed on the first image group and the second image group based on spatial dimension, to obtain spatially aligned first image group and second image group. Next, registration is performed on the spatially aligned first image group and second image group based on time dimension, to obtain at least one image pair, wherein any image pair includes a first image and a second image aligned based on spatial dimension and time dimension, the first image belongs to the first image group, and the second image belongs to the second image group. Wherein the time alignment is performed after the spatial alignment, which can effectively improve the effectiveness of image alignment. At least one image pair is input into an image fusion model, to perform feature fusion on the associated information and complementary information between the first image and the second image through the image fusion model, to obtain a fusion image of the image pair, and the image fusion model is a neighbor attention transformation model. The neighbor attention transformation model has both the local invariance of neighbor attention and the large receptive field of transformation model, and through the image fusion model, effective fusion can be realized on the spatio-temporally aligned image pair, and the effect of image fusion is optimized. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are included to provide a further understanding of the present application and are incorporated in and constitute a part of this application, illustrate embodiments of the present application and together with the description serve to explain the present application. In the drawings:

[0020] Figure 1a is one of the flowcharts of the multi-waveband image registration and fusion method of an embodiment of the present application;

[0021] Figure 1b is a schematic diagram of window attention and neighborhood attention of the multi-waveband image registration and fusion method of an embodiment of the present application;

[0022] Figure 2a is another flowchart of the multi-waveband image registration and fusion method of an embodiment of the present application;

[0023] Figure 2b is a schematic diagram of registration of a visible light image and an infrared light image of the multi-waveband image registration and fusion method of an embodiment of the present application;

[0024] Figure 2c is a modal conversion schematic diagram of the multi-waveband image registration and fusion method of an embodiment of the present application;

[0025] Figure 3a is a flowchart of a multi-band image registration fusion method according to an embodiment of the present application;

[0026] Figure 3b is a result diagram of a first image mask and inflation operation according to a multi-band image registration fusion method of an embodiment of the present application;

[0027] Figure 4a is a flowchart of a multi-band image registration fusion method according to an embodiment of the present application;

[0028] Figure 4b is a forward-backward difference calculation diagram of a Lucas-Kanade optical flow method according to a multi-band image registration fusion method of an embodiment of the present application;

[0029] Figure 5a is a flowchart of a multi-band image registration fusion method according to an embodiment of the present application;

[0030] Figure 5b is a feature fusion diagram according to a multi-band image registration fusion method of an embodiment of the present application;

[0031] Figure 6a is a flowchart of a multi-band image registration fusion method according to an embodiment of the present application;

[0032] Figure 6b is a structure diagram of an image fusion model according to a multi-band image registration fusion method of an embodiment of the present application;

[0033] Figure 7 is a structure diagram of a multi-band image registration fusion device according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application. The figure numbers in the present application are only used to distinguish each step in the scheme, and are not used to limit the execution order of each step, and the specific execution order is subject to the description in the specification.

[0035] In the field of image acquisition, image acquisition can be realized through different wave bands. For example, visible light imaging is the most common imaging method, and the obtained image can provide a clear and intuitive scene description for the observer. The visible light image conforms to the visual characteristics of the human eye, but the imaging effect is poor in the case of dark light, strong light, overexposure, underexposure, rain, fog, etc. An infrared image is formed by detecting the thermal radiation of an object, is sensitive to temperature, has good thermal target detection capability, and has strong anti-interference ability, and can work in harsh environments. However, the image formed by the infrared sensor is overall fuzzy, has low resolution, and the texture of the imaged target is weak compared to the visible light image, resulting in poor human and machine perception effect.

[0036] Considering the imaging characteristics and complementary characteristics of infrared and visible light images, using both for cooperative imaging and fusion can obtain a fusion image with prominent target and rich background information in the case of dark light, strong light, overexposure, underexposure, rain, fog, etc., thereby improving the accuracy of subsequent target detection, recognition and tracking. However, there are many shortcomings in using multiple sensors to describe an actual scene, for example, compared with a single modality sensor, the data of multiple sensors is large and redundant. In order to solve the problems brought by the joint use of multiple sensors and retain their advantages, people have proposed image fusion technology.

[0037] Image fusion technology aims to fuse the effective information in different modal images of the same scene obtained by multiple sensors to generate a fusion image. In an ideal state, the fusion image can contain complementary effective information in each modality, discard redundant information, accurately and comprehensively describe the actual scene, and effectively alleviate the problem of increased data volume. Based on the fused image, the accuracy of subsequent perception tasks can also be improved. Therefore, this technology has been widely applied to practical engineering fields such as forest fire prevention, agricultural monitoring, traffic monitoring and aerial reconnaissance.

[0038] However, in order to fuse visible light images and infrared images, it is required that the visible light image and the infrared image are aligned. In currently applicable dual-spectrum (visible light and infrared dual-spectrum) cameras, the visible light image and the infrared image often come from different sensors. Different sensor positions are different, and it is difficult to ensure that the angles, positions and focusing conditions of images taken by different sensors are consistent, which often leads to misalignment of the images taken in the spatial dimension.

[0039] In the time dimension, although the visible light and infrared images can be synchronized in time, such synchronization will also accumulate errors. In addition, network instability can also cause different synchronization in time, and the visible light and infrared images received by the back-end algorithm are usually not aligned in time.

[0040] The misalignment of the spatial dimension and the time dimension often causes the feature of the same scene at the same time to be not accurately expressed in the fused image, and the fused image effect is poor.

[0041] To solve the problems in the related art, an embodiment of the present application provides a multi-band image registration fusion method, as shown in the formula (1), comprising the steps of: Figure 1a

[0042] S11: acquiring a first image group to be registered and a second image group to be registered, wherein any image group contains at least one image arranged based on time sequence, the images belonging to the same image group have the same imaging band, and there is at least partial non-coincidence between the imaging band of the first image group and the imaging band of the second image group.

[0043] The first image group and the second image group can be collected by the same image device or different devices, and any image group contains at least one image. In actual application, the first image group and the second image group can be images collected in the same time period and for the same region. The imaging band of the first image group and the imaging band of the second image group have at least partial non-coincidence, for example, the imaging band of the first image group is the visible light band, and the imaging band of the second image group is the infrared band.

[0044] The images contained in any image group are arranged based on time sequence, and the time intervals between the multiple images can be uniform or non-uniform. In actual application, the image group can be multiple image frames in a video, for example.

[0045] S12: performing registration on the first image group and the second image group based on the spatial dimension to obtain a spatially aligned first image group and a spatially aligned second image group.

[0046] In the scheme provided by the embodiment of the present application, registration is performed on the images of the first image group and the second image group. Specifically, registration is performed based on the features contained in the images, the corresponding relationship between the same-name features is used for feature matching to determine the paired images.

[0047] ​In this step, first, registration is performed on the first image group and the second image group from the spatial dimension, so as to ensure that the images present the content of the same area. In actual application, the images may contain moving objects, which may affect the accuracy of spatial registration. Based on this, in this step, registration can be performed based on the area that is fixed and unchanged in the images. For example, in a road scene, vehicles may travel on the road, and registration can be performed based on the unchanged environmental area in the road in the step of spatial registration. For example, the area where the road markers such as green belts and traffic lines are located is taken as the reference for spatial registration, which can effectively improve the accuracy of spatial registration.

[0048] S13: performing registration on the spatially aligned first image group and the second image group based on the time dimension to obtain at least one image pair, wherein any image pair includes a first image and a second image that are aligned based on the spatial dimension and the time dimension, the first image belongs to the first image group, and the second image belongs to the second image group.

[0049] In actual application scenarios, visible light and infrared images can be obtained by a commercial dual-lens dual-spectrum camera. Different lenses obtain images of different modalities, although time synchronization is performed on different lenses, but the problem of time asynchronization is still unavoidable. In this regard, although registration can be performed on the images that are not time-synchronized, the positions of the same object in the visible light and infrared images after registration will be different, thereby affecting the subsequent image fusion effect. Specifically, the visible light features and infrared features of an object will appear at different positions, that is, an object appears twice in the fused image.

[0050] To realize registration of images that are not time-synchronized and spatially aligned, first, the spatial positions of images taken by different lenses need to be aligned. Only after the spatial positions of different images are aligned, is the image registration in time meaningful, otherwise, the spatial positions of different images are not aligned, and time alignment has no reference. By performing spatial alignment first and then time alignment, the matching degree of the aligned image pair can be effectively improved, and the subsequent image fusion effect is improved.

[0051] S14: inputting the at least one image pair into an image fusion model to perform feature fusion on the associated information and complementary information between the first image and the second image by the image fusion model, to obtain a fused image of the image pair, and the image fusion model is a neighbor attention transformation model.

[0052] In this step, at least one image pair is input into an image fusion model, so that the features of the image pair are fused by the model, so that the output fused image can contain the features of the two images in the pair. Specifically, the model can perform feature extraction and feature reconstruction on the input images, and these two processes can also be called encoding and decoding.

[0053] For the feature fusion stage, the required feature fusion can be implemented based on convolutional neural network (CNN) feature fusion, self-attention-based feature fusion, and cross-attention-based feature fusion. Among them, due to the large difference in feature distribution between infrared images and visible light images, if CNN or transformer is used for direct fusion of visible light and infrared image features based on self-attention-based feature fusion, information loss and information confusion will often occur. Cross-attention-based feature fusion can only focus on the relevant information of different modal image features, and cannot focus on the irrelevant features (such as complementary features) of different modal image features.

[0054] In terms of model structure, the CNN structure can only obtain a local receptive field. In the present embodiment, a transformer model is applied, which has a longer range of receptive field. However, the transformer model does not have local invariance. To this end, the scheme provided in the present embodiment combines the transformer model with neighborhood attention, so that the image fusion model has both a longer range of receptive field and local invariance.

[0055] Figure 1b The schematic diagram of window attention and neighborhood attention is shown. In the implementation of information extraction, window attention can be used for large-scale information extraction, in which attention is performed on all feature points in the window and the feature points in the fixed window. In the scheme provided in the present embodiment, the neighborhood-invariant attention mechanism is used to perform attention on the feature points in the nearest neighbor region of any point, thereby ensuring local invariance.

[0056] In the transformer structure, the forward neural network structure plays a role in channel information mixing, but the forward neural network structure of the basic neighborhood attention mechanism does not have the ability of local information mixing. In the image fusion model of the present embodiment, the multi-scale forward neural network structure with local information mixing is provided. Specifically, the model divides the input features into two parts, and then inputs the two features into different convolution modules for processing. Subsequently, the features processed by the convolution modules are superimposed, and then convolution is used for information mixing.

[0057] By the scheme provided in the embodiments of the present application, the image fusion model performs feature extraction and reconstruction based on a nearest neighbor attention transformer, and has a large receptive field and local invariance. Among them, the complete attention mechanism is adopted, so that when the features are fused, both relevant information and complementary information are paid attention to, thereby obtaining better fusion effect.

[0058] In the embodiments of the present application, the first image group and the second group have different imaging wavebands, and images of different imaging wavebands can show different image details from different wavebands. Then, spatial dimension registration and time dimension registration are performed on the first image group and the second image group, the image registration effectiveness is improved, and at least one image pair is obtained. The at least one image pair is input into the image fusion model to perform feature fusion on the associated information and complementary information between the first image and the second image by the image fusion model, and a fused image of the image pair is obtained, and the image fusion model is a neighbor attention transformation model. The neighbor attention transformation model has local invariance of neighbor attention and a large receptive field of the transformation model, and can effectively fuse the spatio-temporal aligned image pair through the image fusion model, and optimize the image fusion effect.

[0059] Based on the scheme provided in the above embodiments, optionally, as shown in Figure 2a In the above step S12, registration is performed on the first image group and the second image group based on the spatial dimension, and the spatially aligned first image group and the second image group are obtained, including:

[0060] S21: performing modal conversion processing on the images in the first image group based on the imaging style of the second image group.

[0061] Among them, due to the large modal difference between visible light images and infrared images, there are fewer image matching points, which makes it difficult to register subsequently and affects the accuracy of image matching. To this end, in the scheme provided in the embodiments of the present application, modal conversion processing is performed on the images in the first image group, so that the images in the converted first image group can present the same imaging style as the second image group. Further, the converted first image group and the second image group have higher similarity, narrowing the modal difference between visible light images and infrared images, and further improving the effectiveness of image registration.

[0062] S22: performing registration on the first image group after modal conversion processing and the second image group based on the spatial dimension, and obtaining the spatially aligned first image group and the second image group.

[0063] Figure 2bA schematic diagram of performing registration between a visible light image and an infrared light image is shown. The upper left image is a first image of visible light, the upper right image is a second image of infrared light, and the matching feature points of direct registration are shown as hollow circles connected by line segments. The image in the lower left is obtained by modal conversion of the first image in the upper left based on the imaging style of the second image. Performing registration between the modal-converted first image and the second image shown in the lower right can significantly improve the similarity between the images, increase the matching feature points, and thus improve the accuracy of spatial registration.

[0064] Figure 2c A modal conversion schematic diagram is shown. In this example, the first image of visible light is input into a modal conversion neural network to obtain a modal-converted first image output by the model. The modal conversion neural network can be a pre-trained neural network, which can be trained using image samples containing visible light images and infrared images.

[0065] The imaging styles of the modal-converted first image and the second image are closer, so that the number of matching points is larger, which is beneficial to spatial dimension registration between images based on matching feature points.

[0066] In this example, performing modal conversion can determine the matching feature points between the first image and the second image. Based on the correspondence of these feature points, spatial registration is performed between the first image and the second image, thereby obtaining a spatially aligned first image group and a second image group. It should be noted that the modal-converted first image described above is only used to determine the matching feature points between the first image and the second image, and the first image in the spatially aligned first image group obtained subsequently is the visible light image before modal conversion.

[0067] Through the scheme provided by the embodiments of the present application, the similarity between the first image and the second image is determined by modal conversion, and a larger number of matching feature points are determined, thereby improving the effectiveness of spatial registration and optimizing the quality of spatial alignment of the first image group and the second image group.

[0068] Based on the scheme provided in the above embodiments, optionally, as Figure 3a shown, in step S13, the spatially aligned first image group and the second image group are registered based on the time dimension to obtain at least one pair of image pairs, including:

[0069] S31: performing background separation processing on each image in the spatially aligned first image group and each image in the second image group, respectively, to obtain a first background region corresponding to the image.

[0070] In the embodiments of the present application, the first image group and the second image group after spatial alignment are further subjected to time alignment. In this step, background separation processing is performed on each image in the first image group and the second image group to determine the first background region corresponding to each image.

[0071] In actual applications, the image may contain moving people, vehicles, etc., and also contains non-moving environmental backgrounds such as roads. For two images taken at the same time, the presentation of moving objects in the two images also differs. In this step, the first background region is extracted from each image through background separation processing. On the one hand, the extracted first background region is affected by the moving object, and as the object moves, the first background region obtained through the background separation processing will also change accordingly, that is, the first background region can reflect the characteristics of the foreground moving object to a certain extent. On the other hand, the background contains objects that are not easy to move. Time registration based on the first background region can improve the efficiency of time registration alignment and avoid the problem of difficult time alignment due to the change of moving objects in a short time.

[0072] S32: Performing mask dilation processing on the foreground of the image based on the first background region to obtain a second background region corresponding to the image.

[0073] In this step, based on the first background region obtained in the above step, mask dilation processing is performed on the foreground of each image to obtain a second background region corresponding to each image. The mask dilation processing can expand the foreground region in the image, reduce the characteristics of the moving object presented in the image, and make the second background region reduce the details of the foreground object while retaining the influence of the moving object on the background to a certain extent, which is conducive to improving the efficiency of time matching.

[0074] S33: Performing registration on the second background region of the image in the first image group and the second background region of the image in the second image group based on the time dimension to obtain at least one image pair.

[0075] In this step, the first image group or the second image group can be selected as the reference image group, and one image to be registered is determined from the reference image. Registration is performed on each image in the other image group based on the one image to be registered, and the image with the highest similarity is determined as the image pair.

[0076] Among them, the image pair contains a first image and a second image, and the second background region of the second image has high similarity with the second background region of the first image. The background region is used to compare the similarity between images, so as to determine the closest first image and second image to form an image pair.

[0077] The scheme provided by the embodiments of the present application can extract appropriate features from the image through background separation and foreground mask expansion, retain the features of the foreground object to a certain extent, avoid excessive influence of the edge details of the foreground object on the registration efficiency, and thus improve the efficiency of image time alignment.

[0078] Specifically, to avoid negative influence of object movement on image registration, the scheme uses a mask to filter out the moving object in the image. The moving object needs to be filtered out first. To filter out the moving object, the moving object can be masked. Figure 3b A result diagram of the first image after the mask and expansion operations is shown. For example, the Gaussian Mixture Model background separation algorithm (MOG2 algorithm) can be used to perform background extraction on the first image (I1) Figure 3b left), and a mask image (I1 Figure 3b middle) is obtained. The mask image can be obtained by taking the inverse of the foreground mask. Then, to avoid insufficient background extraction, a 3*3 expansion morphological operation is performed on the first image based on the first background region, and a mask image after the expansion operation (I1 Figure 3b right) is obtained. In comparison, the mask image after the expansion operation removes the detailed features of the foreground edge in the image, and reduces the negative influence of the edge of the moving object on image time registration.

[0079] Based on the scheme provided in the above embodiments, as shown in Figure 4a in the step S13, registration is performed on the first image group and the second image group that are aligned in space based on the time dimension, and at least one pair of image pairs is obtained, including:

[0080] S41: determining a first image to be registered in time from the first image group that is aligned in space.

[0081] In this step, the first image to be registered in time is determined from the first image group that is aligned in space. Optionally, the first image that has not yet been constructed into an image pair is selected in the first image group in the time sequence, so as to select the most similar second image from the second image group to construct an image pair through the subsequent steps. The first image group and the second image group that are aligned in space can be pre-stored in the cache, so as to be called when the time registration is performed.

[0082] S42: performing similarity calculation on the first image to be registered in time and each second image in the second image group that is aligned in space based on the optical flow method.

[0083] In this step, time registration is performed using the optical flow method. For example, assume that the first image group after spatial alignment includes images taken continuously within 10 seconds. Then, the first image in the middle (for example, the image at the 5th second) can be taken out in time sequence as a first image to be registered. Subsequently, each second image is taken out from the second image group, and a forward-backward difference is calculated based on the optical flow method, so as to determine the difference between each second image and the first image to be registered.

[0084] Figure 4b A schematic diagram of forward-backward difference calculation of the Lucas-Kanade optical flow method is shown, where d is the forward-backward difference value, which is determined based on the difference between the forward trajectory and the backward trajectory.

[0085] S43: Construct the second image with the highest similarity and the first image to be registered in time as an image pair.

[0086] Based on the difference calculated in the above steps, it can be directly determined that the second image with the smallest difference matches the first image to be registered, and then the matched first image and second image are stored in the cache.

[0087] Optionally, after determining the second image with the smallest difference from the first image, it is further judged whether the difference is smaller than a preset difference. If the difference is smaller than the preset difference, the second image with the smallest difference is constructed as an image pair with the first image. If the difference is greater than or equal to the preset difference, it is determined that there is no second image in the second image group that matches the first image to be registered, and other first images in the first image group are selected to continue to attempt time registration.

[0088] In an application scenario, if the first image group and the second image group are continuous images taken uniformly in the same period, then based on the determination of the 5th second image pair through the above steps, the images of the first image group and the second image group can be sequentially paired in time sequence, so as to efficiently realize time registration of the first image group and the second image group.

[0089] Based on the scheme provided in the above embodiment, optionally, the image fusion model includes a cross-attention (CA) module and a spatial-embedded single-head channel-cross attention (SSCA) module.

[0090] Wherein, CA refers to cross-attention, and SSCA refers to Spatial-embedded single-head channel-cross attention.

[0091] The CA module is configured to perform fusion on the associated information in the input features, and the SSCA module is configured to perform fusion on the complementary information in the input features.

[0092] As shown in Figure 5a In step S14, the at least one image pair is input into the image fusion model, and the image fusion model performs feature fusion on the associated information and the complementary information between the first image and the second image to obtain a fused image of the image pair.

[0093] S51: input the at least one image pair into the image fusion model, and perform feature fusion on the associated information between the first image and the second image by the CA module, and perform feature fusion on the complementary information between the first image and the second image by the SSCA module to obtain a fused image of the image pair.

[0094] In this embodiment, the image pair is input into the image fusion model, the image fusion model performs feature extraction on the input image, at least part of the extracted features are input into the CA module, and at least part of the extracted features are input into the SSCA module. The features input into the CA module and the SSCA module can be different. The CA module performs fusion on the associated features of the two input images, the SSCA module performs fusion on the complementary features of the two input images, and the convolution module further performs superposition fusion on the features processed by the CA module and the SSCA module to obtain a fused image containing both the associated feature fusion result and the complementary feature fusion result of the image pair.

[0095] Feature fusion is an important step in image fusion. Unlike previous feature fusion using CNN and self-attention, which is prone to information loss and confusion, cross-attention (CA) is used for feature fusion, which only focuses on the relevant information of different modal images and ignores the complementary information. The scheme provided in the embodiments of the present application adopts a complete attention mechanism, so that during feature fusion, both relevant information and complementary information are focused on, thereby obtaining better fusion effect.

[0096] As shown in Figure 5b As shown in (a) of the upper left corner, the CNN-based feature fusion performs merging on A features and B features through MERGE, and obtains fused features Fused features through a convolution module.

[0097] The upper right (b) shows feature fusion based on a self-attention mechanism. A features and B features are first merged through MERGE, and then self-attention SA (self-attention) feature fusion is performed based on Q (query), K (key), and V (value) to obtain fused features. Q represents a query, which determines the part that the model needs to pay attention to. K represents a key value, which represents the importance of each part. V represents a value, which is represented by the numerical value of each input part and can be used for subsequent weighted summation.

[0098] The lower left (c) shows feature fusion based on a cross-attention mechanism. The Q, K, and V based on the attention mechanism are fused through the CA module, and then merged through MERGE to obtain the fused features.

[0099] The lower right (d) shows feature fusion based on the complete cross-attention mechanism provided by the present application. The Q, K, and V based on the attention mechanism are split into the SSCA module and the CA module, and then the feature processing results output by the SSCA module and the CA module are merged through MERGE to obtain the fused features.

[0100] Based on the scheme provided in the above embodiments, as shown in Figure 6a The image fusion model includes a plurality of complete cross-attention CCA modules, wherein any CCA module includes a CA module and an SSCA module.

[0101] In the step S51, the at least one image pair is input into the image fusion model to perform feature fusion on the associated information between the first image and the second image through the CA module, and perform feature fusion on the complementary information between the first image and the second image through the SSCA module, to obtain a fused image of the image pair, including:

[0102] S61: performing multi-dimensional feature splitting on the first image and the second image to obtain a plurality of sub-dimensional features corresponding to the image pair.

[0103] In this step, feature extraction and splitting are performed on the first image and the second image to obtain a plurality of sub-dimensional features corresponding to the first image and a plurality of sub-dimensional features corresponding to the second image. Optionally, the input features are split into two parts and input into the CA module and the SSCA module, respectively.

[0104] S62: Randomly input the plurality of sub-dimension features corresponding to the image pair into the plurality of CCA modules, and perform feature fusion on the feature information output by the plurality of CCA modules through the convolution module.

[0105] In the image fusion model, a plurality of complete cross attention CCA modules are included. Figure 6b The model structure is shown, and the above-mentioned CCA modules can be connected at the CCA Block position shown in the figure. Among them, (a) shows the overall architecture of the model, (b) shows the structure of the neighborhood attention conversion module, (c) shows the structure of the complete cross attention CCA module, and (d) shows the complete cross attention structure in the CCA module.

[0106] The image fusion model applied in the embodiments of the present application is based on an end-to-end network of an iNAT (improve neighborhood attention) transformer structure and a CCA module. The transformer of the traditional neural network is usually stacked in a large number, for example, 4-12 blocks per block. The network of the present patent contains fewer blocks and is a lightweight neural network.

[0107] Optionally, in the structure of the image fusion model provided in the embodiments of the present application, CCA modules are arranged at higher and lower layers of the model to perform feature fusion. The feature fusion at the high layer can ensure fusion at the semantic layer, and the fusion at the low layer can make up for the information loss during feature reconstruction, so that the final fused image has more detailed information.

[0108] Through the scheme provided in the embodiments of the present application, the relevant features and complementary features of the image pair are fused based on the CCA module, and then merged through the convolution module, so that the fused image has both the relevant features and the complementary features after fusion.

[0109] In consideration of the related image registration technology, only the spatial misalignment of visible light and infrared images is considered, and the time misalignment of images in actual application scenarios is not considered. In the scheme provided in the embodiments of the present application, background extraction, morphological operation, modal conversion, forward-backward difference calculation based on optical flow, and feature matching algorithm are applied to realize registration of visible light and infrared images in time and space.

[0110] In the feature extraction and reconstruction process, the related technology often cannot simultaneously have a large receptive field and local invariance. The embodiments of the present application apply an optimized transformer based on the nearest neighbor attention for feature extraction and reconstruction, which has a large receptive field and local invariance.

[0111] In the feature fusion process, if direct fusion is performed, information loss and information confusion often occur, or the cross-attention mechanism is used to pay attention to the related information of different modal images and ignore the complementary information. Embodiments of the present application apply a complete attention mechanism, pay attention to both related information and complementary information in the feature fusion process, and can effectively improve the image fusion effect.

[0112] The scheme provided by the embodiments of the present application can realize image registration in time and space and image fusion with complete attention, and can effectively solve the fusion problem of visible light and infrared images that are not aligned in time and space. By extracting the background of the image, the negative impact of moving objects on image registration is avoided, and the robustness of image registration based on the time dimension can be effectively improved. In the mask expansion process, multiple expansion operations can be performed until the mask expansion result eliminates the impact of foreground edge details on image registration, thereby improving the robustness of image registration to moving objects and background extraction. In addition, in the time registration process, the forward-backward difference of LK optical flow can be calculated to effectively register the visible light and infrared images that are not aligned in time.

[0113] In the image registration process, the scheme provided by the embodiments of the present application first converts the visible light image modal, then performs spatial alignment on the converted visible light image and infrared image, and finally performs background extraction, multiple expansion, and forward-backward difference calculation of LK optical flow on the spatially aligned images, thereby effectively registering the images that are not aligned in time and space.

[0114] In terms of image fusion, the embodiments of the present application apply a transformer with neighborhood attention and a multi-scale forward neural network. The model has both local invariance and wide-range information extraction capability, effectively improving the ability to extract image features and fuse visible light and infrared images. In the model, the SSCA mechanism is applied to pay attention to and fuse the complementary information of visible light and infrared images, thereby improving the ability to fuse visible light and infrared images. In the model, the cross-attention mechanism and the SSCA mechanism are combined to simultaneously fuse the related information and complementary information of visible light and infrared images, thereby improving the ability to fuse visible light and infrared images. The model has the advantages of lightweight, fast speed, and good image fusion effect.

[0115] To solve the problems in the related art, the embodiments of the present application also provide a multi-band image registration and fusion device 70, as shown in Figure 7 , which includes

[0116] The acquisition module 71 acquires a first image group to be registered and a second image group to be registered, wherein any image group contains at least one image arranged based on time sequence, the imaging wavebands of the images belonging to the same image group are the same, and there is at least partial non-overlapping between the imaging wavebands of the first image group and the imaging wavebands of the second image group.

[0117] The spatial registration module 72 performs registration on the first image group and the second image group based on the spatial dimension, to obtain a spatially aligned first image group and a spatially aligned second image group.

[0118] The temporal registration module 73 performs registration on the spatially aligned first image group and the spatially aligned second image group based on the time dimension, to obtain at least one image pair, wherein any image pair includes a first image and a second image aligned based on the spatial dimension and the time dimension, the first image belongs to the first image group, and the second image belongs to the second image group.

[0119] The fusion module 74 inputs the at least one image pair into an image fusion model, to perform feature fusion on the associated information and the complementary information between the first image and the second image by the image fusion model, to obtain a fused image of the image pair, and the image fusion model is a neighbor attention transformation model.

[0120] The device provided by the embodiment of the present application first acquires a first image group to be registered and a second image group to be registered, wherein any image group contains at least one image arranged based on time sequence, the imaging wavebands of the images belonging to the same image group are the same, and there is at least partial non-overlapping between the imaging wavebands of the first image group and the imaging wavebands of the second image group. Wherein, the images of different imaging wavebands can show different image details from different wavebands. Then, registration is performed on the first image group and the second image group based on the spatial dimension, to obtain a spatially aligned first image group and a spatially aligned second image group. Next, registration is performed on the spatially aligned first image group and the spatially aligned second image group based on the time dimension, to obtain at least one image pair, wherein any image pair includes a first image and a second image aligned based on the spatial dimension and the time dimension, the first image belongs to the first image group, and the second image belongs to the second image group. Wherein, after spatial alignment, time alignment is performed, which can effectively improve the effectiveness of image alignment. At least one image pair is input into an image fusion model, to perform feature fusion on the associated information and the complementary information between the first image and the second image by the image fusion model, to obtain a fused image of the image pair, and the image fusion model is a neighbor attention transformation model. The neighbor attention transformation model has both the local invariance of neighbor attention and the large receptive field of the transformation model, and can effectively fuse the spatio-temporally aligned image pair through the image fusion model, to optimize the image fusion effect.

[0121] The modules in the apparatus provided by the embodiments of the present application can also implement the method steps provided by the method embodiments. Alternatively, the apparatus provided by the embodiments of the present application can further include other modules in addition to the modules described above, to implement the method steps provided by the method embodiments. The apparatus provided by the embodiments of the present application can achieve the technical effects achievable by the method embodiments.

[0122] Preferably, the embodiments of the present application further provide an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable in the processor. The computer program, when executed by the processor, implements each process of the above-mentioned multi-band image registration fusion method embodiments and achieves the same technical effects. To avoid repetition, details are not described herein.

[0123] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program, when executed by a processor, implements each process of the above-mentioned multi-band image registration fusion method embodiments and achieves the same technical effects. To avoid repetition, details are not described herein. The computer readable storage medium can be, for example, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk.

[0124] The embodiments of the present application further provide a computer program product, which includes a non-transitory computer readable storage medium storing a computer program. The computer program is operable to cause a computer to implement some or all of the steps of the above-mentioned multi-band image registration fusion method embodiments and achieve the same technical effects. To avoid repetition, details are not described herein.

[0125] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage media, etc.) containing computer-usable program code.

[0126] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

[0127] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks.

[0128] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

[0129] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0130] The memory can include non-persistent memory and / or persistent memory, such as flash memory, or other non-volatile memory, among others. The memory is an example of computer readable media.

[0131] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0132] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0133] Those skilled in the art will appreciate that embodiments of the present application can be provided as a method, system or computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer usable program code.

[0134] The above description is only an embodiment of the present application and is not intended to limit the present application. Those skilled in the art can make various modifications and changes to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of the claims of the present application.

Claims

1. A method of registering and fusing multi-band images, characterized by, The method comprises: obtaining a first image group to be registered and a second image group to be registered, wherein any image group comprises at least one image arranged based on time sequence, and the imaging bands of the images belonging to the same image group are the same, and there is at least partial non-overlapping between the imaging bands of the first image group and the imaging bands of the second image group; performing registration on the first image group and the second image group based on the spatial dimension to obtain a spatially aligned first image group and a spatially aligned second image group; performing registration on the spatially aligned first image group and the spatially aligned second image group based on the time dimension to obtain at least one image pair, wherein any image pair comprises a first image and a second image aligned based on the spatial dimension and the time dimension, the first image belongs to the first image group, and the second image belongs to the second image group; inputting the at least one image pair into an image fusion model to perform feature fusion on the associated information and the complementary information between the first image and the second image by the image fusion model, and obtaining a fused image of the image pair, wherein the image fusion model is a neighboring attention transformation model; the image fusion model comprises a plurality of complete cross-attention (CCA) modules, wherein any CCA module comprises a CA module and an SSCA module; wherein the at least one image pair is input into the image fusion model to perform feature fusion on the associated information between the first image and the second image by the CA module, and perform feature fusion on the complementary information between the first image and the second image by the SSCA module, and obtain a fused image of the image pair, comprising: performing multi-dimensional feature segmentation on the first image and the second image to obtain a plurality of sub-dimensional features corresponding to the image pair; randomly inputting the plurality of sub-dimensional features corresponding to the image pair into a plurality of CCA modules, and performing feature fusion on the feature information output by the plurality of CCA modules by a convolution module.

2. The method of claim 1, wherein, performing registration on the first image group and the second image group based on the spatial dimension to obtain a spatially aligned first image group and a spatially aligned second image group, comprising: performing modal conversion processing on the images in the first image group based on the imaging style of the second image group; performing registration on the first image group after modal conversion processing and the second image group based on the spatial dimension to obtain a spatially aligned first image group and a spatially aligned second image group.

3. The method of claim 1, wherein, performing registration on the spatially aligned first image group and the spatially aligned second image group based on the time dimension to obtain at least one image pair, comprising: performing background separation processing on each image in the spatially aligned first image group and each image in the second image group respectively to obtain a first background region corresponding to the image; performing mask inflation processing on the foreground of the image based on the first background region to obtain a second background region corresponding to the image; performing registration on the second background region of the image in the spatially aligned first image group and the second background region of the image in the second image group based on the time dimension to obtain at least one image pair.

4. The method of claim 1, wherein, performing registration on the spatially aligned first image group and the spatially aligned second image group based on the time dimension to obtain at least one image pair, comprising: determining a first image to be time registered from a first set of spatially aligned images; performing a similarity calculation based on an optical flow method between the first image to be time registered and each second image in a second set of spatially aligned images; constructing the second image with the highest similarity and the first image to be time registered as an image pair.

5. An apparatus for registering and fusing multi-band images, characterized by comprising: an acquisition module, acquiring a first set of images to be registered and a second set of images to be registered, wherein any image set comprises at least one image arranged based on a time sequence, and the images belonging to the same image set have the same imaging band, and there is at least partial non-overlapping between the imaging band of the first image set and the imaging band of the second image set; a spatial registration module, performing registration on the first image set and the second image set based on a spatial dimension, to obtain a first set of spatially aligned images and a second set of spatially aligned images; a time registration module, performing registration on the first set of spatially aligned images and the second set of spatially aligned images based on a time dimension, to obtain at least one image pair, wherein any image pair comprises a first image and a second image aligned based on the spatial dimension and the time dimension, the first image belongs to the first image set, and the second image belongs to the second image set; a fusion module, inputting the at least one image pair into an image fusion model, to perform feature fusion on the associated information and the complementary information between the first image and the second image by the image fusion model, to obtain a fused image of the image pair, and the image fusion model is a neighbor attention transformation model; the image fusion model comprises a plurality of complete cross-attention (CCA) modules, wherein any CCA module comprises a CA module and an SSCA module; wherein the at least one image pair is input into the image fusion model, to perform feature fusion on the associated information between the first image and the second image by the CA module, and perform feature fusion on the complementary information between the first image and the second image by the SSCA module, to obtain a fused image of the image pair, comprising: performing multi-dimensional feature segmentation on the first image and the second image, to obtain a plurality of sub-dimensional features corresponding to the image pair; randomly inputting the plurality of sub-dimensional features corresponding to the image pair into a plurality of CCA modules, and performing feature fusion on the feature information output by the plurality of CCA modules by a convolution module.

6. An electronic device, comprising: comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, when the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium, characterized in that, a computer program is stored on the computer readable storage medium, and when the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 4 are implemented.

8. A computer program product, characterised in that, the computer program product comprises a non-transitory computer readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps of the method according to any one of claims 1 to 4.

Images