Image fusion method and device

Abstract

The image fusion method and device provided in the embodiment of the present application include: obtaining multiple images to be fused, wherein the multiple images to be fused are images of multiple viewing angles and / or multiple spectra for the same scene. Performing coarse registration processing on the multiple images to be fused to obtain multiple images to be fused after coarse registration. Inputting the multiple images to be fused after coarse registration into a preset model so that the preset model outputs a fused target image. Among them, the preset model is a model for fusing images, and the preset model is obtained by learning based on multiple groups of samples, each group of samples includes sample images to be fused and sample target images, and the sample images to be fused in the multiple groups of samples are images of multiple viewing angles and / or multiple spectra for at least one scene. By successively performing coarse registration and fine registration processing on multiple images to be fused, the technical effect of image fusion of multi-view and multi-spectral images in any scene is achieved.

Description

Technical Field

[0001] The embodiments of the present application relate to image processing technology, and more particularly to an image fusion method and device. Background Art

[0002] Image fusion refers to the process of extracting useful information from each channel to the maximum extent through image processing and computer technology from image data of the same target collected by multiple source channels, and finally fusing them into a high-quality image to reduce the uncertainty of the scene description in the image. Image fusion can effectively improve the utilization rate of the original image information.

[0003] In order to obtain a high-quality fused image, selecting a suitable image fusion method has a great influence on the image fusion effect. At present, in the existing technology of image fusion based on deep learning, an image fusion method based on a convolutional neural network is usually used for image fusion processing.

[0004] However, existing deep learning-based image fusion methods can only perform image fusion for specific scenes or specific multi-spectral situations, which leads to a limited application scope of image fusion. Summary of the invention

[0005] The embodiments of the present application provide an image fusion method and device to overcome the problem of limited application scope of image fusion.

[0006] In a first aspect, an embodiment of the present application provides an image fusion method, comprising:

[0007] Acquire multiple images to be fused, wherein the multiple images to be fused are images of multiple viewing angles and / or multiple spectra of the same scene;

[0008] Performing a coarse registration process on the multiple images to be fused to obtain a plurality of images to be fused after coarse registration;

[0009] Inputting the plurality of images to be fused after the rough registration into a preset model, so that the preset model outputs a fused target image;

[0010] Among them, the preset model is a model for fusing images, and the preset model is obtained by learning based on multiple groups of samples, each group of samples includes sample images to be fused and sample target images, wherein the sample images to be fused in the multiple groups of samples are images of multiple perspectives and / or multiple spectra for at least one scene.

[0011] In one possible design, the preset model includes a feature extraction unit, and the feature extraction unit includes N feature extraction layers, wherein the feature extraction unit is used to extract feature images, and the preset model also includes N gate units and 1 fusion unit, the gate unit is used to perform preliminary fusion of images, and the fusion unit is used to perform final fusion of the outputs of the gate units, and N is an integer greater than or equal to 1.

[0012] In a possible design, the inputting the plurality of images to be fused after the rough registration into a preset model so that the preset model outputs a fused target image includes:

[0013] Inputting the plurality of images to be fused after the rough registration into the feature extraction unit of the preset model, and outputting feature images of N scales corresponding to each of the images to be fused through N feature extraction layers in the feature extraction unit;

[0014] Inputting the feature images of N scales corresponding to each of the images to be fused into the N gate units respectively, so that each of the gate units preliminarily fuses the feature images of N scales corresponding to each of the images to be fused, and obtains fusion results of N scales;

[0015] The fusion results of the N scales are upsampled, and the fusion results of the N scales after the upsampling are input into the fusion unit, so that the fusion unit outputs a fused target image.

[0016] In one possible design, the upsampling process includes:

[0017] Image interpolation processing is performed on the fusion results of the N scales, and the resolution of the fusion results of the N scales is adjusted to the target resolution.

[0018] In a possible design, performing a rough registration process on the multiple images to be fused to obtain the multiple images to be fused after rough registration includes:

[0019] Determine the homography matrix corresponding to the multiple images to be fused;

[0020] According to the homography matrices corresponding to the multiple images to be fused, each of the images to be fused is transformed to obtain multiple images to be fused after rough registration, wherein the multiple images to be fused after rough registration have the same viewing angle.

[0021] In one possible design, the method further includes:

[0022] The preset model is obtained by learning according to the multiple groups of samples.

[0023] In a possible design, learning according to the multiple groups of samples to obtain the preset model includes:

[0024] Processing the sample images to be fused in the multiple groups of samples by using the initial model to obtain fused images corresponding to the sample images to be fused;

[0025] Processing the fused image and the sample target image according to the loss function to obtain a loss function value;

[0026] Performing back propagation on the initial model according to the loss function value to adjust the model parameters of the initial model;

[0027] The above two steps are repeated until the initial model meets the convergence condition to obtain the preset model.

[0028] In a second aspect, an embodiment of the present application provides an image fusion device, including:

[0029] An acquisition module, used for acquiring a plurality of images to be fused, wherein the plurality of images to be fused are images of multiple viewing angles and / or multiple spectra of the same scene;

[0030] A first processing module is used to perform a rough registration process on the multiple images to be fused to obtain a plurality of images to be fused after rough registration;

[0031] A second processing module, used for inputting the plurality of images to be fused after the rough registration into a preset model, so that the preset model outputs a fused target image;

[0032] Among them, the preset model is a model for fusing images, and the preset model is obtained by learning based on multiple groups of samples, each group of samples includes sample images to be fused and sample target images, wherein the sample images to be fused in the multiple groups of samples are images of multiple perspectives and / or multiple spectra for at least one scene.

[0033] In one possible design, the preset model includes a feature extraction unit, and the feature extraction unit includes N feature extraction layers, wherein the feature extraction unit is used to extract feature images, and the preset model also includes N gate units and 1 fusion unit, the gate unit is used to perform preliminary fusion of images, and the fusion unit is used to perform final fusion of the outputs of the gate units, and N is an integer greater than or equal to 1.

[0034] In a possible design, the first processing module is specifically used to:

[0035] Inputting the plurality of images to be fused after the rough registration into the feature extraction unit of the preset model, and outputting feature images of N scales corresponding to each of the images to be fused through N feature extraction layers in the feature extraction unit;

[0036] Inputting the feature images of N scales corresponding to each of the images to be fused into the N gate units respectively, so that each of the gate units preliminarily fuses the feature images of N scales corresponding to each of the images to be fused, and obtains fusion results of N scales;

[0037] The fusion results of the N scales are upsampled, and the fusion results of the N scales after the upsampling are input into the fusion unit, so that the fusion unit outputs a fused target image.

[0038] In a possible design, the first processing module is specifically used to:

[0039] Image interpolation processing is performed on the fusion results of the N scales, and the resolution of the fusion results of the N scales is adjusted to the target resolution.

[0040] In a possible design, the first processing module is specifically used to:

[0041] Determine the homography matrix corresponding to the multiple images to be fused;

[0042] According to the homography matrices corresponding to the multiple images to be fused, each of the images to be fused is transformed to obtain multiple images to be fused after rough registration, wherein the multiple images to be fused after rough registration have the same viewing angle.

[0043] In a possible design, the second processing module is further used for:

[0044] The preset model is obtained by learning according to the multiple groups of samples.

[0045] In a possible design, the second processing module is specifically used for:

[0046] Processing the sample images to be fused in the multiple groups of samples by using the initial model to obtain fused images corresponding to the sample images to be fused;

[0047] Processing the fused image and the sample target image according to the loss function to obtain a loss function value;

[0048] Performing back propagation on the initial model according to the loss function value to adjust the model parameters of the initial model;

[0049] The above two steps are repeated until the initial model meets the convergence condition to obtain the preset model.

[0050] In a third aspect, an embodiment of the present application provides an image fusion device, including:

[0051] Memory, used to store programs;

[0052] A processor is used to execute the program stored in the memory. When the program is executed, the processor is used to execute the method described in the first aspect and any one of the various possible designs of the first aspect.

[0053] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enables the computer to execute the method described in the first aspect and any one of the various possible designs of the first aspect.

[0054] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising: a computer program, the computer program being stored in a readable storage medium, at least one processor of an electronic device being able to read the computer program from the readable storage medium, and the at least one processor executing the computer program so that the electronic device executes the method described in the first aspect above and any one of the various possible designs of the first aspect.

[0055] The image fusion method and device provided in the embodiment of the present application include: obtaining multiple images to be fused, wherein the multiple images to be fused are images of multiple viewing angles and / or multiple spectra for the same scene. Performing coarse registration processing on the multiple images to be fused to obtain multiple images to be fused after coarse registration. Inputting the multiple images to be fused after coarse registration into a preset model so that the preset model outputs a fused target image. Among them, the preset model is a model for fusing images, and the preset model is obtained by learning based on multiple groups of samples, each group of samples includes sample images to be fused and sample target images, wherein the sample images to be fused in the multiple groups of samples are images of multiple viewing angles and / or multiple spectra for at least one scene. By successively performing coarse registration on multiple images to be fused and performing image fusion using a preset model to complete fine registration, the technical effect of image fusion of multi-view and multi-spectral images in any scene is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0057] Figure 1 The process of the image fusion method provided in the embodiment of the present application Figure 1 ;

[0058] Figure 2 The process of the image fusion method provided in the embodiment of the present application Figure 2 ;

[0059] Figure 3 A schematic diagram of the structure of the image fusion method provided in the embodiment of the present application;

[0060] Figure 4 A schematic diagram of a process for training a prediction model provided in an embodiment of the present application;

[0061] Figure 5 A schematic diagram of a structure for training a prediction model provided in an embodiment of the present application;

[0062] Figure 6 A schematic diagram of the structure of an image fusion device provided in an embodiment of the present application;

[0063] Figure 7 A schematic diagram of the hardware structure of the image fusion device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0065] In order to facilitate understanding of the technical solution of the present application, the relevant concepts involved in the present application are first explained:

[0066] Image fusion refers to the process of image processing and computer technology to extract the useful information in each channel from the image data collected by multiple source channels on the same target, and finally integrate them into high-quality images to improve the utilization rate of image information, improve the accuracy and reliability of computer interpretation, and enhance the spatial resolution and spectral resolution of the original image, which is conducive to monitoring. Among them, for the multi-source channels of the same observation target, at least the following factors should be considered: multi-angle observation and multi-spectral observation.

[0067] Image fusion methods can be performed at three different levels: feature level, decision level, and pixel level. Feature-level fusion methods generally use visual feature extraction operators to obtain significant features, then assign different weights to the acquired features, and finally fuse the source images. Decision-level fusion is a cognitive-based method, and its main process is to identify and understand the extracted features and make corresponding decisions. Pixel-level image fusion methods directly combine the original pixel information, which can retain more details of the source image.

[0068] Deep learning is the process of learning the inherent laws and representation levels of sample data. The information obtained in the learning process is of great help in interpreting data such as text, images and sounds. Its ultimate goal is to enable robots to have analytical learning capabilities like humans and to recognize data such as text, images and sounds. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition that far exceed previous related technologies.

[0069] Deep learning has achieved many results in search technology, data mining, machine learning, natural language processing, image recognition and other related fields. Deep learning enables machines to imitate human activities such as listening and thinking, solves many complex pattern recognition problems, and makes great progress in artificial intelligence related technologies.

[0070] In order to obtain high-quality fused images, selecting a suitable image fusion method has a great influence on the image fusion effect. In recent years, researchers have begun to try to use deep learning methods to train the convolutional neural network model using relevant image sample data to obtain a convolutional neural network model for image processing. Usually, the neural network-based method divides the source image into image blocks and feeds them into the trained convolutional neural network model. Since the convolutional neural network is data-driven, this method is more robust and has better results than traditional algorithm methods.

[0071] However, existing deep learning-based image fusion methods have the problem of limited application scope because they only perform image fusion for specific scenes or specific multi-spectral situations. The specific scenes may be, for example, medical scenes, office scenes, outdoor scenes, traffic road scenes, etc.

[0072] Based on the above-mentioned problems, this application proposes the following technical ideas: a new image fusion neural network is designed, the multi-scale features of the image are extracted through the pyramid feature extraction substructure, the image fusion is performed using the graph neural network fusion module in the proposed new image fusion neural network, and a special loss function is constructed for neural network back propagation. Through the designed new image fusion neural network, multi-spectral and multi-view image fusion processing can be realized for images in any scene, overcoming the problem of limited application scope in the prior art that image fusion is only performed for specific scenes or specific multi-spectral situations.

[0073] Based on the above introduction, the following Figure 1 The image fusion method provided in this application is introduced in detail. Figure 1 The process of the image fusion method provided in the embodiment of the present application Figure 1 .

[0074] like Figure 1 As shown, the method includes:

[0075] S101. Acquire multiple images to be fused, wherein the multiple images to be fused are images of multiple viewing angles and / or multiple spectra of the same scene.

[0076] In order to realize image fusion processing for multiple images, multiple images to be fused need to be obtained before processing. The multiple images to be fused are images of multiple viewing angles or multiple spectra of the same target scene, that is, multi-view images and / or multi-spectral images.

[0077] Multi-view images refer to image information of the same scene taken from multiple angles. Correspondingly, multi-view image fusion is to fuse the image information of the same scene taken from multiple angles into one image, so that the fused image contains more information, reduces the uncertainty of scene description, and facilitates human observation or computer processing.

[0078] A multispectral image is an image that contains many bands. Each band is a grayscale image that represents the brightness of the scene according to the sensitivity of the sensor used for that band. In a multispectral image, each pixel is associated with a string of values ​​of the pixel in different bands, i.e., a vector. This string of values ​​is called the spectral label of the pixel. Multispectral, for example, can be infrared, ultraviolet and visible light. Correspondingly, multispectral image fusion requires scientific fusion of complementary information of different spectral bands in a multispectral image according to certain criteria.

[0079] S102 , performing a coarse registration process on the multiple images to be fused to obtain the multiple images to be fused after coarse registration.

[0080] After a plurality of images to be fused are acquired in step S101, it is necessary to perform a rough registration process on the plurality of images to be fused.

[0081] In this implementation, a dual-spectrum image of a scene, namely an ultraviolet image and a visible light image, collected by an ultraviolet camera and a visible light camera is taken as an example to introduce the process involved in the coarse registration process.

[0082] The rough registration process refers to first performing scale-invariant feature transform (SIFI) feature detection and extraction on the bispectral image, matching feature points using the nearest neighbor / next nearest neighbor method and KD tree (K-DistanceTree, KD Tree) data structure, removing outliers using the random sample consensus (RANSAC) algorithm, and calculating the homography matrix based on the remaining matching points. The homography matrix is ​​a planar homography matrix that is used to indicate the mapping relationship between points on different image planes.

[0083] The homography matrix is ​​mainly used to solve the following two problems: 1. To describe the perspective transformation between a plane and its corresponding image in the real world; 2. To transform an image from one viewing angle to another through perspective transformation. Specifically, the implementation method of solving the homography matrix can refer to any possible implementation method in the prior art, which will not be described here.

[0084] The image to be fused is transformed according to the homography matrix so that multiple multi-view and multi-spectral images are realized at the same viewing angle, that is, the rough registration processing of multiple images to be fused is completed.

[0085] Among them, the implementation method of the related processing involved in the rough alignment process (for example, SIFI feature detection and extraction, matching feature points using the nearest neighbor / next nearest neighbor method and kd tree data structure, etc.) can refer to any possible implementation method in the prior art, and will not be repeated here.

[0086] In this embodiment, the coarse registration process is merely introduced as an example, and the specific process of the coarse registration process is not limited. The specific processing settings of the coarse registration process can be selected according to actual needs.

[0087] S103, inputting the multiple images to be fused after rough registration into a preset model, so that the preset model outputs a fused target image.

[0088] After obtaining the multiple images to be registered and fused after the rough registration processing based on step S102, the multiple images to be fused after the rough registration processing are then input into the preset model, so that the preset model outputs the fused target image.

[0089] The preset model is a model for image fusion, and the preset model is learned based on multiple groups of samples, each group of samples includes sample images to be fused and sample target images. The sample images to be fused in the multiple groups of samples are images of multiple viewing angles and / or multiple spectra for at least one scene.

[0090] In this implementation, because the preset model is learned based on multiple groups of samples, and because the multiple groups of samples are images of multiple perspectives and / or multiple spectra of at least one scene, the preset model can effectively realize the fusion of images of multiple perspectives and / or multiple spectra in various scenes, and in specific implementation scenarios, the specific scenes in the multiple samples can be selected and expanded according to actual needs to ensure that the trained preset model can be effectively applied to multiple scenes to ensure the diversity of images to be fused that can be processed by the preset model, thereby effectively expanding the application scenarios of image fusion.

[0091] In one possible implementation, the preset model includes a feature extraction unit, the feature extraction unit includes N feature extraction layers, wherein the feature extraction unit is used to extract feature images, and the preset model also includes N gate units and 1 fusion unit, the gate unit is used to perform preliminary fusion of images, and the fusion unit is used to perform final fusion of the outputs of the gate units, and N is an integer greater than or equal to 1.

[0092] The image fusion method provided in the embodiment of the present application includes: obtaining multiple images to be fused, wherein the multiple images to be fused are images of multiple viewing angles and / or multiple spectra for the same scene. Performing rough registration processing on the multiple images to be fused to obtain multiple images to be fused after rough registration. Inputting the multiple images to be fused after rough registration into a preset model so that the preset model outputs a fused target image. Among them, the preset model is a model for fusing images, and the preset model is obtained by learning based on multiple groups of samples, each group of samples includes sample images to be fused and sample target images, wherein the sample images to be fused in the multiple groups of samples are images of multiple viewing angles and / or multiple spectra for at least one scene. By successively performing rough registration on multiple multi-view multi-spectral images to be fused, and using the preset model to perform image fusion to complete the fine registration of image fusion, and the preset model in this embodiment is obtained by training sample images of multiple viewing angles and / or multiple spectra under at least one scene, it is possible to effectively realize the image fusion of multi-view multi-spectral images under any scene to expand the application scenario of image fusion.

[0093] Based on the above embodiments, the following is a specific embodiment and Figure 2 , Figure 3The image fusion method provided in this application is further introduced in detail. Figure 2 The process of the image fusion method provided in the embodiment of the present application Figure 2 , Figure 3 A schematic diagram of the structure of the image fusion method provided in an embodiment of the present application.

[0094] The method comprises:

[0095] S201. Acquire multiple images to be fused, wherein the multiple images to be fused are images of multiple viewing angles and / or multiple spectra of the same scene.

[0096] The implementation of step S201 is similar to that of step S101 and will not be described in detail here.

[0097] In order to make the image fusion method of the embodiment of the present application more clear and explicit, the following describes the image fusion method in combination with Figure 3 The image fusion method in the present application is further introduced. Next, the structural composition of the image fusion method provided in the embodiment of the present application is first introduced.

[0098] like Figure 3 As shown, the preset model provided in the embodiment of the present application includes a feature extraction unit, a preliminary image fusion processing module, and a fine image fusion processing module. Among them, the feature extraction unit includes 4 feature extraction layers. The preliminary image fusion processing module includes 4 gate units and upsampling the results output by the gate units. It should be emphasized that the number of feature extraction layers in the feature extraction unit is consistent with the number of gate units.

[0099] Corresponds to Figure 3 In the process, the obtained multiple images to be fused are input as input images into the feature extraction unit in the preset model.

[0100] In the embodiments of the present application, only an exemplary introduction is given to the composition of the preset model, and no limitation is imposed on the composition of the preset model. The specific setting of the composition of the preset model can be selected according to actual needs.

[0101] S202: Determine homography matrices corresponding to the multiple images to be fused.

[0102] Based on step S201 , a plurality of images to be fused are obtained. In order to achieve a better effect after image fusion, a rough registration process needs to be performed before image fusion so that the plurality of images to be fused are located at the same viewing angle.

[0103] Next, the specific implementation method of the rough registration processing of the image to be fused is further introduced. First, SIFI feature detection and extraction are performed on the multiple multi-view multi-spectral images to be fused that have been acquired. Next, the feature points are matched using the nearest neighbor / next nearest neighbor method and the kd tree data structure. After the matching feature points are processed, the RANSAC algorithm is used to remove the matching outliers in the matching feature points. Finally, based on the remaining matching feature points, the homography matrix corresponding to the multiple images to be fused is determined. Specifically, the implementation method of determining the homography matrix can, for example, refer to any possible implementation method in the prior art, which will not be repeated here.

[0104] Corresponds to Figure 3 In the process of coarse registration of the input image, a homography matrix indicating the mapping relationship between points on different image planes is obtained.

[0105] S203 , transforming each image to be fused according to the homography matrices corresponding to the multiple images to be fused to obtain a plurality of roughly registered images to be fused, wherein the multiple roughly registered images to be fused have the same viewing angle.

[0106] After the homography matrix corresponding to the multiple images to be fused is determined based on step S202, the perspective transformation processing is performed on each image to be fused according to the homography matrix to obtain the multiple images to be fused after rough registration. The multiple images to be fused obtained after the rough registration processing are in the same perspective.

[0107] S204, inputting the multiple images to be fused after rough registration into the feature extraction unit of the preset model, and outputting feature images of N scales corresponding to each of the images to be fused through N feature extraction layers in the feature extraction unit.

[0108] After obtaining a plurality of images to be fused under the same viewing angle based on step S203, the specific implementation process of performing image fusion processing on the plurality of images to be fused under the same viewing angle using a preset model is described in detail below.

[0109] First, multiple images to be fused at the same viewing angle after rough registration are input into the feature extraction unit of the preset model to extract feature images of multiple scales from the multiple images to be fused at the same viewing angle. The feature extraction unit in the preset model includes N feature extraction layers, and different feature extraction layers output feature images of different scales. The image scale can be, for example, image resolution.

[0110] Multiple images to be fused under the same viewing angle are used as inputs of the feature extraction unit. Specifically, multiple images to be fused pass through N feature extraction layers in sequence. Each time through a feature extraction layer, the scale of the output feature image is continuously reduced. N feature images of different scales are output from the N feature extraction layers in the feature extraction unit.

[0111] Corresponds to Figure 3 In the method, the multiple images to be fused after rough registration are input into the feature extraction unit of the preset model, and the feature images of four scales corresponding to each image to be fused are output through the four feature extraction layers in the feature extraction unit.

[0112] In this embodiment, the feature extraction layer can be, for example, a pyramid extraction substructure. The four feature extraction layers correspond to the four pyramid feature extraction substructures, and the feature images of the multiple images to be fused at four different scales are extracted through the four pyramid features after rough registration processing. Structurally, the N pyramid feature extraction substructures form an image pyramid convolutional coding network. Among them, each pyramid feature extraction substructure is composed of two 3×3 convolutional layers with PReLU activation function and a maximum pooling layer. The number of convolution kernels in the convolution layer is 64. The maximum pooling layer is generally used after the convolution layer. Pooling is used to reduce the scale of the feature map output by the convolution layer, which can effectively reduce the network parameters and prevent overfitting.

[0113] In this embodiment, only an exemplary introduction is given to the feature extraction unit and the feature extraction layer in the preset model, and no limitation is imposed on the specific settings of the feature extraction unit and the feature extraction layer. The specific settings of the feature extraction unit and the feature extraction layer can be set according to actual needs.

[0114] S205, inputting the feature images of N scales corresponding to each image to be fused into N gate units respectively, so that each gate unit preliminarily fuses the feature images of N scales corresponding to each image to be fused, and obtains fusion results of N scales.

[0115] S206 , performing upsampling processing on the fusion results of the N scales, and inputting the upsampling fusion results of the N scales into a fusion unit, so that the fusion unit outputs a fused target image.

[0116] Next, steps S205 and S206 are introduced in detail.

[0117] Based on the above step S204, feature images of N scales corresponding to the images to be fused are obtained. Next, preliminary image fusion processing is performed on the images to be fused according to the feature images of N scales corresponding to the images to be fused obtained above.

[0118] In the preliminary image fusion processing, the gate unit and the upsampling operation are combined to realize the function of preliminary fusion of multiple images to be fused. Among them, the gate unit is composed of a convolution layer with an activation function. The upsampling operation is used to enlarge the scale of the image.

[0119] In the process of preliminary image fusion, the feature images of N scales corresponding to each image to be fused are respectively input into N gate units, so that each gate unit performs preliminary fusion on the feature images of multiple scales corresponding to each image to be fused, and obtains image fusion results of N scales.

[0120] After the gate unit outputs the image fusion results of N scales, in order to unify the scales of the image to be fused before the next fine registration process, it is necessary to perform upsampling processing on the image fusion results of the N scales. Specifically, the implementation method of performing upsampling processing on the image fusion results of the N scales can refer to any possible implementation method in the prior art, and will not be repeated here.

[0121] After the initial fusion processing of the image, the fusion results of N scales are obtained, and then the N scale fusion results after the upsampling processing are finely and roughly registered. Specifically, the N scale fusion results after the upsampling processing are input into the fusion unit, so that the fusion unit outputs the fused target image.

[0122] Corresponds to Figure 3 In the above, the feature images of the four scales corresponding to each image to be fused are input into four gate units respectively. Each gate unit is composed of a 1×1 convolution layer with a PReLU activation function. Each gate unit preliminarily fuses the feature images of the four scales corresponding to each image to be fused to obtain fusion results of the four scales. The fusion results of the four scales are upsampled. Specifically, the resolution of the fusion results of the four scales is adjusted to the target resolution by performing image interpolation processing on the fusion results of the four scales to complete the upsampling processing.

[0123] Next, the four scale fusion results after upsampling are input into the fine image fusion processing module, so that the fine image fusion processing module outputs the fused target image. The fine image fusion processing module corresponds to the image fusion module.

[0124] The image fusion method provided by the embodiment of the present application includes: obtaining multiple images to be fused, wherein the multiple images to be fused are images of multiple viewing angles and / or multiple spectra for the same scene. Determine the homography matrix corresponding to the multiple images to be fused. According to the homography matrix corresponding to the multiple images to be fused, transform each image to be fused to obtain multiple images to be fused after rough registration, wherein the viewing angles of the multiple images to be fused after rough registration are the same. Input the multiple images to be fused after rough registration into the feature extraction unit of the preset model, and output the feature images of N scales corresponding to each image to be fused through the N feature extraction layers in the feature extraction unit. Input the feature images of N scales corresponding to each image to be fused into N gate units respectively, so that each gate unit preliminarily fuses the feature images of N scales corresponding to each image to be fused, and obtains N scale fusion results. Upsample the N scale fusion results, and input the N scale fusion results after upsampling into the fusion unit, so that the fusion unit outputs the fused target image. Among them, by advancing the gate unit in the neural network, the gate unit combined with the upsampling operation realizes the preliminary fusion of multi-view and multi-spectral images.

[0125] Based on the above embodiment, the prediction model needs to be trained before it is applied. Figure 4 and Figure 5 The implementation process of the model training of this application is introduced. Figure 4 A schematic diagram of a process for training a prediction model provided in an embodiment of the present application, Figure 5 A schematic diagram of the structure of the model training provided in the embodiment of the present application.

[0126] S401 , processing sample images to be fused in multiple groups of samples by using an initial model to obtain fused images corresponding to the sample images to be fused.

[0127] In an embodiment of the present application, the sample images to be fused in the multiple groups of samples are images of multiple viewing angles and / or multiple spectra for at least one scene.

[0128] The sample images to be fused in the multiple groups of samples are processed by the initial module to obtain fused images corresponding to the sample images to be fused.

[0129] like Figure 5 As shown, the initial model includes a feature extraction unit, a preliminary image fusion processing module, a fine image fusion processing module, a fusion output module and a loss function.

[0130] It should be emphasized here that in the fine image fusion processing module, in order to adaptively evaluate the contribution of multi-view image priors, it is necessary to construct a graph for the pixel similarity of each feature image. For example, a graph G is constructed for the pixel feature similarity of feature image F. F = {V F , E F}, where, in the graph G, each pixel in the feature image F is a graph node, V F is a graph node set consisting of all pixels in the feature image F. The connection between each node is established based on the similarity (i.e., correlation) of the pixel direction between all pixels in the feature image F, where E F is the feature image G F The set of edges connecting each pixel in , where the length of the edge between each two pixels is determined by the similarity between the two pixel directions. For example, pixel p and pixel q are two graph nodes in the feature image F. The similarity between the two pixel directions can be determined by calculating the visual feature difference between the graph node p and the graph node q. Specifically, the similarity calculation method between the graph node p and the graph node q can refer to Formula 1:

[0131] S p,q =||F p -F q || 2 Formula 1

[0132] Among them, S p,q is the similarity between graph node p and graph node q, F p ,F q are the pixel sizes of graph nodes p and q respectively. The similarity between graph nodes p and q is obtained by taking the difference, taking the absolute value and squaring the absolute value of the pixel sizes of graph nodes p and q.

[0133] For the feature image G F For example, the feature image G F Each node in is only connected to the K graph nodes with the smallest distance from the top K in the feature space, which can be regarded as a K nearest neighbor graph. It should be noted that the K nearest neighbor graph is constantly and dynamically adjusted, and each node can adaptively select more useful priors from the dynamically updated adjacent nodes, while breaking away from the intensity consistency restrictions brought by the optical flow strategy.

[0134] Multiple groups of sample images to be fused are sequentially input into the feature extraction unit, preliminary image fusion processing module and fine image fusion processing module in the initial model for processing. After the processing is completed, they are output through the fusion output module to obtain the image fusion results corresponding to the sample images to be fused.

[0135] S402: Process the fused image and the sample target image according to the loss function to obtain a loss function value.

[0136] The loss function maps the value of a random event or its related random variables to a non-negative real number to represent the "risk" or "loss" of the random event. In general, the loss function is usually associated with the optimization problem as a learning criterion, that is, the model is solved and evaluated by minimizing the loss function. For example, it is used for model parameter estimation in statistics and machine learning.

[0137] In order to make the image fusion results of the training model more and more ideal during the training process of the initial model, a loss function is constructed as feedback for the back propagation of the graph neural network, so as to continuously train the model until the training model converges, that is, the preset model obtained after the training is completed can achieve the ideal image fusion effect when performing image fusion.

[0138] In this implementation, the constructed loss function mainly consists of two parts: 1. The structural level loss term L CONS =1-MSSIM; where SSIM(x j ,y j ) is the fused image pixel (x j ,y j ) is the structural similarity of the corresponding pixels in the original image, and M is the number of structured blocks in the image. 2. Pixel-level loss term L MSE , where L MSE is the mean square error loss function.

[0139] The loss function Loss is composed of the structure-level loss term L CONS And the pixel-level loss term is weighted and superimposed, which can be obtained by referring to Formula 2:

[0140] Loss = λ1L CONS +λ2L MSE Formula 2

[0141] Among them, λ1 and λ2 are both loss weights, and satisfy λ1+λ2=1.

[0142] In this embodiment, the loss function is only introduced as an example, and the loss function is not limited. The specific implementation method of the loss function can be set according to actual needs.

[0143] S403: Back-propagate the initial model according to the loss function value to adjust the model parameters of the initial model.

[0144] Based on the loss function obtained in step S402, the loss function is then used in the training process of the initial model, and the graph neural network fusion module is back-propagated according to the image fusion result to continuously correct the model parameters of the initial model.

[0145] S404: Determine whether the initial model meets the convergence condition. If yes, execute S405; if no, execute S402.

[0146] The convergence condition is used to indicate that the image fusion result processed by the initial model has reached the ideal image fusion result.

[0147] During each round of model training, the initial model is subjected to convergence judgment.

[0148] If it is determined that the initial model does not meet the convergence condition, steps S402 and S403 are repeated. If it is determined that the initial model meets the convergence condition, step S405 is executed.

[0149] S405: Obtain a preset model.

[0150] When it is judged that the initial model meets the convergence conditions, it indicates that the initial model training is completed and the preset model is obtained.

[0151] The image fusion method provided in the embodiment of the present application comprises: constructing a preset model. Processing the sample images to be fused in multiple groups of samples through the initial model to obtain a fused image corresponding to the sample images to be fused. Processing the fused image and the sample target image according to the loss function to obtain the loss function value. Back-propagating the initial model according to the loss function value to adjust the model parameters of the initial model. Repeating the above two steps until the initial model meets the convergence condition, the preset model is obtained.

[0152] Figure 6 This is a schematic diagram of the structure of the image fusion device provided in the embodiment of the present application. Figure 6 As shown, the device 60 includes: an acquisition module 601 , a first processing module 602 , and a second processing module 603 .

[0153] An acquisition module 601 is used to acquire multiple images to be fused, wherein the multiple images to be fused are images of multiple viewing angles and / or multiple spectra of the same scene;

[0154] A first processing module 602 is used to perform a rough registration process on the multiple images to be fused to obtain a plurality of images to be fused after rough registration;

[0155] The second processing module 603 is used to input the plurality of images to be fused after the rough registration into a preset model, so that the preset model outputs a fused target image;

[0156] Among them, the preset model is a model for fusing images, and the preset model is obtained by learning based on multiple groups of samples, each group of samples includes sample images to be fused and sample target images, wherein the sample images to be fused in the multiple groups of samples are images of multiple perspectives and / or multiple spectra for the same scene.

[0157] In one possible design, the preset model includes a feature extraction unit, and the feature extraction unit includes N feature extraction layers, wherein the feature extraction unit is used to extract feature images, and the preset model also includes N gate units and 1 fusion unit, the gate unit is used to perform preliminary fusion of images, and the fusion unit is used to perform final fusion of the outputs of the gate units, and N is an integer greater than or equal to 1.

[0158] In a possible design, the first processing module 602 is specifically configured to:

[0159] Inputting the plurality of images to be fused after the rough registration into the feature extraction unit of the preset model, and outputting N scale feature images corresponding to each of the images to be fused through N feature extraction layers in the feature extraction unit;

[0160] Inputting the N scale feature images corresponding to each of the images to be fused into the N gate units respectively, so that each of the gate units preliminarily fuses the N scale feature images corresponding to each of the images to be fused, and obtains N scale fusion results;

[0161] The N scale fusion results are upsampled, and the N scale fusion results after the upsampling process are input into the fusion unit, so that the fusion unit outputs a fused target image.

[0162] In a possible design, the first processing module 602 is specifically configured to:

[0163] Image interpolation processing is performed on the N scale fusion results to adjust the resolution of the N scale fusion results to a target resolution.

[0164] In a possible design, the first processing module 602 is specifically configured to:

[0165] Determine the homography matrix corresponding to the multiple images to be fused;

[0166] According to the homography matrices corresponding to the multiple images to be fused, each of the images to be fused is transformed to obtain multiple images to be fused after rough registration, wherein the multiple images to be fused after rough registration have the same viewing angle.

[0167] In a possible design, the second processing module 603 is further configured to:

[0168] The preset model is obtained by learning according to the multiple groups of samples.

[0169] In a possible design, the second processing module 603 is specifically used to:

[0170] Processing the sample images to be fused in the multiple groups of samples by using the initial model to obtain fused images corresponding to the sample images to be fused;

[0171] Processing the fused image and the sample target image according to the loss function to obtain a loss function value;

[0172] Performing back propagation on the initial model according to the loss function value to adjust the model parameters of the initial model;

[0173] The above two steps are repeated until the initial model meets the convergence condition to obtain the preset model.

[0174] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0175] Figure 7 A schematic diagram of the hardware structure of the image fusion device provided in the embodiment of the present application is shown in FIG. Figure 7 As shown, the image fusion device 70 of this embodiment includes: a processor 701 and a memory 702;

[0176] Memory 702, used to store computer-executable instructions;

[0177] The processor 701 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the image fusion method in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0178] Optionally, the memory 702 may be independent or integrated with the processor 701 .

[0179] When the memory 702 is independently provided, the image fusion device further includes a bus 703 for connecting the memory 702 and the processor 701 .

[0180] An embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the image fusion method performed by the above image fusion device is implemented.

[0181] An embodiment of the present application also provides a computer program product, which includes: a computer program, which is stored in a readable storage medium, and at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0182] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0183] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present application.

[0184] It should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented by a hardware processor or implemented by a combination of hardware and software modules in the processor.

[0185] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0186] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0187] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0188] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0189] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image fusion method, characterized in that: include: Acquire multiple images to be fused, wherein the multiple images to be fused are images of multiple viewing angles and / or multiple spectra of the same scene; Performing a coarse registration process on the multiple images to be fused to obtain a plurality of images to be fused after coarse registration; Inputting the plurality of images to be fused after the rough registration into a preset model, so that the preset model outputs a fused target image; The preset model is a model for fusing images, and the preset model is obtained by learning based on multiple groups of samples, each group of samples includes sample images to be fused and sample target images, and the sample images to be fused in the multiple groups of samples are images of multiple viewing angles and / or multiple spectra for at least one scene; The step of performing a rough registration process on the multiple images to be fused to obtain the multiple images to be fused after rough registration includes: Determine the homography matrix corresponding to the multiple images to be fused; According to the homography matrices corresponding to the multiple images to be fused, transform each of the images to be fused to obtain a plurality of images to be fused after rough registration, wherein the multiple images to be fused after rough registration have the same viewing angle; Among them, the preset model includes a feature extraction unit, and the feature extraction unit includes N feature extraction layers, wherein the feature extraction unit is used to extract feature images, and the preset model also includes N gate units and 1 fusion unit, the gate unit is used to perform preliminary fusion of images, and the fusion unit is used to perform final fusion of the outputs of the gate units, and N is an integer greater than or equal to 1.

2. The method according to claim 1, characterized in that The step of inputting the plurality of images to be fused after the rough registration into a preset model so that the preset model outputs a fused target image comprises: Inputting the plurality of images to be fused after the rough registration into the feature extraction unit of the preset model, and outputting N scale feature images corresponding to each of the images to be fused through N feature extraction layers in the feature extraction unit; Inputting the feature images of N scales corresponding to each of the images to be fused into the N gate units respectively, so that each of the gate units preliminarily fuses the feature images of N scales corresponding to each of the images to be fused, and obtains fusion results of N scales; The fusion results of the N scales are upsampled, and the fusion results of the N scales after the upsampling are input into the fusion unit, so that the fusion unit outputs a fused target image.

3. The method according to claim 2, characterized in that The upsampling process comprises: Image interpolation processing is performed on the fusion results of the N scales, and the resolution of the fusion results of the N scales is adjusted to the target resolution.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: The preset model is obtained by learning according to the multiple groups of samples.

5. The method according to claim 4, characterized in that The learning according to the multiple groups of samples to obtain the preset model includes: Processing the sample images to be fused in the multiple groups of samples by using the initial model to obtain fused images corresponding to the sample images to be fused; Processing the fused image and the sample target image according to the loss function to obtain a loss function value; Performing back propagation on the initial model according to the loss function value to adjust the model parameters of the initial model; The above two steps are repeated until the initial model meets the convergence condition to obtain the preset model.

6. An image fusion device, characterized in that: include: An acquisition module, used for acquiring a plurality of images to be fused, wherein the plurality of images to be fused are images of multiple viewing angles and / or multiple spectra of the same scene; A first processing module is used to perform a rough registration process on the multiple images to be fused to obtain a plurality of images to be fused after rough registration; A second processing module, used for inputting the plurality of images to be fused after the rough registration into a preset model, so that the preset model outputs a fused target image; The preset model is a model for fusing images, and the preset model is obtained by learning based on multiple groups of samples, each group of samples includes sample images to be fused and sample target images, and the sample images to be fused in the multiple groups of samples are images of multiple viewing angles and / or multiple spectra for at least one scene; Wherein, the second processing module is specifically used for: Determine the homography matrix corresponding to the multiple images to be fused; According to the homography matrices corresponding to the multiple images to be fused, transform each of the images to be fused to obtain a plurality of images to be fused after rough registration, wherein the multiple images to be fused after rough registration have the same viewing angle; Among them, the preset model includes a feature extraction unit, and the feature extraction unit includes N feature extraction layers, wherein the feature extraction unit is used to extract feature images, and the preset model also includes N gate units and 1 fusion unit, the gate unit is used to perform preliminary fusion of images, and the fusion unit is used to perform final fusion of the outputs of the gate units, and N is an integer greater than or equal to 1.

7. An image fusion device, characterized in that: include: Memory, used to store programs; A processor, configured to execute the program stored in the memory; when the program is executed, the processor is configured to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The method comprises instructions which, when executed on a computer, cause the computer to perform the method as claimed in any one of claims 1 to 5.

Images