Image fusion method and device based on generative adversarial dual-channel weight allocation

By generating an image fusion method with adversarial dual-channel weight allocation, the problem of information imbalance between infrared and visible light images is solved, and a balanced fused image containing heatmap and texture information is generated.

CN114972146BActive Publication Date: 2024-11-12WUHAN INST OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210603129.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-25
Publication Date
2024-11-12
Estimated Expiration
2042-05-25

AI Technical Summary

Technical Problem

Existing technologies struggle to balance the information from infrared and visible light images, resulting in the inability to obtain a fused image with balanced information.

Method used

An image fusion method with generative adversarial dual-channel weight allocation is adopted. By constructing an initial image fusion model, the features of infrared and visible light images are extracted using a generator module and a twin discriminator module. The target image fusion model is then trained through feature fusion and similarity calculation to generate a fused image with balanced information.

Benefits of technology

The generated fused image contains more thermal map information from the infrared image and texture information from the visible light image, achieving full extraction of the main salient target information from the infrared image and the main detail information from the visible light image, generating an information-balanced image with clear targets and rich detail textures.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114972146B_ABST
    Figure CN114972146B_ABST
Patent Text Reader

Abstract

The present invention provides an image fusion method and device based on generative adversarial dual-channel weight allocation, the method comprising: constructing an initial image fusion model, including a generator module and a twin discriminator module, the generator module including a pseudo twin feature extraction module and a feature fusion module; obtaining infrared image samples and visible light image samples, and obtaining infrared feature maps and visible light feature maps based on the pseudo twin feature extraction module; generating fusion feature maps based on the feature fusion module; obtaining the first similarity of the fusion feature map with the infrared image sample and the second similarity with the visible light image sample based on the twin discriminator module; training the initial image fusion model to obtain a target image fusion model; fusing the infrared image to be fused and the visible light image to be fused based on the target image fusion model to obtain a target fusion image. The present invention can obtain a target fusion image with a clear target and rich detail texture.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to an image fusion method and device based on generative adversarial dual-channel weight allocation. Background Art

[0002] Image fusion is to extract meaningful information from images collected by different sensors and then combine them to generate a single image, which contains richer information and is more conducive to subsequent applications. Among them, the fusion of infrared images and visible light images is the most widely used. Visible light images are generated by visible light sensors that capture reflected light. They are characterized by rich texture detail information and conform to the observation rules of the human eye. Infrared sensors can sense infrared bands and convert them into thermal radiation information to generate grayscale images. Infrared images have obvious contrast and can effectively distinguish between background and target even at night and in bad weather.

[0003] The key to image fusion is to extract and reconstruct the most meaningful information. For the fusion of visible light images and infrared images, the most meaningful information is significant contrast and rich texture information. To achieve this goal, researchers have proposed many fusion methods, which can be divided into traditional methods and deep learning-based methods. The traditional method is to achieve image fusion according to specific fusion rules by measuring the activity of pixels or regions in the spatial domain or transform domain. Typical traditional methods are sparse representation-based methods, multi-scale transformation-based methods, subspace-based methods, saliency-based methods, and hybrid methods. The deep learning-based method uses the powerful nonlinear fitting ability of neural networks to make the fused image have an ideal distribution, but the fused image generated by the current fusion method is difficult to balance the information of infrared images and visible light images, resulting in the fused image being biased towards a certain source image.

[0004] Therefore, it is urgent to propose an image fusion method and device based on generative adversarial dual-channel weight allocation to solve the technical problem in the prior art that it is difficult to balance the information of infrared images and visible light images, resulting in the inability to obtain a fused image with balanced information. Summary of the invention

[0005] In view of this, it is necessary to provide an image fusion method and device based on generative adversarial dual-channel weight allocation to solve the technical problem in the prior art that it is difficult to balance the information of infrared images and visible light images, resulting in the inability to obtain a fused image with balanced information.

[0006] On the one hand, the present invention provides an image fusion method based on generative adversarial dual-channel weight allocation, comprising:

[0007] Constructing an initial image fusion model, wherein the initial image fusion model includes a generator module and a twin discriminator module, wherein the generator module includes a pseudo-twin feature extraction module and a feature fusion module;

[0008] Acquire an infrared image sample and a visible light image sample, and extract features of the infrared image sample and the visible light image sample respectively based on the pseudo-twin feature extraction module, and obtain an infrared feature map and a visible light feature map correspondingly;

[0009] Based on the feature fusion module, the infrared feature map and the visible light feature map are fused to generate a fused feature map;

[0010] Based on the twin identification module, respectively obtain a first similarity between the infrared image sample and the fused feature map and a second similarity between the visible light image sample and the fused feature map;

[0011] The initial image fusion model is trained based on a preset loss function, the first similarity, the second similarity, the infrared image sample, and the visible light image sample to obtain a fully trained target image fusion model;

[0012] The infrared image to be fused and the visible light image to be fused are fused based on the target image fusion model to obtain a target fused image.

[0013] In some possible implementations, the training of the initial image fusion model based on a preset loss function, the first similarity, the second similarity, the infrared image sample, and the visible light image sample to obtain a fully trained target image fusion model includes:

[0014] Determine whether a difference between the first similarity and the second similarity is less than a preset difference;

[0015] When the difference between the first similarity and the second similarity is less than the preset difference, the initial image fusion model is the target image fusion model;

[0016] When the difference between the first similarity and the second similarity is greater than or equal to the preset difference, the initial image fusion model continues to be trained based on a preset loss function, the infrared image samples, and the visible light image samples.

[0017] In some possible implementations, the pseudo-twin feature extraction module includes an infrared feature extraction unit and a visible light feature extraction unit, the infrared feature extraction unit and the visible light feature extraction unit each include a first feature extraction convolution unit and four second feature extraction convolution units, the first feature extraction convolution unit includes a first convolution layer with a convolution kernel of 5×5 and a first leaky ReLU activation layer, the second feature extraction convolution unit includes a second convolution layer with a convolution kernel of 3×3, a batch normalization layer and a second leaky ReLU activation layer.

[0018] In some possible implementations, the feature fusion module includes a first fused convolution unit, a second fused convolution unit, a third fused convolution unit, a cross-connected convolution layer, and a fusion target fused leakyReLU activation layer, the first fused convolution unit includes a first fused convolution layer with a convolution kernel of 1×1 and a first fused leakyReLU activation layer, the second fused convolution unit includes a second fused convolution layer with a convolution kernel of 3×3 and a second fused leakyReLU activation layer, the third fused convolution unit includes a third fused convolution layer with a convolution kernel of 1×1 and a third fused Tanh activation layer, and the convolution kernel of the cross-connected convolution layer is 1×1.

[0019] In some possible implementations, the twin identification module includes an infrared twin identification unit and a visible light twin identification unit. The infrared twin identification unit is used to determine a first similarity between the infrared image sample and the fused feature map, and the visible light twin identification unit is used to determine a second similarity between the visible light image sample and the fused feature map.

[0020] In some possible implementations, the first similarity is:

[0021]

[0022] The second similarity is:

[0023]

[0024] In the formula, S1(I f ,I ir ) is the first similarity; S2(I f ,I vi ) is the second similarity; I f is the fusion feature map; I ir is the infrared image sample; I vi is a visible light image sample; P is the characteristic dimension of an infrared image sample or a visible light image sample; is the i-th dimension feature of the fused feature map; is the i-th dimension feature of the infrared image sample; is the i-th dimension feature of the visible light image sample; || ||2 is the binary norm.

[0025] In some possible implementations, the loss function includes a feature extraction loss function, and the feature extraction loss function is:

[0026]

[0027] Where L(W) is the feature extraction loss function; ω ir is the weight coefficient of the infrared feature extraction unit; ω vi is the weight coefficient of the visible light feature extraction unit; is the label indicating whether the k-th visible light image sample matches the k-th infrared image sample; is the Euclidean distance between the kth infrared feature map and the kth visible light feature map; m is the spacing threshold between different input categories.

[0028] In some possible implementations, the loss function includes a generator loss function, and the generator loss function is:

[0029] L G =β(L Gir +L Gvi )+L Gadv

[0030]

[0031]

[0032]

[0033] Where, L G is the generator loss function; L Gir is the content loss of infrared image; L Gvi is the content loss of the visible light image; L Gadv is the adversarial loss between the generator module and the twin discriminator module; β is the regularization parameter; H is the height of the infrared image sample and the visible light image sample; W is the width of the infrared image sample and the visible light image sample; || || F is the Frobenius norm; is the gradient operator; α1 is the first positive parameter; α2 is the second positive parameter; N is the total number of fused images; is the classification result of the infrared twin identification unit for the nth fused image, is the classification result of the visible light twin discriminator for the nth fused image; c is the false data value that the generator module wants the twin discriminator to believe.

[0034] In some possible implementations, the loss function further includes a discriminator loss function, and the discriminator loss function is:

[0035] L D =L Dvi +L Dir

[0036]

[0037]

[0038] Where, L D is the discriminator loss function; L Dvi is the visible light image judgment loss; L Dir Determine the loss for infrared images; is the judgment result of the infrared twin identification unit on the nth infrared image sample; is the judgment result of the infrared twin identification unit on the nth fused image; is the judgment result of the visible light twin identification unit on the nth visible light image sample; is the judgment result of the visible light twin identification unit on the nth fused image; p1 is the first probability label; p2 is the second probability label; b1 is the third probability label; b2 is the fourth probability label.

[0039] On the other hand, the present invention also provides an image fusion device based on generative adversarial dual-channel weight allocation, comprising:

[0040] An initial model construction module, used to construct an initial image fusion model, wherein the initial image fusion model includes a generator module and a twin discriminator module, wherein the generator module includes a pseudo-twin feature extraction module and a feature fusion module;

[0041] A feature extraction module, used to obtain infrared image samples and visible light image samples, and extract features of the infrared image samples and the visible light image samples respectively based on the pseudo-twin feature extraction module, and correspondingly obtain an infrared feature map and a visible light feature map;

[0042] A feature fusion module, used for fusing the infrared feature map and the visible light feature map based on the feature fusion module to generate a fused feature map;

[0043] an identification module, used for respectively obtaining a first similarity between the infrared image sample and the fused feature map and a second similarity between the visible light image sample and the fused feature map based on the twin identification module;

[0044] A model training module, used for training the initial image fusion model based on a preset loss function, the first similarity, the second similarity, the infrared image samples and the visible light image samples to obtain a fully trained target image fusion model;

[0045] The image fusion module is used to fuse the infrared image to be fused and the visible light image to be fused based on the target image fusion model to obtain a target fused image.

[0046] The beneficial effect of adopting the above-mentioned embodiment is: the image fusion method based on generative adversarial dual-channel weight allocation provided by the present invention utilizes a generative adversarial deep learning method to make the fused image contain more thermal map information of the infrared image and texture information of the visible light image, and on this basis, utilizes a pseudo-twin feature extraction module and a twin discriminator module to realize dual-channel weight learning and sharing of infrared images and visible light images, fully extracting the main salient target information and secondary texture information of the infrared image, as well as the main detail information and secondary salient target information of the visible light image, to generate an information-balanced fused image with clear targets and rich detail textures.

[0047] Furthermore, the present invention uses the fused image, the infrared image and the visible light image as two sets of image pairs, and inputs each into the twin identification module. The twin identification module maps the input infrared image and visible light image to the feature space, and determines the first similarity and the second similarity by calculating the Euclidean distance between the feature vectors. Since the twin identification module uses the idea of ​​weight distribution, when calculating the first similarity and the second similarity, it not only retains the specific information of the infrared image and the visible light image by using dual channels, but also reduces the complexity of parameter adjustment by weight sharing, and is not prone to overfitting. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 A schematic flow chart of an embodiment of the image fusion method based on generative adversarial dual-channel weight allocation provided by the present invention;

[0050] Figure 2 A schematic diagram of the structure of an embodiment of the initial image fusion model provided by the present invention;

[0051] Figure 3 For the present invention Figure 1A schematic flow chart of an embodiment of S105;

[0052] Figure 4 A schematic diagram of the structure of an embodiment of a pseudo-twin feature extraction module provided by the present invention;

[0053] Figure 5 A schematic diagram of the structure of an embodiment of the feature fusion module provided by the present invention;

[0054] Figure 6 A schematic diagram of the structure of an embodiment of an image fusion device based on generative adversarial dual-channel weight allocation provided by the present invention;

[0055] Figure 7 A schematic structural diagram of an embodiment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0057] It should be understood that the schematic drawings are not drawn to scale. The flow chart used in the present invention shows the operations implemented according to some embodiments of the present invention. It should be understood that the operations of the flow chart can be implemented out of order, and the steps without logical context can be reversed or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flow chart under the guidance of the content of the present invention, and can also remove one or more operations from the flow chart.

[0058] Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor systems and / or microcontroller systems.

[0059] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0060] The embodiments of the present invention provide an image fusion method and device based on generative adversarial dual-channel weight allocation, which are described below respectively.

[0061] Figure 1 A schematic flow chart of an embodiment of the image fusion method based on generative adversarial dual-channel weight allocation provided by the present invention is shown in FIG. Figure 2 A schematic diagram of the structure of an embodiment of the initial image fusion model provided by the present invention is shown in FIG. Figure 1 and Figure 2 As shown, the image fusion method based on generative adversarial dual-channel weight allocation includes:

[0062] S101, constructing an initial image fusion model, the initial image fusion model including a generator module and a twin discriminator module, the generator module including a pseudo twin feature extraction module and a feature fusion module;

[0063] S102, obtaining infrared image samples and visible light image samples, and extracting features of the infrared image samples and the visible light image samples respectively based on the pseudo-twin feature extraction module, and correspondingly obtaining an infrared feature map and a visible light feature map;

[0064] S103, fusing the infrared feature map and the visible light feature map based on a feature fusion module to generate a fused feature map;

[0065] S104, obtaining a first similarity between the infrared image sample and the fused feature map and a second similarity between the visible light image sample and the fused feature map based on the twin identification module;

[0066] S105, training the initial image fusion model based on a preset loss function, the first similarity, the second similarity, the infrared image samples, and the visible light image samples to obtain a fully trained target image fusion model;

[0067] S106 , fusing the infrared image to be fused and the visible light image to be fused based on the target image fusion model to obtain a target fused image.

[0068] Compared with the prior art, the image fusion method based on generative adversarial dual-channel weight allocation provided in an embodiment of the present invention utilizes a generative adversarial deep learning method to make the fused image contain more thermal map information of the infrared image and texture information of the visible light image, and on this basis utilizes a pseudo-twin feature extraction module and a twin discriminator module to realize dual-channel weight learning and sharing of the infrared image and the visible light image, fully extracting the main salient target information and secondary texture information of the infrared image, as well as the main detail information and secondary salient target information of the visible light image, to generate an information-balanced fused image with clear targets and rich detail textures.

[0069] Furthermore, the embodiment of the present invention uses the fused image, the infrared image and the visible light image as two sets of image pairs, and inputs each into the twin identification module. The twin identification module maps the input infrared image and visible light image to the feature space, and determines the first similarity and the second similarity by calculating the Euclidean distance between the feature vectors. Since the twin identification module uses the idea of ​​weight distribution, when calculating the first similarity and the second similarity, it not only retains the specific information of the infrared image and the visible light image by using dual channels, but also reduces the complexity of parameter adjustment by weight sharing, and is not prone to overfitting.

[0070] In some embodiments of the present invention, Figure 3 As shown, step S105 includes:

[0071] S301, determining whether the difference between the first similarity and the second similarity is less than a preset difference;

[0072] S302: when the difference between the first similarity and the second similarity is less than a preset difference, the initial image fusion model is the target image fusion model;

[0073] S303: When the difference between the first similarity and the second similarity is greater than or equal to a preset difference, continue to train the initial image fusion model based on a preset loss function, infrared image samples, and visible light image samples.

[0074] In the embodiment of the present invention, when the difference between the first similarity and the second similarity is less than a preset difference, the initial image fusion model is the target image fusion model, that is, the probability of the fused image being identified as an infrared image or a visible light image by the twin identification module is basically the same, which means that the image information in the fused image is relatively balanced. This can further ensure the balance of the image information in the generated target fused image.

[0075] It should be understood that the preset difference can be adjusted according to experience or actual application scenarios, which will not be elaborated here.

[0076] In a specific embodiment of the present invention, Figure 2 and Figure 4 As shown, the pseudo-twin feature extraction module includes an infrared feature extraction unit and a visible light feature extraction unit. The infrared feature extraction unit and the visible light feature extraction unit each include a first feature extraction convolution unit and four second feature extraction convolution units. The first feature extraction convolution unit includes a first convolution layer with a convolution kernel of 5×5 and a first leaky ReLU activation layer. The second feature extraction convolution unit includes a second convolution layer with a convolution kernel of 3×3, a batch normalization layer, and a second leaky ReLU activation layer.

[0077] In a specific embodiment of the present invention, Figure 2and Figure 5 As shown, the feature fusion module includes a first fused convolution unit, a second fused convolution unit, a third fused convolution unit, a cross-connected convolution layer, and a fusion target fused leakyReLU activation layer. The first fused convolution unit includes a first fused convolution layer with a convolution kernel of 1×1 and a first fused leakyReLU activation layer. The second fused convolution unit includes a second fused convolution layer with a convolution kernel of 3×3 and a second fused leakyReLU activation layer. The third fused convolution unit includes a third fused convolution layer with a convolution kernel of 1×1 and a third fused Tanh activation layer. The convolution kernel of the cross-connected convolution layer is 1×1.

[0078] The embodiment of the present invention can ensure that the variation range of the fused image is consistent with the variation range of the input image (infrared image samples and visible light image samples) by setting the third fused convolution unit including the third fused Tanh activation layer. The problem of inconsistent input and output dimensions of the feature fusion module is overcome by setting the cross-connected convolution layer.

[0079] In a specific embodiment of the present invention, Figure 2 As shown, the twin identification module includes an infrared twin identification unit and a visible light twin identification unit. The infrared twin identification unit is used to determine the first similarity between the infrared image sample and the fused feature map, and the visible light twin identification unit is used to determine the second similarity between the visible light image sample and the fused feature map.

[0080] Among them, the first similarity is:

[0081]

[0082] The second similarity is:

[0083]

[0084] In the formula, S1(I f ,I ir ) is the first similarity; S2(I f ,I vi ) is the second similarity; I f is the fusion feature map; I ir is the infrared image sample; I vi is a visible light image sample; P is the characteristic dimension of an infrared image sample or a visible light image sample; is the i-th dimension feature of the fused feature map; is the i-th dimension feature of the infrared image sample; is the i-th dimension feature of the visible light image sample; || ||2 is the binary norm.

[0085] In the pseudo-twin feature extraction module, an infrared feature extraction unit and a visible light feature extraction unit are used to extract the features of infrared image samples and visible light image samples respectively. Since the input image samples are of different categories, the pseudo-twin feature extraction module hopes that the loss function is as small as possible when the two input image samples are of the same category, and as large as possible when the two input image samples are of different categories. Specifically, the loss function includes a feature extraction loss function, and the feature extraction loss function is:

[0086]

[0087] Where L(W) is the feature extraction loss function; ω ir is the weight coefficient of the infrared feature extraction unit; ω vi is the weight coefficient of the visible light feature extraction unit; is the label indicating whether the k-th visible light image sample matches the k-th infrared image sample; is the Euclidean distance between the kth infrared feature map and the kth visible light feature map; m is the spacing threshold between different input categories.

[0088] By setting a feature extraction loss function, the embodiment of the present invention can adjust the weight distribution of the infrared feature extraction unit and the visible light feature extraction unit in the pseudo-twin feature extraction module through the feature extraction loss function when training the initial image fusion model, so that the fused image can highlight the significant targets in the infrared image while retaining as much detail information of the visible light image as possible.

[0089] It should be understood that the generator module is used to generate a fused image. The generator module hopes that the generated fused image can be judged as true by the twin discriminator module, while the twin discriminator module needs to judge the infrared image samples and the visible light image samples as true. Therefore, the generator module and the twin discriminator module perform the maximum and minimum game according to the following formula:

[0090]

[0091] Where x is a visible light image sample; D(x) is the output of the twin discriminator module; z is the input of the generator module; G(z) is the output of the generator module; pdata is the real image data distribution; pz is the fused image data distribution; E x~pdata Represents mathematical expectation, x obeys probability distribution pdata; E z~pz Represents the mathematical expectation, x obeys the probability pz respectively.

[0092] After the generator module generates the fused image, it is necessary to use a loss function to determine the loss value of the fused image and the infrared image sample and the visible light image sample. Therefore, in some embodiments of the present invention, the loss function includes a generator loss function, and the generator loss function is:

[0093] L G =β(L Gir +L vi )+L Gadv

[0094]

[0095]

[0096]

[0097] Where, L G is the generator loss function; L Gir is the content loss of infrared image; L vi is the content loss of the visible light image; L Gadv is the adversarial loss between the generator module and the twin discriminator module; β is the regularization parameter; H is the height of the infrared image sample and the visible light image sample; W is the width of the infrared image sample and the visible light image sample; || || F is the Frobenius norm; is the gradient operator; α1 is the first positive parameter; α2 is the second positive parameter; N is the total number of fused images; is the classification result of the infrared twin identification unit for the nth fused image, is the classification result of the visible light twin discriminator for the nth fused image; c is the false data value that the generator module wants the twin discriminator to believe. Specifically, c is set to 1.

[0098] Since the twin discriminator module uses an infrared feature extraction unit and a visible light feature extraction unit to respectively calculate the first similarity between the fused image and the infrared image sample and the second similarity between the fused image and the visible light image sample, in order to improve the identification ability of the twin discriminator module, in some embodiments of the present invention, the loss function also includes a discriminator loss function, and the discriminator loss function is:

[0099] L D =L Dvi +L Dir

[0100]

[0101]

[0102] Where, LD is the discriminator loss function; L Dvi is the visible light image judgment loss; L Dir Determine the loss for infrared images; is the judgment result of the infrared twin identification unit on the nth infrared image sample; is the judgment result of the infrared twin identification unit on the nth fused image; is the judgment result of the visible light twin identification unit on the nth visible light image sample; is the judgment result of the visible light twin identification unit on the nth fused image; p1 is the first probability label; p2 is the second probability label; b1 is the third probability label; b2 is the fourth probability label.

[0103] Specifically: p1 is 1; p2 is 0; b1 is 1; b2 is 0.

[0104] In order to better implement the image fusion method based on generative adversarial dual-channel weight allocation in the embodiment of the present invention, on the basis of the image fusion method based on generative adversarial dual-channel weight allocation, the embodiment of the present invention also provides an image fusion device based on generative adversarial dual-channel weight allocation, such as Figure 6 As shown, the image fusion device 600 based on generative adversarial dual-channel weight allocation includes:

[0105] An initial model construction module 601 is used to construct an initial image fusion model, the initial image fusion model includes a generator module and a twin discriminator module, the generator module includes a pseudo twin feature extraction module and a feature fusion module;

[0106] A feature extraction module 602 is used to obtain infrared image samples and visible light image samples, and respectively extract features of the infrared image samples and the visible light image samples based on the pseudo-twin feature extraction module, and correspondingly obtain an infrared feature map and a visible light feature map;

[0107] A feature fusion module 603 is used to fuse the infrared feature map and the visible light feature map based on the feature fusion module to generate a fused feature map;

[0108] An identification module 604, used to obtain a first similarity between the infrared image sample and the fused feature map and a second similarity between the visible light image sample and the fused feature map based on the twin identification module;

[0109] A model training module 605 is used to train the initial image fusion model based on a preset loss function, a first similarity, a second similarity, infrared image samples, and visible light image samples to obtain a fully trained target image fusion model;

[0110] The image fusion module 606 is used to fuse the infrared image to be fused and the visible light image to be fused based on the target image fusion model to obtain a target fused image.

[0111] The image fusion device 600 based on generative adversarial dual-channel weight allocation provided in the above embodiment can implement the technical solution described in the above embodiment of the image fusion method based on generative adversarial dual-channel weight allocation. The specific implementation principles of the above modules or units can refer to the corresponding contents in the above embodiment of the image fusion method based on generative adversarial dual-channel weight allocation, which will not be repeated here.

[0112] like Figure 7 As shown, the present invention also provides an electronic device 700. The electronic device 700 includes a processor 701, a memory 702 and a display 703. Figure 7 Only some components of the electronic device 700 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0113] In some embodiments, the processor 701 can be a central processing unit (CPU), a microprocessor or other data processing chip, which is used to run the program code or process data stored in the memory 702, such as the image fusion method based on generative adversarial dual-channel weight allocation in the present invention.

[0114] In some embodiments, the processor 701 may be a single server or a server group. The server group may be centralized or distributed. In some embodiments, the processor 701 may be local or remote. In some embodiments, the processor 701 may be implemented in a cloud platform. In one embodiment, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-cloud, etc., or any combination thereof.

[0115] In some embodiments, the memory 702 may be an internal storage unit of the electronic device 700, such as a hard disk or memory of the electronic device 700. In other embodiments, the memory 702 may also be an external storage device of the electronic device 700, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 700.

[0116] Furthermore, the memory 702 may include both an internal storage unit of the electronic device 700 and an external storage device. The memory 702 is used to store application software installed in the electronic device 700 and various data.

[0117] In some embodiments, the display 703 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 703 is used to display information on the electronic device 700 and to display a visual user interface. The components 701-703 of the electronic device 700 communicate with each other via a system bus.

[0118] In one embodiment, when the processor 701 executes the image fusion program based on generative adversarial dual-channel weight allocation in the memory 702, the following steps may be implemented:

[0119] Constructing an initial image fusion model, the initial image fusion model includes a generator module and a twin discriminator module, the generator module includes a pseudo twin feature extraction module and a feature fusion module;

[0120] Obtain infrared image samples and visible light image samples, and extract features of the infrared image samples and visible light image samples respectively based on the pseudo-twin feature extraction module, and obtain infrared feature maps and visible light feature maps accordingly;

[0121] Based on the feature fusion module, the infrared feature map and the visible light feature map are fused to generate a fused feature map;

[0122] Based on the twin identification module, a first similarity between the infrared image sample and the fused feature map and a second similarity between the visible light image sample and the fused feature map are respectively obtained;

[0123] The initial image fusion model is trained based on a preset loss function, a first similarity, a second similarity, infrared image samples, and visible light image samples to obtain a fully trained target image fusion model;

[0124] The infrared image to be fused and the visible light image to be fused are fused based on the target image fusion model to obtain the target fused image.

[0125] It should be understood that: when the processor 701 executes the image fusion program based on generative adversarial dual-channel weight allocation in the memory 702, in addition to the above functions, other functions can also be implemented. For details, please refer to the description of the corresponding method embodiment above.

[0126] Furthermore, the embodiment of the present invention does not specifically limit the type of the electronic device 700 mentioned, and the electronic device 700 may be a portable electronic device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with IOS, Android, Microsoft or other operating systems. The above-mentioned portable electronic device may also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 700 may not be a portable electronic device, but a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0127] Accordingly, an embodiment of the present application also provides a computer-readable storage medium, which is used to store computer-readable programs or instructions. When the program or instructions are executed by a processor, it can implement the steps or functions of the image fusion method based on generative adversarial dual-channel weight allocation provided in the above-mentioned method embodiments.

[0128] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0129] The above is a detailed introduction to the image fusion method and device based on generative adversarial dual-channel weight allocation provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An image fusion method based on generative adversarial dual-channel weight allocation, characterized in that: include: Constructing an initial image fusion model, wherein the initial image fusion model includes a generator module and a twin discriminator module, and the generator module includes a pseudo-twin feature extraction module and a feature fusion module; Acquire an infrared image sample and a visible light image sample, and extract features of the infrared image sample and the visible light image sample respectively based on the pseudo-twin feature extraction module, and obtain an infrared feature map and a visible light feature map correspondingly; Based on the feature fusion module, the infrared feature map and the visible light feature map are fused to generate a fused feature map; Based on the twin discriminator module, respectively obtain a first similarity between the infrared image sample and the fused feature map and a second similarity between the visible light image sample and the fused feature map; The initial image fusion model is trained based on a preset loss function, the first similarity, the second similarity, the infrared image sample, and the visible light image sample to obtain a fully trained target image fusion model; Based on the target image fusion model, the infrared image to be fused and the visible light image to be fused are fused to obtain a target fused image; The initial image fusion model is trained based on a preset loss function, the first similarity, the second similarity, the infrared image sample, and the visible light image sample to obtain a fully trained target image fusion model, including: Determine whether a difference between the first similarity and the second similarity is less than a preset difference; When the difference between the first similarity and the second similarity is less than the preset difference, the initial image fusion model is the target image fusion model; When the difference between the first similarity and the second similarity is greater than or equal to the preset difference, continuing to train the initial image fusion model based on a preset loss function, the infrared image sample and the visible light image sample; The first similarity is: The second similarity is: In the formula, is the first similarity; is the second similarity; is the fusion feature map; is the infrared image sample; is a visible light image sample; P is the characteristic dimension of an infrared image sample or a visible light image sample; is the i-th dimension feature of the fused feature map; is the i-th dimension feature of the infrared image sample; is the i-th dimension feature of the visible light image sample; is the two-norm.

2. The image fusion method based on generative adversarial dual-channel weight allocation according to claim 1 is characterized in that: The pseudo-twin feature extraction module includes an infrared feature extraction unit and a visible light feature extraction unit. The infrared feature extraction unit and the visible light feature extraction unit each include a first feature extraction convolution unit and four second feature extraction convolution units. The first feature extraction convolution unit includes a first convolution layer with a convolution kernel of 5×5 and a first leaky ReLU activation layer. The second feature extraction convolution unit includes a second convolution layer with a convolution kernel of 3×3, a batch normalization layer, and a second leaky ReLU activation layer.

3. The image fusion method based on generative adversarial dual-channel weight allocation according to claim 2 is characterized in that: The feature fusion module includes a first fused convolution unit, a second fused convolution unit, a third fused convolution unit, a cross-connected convolution layer, and a fusion target fused leaky ReLU activation layer. The first fused convolution unit includes a first fused convolution layer with a convolution kernel of 1×1 and a first fused leaky ReLU activation layer. The second fused convolution unit includes a second fused convolution layer with a convolution kernel of 3×3 and a second fused leaky ReLU activation layer. The third fused convolution unit includes a third fused convolution layer with a convolution kernel of 1×1 and a third fused Tanh activation layer. The convolution kernel of the cross-connected convolution layer is 1×1.

4. The image fusion method based on generative adversarial dual-channel weight allocation according to claim 3 is characterized in that: The twin identifier module includes an infrared twin identification unit and a visible light twin identification unit. The infrared twin identification unit is used to determine a first similarity between the infrared image sample and the fused feature map, and the visible light twin identification unit is used to determine a second similarity between the visible light image sample and the fused feature map.

5. The image fusion method based on generative adversarial dual-channel weight allocation according to claim 4 is characterized in that: The loss function includes a feature extraction loss function, and the feature extraction loss function is: In the formula, Extract loss function for features; is the weight coefficient of the infrared feature extraction unit; is the weight coefficient of the visible light feature extraction unit; is the label indicating whether the k-th visible light image sample matches the k-th infrared image sample; is the Euclidean distance between the kth infrared feature map and the kth visible light feature map; m is the spacing threshold between different input categories.

6. The image fusion method based on generative adversarial dual-channel weight allocation according to claim 5 is characterized in that: The loss function includes a generator loss function, and the generator loss function is: In the formula, is the generator loss function; It is the content loss of infrared image; It is the content loss of the visible light image; is the adversarial loss between the generator module and the twin discriminator module; is the regularization parameter; H is the height of the infrared image sample and the visible light image sample; W is the width of the infrared image sample and the visible light image sample; is the Frobenius norm; is the gradient operator; is the first positive parameter; is the second positive parameter; N is the total number of fused images; is the classification result of the infrared twin identification unit for the nth fused image, is the classification result of the visible light twin identification unit on the nth fused image; c is the fake data value that the generator module wants the twin discriminator to believe.

7. The image fusion method based on generative adversarial dual-channel weight allocation according to claim 6 is characterized in that: The loss function also includes a discriminator loss function, which is: In the formula, is the discriminator loss function; Determine the loss for visible light images; Determine the loss for infrared images; is the judgment result of the infrared twin identification unit on the nth infrared image sample; is the judgment result of the infrared twin identification unit on the nth fused image; is the judgment result of the visible light twin identification unit on the nth visible light image sample; is the judgment result of the visible light twin identification unit on the nth fused image; is the first probability label; is the second probability label; is the third probability label; is the fourth probability label.

8. An image fusion device based on generative adversarial dual-channel weight allocation, characterized in that: include: An initial model construction module, used to construct an initial image fusion model, wherein the initial image fusion model includes a generator module and a twin discriminator module, wherein the generator module includes a pseudo-twin feature extraction module and a feature fusion module; A feature extraction module, used to obtain infrared image samples and visible light image samples, and extract features of the infrared image samples and the visible light image samples respectively based on the pseudo-twin feature extraction module, and correspondingly obtain an infrared feature map and a visible light feature map; A feature fusion module, used for fusing the infrared feature map and the visible light feature map based on the feature fusion module to generate a fused feature map; an identification module, used for respectively obtaining a first similarity between the infrared image sample and the fused feature map and a second similarity between the visible light image sample and the fused feature map based on the twin discriminator module; A model training module, used for training the initial image fusion model based on a preset loss function, the first similarity, the second similarity, the infrared image samples and the visible light image samples to obtain a fully trained target image fusion model; An image fusion module, used for fusing the infrared image to be fused and the visible light image to be fused based on the target image fusion model to obtain a target fused image; The initial image fusion model is trained based on a preset loss function, the first similarity, the second similarity, the infrared image sample, and the visible light image sample to obtain a fully trained target image fusion model, including: Determine whether a difference between the first similarity and the second similarity is less than a preset difference; When the difference between the first similarity and the second similarity is less than the preset difference, the initial image fusion model is the target image fusion model; When the difference between the first similarity and the second similarity is greater than or equal to the preset difference, continuing to train the initial image fusion model based on a preset loss function, the infrared image sample and the visible light image sample; The first similarity is: The second similarity is: In the formula, is the first similarity; is the second similarity; is the fusion feature map; is the infrared image sample; is a visible light image sample; P is the characteristic dimension of an infrared image sample or a visible light image sample; is the i-th dimension feature of the fused feature map; is the i-th dimension feature of the infrared image sample; is the i-th dimension feature of the visible light image sample; is the two-norm.