Image fusion method and device based on GAN network

Abstract

The present application discloses an image fusion method based on a GAN network, which relates to the field of image processing technology. The solution of the present application first inputs the fusion material image into the generator sub-model in the GAN network architecture, generates a first fusion image through a specific generator sub-model, and then performs illumination enhancement processing on the first fusion image so that it can retain visible light illumination details. The processed first fusion image is then sent to the discriminator for discrimination. According to the discrimination result output by the discriminator sub-model, when the discrimination result meets the preset fusion image output requirements, the first fusion image is output. If not, the discrimination result is returned to the generator sub-model for optimization and updating of the parameters of the generator sub-model. The fusion image obtained by this solution can restore the rich information of the scene and the information to be highlighted by the infrared radiation image to a greater extent, solving the technical problem of low fusion quality in the existing technology.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a GAN network-based image fusion method and device. Background Art

[0002] IVIF (Infrared and Visible Image Fusion) technology is a common branch of image fusion. Infrared images capture thermal radiation information, while visible light images capture reflected light, providing richer texture details. Their complementary properties allow them to be fused to produce an image that incorporates complementary information from the source images, further supporting advanced vision tasks. The fusion results obtained using IVIF technology can effectively display infrared targets and detailed scene backgrounds, finding applications in fields such as fault diagnosis, military reconnaissance, and target tracking.

[0003] Existing GAN-based fusion methods utilize adversarial learning to achieve the IVIF task. A generator is used to generate the fusion result, and a discriminator is used to distinguish the authenticity of the fused image from the source image. This method establishes an adversarial game between the fused source image and the visible light image. FusionGAN is the first method to use GAN-based images. This method describes the fusion as an adversarial game between preserving infrared thermal radiation information and maintaining visible appearance texture information. It can be viewed as a minimax problem between the generator and the discriminator. This method first pre-trains a discriminator capable of determining basic authenticity. The image is then fed into the generator. The generator and discriminator then form an adversarial relationship, forming a minimax game between the two. However, as the adversarial process progresses, the data generated by the FusionGAN generator becomes increasingly difficult to distinguish from the actual data, leading to the low fusion quality that currently exists in existing infrared and visible light image fusion techniques. Summary of the Invention

[0004] The present application provides an image fusion method and device based on a GAN network, which is used to solve the technical problem of low fusion quality in existing infrared image and visible light image fusion technology.

[0005] To solve the above technical problems, the first aspect of the present application provides an image fusion method based on a GAN network, comprising:

[0006] Acquire a plurality of groups of fused material images, wherein each group of fused material images includes: an infrared light image and a visible light image;

[0007] Inputting the fused source image into a generator sub-model to obtain a first fused image by combining the infrared image and the visible light image in the fused source image according to a generator loss function in the generator sub-model, wherein the generator loss function is a weighted sum of a content loss, a perceptual loss, and an adversarial loss;

[0008] The first fused image is input into the illumination perception sub-model, illumination enhancement processing is performed on the first fused image through the illumination perception sub-model, and the processed first fused image is input into the discriminator sub-model, so that the discriminator sub-model discriminates the first fused image, and according to the discrimination result output by the discriminator sub-model, the first fused image is output or the parameters of the generator sub-model are optimized and updated.

[0009] Preferably, the adversarial loss is expressed as follows:

[0010]

[0011] Where, L G is the adversarial loss, I G is the first fused image output after being processed by the illumination perception sub-model, I R is an infrared image, D represents the discriminator sub-model, Ex~P fake [] represents that all data in the equation are the expectations of generated data, Ex~P real [] represents that all data in the equation are expectations of real data.

[0012] Preferably, the expression of the content loss is specifically:

[0013]

[0014] Where, L C is the content loss, ζ and ζ' are edge maps obtained by convolving the infrared light image and the first fused image with the Laplacian operator, and τ is a balancing weighting coefficient.

[0015] Preferably, the expression of the perceptual loss is specifically:

[0016]

[0017] Where H and W are the height and width of the perception area, C is the number of channels, For the VGG network, i represents the layer index number.

[0018] Preferably, the loss function expression of the discriminator sub-model is:

[0019]

[0020] Where, I G is the first fused image output after being processed by the illumination perception sub-model, I R is an infrared image, D represents the discriminator sub-model, is the loss function based on LSGAN, Ex~P fake [] represents that all data in the equation are the expectations of generated data, Ex~P real [] represents that all data in the equation are expectations of real data.

[0021] At the same time, the second aspect of the present application provides an image fusion device based on a GAN network, comprising:

[0022] A fusion material image acquisition unit, configured to acquire a plurality of groups of fusion material images, wherein each group of fusion material images includes: an infrared light image and a visible light image;

[0023] an image fusion unit, configured to input the fused source image into a generator sub-model to combine the infrared image and the visible light image in the fused source image according to a generator loss function in the generator sub-model to obtain a first fused image, wherein the generator loss function is a weighted sum of a content loss, a perceptual loss, and an adversarial loss;

[0024] The image optimization and adversarial discrimination unit is used to input the first fused image into the illumination perception sub-model, perform illumination enhancement processing on the first fused image through the illumination perception sub-model, and input the processed first fused image into the discriminator sub-model so that the discriminator sub-model discriminates the first fused image, and output the first fused image or optimize and update the parameters of the generator sub-model based on the discrimination result output by the discriminator sub-model.

[0025] Preferably, the adversarial loss is expressed as follows:

[0026]

[0027] Where, L G is the adversarial loss, I G is the first fused image output after being processed by the illumination perception sub-model, I R is an infrared image, D represents the discriminator sub-model, Ex~P fake [] represents that all data in the equation are the expectations of generated data, Ex~P real [] represents that all data in the equation are expectations of real data.

[0028] Preferably, the expression of the content loss is specifically:

[0029]

[0030] Where, L C is the content loss, ζ and ζ' are edge maps obtained by convolving the infrared light image and the first fused image with the Laplacian operator, and τ is a balancing weighting coefficient.

[0031] Preferably, the expression of the perceptual loss is specifically:

[0032]

[0033] Where H and W are the height and width of the perception area. C is the number of channels. For the VGG network, i represents the layer index number.

[0034] Preferably, the loss function expression of the discriminator sub-model is:

[0035]

[0036] Where, I G is the first fused image output after being processed by the illumination perception sub-model, I R is an infrared image, D represents the discriminator sub-model, is the loss function based on LSGAN, Ex~P fake [] represents that all data in the equation are the expectations of generated data, Ex~P real [] represents that all data in the equation are expectations of real data.

[0037] It can be seen from the above technical solutions that this application has the following advantages:

[0038] The technical solution provided by this application is based on the GAN network architecture. First, the fusion material image is input into the generator sub-model in the GAN network architecture, and a first fusion image is generated through a specific generator sub-model. Then, the obtained first fusion image is subjected to illumination enhancement processing so that the first fusion image can retain the visible light illumination details. The processed first fusion image is then sent to the discriminator for discrimination. According to the discrimination result output by the discriminator sub-model, when the discrimination result meets the preset fusion image output requirements, the first fusion image is output. If not, the discrimination result of the discriminator sub-model is returned to the generator sub-model for optimization and updating of the parameters of the generator sub-model. The fusion image obtained by the technical solution of this application can restore the rich information of the scene and the information to be highlighted by the infrared radiation image to a greater extent, solving the technical problem of low fusion quality in the existing infrared image and visible light image fusion technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0040] Figure 1 This is a flowchart of an embodiment of an image fusion method based on a GAN network provided in this application.

[0041] Figure 2 A schematic diagram of the network architecture of a GAN network for an image fusion method based on a GAN network provided in this application.

[0042] Figure 3 A comparison chart of the fusion image effects of a GAN network-based image fusion method provided in this application and other existing algorithms.

[0043] Figure 4 This is a structural diagram of an embodiment of an image fusion device based on a GAN network provided in this application. DETAILED DESCRIPTION

[0044] The embodiments of the present application provide an image fusion method and device based on a GAN network, which are used to solve the technical problem of low fusion quality in the existing infrared image and visible light image fusion technology.

[0045] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0046] First, a detailed description of an embodiment of an image fusion method based on a GAN network provided by this application is as follows:

[0047] See also Figure 1 This embodiment provides an image fusion method based on a GAN network, including:

[0048] Step 101: Acquire several groups of fused material images.

[0049] It should be noted that, first, several groups of fusion material images to be fused are obtained, and each group of fusion material images includes: a pair of infrared light images and visible light images.

[0050] Step 102: Input the fused material image into the generator sub-model to obtain a first fused image by combining the infrared light image and the visible light image in the fused material image according to the generator loss function in the generator sub-model.

[0051] It should be noted that if Figure 2 As shown, the fused material image obtained in step 101 is fed into the generator sub-model using the concat method, i.e., direct accumulation on the channel, to generate the first fused image. The generator loss function is the weighted sum of content loss, perceptual loss, and adversarial loss, i.e., L total =L G +λL c +ελL perr , where L c is the content loss, L per is the perceptual loss, L G is the adversarial loss, λ and ξ are the trade-off parameters that balance the three terms. total Indicates the total loss.

[0052] The content loss aims to generate an image with a pixel intensity distribution similar to the true value. Therefore, this patent expects to minimize the generated output I G The network is trained by the l2 norm between the image and the infrared image, which can be described as:

[0053]

[0054] Where τ is the weight coefficient that balances the two terms. ζ and ζ′ are edge maps obtained by convolving the infrared image and the fused image with the Laplacian operator. τ is set to 1.0×10 -3 .

[0055] Perceptual loss helps measure differences in the feature domain, such as the VGG network This embodiment defines the perceptual loss as:

[0056]

[0057] Among them, the height and width of the perception area are represented as H and W, C is the number of channels; I G is the output image; I R is an infrared image; i represents the layer index, which is generally set to 4 by default. For the VGG network.

[0058] Loss function L G Defined as:

[0059]

[0060] Where, IG is the first fused image output after being processed by the illumination perception sub-model, I R is an infrared image, D represents the discriminator sub-model, D(I G , I R ) represents the discrimination result between the first fusion image and the infrared image output by the discriminator sub-model, Ex~P fake [] represents that all data in the equation are the expectations of generated data, Ex~P real [] represents that all data in the equation are expectations of real data.

[0061] Step 103: Input the first fused image into the illumination perception sub-model, perform illumination enhancement processing on the first fused image through the illumination perception sub-model, and input the processed first fused image into the discriminator sub-model, so that the discriminator sub-model discriminates the first fused image, and outputs the first fused image or optimizes and updates the parameters of the generator sub-model based on the discrimination result output by the discriminator sub-model.

[0062] It should be noted that the generated fused image is imported into the illumination perception sub-model for illumination enhancement. After illumination enhancement through the illumination perception network, more visible light illumination details of the power equipment can be retained, while avoiding phenomena such as image tearing. The processed image is then sent to the discriminator for discrimination, and the result of the discriminator's discrimination is returned to the generator, the generator weight is updated, and the loss value of the discriminator is continuously reduced, that is, the result of the discriminator's discrimination is closer to the label value, so that it can better identify the fused image generated by the generator and the source image, until the picture generated by the generator can pass the discrimination of the discriminator and the fused image is output.

[0063] In the discriminator sub-model of this embodiment, the discriminator is used to force the retention of more information and select the same parts in the fused image and the true value. The loss function of the discriminator sub-model can be expressed as:

[0064]

[0065] Where, is the loss function based on LSGAN.

[0066] To verify the performance of the method of this embodiment, this embodiment conducted subjective and objective experiments on the publicly available TNO dataset. The TNO dataset contains various scenes with different multi-spectral characteristics, as well as high-quality reference true values. Each image sequence is pixel-level. However, the number of images contained in each sequence is different. For the method of this embodiment, this embodiment only requires one infrared and one visible light image pair for training. In order to better reflect the ability of the method of this embodiment to handle extreme scenes, this embodiment selected 360 image pairs for network training, which include visible light images and infrared images in the sequence. This embodiment selected the remaining 229 sequences in the same way as the test set. In addition, this embodiment introduced 19 image sequences from the dataset that did not have reference true values, and selected extreme infrared visible image pairs in the same way as the test set supplement.

[0067] This experiment primarily used a combined image set from the TNO dataset and the infrared detection dataset to train the model. Eight loss functions, including gradient and pixel loss, were designed for training, and the MS-SSIM loss function, which incorporates subjective judgment criteria, was ultimately chosen:

[0068]

[0069]

[0070] Where M represents different scales. p , μ g Represents the mean of the predicted image and the ground truth. p ,σ g Represents the standard deviation between the predicted image and the ground truth. pg represents the covariance between the two. m and γ m Indicates the relative importance of two terms. c1 and c2 are constants set to prevent division by zero.

[0071] First, the model is randomly initialized, and then the network parameters are iteratively updated using a loss function containing MS-SSIM. Since the CNN-based deep learning network has been pre-trained on ImageNet and has good information absorption capabilities, this training is no longer targeted at the feature extraction network. Its network weights for feature extraction are fixed, and only its reconstruction network needs to be trained. The number of iterations is manually set. In this embodiment, it is set to 1500 times for a single image, and the batch_size is set to 32, and 20 rounds of training are performed. After the iteration is completed, the network parameters are fixed to produce the final CNN model. Based on the results of several training sessions. The importance parameters in the loss function are fixed at a weight value of 4:6, and the initial fusion effect obtained is better.

[0072] This algorithm uses Pytorch 2.0.0 and Matlab 2023 platforms to build the network. The experimental operating system is Windows 11, the CPU is Intel(R) Core(TM) i9-13980HX 2.20GHz, and the GPU is NVIDIA GeForce RTX 4070 Laptop GPU.

[0073] Implementation details: To train the generator, the training dataset is resized to 320×320 square images, and the batch size is set to 4 before entering the network. This embodiment uses Adam as the optimizer and uses an initial learning rate of 1e-4 to optimize the parameters. This embodiment alternately trains the generator and discriminator for a total of 200 epochs. For the generated images and the reference truth, this embodiment cuts 5 groups of 64×64 patches at the same position to train the local discriminator. In addition, the learning rate decay strategy is also applied to the training process. Specifically, after 100 epochs, the learning rate will be linearly reduced until it drops to 0. The entire training process is implemented on the Pytorch framework and trained on a personal computer equipped with NVIDIA3080ti (GPU) and 32GBRAM (memory).

[0074] Conclusion: Comparing the results of this method with other methods, it can be clearly seen that the effect is better than the traditional methods and deep learning methods originally selected. Compared with CNN and LP-SR methods, this method has a significant improvement in highlighting the overall pixel and infrared information, and also has a good performance improvement in detail contour information. The fused images obtained by each network are as follows Figure 3 As shown, the leftmost image in the first row is the infrared image and visible light image serving as the source images. The method used in this embodiment clearly achieves significantly higher brightness contrast on the background than other methods. This is due to the illumination-aware network, which enhances the network's illumination perception capabilities. For example, more visible light details are retained in the background sky, meaning the weight of the visible light image is increased while retaining the temperature information of the infrared image. The texture on the circuit breaker is also clearer. Compared to FusionGAN, which only has a simple GAN network, the illumination-aware network has a more pronounced effect. In dark scenes, the contour information of the infrared target is also clearly preserved, making it easy to identify its position and even state. Overall, it can largely restore the rich information of the scene and the information that the infrared radiation image wants to highlight.

[0075] The above is a detailed description of an embodiment of an image fusion method based on a GAN network provided by this application. The following is a detailed description of an embodiment of an image fusion device based on a GAN network provided by this application.

[0076] See also Figure 4 In a second aspect, the present application provides an image fusion device based on a GAN network, comprising:

[0077] The fusion material image acquisition unit 201 is used to acquire a plurality of groups of fusion material images, wherein each group of fusion material images includes: an infrared light image and a visible light image;

[0078] an image fusion unit 202 configured to input the fused source image into a generator sub-model to combine the infrared image and the visible light image in the fused source image according to a generator loss function in the generator sub-model to obtain a first fused image, wherein the generator loss function is a weighted sum of a content loss, a perceptual loss, and an adversarial loss;

[0079] The image optimization and adversarial discrimination unit 203 is used to input the first fused image into the illumination perception sub-model, perform illumination enhancement processing on the first fused image through the illumination perception sub-model, and input the processed first fused image into the discriminator sub-model, so that the discriminator sub-model discriminates the first fused image, and outputs the first fused image or optimizes and updates the parameters of the generator sub-model based on the discrimination result output by the discriminator sub-model.

[0080] Furthermore, the expression of adversarial loss is specifically:

[0081]

[0082] Where, L G is the adversarial loss, I G is the first fused image output after being processed by the illumination perception sub-model, I R is an infrared image, D represents the discriminator sub-model, Ex~P fake [] represents that all data in the equation are the expectations of generated data, Ex~P real [] represents that all data in the equation are expectations of real data.

[0083] Furthermore, the expression of content loss is specifically:

[0084]

[0085] Where, L C is the content loss, ζ and ζ' are the edge maps obtained by convolving the infrared image and the first fused image with the Laplacian operator, and τ is the balance weighting coefficient.

[0086] Furthermore, the expression of perceptual loss is specifically:

[0087]

[0088] Where H and W are the height and width of the perception area. C is the number of channels. For the VGG network, i represents the layer index number.

[0089] Furthermore, the loss function expression of the discriminator sub-model is:

[0090]

[0091] Where, I G is the first fused image output after being processed by the illumination perception sub-model, I R is an infrared image, D represents the discriminator sub-model, is the loss function based on LSGAN, Ex~P fake [] represents that all data in the equation are the expectations of generated data, Ex~P real [] represents that all data in the equation are expectations of real data.

[0092] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0093] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0094] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0095] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0096] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0097] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0098] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0099] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image fusion method based on a GAN network, characterized in that: include: Acquire a plurality of groups of fused material images, wherein each group of fused material images includes: an infrared light image and a visible light image; Inputting the fused source image into a generator sub-model to obtain a first fused image by combining the infrared image and the visible light image in the fused source image according to a generator loss function in the generator sub-model, wherein the generator loss function is a weighted sum of a content loss, a perceptual loss, and an adversarial loss; Inputting the first fused image into the illumination perception sub-model, performing illumination enhancement processing on the first fused image through the illumination perception sub-model, and inputting the processed first fused image into the discriminator sub-model so that the discriminator sub-model discriminates the first fused image, and outputting the first fused image or optimizing and updating the parameters of the generator sub-model based on the discrimination result output by the discriminator sub-model; The expression of the adversarial loss is specifically: ； Where, L G is the adversarial loss, I G is the first fused image output after being processed by the illumination perception sub-model, I R is an infrared image, D represents the discriminator sub-model, Ex~P fake [] represents that all data in the equation are the expectations of generated data, Ex~P real [] represents the expectation that all data in the equation are real data; The expression of the content loss is specifically: ； Where, L C is the content loss, ζ and ζ' are edge maps obtained by convolving the infrared light image and the first fused image with the Laplacian operator, is the balance weighting coefficient; The expression of the perceptual loss is specifically: ； Where H and W are the height and width of the perception area, C is the number of channels, For the VGG network, i represents the layer index number.

2. The image fusion method based on GAN network according to claim 1, characterized in that: The loss function expression of the discriminator sub-model is: ； Where, I G is the first fused image output after being processed by the illumination perception sub-model, I R is an infrared image, D represents the discriminator sub-model, is the loss function based on LSGAN, Ex~P fake [] represents that all data in the equation are the expectations of generated data, Ex~P real [] represents that all data in the equation are expectations of real data.

3. An image fusion device based on a GAN network, characterized in that: include: A fusion material image acquisition unit, configured to acquire a plurality of groups of fusion material images, wherein each group of fusion material images includes: an infrared light image and a visible light image; an image fusion unit, configured to input the fused source image into a generator sub-model to combine the infrared image and the visible light image in the fused source image according to a generator loss function in the generator sub-model to obtain a first fused image, wherein the generator loss function is a weighted sum of a content loss, a perceptual loss, and an adversarial loss; an image optimization and adversarial discrimination unit, configured to input the first fused image into a lighting perception sub-model, perform lighting enhancement processing on the first fused image through the lighting perception sub-model, and input the processed first fused image into a discriminator sub-model so that the discriminator sub-model discriminates the first fused image, and output the first fused image or optimize and update the parameters of the generator sub-model based on the discrimination result output by the discriminator sub-model; The expression of the adversarial loss is specifically: ； Where, L G is the adversarial loss, I G is the first fused image output after being processed by the illumination perception sub-model, I R is an infrared image, D represents the discriminator sub-model, Ex~P fake [] represents that all data in the equation are the expectations of generated data, Ex~P real [] represents the expectation that all data in the equation are real data; The expression of the content loss is specifically: ； Where, L C is the content loss, ζ and ζ' are edge maps obtained by convolving the infrared light image and the first fused image with the Laplacian operator, is the balance weighting coefficient; The expression of the perceptual loss is specifically: ； Where H and W are the height and width of the perception area, C is the number of channels, For the VGG network, i represents the layer index number.

4. The image fusion device based on the GAN network according to claim 3, characterized in that: The loss function expression of the discriminator sub-model is: ； Where, I G is the first fused image output after being processed by the illumination perception sub-model, I R is an infrared image, D represents the discriminator sub-model, is the loss function based on LSGAN, Ex~P fake [] represents that all data in the equation are the expectations of generated data, Ex~P real [] represents that all data in the equation are expectations of real data.

Images