A multi-source image fusion method based on a three-discriminator generative adversarial network

Abstract

The application provides a multi-source image fusion method based on a three-discriminator generative adversarial network, which comprises the following steps: first, performing color space conversion on a visible light RGB image to obtain YCbCr, and concatenating the Y channel and a single-channel infrared image into a double-channel combined image; then inputting the combined image into a generator to obtain a single-channel fused luminance image with the same spatial size as the input image; thereafter, converting the fused luminance image as the Y channel and the CbCr channel to obtain a fused color image in RGB format; the generator is trained based on three discriminators before image fusion, the first discriminator discriminates the authenticity of the fused color image and the visible light color image, the second discriminator discriminates the authenticity of the fused luminance image and the infrared image, and the third discriminator discriminates the authenticity of the fused luminance image and the Y channel; the application can improve the problem of insufficient information expression of visible light in poor light, and solve the challenges of insufficient clarity of the fused image and weak correlation with the source image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a multi-source image fusion method based on a three-discriminator generative adversarial network. Background Technology

[0002] In image-based intelligent monitoring of perimeter intrusion and obstacles in rail transit, single-source image detection often fails to provide a comprehensive description of the object to be detected due to weather changes such as rain, snow, fog, haze, or differences in day and night lighting. This can lead to false alarms and missed detections in high-speed rail perimeter intrusion detection systems. Therefore, multi-source image detection methods, such as combining multispectral cameras, infrared cameras with lidar, or infrared and visible light color images, are more effective than single-source image detection methods and provide valuable insights for this research.

[0003] However, existing multi-source image fusion methods, especially those that fuse visible light color images with infrared images, suffer from insufficient information representation by visible light in poor lighting conditions, resulting in insufficient clarity of the fused image and weak correlation with the source images. Summary of the Invention

[0004] In view of this, embodiments of this application provide a multi-source image fusion method based on a three-discriminator generative adversarial network, which can improve the problem of insufficient visible light information expression under poor lighting conditions, and solve challenges such as insufficient clarity of the fused image and weak correlation with the source image.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a multi-source image fusion method based on a three-discriminator generative adversarial network, the method comprising:

[0007] The visible light color image is subjected to color space conversion processing to obtain the color space YCbCr corresponding to the visible light color image, and the Y channel is concatenated with the single-channel infrared image in the channel dimension to obtain a combined image; wherein, the visible light color image is an RGB three-channel color image, and the combined image is a two-channel image;

[0008] The combined image is used as the input image to the generator to obtain a fused brightness image; wherein the fused brightness image is a single-channel image and the spatial size of the fused brightness image is the same as the spatial size of the input image;

[0009] The fused brightness image is used as the Y channel, and color space conversion processing is performed with the CbCr channel to obtain a fused color image; wherein, the fused color image is an RGB image with the same color space as the visible light color image; the generator is trained based on a discriminator before image fusion, the discriminator includes a first discriminator, a second discriminator and a third discriminator, the first discriminator is used to identify the authenticity of the fused color image and the visible light color image, the second discriminator is used to identify the authenticity of the fused brightness image and the infrared image, and the third discriminator is used to identify the authenticity of the fused brightness image and the Y channel.

[0010] Secondly, embodiments of this application also provide a multi-source image fusion apparatus based on a three-discriminator generative adversarial network, the apparatus comprising:

[0011] The combination module is used to perform color space conversion processing on the visible light color image to obtain the color space YCbCr corresponding to the visible light color image, and to cascade the Y channel with the single-channel infrared image in the channel dimension to obtain the combined image; wherein, the visible light color image is an RGB three-channel color image, and the combined image is a two-channel image;

[0012] A generation module is used to input the combined image as an input image into a generator to obtain a fused brightness image; wherein the fused brightness image is a single-channel image and the spatial size of the fused brightness image is the same as the spatial size of the input image;

[0013] A fusion module is used to perform color space conversion processing on the fused brightness image as the Y channel and the CbCr channel to obtain a fused color image; wherein, the fused color image is an RGB image with the same color space as the visible light color image; the generator is trained based on a discriminator before performing image fusion, the discriminator includes a first discriminator, a second discriminator and a third discriminator, the first discriminator is used to identify the authenticity of the fused color image and the visible light color image, the second discriminator is used to identify the authenticity of the fused brightness image and the infrared image, and the third discriminator is used to identify the authenticity of the fused brightness image and the Y channel.

[0014] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the multi-source image fusion method based on a three-discriminator generative adversarial network as described in any of the first aspects.

[0015] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the multi-source image fusion method based on a three-discriminator generative adversarial network as described in any one of the first aspects.

[0016] The embodiments of this application have the following beneficial effects:

[0017] This application improves upon existing deep learning-based image fusion techniques by modifying the network structure and loss function. Regarding the network structure, two discriminators are used, combining a residual network and a densely connected network. Simultaneously, the image color space transformation YCBCr2RGB is utilized to fuse the color information of the source image along with the visible image. For the loss function, the generator's loss function combines content loss and adversarial loss to ensure that a large amount of the main information of the source image is preserved. The discriminator's network structure and loss function are relatively simple, primarily aiming to enrich the details of visible light information in the fused image and integrate the high-brightness thermal radiation information of the infrared portion of the fused image. This method can improve the problem of insufficient visible light information representation in poor lighting conditions and address challenges such as insufficient clarity of the fused image and weak correlation with the source image. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating steps S101-S103 provided in the embodiments of this application;

[0020] Figure 2 The principle diagram of the fusion process based on a three-discriminator generative adversarial network provided in this application embodiment;

[0021] Figure 3 This is a generator network structure diagram provided in the embodiments of this application;

[0022] Figure 4 This is a schematic diagram of REFLECT reflection filling provided in the embodiments of this application;

[0023] Figure 5 This is the specific network structure of the residual module provided in the embodiments of this application;

[0024] Figure 6 This is a diagram of the discriminator network structure provided in the embodiments of this application;

[0025] Figure 7 This is a diagram illustrating the convolution calculation process using VALID padding provided in an embodiment of this application.

[0026] Figure 8 This is a flowchart of adversarial training provided in an embodiment of this application;

[0027] Figure 9 This is a schematic diagram of the structure of the multi-source image fusion device based on a three-discriminator generative adversarial network provided in the embodiments of this application;

[0028] Figure 10 This is a schematic diagram of the composition structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0030] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0031] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0032] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0033] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application and is not intended to limit this application.

[0035] See Figure 1 , Figure 1 This is a flowchart illustrating steps S101-S103 of the multi-source image fusion method based on a three-discriminator generative adversarial network provided in this application embodiment. Figure 1 Steps S101-S103 shown will be explained.

[0036] In step S101, the visible light color image is subjected to color space conversion processing to obtain the color space YCbCr corresponding to the visible light color image, and the Y channel is concatenated with the single-channel infrared image in the channel dimension to obtain a combined image; wherein, the visible light color image is an RGB three-channel color image, and the combined image is a two-channel image.

[0037] In step S102, the combined image is used as the input image to the generator to obtain a fused brightness image; wherein the fused brightness image is a single-channel image and the spatial size of the fused brightness image is the same as the spatial size of the input image.

[0038] In step S103, the fused brightness image is used as the Y channel and subjected to color space conversion processing with the CbCr channel to obtain a fused color image; wherein, the fused color image is an RGB image with the same color space as the visible light color image; the generator is trained based on a discriminator before performing image fusion, the discriminator includes a first discriminator, a second discriminator and a third discriminator, the first discriminator is used to identify the authenticity of the fused color image and the visible light color image, the second discriminator is used to identify the authenticity of the fused brightness image and the infrared image, and the third discriminator is used to identify the authenticity of the fused brightness image and the Y channel.

[0039] Please see Figure 2 , Figure 2 This is a schematic diagram of the fusion process based on a three-discriminator generative adversarial network provided in an embodiment of this application, as shown below. Figure 2 As shown, the entire network consists of one generator network and three discriminator networks. The image fusion process can be divided into three steps:

[0040] Step 1: Visible light color images typically use RGB three-channel color images as input. First, the three-channel visible light color image is converted to a color space, converting the RGB three channels to YCbCr. The CbCr channel containing color information is retained for subsequent restoration of image color information through fusion. The Y channel information of the visible light color image is concatenated with the single-channel infrared pixel intensity image in the channel dimension to form a combined image of two channels, which is then fed into the generator as input.

[0041] Step 2: The generator performs convolution, feature extraction, and feature fusion on the input dual-channel image containing the visible light Y channel and the infrared image, and finally outputs a fused two-dimensional image, which is a single-channel image with the same spatial size as the input image.

[0042] Step 3: Take the obtained fused luminance image as the Y channel, and use the CbCr channel retained in Step 1 to perform color space conversion again to restore it to an RGB image with the same color space as the input visible light source image, as the fused color image.

[0043] Before image fusion, the generator network needs to be trained to learn the features of infrared and visible light color images for high-quality fusion. First, the network needs to be trained using pre-registered datasets of visible light color images and infrared images. The training process can be viewed as an adversarial process between the generator and the discriminator. The generator needs to generate a fused image based on the visible light color image and its corresponding infrared image. The discriminator needs to determine whether the input image is a real visible light color image or a generated fused image. The goal of training is to continuously optimize the generator and discriminator alternately until the fused image generated by the generator can deceive the discriminator, making it difficult for the discriminator to distinguish between the real image and the generated fused image. This indicates that the generator has good fusion performance. After training, the discriminator can be ignored. The generator can then be used to fuse visible light color images and infrared images.

[0044] In some embodiments, the first discriminator uses the visible light color image as the real label and the fused color image as fake data; the second discriminator uses the infrared image as the real label and the fused brightness image as fake data; the third discriminator uses the Y channel as the real label and the fused brightness image as fake data.

[0045] The first discriminator is used to make the fused color image contain more color features of the visible light color image; the second discriminator is used to make the fused brightness image contain more pixel intensity information of the infrared image; the third discriminator is used to make the fused brightness image contain more detailed texture features, the detailed texture features including detailed texture information of the intrusion target;

[0046] The first discriminator, the second discriminator, and the third discriminator force the generator to optimize the fusion effect through adversarial training, and the generated image is close to the real image in terms of color, infrared intensity, and brightness details.

[0047] In this embodiment, three discriminators are used. The first discriminator, discriminator 1 (visible light color image discriminator Dv), is used to distinguish the authenticity of the fused color image from the input color image. Dv treats the input visible light color image as the true label and the fused color image as fake data. The second discriminator, discriminator 2 (infrared image discriminator Di), is used to distinguish the authenticity of the comparison between the fused luminance image and the infrared image. Di treats the input infrared image as the true label and the fused luminance image as fake data. The third discriminator, discriminator 3 (luminance channel discriminator Dy), is used to distinguish the authenticity of the comparison between the fused luminance image and the visible light luminance channel. Dy treats the input visible light color image Y channel as the true label and the fused luminance image as fake data. The training objective of the three discriminators is to accurately distinguish whether the input is a true label or fake data.

[0048] The three discriminators each serve a different purpose. The visible light color image discriminator Dv aims to include more color features from the visible light color image in the fused image. The infrared image discriminator Di aims to include more pixel intensity information from the infrared image in the fused image, especially the thermal radiation information of the intruding target. The luminance channel discriminator Dy aims to include more detailed texture features in the fused image, especially the detailed texture information of the intruding target.

[0049] Finally, the parameters of the generator and the three discriminator networks are updated through backpropagation under the constraints of their respective loss functions.

[0050] When performing image fusion, since the generator has already been adjusted by the discriminator during training to produce a fused image that includes infrared intensity and visible light image details, only the trained generator is needed and the discriminator does not need to participate.

[0051] In some embodiments, the generator uses dense blocks, residual blocks, skip connections, and multiple convolutional layers to extract, fuse, and reconstruct features from visible light color images and infrared images. The generator employs 3×3 convolutional kernels and REFLECT reflection fill to preserve image details, combines batch normalization layers and ReLU / tanh activation functions to improve network stability and output quality, and optimizes the fusion effect through adversarial loss and content loss.

[0052] The image fusion network provided in this application includes a generator that utilizes multiple convolutional layers to increase network depth and alters the feature channel transformation strategy to ensure the formation of multi-scale features before decoding output. Dense blocks are then introduced to enrich the image's feature extraction, while bypasses enhance feature reuse. Residual blocks are added and concatenated with the dense blocks. This preserves more residual information and provides a certain degree of regularization, making the network easier to train and mitigating gradient vanishing and model degradation problems. Furthermore, to transmit and preserve the characteristics of previous layers and further integrate the characteristics before decoding, additional skip connections are added before and after the dense and residual blocks, further optimizing and upgrading the network. The generator network structure diagram is shown below. Figure 3 As shown. Concat represents concatenating visible light Y-channel data with infrared image data on the channel, Conv represents convolution operation, BN represents batch normalization operation, and ReLU and tanh represent activation functions.

[0053] In some embodiments, the kernel size setting of the generator includes: except for the transition layer conv3 and the output convolutional layer conv5 which use 1×1 kernels, all other convolutional layers use 3×3 kernels, thereby achieving the function of large-size convolutional kernels and reducing the number of parameters by stacking.

[0054] The generator's stride setting includes: setting the stride of all convolutional layers to 1 to fully extract features and reduce information loss;

[0055] The generator introduces a normalization layer (BN) after each convolutional layer to alleviate the gradient vanishing problem and accelerate network convergence.

[0056] The activation function settings for the generator include: except for the output convolutional layer which uses the tanh activation function, all other convolutional layers use the ReLU activation function to enhance nonlinear expressive power;

[0057] The generator's padding method includes: all convolutional layers are padded with REFLECT reflections to ensure feature map size matching and maintain the output image size unchanged;

[0058] The generator's input and feature extraction convolutional layers upscale the input two-channel image to a 64-channel feature map through two convolutional layers to avoid distortion caused by a sudden increase in the number of channels.

[0059] The dense block function of the generator includes: maximizing feature reuse and information flow through dense connections, reducing parameters and mitigating gradient vanishing;

[0060] The generator's transition layer uses a 1×1 convolutional kernel to reduce the number of channels to 64 in order to control network complexity and connect residual blocks;

[0061] The generator's residual blocks maintain the integrity of the input data through identity mapping and learn residual information to supplement multi-scale features;

[0062] The skip connection of the generator is used to directly pass the output of the shallow feature extraction layer to the residual block, thereby enhancing feature fusion and reuse.

[0063] The generator’s feature reconstruction and output convolutional layer reduces the features to a single channel and outputs a fused image through two convolutional layers. The last layer uses a 1×1 convolutional kernel to integrate multi-channel features.

[0064] The generator's loss function is composed of adversarial loss. With content loss The adversarial loss is optimized for realism through adversarial training against the discriminator, and the content loss preserves key information by constraining the similarity between the fused image and the source image.

[0065] As an example, the generator network consists of a dense block, a residual block, a transition layer, a skip connection, and four convolutional layers. The first two convolutional layers serve as input and feature extraction layers, while the last two serve as feature reconstruction and output convolutional layers. The introduction of the dense block and residual block improves network performance. All these modules and structural improvements ensure that the generator designed in this chapter can produce high-quality, information-rich, and diverse fused images. The network setup and the function of each module (layer) are described below.

[0066] (1) Convolutional kernel size setting. In this network, except for the intermediate transition layer conv3 and the last convolutional output layer conv5 which use 1×1 convolutional kernels, all other layers use 3×3 convolutional kernels. In actual processing, the 3×3 size is the smallest, and it can obtain the eight-neighbor information of a certain pixel in the image when calculating convolution, with the smallest volume. By stacking multiple 3×3 convolutional kernels, the function of large-size convolutional kernels can be achieved with multiple angles and small receptive fields, and fewer parameters are required. At the same time, multiple 3×3 convolutional layers have more nonlinearity, which increases the nonlinear expression capability of the network. The last convolutional layer uses a single-channel 1×1 convolutional layer, which can integrate the features on different channels of the previous layer. Through multi-channel fusion, the dimensionality of multi-channel feature maps is reduced, and the single-channel fusion is output, thereby realizing feature normalization.

[0067] (2) Stride setting. In convolution operations, the stride of the convolution kernel determines the accuracy of feature extraction. The smaller the stride, the more comprehensive the extracted features will be, ensuring that as many features as possible are extracted, with less missing information, and which are crucial for generating the fused image. On the other hand, a larger stride setting will result in fewer computational parameters. In this network, in order to extract as many features as possible, the stride of all layers is set to 1.

[0068] (3) The role of batch normalization (BN). Adding BN to the network can normalize the output of convolutional layers, effectively alleviating the gradient vanishing problem, improving model efficiency, increasing generator stability, and accelerating model convergence. It can accelerate the training of neural networks and improve the generalization performance of the model. In addition, BN can reduce the model's dependence on initialization, thereby improving training efficiency and achieving a larger learning rate in actual training.

[0069] (4) Activation function settings. The ReLU activation function is used in all layers except the last output convolutional layer. For the last convolutional layer, the tanh activation function is used to ensure the stability of the output result.

[0070] (5) Fill Settings. Convolution operations change the image size. However, the method proposed in this chapter contains dense blocks. To ensure that the feature map sizes within the dense blocks match, and to prevent the final fused image size from changing, image fill is necessary. The fill method used in this chapter is REFLECT reflection fill. This fill method fills the boundary pixel values ​​in a mirror manner. When the image size is enlarged, the fill is performed by symmetrically copying along the axis of symmetry (here, the image boundary). New pixels are obtained from pixels next to the axis of symmetry, excluding the axis of symmetry during copying. This fill method makes the filled image more complete and looks more natural. Compared to fill methods using fixed values, REFLECT reflection fill achieves better convolution results. Figure 4 This is a schematic diagram of REFLECT reflection filling.

[0071] (6) Input and feature extraction convolutional layer. To avoid image distortion caused by a sudden increase in the number of channels, the input two-channel image is convolved twice to increase the number of channels. The two-channel information composed of the channel-cascaded infrared image and the visible image is sent to the network. After one convolution operation, a 32-channel feature image is output, and after another convolution, a 64-channel feature image is output.

[0072] (7) Dense Block Function. In this embodiment, to ensure that the fused image contains more source image information, a dense connection network for feature extraction and a feature network for feature reconstruction are applied in the fusion structure. The specific connection form is shown in the figure below. The use of dense connection blocks enables the bypass dense connection to increase feature reuse, maximize information flow, reduce network parameters, simplify training, and alleviate the gradient vanishing problem.

[0073] (8) Role of the transition layer. Because each dense block increases the number of channels, excessive use can lead to excessive model complexity. The transition layer is used to control and reduce the complexity of the network model. It reduces the number of channels to 64 through a 1×1 convolutional layer, while connecting to the subsequent residual blocks.

[0074] (9) Residual Block Function. The addition of residual blocks is the main feature and highlight of this method. The residual block contains two convolutional layers. After the first convolutional layer, a batch normalization layer, and a ReLU activation function, it goes through a second convolutional layer and a batch normalization layer. Then it is connected to the input x residual before activation. The specific network structure of the residual module is as follows: Figure 5 As shown.

[0075] The generator network is deepened through residual blocks, significantly improving network performance. The identity mappings in the residual blocks preserve the integrity of the input data, while the learned residual information helps supplement multi-scale features, laying the groundwork for further feature extraction or dimensionality reduction. This is crucial for image fusion tasks, as it preserves the source image features extracted from previous layers while learning more internal interprise connections. Furthermore, residual blocks make deeper networks easier to train, and the gradient vanishing problem is effectively resolved during backpropagation.

[0076] (10) Skip connection effect. The output of the shallow feature extraction layer conv2 is skipped to the residual block. This can further enhance the fusion of the features extracted by the previous layer and the features of the later layer, so as to realize the reuse of features. At the same time, it also promotes the optimization and training of the network.

[0077] (11) Feature Reconstruction and Output Convolutional Layer. To achieve feature dimensionality reduction and reconstruct the output into a fused image, two convolutions are used. The first convolutional layer reduces the number of channels to 32, and the second convolutional layer outputs the fused image through a 1×1 convolutional kernel.

[0078] (12) Loss function. The generator loss consists of adversarial loss L_adv and content loss L_con. The adversarial loss is for adversarial training with the discriminator, and the content loss is for making the fused image fuse in the direction of the set constraint. The two together affect the image fusion result.

[0079] In summary, residual blocks and skip connections deepen the network. They also effectively utilize features from previous layers, helping to obtain more deep features, which greatly aids image fusion. The network as a whole is optimized, and the convergence speed is accelerated. Meanwhile, batch normalization (BN) and the ReLU activation function further improve model training efficiency and model stability. To more clearly illustrate the generator network structure and parameter information, Table 1 lists the detailed calculations of each layer. Conv represents convolution, Add represents addition, Net represents the network layer input or output feature map, ir and vi represent infrared and visible light source input images, respectively, and f represents the fused image.

[0080] Table 1

[0081]

[0082] In some embodiments, the network structures of the first discriminator, the second discriminator, and the third discriminator are identical; the network structure includes a three-layer convolutional layer and a fully connected layer structure, which achieves feature extraction and classification through strided convolution, VALID padding, and linear fully connected layers, and combines a customized loss function to drive the generator to optimize the fusion effect.

[0083] The image fusion network described above includes three discriminators. Discriminator 1 distinguishes between the fused color image and the visible light color image. Discriminators 2 and 3 distinguish between the fused luminance image and the Y-channel of the infrared and visible light color images, respectively. The quality of the fused image generated by the generator is affected by the recognition performance of these two discriminators. Furthermore, if the discrimination performance of any one of the discriminators—the luminance channel discriminator, the infrared image discriminator, or the visible light color image discriminator—is too strong, the fused image will be more biased towards the style of that discriminator, thus weakening or even rendering the other discriminators ineffective. Therefore, the discriminators in this invention are designed with identical network structures. Figure 6 As shown, Figure 6 The discriminator network structure diagram provided in this application embodiment is shown. "Linear" represents feature linear transformation, which expands each feature map containing multiple neurons into a linear form, and then concatenates all the linear feature maps to form a "long column" of feature output, which facilitates matching the access of the subsequent fully connected layer.

[0084] In some embodiments, the network structure uses a 3×3 convolutional kernel in the first convolutional layer to extract primary features of the input image through a large receptive field and perform channel upscaling; the second and third convolutional layers also use 3×3 convolutional kernels, stacking small receptive fields to achieve multi-scale feature extraction, while reducing the number of parameters and enhancing nonlinear expressive power; all convolutional layers use the ReLU activation function, and batch normalization (BN) layers are added except for the first layer to accelerate model convergence and stabilize the training process;

[0085] The network structure replaces the traditional pooling layer with a stride of 2 convolution operation, which reduces information loss while decreasing the resolution of the feature map layer by layer. The stride convolution controls the decrease in feature map size, and finally outputs a compact feature vector for processing by the fully connected layer.

[0086] All convolutional layers use VALID padding to avoid introducing extra pixel padding, thus preventing noise interference and ensuring the accuracy of feature extraction.

[0087] The feature map output from the last convolutional layer is unfolded into a linear sequence, and then weighted and fused through a fully connected layer to generate a discriminator that scores the authenticity of the input image. The weights of the fully connected layer are matched with the number of feature points, completing a non-linear mapping from the feature space to the discrimination result, and outputting a binary classification result.

[0088] The first discriminator loss consists of the sum of the authenticity score of the visible light color image and the false score of the fused image, driving the fused image to retain visible light texture details; the second discriminator loss consists of the sum of the authenticity score of the infrared image and the false score of the fused image, ensuring that the fused image contains infrared thermal radiation information; the loss function is jointly optimized with the generator loss by minimizing the discrimination score of the source image and maximizing the discrimination score of the fused image, achieving adversarial balance.

[0089] Please continue reading Figure 6 The discriminator networks all consist of three convolutional layers and one fully connected layer. For the infrared image discriminator Di, the input is the infrared image and the fused image, and its function is to discriminate the pixel intensity information of the infrared image, allowing the fused image to contain more infrared intensity information. For the visible light color image discriminator Dv, the input is the visible light color image and the fused image, and its function is to discriminate the visible light gradient information, allowing the fused image to contain more visible light texture information. The generator designed in this embodiment generates a fused image that can confuse the discriminator, while the discriminator, in contrast to the generator, is designed to distinguish the generated fused image from the original infrared image and the visible light color image. The specific structure of the discriminator network is as follows.

[0090] (1) Kernel Size Setting. The discriminator network outputs a single discrimination result for its input image, so the discriminator is more like a "classifier" that evaluates the input. Unlike the generator, the first convolutional layer extracts features through a larger receptive field and performs dimensionality upscaling, with the convolution kernel set to 3×3. Subsequent convolutions also use 3×3 kernels for internal feature extraction. Simultaneously, all four convolutional layers use ReLU for activation, and except for the first convolutional layer, batch normalization (BN) is added to all layers to improve the model's convergence speed.

[0091] (2) Stride setting. As the number of feature map channels increases and the size of the feature maps decreases, a design similar to the mesh structure of classifiers is used in discriminator networks. Pooling to reduce the size of feature maps is usually applied in conventional convolutional neural networks. However, this method uses strided convolution instead of pooling, mainly because the discriminator network has fewer layers. Although strided convolution increases some parameters, it can reduce feature loss and is more helpful for actual discrimination. In this method, the stride is set to 2.

[0092] (3) Padding Settings. The padding method is VALID padding, which does not use any padding. VALID padding avoids noise interference from other padding methods, ensuring feature extraction. Using VALID padding reduces the image size, but in the dual discriminator network of this chapter, the feature map sizes do not need to be consistent, making VALID padding suitable. For example... Figure 7 As shown, Figure 7This is a diagram illustrating the convolution calculation process using VALID padding provided in the embodiments of this application. It shows the convolution process using a 3×3 convolution kernel and VALID padding.

[0093] (4) Fully connected layer. Based on the number of feature points of the linear transformation, the corresponding weights are matched, and the features extracted after the linear transformation of the previous layer are weighted to complete the identification and classification tasks of the infrared image discriminator for the fused image and the infrared source image, and the visible light color image discriminator for the fused image and the visible light source image.

[0094] (5) Loss function. The loss functions of the two discriminators are designed differently, including the discrimination value of the fused image output by the generator, and the discrimination value of each discriminator for the corresponding source image. The loss of the visible light color image discriminator is obtained by adding the judgment results of the visible light color image discriminator for the visible light source image and the fused image output by the generator. The loss of the infrared image discriminator is obtained by adding the judgment results of the infrared image discriminator for the infrared source image and the fused image output by the generator.

[0095] To more clearly illustrate the discriminator network structure and parameter information, Table 2 describes the layer-by-layer unfolding of the dual discriminator network. Here, Conv represents the convolution operation, Net represents the network layer input or output feature map, ir and vi represent the infrared and visible light source input images, respectively, f represents the fused result image, and scalar represents the discriminant value.

[0096] Table 2

[0097]

[0098] In some embodiments, the generator's loss function is used to deceive the discriminator and maintain content similarity with the source image; the generator's loss function Losses due to confrontation and content loss The structure is as follows, where the weight λ is used to control the balance between the two, and the function expression is:

[0099] ;

[0100] The definition of adversarial loss is:

[0101] ;

[0102] in, The first discriminator is the one that judges the fused color image. The identification results The second discriminator is the fused brightness image The identification results The third discriminator is used to analyze the fused brightness image. The identification results;

[0103] The content loss function is:

[0104] ;

[0105] in, This represents the content loss weights when comparing the generated color image with the real color image;

[0106] This indicates the content loss weights when comparing the generated brightness image with the real infrared image;

[0107] This indicates the content loss weights when comparing the generated brightness image with the Y channel of the real visible light color image;

[0108] This represents the generated brightness image;

[0109] Represents an infrared image;

[0110] y represents the Y channel of the photoluminescent image;

[0111] This represents the generated color image;

[0112] Represents a true color image;

[0113] The loss function of the first discriminator is:

[0114] ;

[0115] The loss function of the second discriminator is:

[0116] ;

[0117] The loss function of the third discriminator is:

[0118] ;

[0119] The generator's loss function is used to confuse the discriminator, while the discriminator's loss function is used to improve the discriminator's performance. The generator and discriminator undergo adversarial training.

[0120] The generator's loss function is used to confuse the discriminator, while the discriminator's loss function is used to improve the discriminator's performance. The generator and discriminator undergo adversarial training.

[0121] The generator loss function of this invention ensures that the generator is not only trained to fool the discriminator, but also maintains content similarity to the source image. Generator Loss Function Losses due to confrontation and content loss Composed of, where weights The function expression for controlling the balance between the two is shown in formula (1). The purpose of content loss is to constrain the generator designed above, so that the fused image generated by the generator designed above is similar to the original infrared image in terms of image pixel intensity and similar to the original visible light color image in terms of image gradient texture details, thereby improving the stability of the entire network training process.

[0122] (1)

[0123] (1) Combating losses

[0124] To enhance the generator's generative capabilities, adversarial loss is typically incorporated into the generator's training process. The core of adversarial loss lies in the competition and interaction between the generator and the discriminator. Through this competition, the generator is forced to produce more realistic images, while the discriminator strives to distinguish between real and generated images, prompting the generator to continuously improve its generative capabilities and ultimately achieve better outputs that are relevant to the input image. This dynamic interaction between the generator and discriminator helps the generator learn more accurate and realistic image representations. The optimization process of this loss function, as described in this section, is essentially a process of supplementing the fused image with gradient information from the visible light color image and pixel intensity information from the infrared image.

[0125] The specific definition of adversarial loss is shown in formula (2).

[0126] (2)

[0127] It is a visible light color image discriminator for fused color images The identification results It is an infrared image discriminator that analyzes the fused brightness image. The identification results It is the luminance channel discriminator for fused luminance images The discrimination results are as follows. From the adversarial loss function, it can be seen that to reduce the loss function, the discrimination results of the three discriminators on the fused image must be larger. In other words, the more the fused image appears "real" to the three discriminators, the closer the fused image will be to the "real fused image," and the better the fusion quality will be.

[0128] (2) Content loss

[0129] Content loss ( Content loss is one of the main loss functions of the generator. It determines the correlation between the fused image and the source image, and which important feature information from different modalities of infrared and visible color images is included in the fused image. The smaller the content loss, the more features of the source image are in the fused image.

[0130] The content loss function is shown in formula (3).

[0131] (3)

[0132] The content loss function comprises three parts: visible light color image gradient loss, infrared pixel intensity loss, and visible light brightness channel loss. On one hand, since thermal radiation information in the infrared image represents pixel intensity, a Frobenius norm constraint is applied between the infrared image and the fused image, causing the fused image to resemble the pixel intensity of the infrared image. On the other hand, texture details in the visible light color image represent gradient changes, and a Frobenius norm constraint is applied between the visible light color image and the fused image, causing the fused image to resemble the gradient texture of the visible light color image. , , These are the weight values ​​for the content loss constraints of the corresponding parts.

[0133] Secondly, the discriminator loss function is defined, characterized by the fact that the three discriminator loss functions are designed differently and play different roles. The purpose of the discriminator loss is to constrain the backpropagation parameter update of the discriminator network, specifically for infrared image discriminators. To ensure the trained discriminator has a strong ability to distinguish between infrared and fused images, and a visible light color image discriminator. The trained discriminator is designed to have a strong ability to distinguish between visible light color images and fused images. The discriminator loss function is given by formulas (4)-(6), which include the adversarial discrimination results of each discriminator on the fused image. The difference is that the two discriminator loss functions include the discrimination results of infrared images and visible light color images, respectively. In the embodiments of this application, the generator loss function and the discriminator loss function play completely opposite roles in the discrimination results of the fused image. One is to confuse the discriminator, and the other is to achieve a good discrimination effect. The two are trained adversarially.

[0134] (4)

[0135] (5)

[0136] (6)

[0137] Finally, using the infrared and visible light fusion dataset, the image fusion network is trained according to the aforementioned generator and discriminator loss functions. The infrared and visible light color image fusion method using a dual-discriminator generative adversarial network incorporates the ideas of GAN networks. The adversarial training strategy also influences the output of the GAN network, especially since this invention uses a dual-discriminator network. The training effect of any one discriminator affects the entire network, as does the training relationship between the generator and the two discriminators. Therefore, developing a good adversarial training strategy is essential. Common GAN network training strategies are: train the generator once, then train the discriminator once; train the generator once, then train the discriminator k times; train the generator k times, then train the discriminator k times. In the training process of this invention, to make the generator, infrared image discriminator, and visible light color image discriminator more efficient in training, the following strategy is developed: Figure 8 The adversarial training process is shown.

[0138] In some embodiments, the generator and discriminator are trained in the following manner:

[0139] With fixed generator network parameters, batch training data is input into the generator to generate fused images. The discriminator distinguishes the fused images from the source images and calculates the discriminator loss function value. If the discriminator loss value is greater than the first threshold or the current training round has not reached the set value, the gradient descent method is used to optimize the discriminator network parameters.

[0140] When the discriminant loss value is less than the second threshold, the generator loss value is greater than the third threshold, or the total number of training iterations reaches the preset threshold, the current training phase is deemed to have met the target, and the process is switched to the generator training phase; otherwise, the discriminant network continues to be iteratively optimized.

[0141] The discriminator network parameters are fixed, and the generator parameters are updated through backpropagation using the generator loss function. If the generator loss value does not reach the convergence threshold or the training rounds are not completed, the alternating training process is repeated.

[0142] During training, the loss function value and total number of training iterations are monitored in real time, and the switching conditions for training phases are dynamically adjusted until the network converges to the optimal state.

[0143] For example, see Figure 8During network training, the first step is to determine if the training epoch has been completed and reached a set value. If it has, training is considered complete. Otherwise, the generator is fixed, and a batch of training data is fed into the generator to produce fused images. The infrared / visible color image discriminator then discriminates these images, calculates the loss function, and optimizes the infrared / visible color image discriminator network using gradient descent. Determining whether training has reached the target is crucial for formulating the training strategy. This strategy is determined by two parameters: the calculated loss function and the number of training iterations. If the calculated loss function is less than the set value or the number of training iterations reaches the set value, training is considered successful, and the next batch of training begins, training the generator. Otherwise, the loss is calculated again, and the network is optimized using gradient descent. The generator training process is similar to the discriminator training process, except that the infrared / visible color image discriminator is fixed during training, and the generator network is trained instead. The specific number of training iterations is set to 20. The infrared / visible color image discriminator is trained when the loss value of the infrared / visible color image discriminator is greater than 1.7, and the generator is trained when the loss value of the infrared / visible color image discriminator is less than 1.4 or the generator loss is greater than 200.

[0144] In summary, the embodiments of this application have the following beneficial effects:

[0145] This application improves upon existing deep learning-based image fusion techniques by modifying the network structure and loss function. Regarding the network structure, two discriminators are used, combining a residual network and a densely connected network. Simultaneously, the image color space transformation YCBCr2RGB is utilized to fuse the color information of the source image along with the visible image. For the loss function, the generator's loss function combines content loss and adversarial loss to ensure that a large amount of the main information of the source image is preserved. The discriminator's network structure and loss function are relatively simple, primarily aiming to enrich the details of visible light information in the fused image and integrate the high-brightness thermal radiation information of the infrared portion of the fused image. This method can improve the problem of insufficient visible light information representation in poor lighting conditions and address challenges such as insufficient clarity of the fused image and weak correlation with the source image.

[0146] Based on the same inventive concept, this application also provides a multi-source image fusion device based on a three-discriminator generative adversarial network, which corresponds to the multi-source image fusion method based on a three-discriminator generative adversarial network in the first embodiment. Since the principle of the device in this application is similar to the multi-source image fusion method based on a three-discriminator generative adversarial network, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0147] like Figure 9 As shown, Figure 9This is a schematic diagram of the structure of a multi-source image fusion device 900 based on a three-discriminator generative adversarial network provided in an embodiment of this application. The multi-source image fusion device 900 based on a three-discriminator generative adversarial network includes:

[0148] The combination module 901 is used to perform color space conversion processing on the visible light color image to obtain the color space YCbCr corresponding to the visible light color image, and to cascade the Y channel with the single-channel infrared image in the channel dimension to obtain the combined image; wherein, the visible light color image is an RGB three-channel color image, and the combined image is a two-channel image;

[0149] The generation module 902 is used to input the combined image as an input image into the generator to obtain a fused brightness image; wherein the fused brightness image is a single-channel image and the spatial size of the fused brightness image is the same as the spatial size of the input image;

[0150] The fusion module 903 is used to perform color space conversion processing on the fused brightness image as the Y channel and the CbCr channel to obtain a fused color image; wherein, the fused color image is an RGB image with the same color space as the visible light color image; the generator is trained based on a discriminator before performing image fusion, the discriminator includes a first discriminator, a second discriminator and a third discriminator, the first discriminator is used to identify the authenticity of the fused color image and the visible light color image, the second discriminator is used to identify the authenticity of the fused brightness image and the infrared image, and the third discriminator is used to identify the authenticity of the fused brightness image and the Y channel.

[0151] Those skilled in the art should understand that Figure 9 The functions of each unit in the multi-source image fusion device 900 based on a three-discriminator generative adversarial network shown can be understood by referring to the relevant description of the multi-source image fusion method based on a three-discriminator generative adversarial network described above. Figure 9 The functions of each unit in the multi-source image fusion device 900 based on a three-discriminator generative adversarial network shown can be implemented by a program running on a processor or by specific logic circuits.

[0152] In one possible implementation, the first discriminator uses the visible light color image as the real label and the fused color image as fake data; the second discriminator uses the infrared image as the real label and the fused brightness image as fake data; and the third discriminator uses the Y channel as the real label and the fused brightness image as fake data.

[0153] The first discriminator is used to make the fused color image contain more color features of the visible light color image; the second discriminator is used to make the fused brightness image contain more pixel intensity information of the infrared image; the third discriminator is used to make the fused brightness image contain more detailed texture features, the detailed texture features including detailed texture information of the intrusion target;

[0154] The first discriminator, the second discriminator, and the third discriminator force the generator to optimize the fusion effect through adversarial training, and the generated image is close to the real image in terms of color, infrared intensity, and brightness details.

[0155] In one possible implementation, the generator uses dense blocks, residual blocks, skip connections, and multiple convolutional layers to extract, fuse, and reconstruct features from visible light color images and infrared images. The generator employs 3×3 convolutional kernels and REFLECT reflection fill to preserve image details, combines batch normalization layers and ReLU / tanh activation functions to improve network stability and output quality, and optimizes the fusion effect through adversarial loss and content loss.

[0156] In one possible implementation, the kernel size setting of the generator includes: except for the transition layer conv3 and the output convolutional layer conv5 which use 1×1 kernels, all other convolutional layers use 3×3 kernels, thereby achieving the function of large-size convolutional kernels and reducing the number of parameters through stacking;

[0157] The generator's stride setting includes: setting the stride of all convolutional layers to 1 to fully extract features and reduce information loss;

[0158] The generator introduces a normalization layer (BN) after each convolutional layer to alleviate the gradient vanishing problem and accelerate network convergence.

[0159] The activation function settings for the generator include: except for the output convolutional layer which uses the tanh activation function, all other convolutional layers use the ReLU activation function to enhance nonlinear expressive power;

[0160] The generator's padding method includes: all convolutional layers are padded with REFLECT reflections to ensure feature map size matching and maintain the output image size unchanged;

[0161] The generator's input and feature extraction convolutional layers upscale the input two-channel image to a 64-channel feature map through two convolutional layers to avoid distortion caused by a sudden increase in the number of channels.

[0162] The dense block function of the generator includes: maximizing feature reuse and information flow through dense connections, reducing parameters and mitigating gradient vanishing;

[0163] The generator's transition layer uses a 1×1 convolutional kernel to reduce the number of channels to 64 in order to control network complexity and connect residual blocks;

[0164] The generator's residual blocks maintain the integrity of the input data through identity mapping and learn residual information to supplement multi-scale features;

[0165] The skip connection of the generator is used to directly pass the output of the shallow feature extraction layer to the residual block, thereby enhancing feature fusion and reuse.

[0166] The generator’s feature reconstruction and output convolutional layer reduces the features to a single channel and outputs a fused image through two convolutional layers. The last layer uses a 1×1 convolutional kernel to integrate multi-channel features.

[0167] The generator's loss function is composed of adversarial loss. With content loss The adversarial loss is optimized for realism through adversarial training against the discriminator, and the content loss preserves key information by constraining the similarity between the fused image and the source image.

[0168] In one possible implementation, the first discriminator, the second discriminator, and the third discriminator have the same network structure; the network structure includes a three-layer convolutional layer and a fully connected layer structure, and feature extraction and classification are achieved through strided convolution, VALID padding, and linear fully connected layers, and a customized loss function is combined to drive the generator to optimize the fusion effect.

[0169] In one possible implementation, the network structure uses a 3×3 convolutional kernel in the first convolutional layer to extract primary features of the input image through a large receptive field and perform channel upscaling; the second and third convolutional layers also use 3×3 convolutional kernels, stacking small receptive fields to achieve multi-scale feature extraction, while reducing the number of parameters and enhancing nonlinear expressive power; all convolutional layers use the ReLU activation function, and batch normalization (BN) layers are added except for the first layer to accelerate model convergence and stabilize the training process;

[0170] The network structure replaces the traditional pooling layer with a stride of 2 convolution operation, which reduces information loss while decreasing the resolution of the feature map layer by layer. The stride convolution controls the decrease in feature map size, and finally outputs a compact feature vector for processing by the fully connected layer.

[0171] All convolutional layers use VALID padding to avoid introducing extra pixel padding, thus preventing noise interference and ensuring the accuracy of feature extraction.

[0172] The feature map output from the last convolutional layer is unfolded into a linear sequence, and then weighted and fused through a fully connected layer to generate a discriminator that scores the authenticity of the input image. The weights of the fully connected layer are matched with the number of feature points, completing a non-linear mapping from the feature space to the discrimination result, and outputting a binary classification result.

[0173] The first discriminator loss consists of the sum of the authenticity score of the visible light color image and the false score of the fused image, driving the fused image to retain visible light texture details; the second discriminator loss consists of the sum of the authenticity score of the infrared image and the false score of the fused image, ensuring that the fused image contains infrared thermal radiation information; the loss function is jointly optimized with the generator loss by minimizing the discrimination score of the source image and maximizing the discrimination score of the fused image, achieving adversarial balance.

[0174] In one possible implementation, the generator's loss function is used to deceive the discriminator and maintain content similarity with the source image; the generator's loss function Losses due to confrontation and content loss The structure is as follows, where the weight λ is used to control the balance between the two, and the function expression is:

[0175] ;

[0176] The definition of adversarial loss is:

[0177] ;

[0178] in, The first discriminator is the one that judges the fused color image. The identification results The second discriminator is the fused brightness image The identification results The third discriminator is used to analyze the fused brightness image. The identification results;

[0179] The content loss function is:

[0180] ;

[0181] in, This represents the content loss weights when comparing the generated color image with the real color image;

[0182] This indicates the content loss weights when comparing the generated brightness image with the real infrared image;

[0183] This indicates the content loss weights when comparing the generated brightness image with the Y channel of the real visible light color image;

[0184] This represents the generated brightness image;

[0185] Represents an infrared image;

[0186] y represents the Y channel of the photoluminescent image;

[0187] This represents the generated color image;

[0188] Represents a true color image;

[0189] The loss function of the first discriminator is:

[0190] ;

[0191] The loss function of the second discriminator is:

[0192] ;

[0193] The loss function of the third discriminator is:

[0194] ;

[0195] The generator's loss function is used to confuse the discriminator, while the discriminator's loss function is used to improve the discriminator's performance. The generator and discriminator undergo adversarial training.

[0196] In one possible implementation, the generator and discriminator are trained in the following manner:

[0197] With fixed generator network parameters, batch training data is input into the generator to generate fused images. The discriminator distinguishes the fused images from the source images and calculates the discriminator loss function value. If the discriminator loss value is greater than the first threshold or the current training round has not reached the set value, the gradient descent method is used to optimize the discriminator network parameters.

[0198] When the discriminant loss value is less than the second threshold, the generator loss value is greater than the third threshold, or the total number of training iterations reaches the preset threshold, the current training phase is deemed to have met the target, and the process is switched to the generator training phase; otherwise, the discriminant network continues to be iteratively optimized.

[0199] The discriminator network parameters are fixed, and the generator parameters are updated through backpropagation using the generator loss function. If the generator loss value does not reach the convergence threshold or the training rounds are not completed, the alternating training process is repeated.

[0200] During training, the loss function value and total number of training iterations are monitored in real time, and the switching conditions for training phases are dynamically adjusted until the network converges to the optimal state.

[0201] The aforementioned multi-source image fusion device based on a three-discriminator generative adversarial network improves upon existing deep learning-based image fusion techniques by refining the network structure and loss function. In terms of network structure, two discriminators are used, combining a residual network and a densely connected network to construct the network structure. Simultaneously, the image color space transformation YCBCr2RGB is utilized to fuse the color information of the source image along with the visible image. Regarding the loss function, the generator's loss function combines content loss and adversarial loss to ensure that a large amount of the main information of the source image is preserved. The discriminator's network structure and loss function are relatively simple, primarily aiming to enrich the details of visible light information in the fused image and integrate the high-brightness thermal radiation information of the infrared portion of the fused image. This method can improve the problem of insufficient visible light information representation under poor lighting conditions and address challenges such as insufficient clarity of the fused image and weak correlation with the source image.

[0202] like Figure 10 As shown, Figure 10 This is a schematic diagram of the composition structure of the electronic device 1000 provided in the embodiments of this application. The electronic device 1000 includes:

[0203] The device 1000 includes a processor 1001, a storage medium 1002, and a bus 1003. The storage medium 1002 stores machine-readable instructions executable by the processor 1001. When the electronic device 1000 is running, the processor 1001 communicates with the storage medium 1002 via the bus 1003. The processor 1001 executes the machine-readable instructions to perform the steps of the multi-source image fusion method based on a three-discriminator generative adversarial network described in this application embodiment.

[0204] In practical applications, the various components in the electronic device 1000 are coupled together via a bus 1003. It is understood that the bus 1003 is used to achieve communication between these components. In addition to a data bus, the bus 1003 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 10 The general labeled all buses as Bus 1003.

[0205] The aforementioned electronic device improves upon existing deep learning-based image fusion techniques by refining the network structure and loss function. Regarding the network structure, it employs two discriminators, combining a residual network and a densely connected network to construct the network structure. Simultaneously, it utilizes the image color space transformation YCBCr2RGB to fuse the color information of the source image along with the visible image. For the loss function, the generator's loss function combines content loss and adversarial loss to ensure that a large amount of the main information of the source image is preserved. The discriminator's network structure and loss function are relatively simple, primarily aiming to enrich the details of the visible light information in the fused image and integrate the high-brightness thermal radiation information of the infrared portion of the fused image. This method can improve the problem of insufficient visible light information representation in poor lighting conditions and address challenges such as insufficient clarity of the fused image and weak correlation with the source image.

[0206] This application also provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed by at least one processor 1001, the multi-source image fusion method based on a three-discriminator generative adversarial network described in this application is implemented.

[0207] In some embodiments, the storage medium may be a magnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.; or it may be a device that includes one or any combination of the above-mentioned memories.

[0208] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0209] As an example, executable instructions may, but do not necessarily, correspond to files in the file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0210] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0211] The aforementioned computer-readable storage medium improves upon existing deep learning-based image fusion techniques by modifying the network structure and loss function. Regarding the network structure, two discriminators are used, combining a residual network and a densely connected network to construct the network structure. Simultaneously, the image color space transformation YCBCr2RGB is utilized to fuse the color information of the image along with the visible image. For the loss function, the generator's loss function combines content loss and adversarial loss to ensure that a large amount of the main information of the source image is preserved. The discriminator's network structure and loss function are relatively simple, primarily aiming to enrich the details of visible light information in the fused image and integrate the high-brightness thermal radiation information of the infrared portion of the fused image. This method can improve the problem of insufficient visible light information expression under poor lighting conditions and solve challenges such as insufficient clarity of the fused image and weak correlation with the source image. In the several embodiments provided in this application, it should be understood that the disclosed methods and electronic devices can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods, such as multiple units or components being combined, integrated into another system, or some features being ignored or not executed. Furthermore, the coupling, direct coupling, or communication connection between the components shown or discussed may be indirect coupling or communication connection through some interface, device, or unit, and may be electrical, mechanical, or other forms.

[0212] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0213] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0214] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a platform server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0215] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multi-source image fusion method based on a three-discriminator generative adversarial network, characterized in that, The method includes: The visible light color image is subjected to color space conversion processing to obtain the color space YCbCr corresponding to the visible light color image, and the Y channel is concatenated with the single-channel infrared image in the channel dimension to obtain a combined image; wherein, the visible light color image is an RGB three-channel color image, and the combined image is a two-channel image; The combined image is used as the input image to the generator to obtain a fused brightness image; wherein the fused brightness image is a single-channel image and the spatial size of the fused brightness image is the same as the spatial size of the input image; The fused brightness image is used as the Y channel, and color space conversion processing is performed with the CbCr channel to obtain a fused color image; wherein, the fused color image is an RGB image with the same color space as the visible light color image; the generator is trained based on a discriminator before image fusion, the discriminator includes a first discriminator, a second discriminator and a third discriminator, the first discriminator is used to identify the authenticity of the fused color image and the visible light color image, the second discriminator is used to identify the authenticity of the fused brightness image and the infrared image, and the third discriminator is used to identify the authenticity of the fused brightness image and the Y channel.

2. The method according to claim 1, characterized in that, The first discriminator uses the visible light color image as the real label and the fused color image as fake data; the second discriminator uses the infrared image as the real label and the fused brightness image as fake data; the third discriminator uses the Y channel as the real label and the fused brightness image as fake data. The first discriminator is used to enable the fused color image to contain more color features of the visible light color image; The second discriminator is used to make the fused brightness image contain more pixel intensity information from the infrared image; the third discriminator is used to make the fused brightness image contain more detailed texture features, including detailed texture information of the intrusion target; The first discriminator, the second discriminator, and the third discriminator force the generator to optimize the fusion effect through adversarial training, and the generated image is close to the real image in terms of color, infrared intensity, and brightness details.

3. The method according to claim 1, characterized in that, The generator uses dense blocks, residual blocks, skip connections, and multiple convolutional layers to extract, fuse, and reconstruct features from visible light color images and infrared images. The generator employs 3×3 convolutional kernels and REFLECT reflection fill to preserve image details, and combines batch normalization layers and ReLU / tanh activation functions to improve network stability and output quality. Furthermore, it optimizes the fusion effect through adversarial loss and content loss.

4. The method according to claim 3, characterized in that, The kernel size settings of the generator include: except for the transition layer conv3 and the output convolutional layer conv5 which use 1×1 kernels, all other convolutional layers use 3×3 kernels. By stacking, the function of large-size convolutional kernels is achieved and the number of parameters is reduced. The generator's stride setting includes: setting the stride of all convolutional layers to 1 to fully extract features and reduce information loss; The generator introduces a normalization layer (BN) after each convolutional layer to alleviate the gradient vanishing problem and accelerate network convergence. The activation function settings for the generator include: except for the output convolutional layer which uses the tanh activation function, all other convolutional layers use the ReLU activation function to enhance nonlinear expressive power; The generator's padding method includes: all convolutional layers are padded with REFLECT reflections to ensure feature map size matching and maintain the output image size unchanged; The generator's input and feature extraction convolutional layers upscale the input two-channel image to a 64-channel feature map through two convolutional layers to avoid distortion caused by a sudden increase in the number of channels. The dense block function of the generator includes: maximizing feature reuse and information flow through dense connections, reducing parameters and mitigating gradient vanishing; The generator's transition layer uses a 1×1 convolutional kernel to reduce the number of channels to 64 in order to control network complexity and connect residual blocks; The generator's residual blocks maintain the integrity of the input data through identity mapping and learn residual information to supplement multi-scale features; The skip connection of the generator is used to directly pass the output of the shallow feature extraction layer to the residual block, thereby enhancing feature fusion and reuse. The generator’s feature reconstruction and output convolutional layer reduces the features to a single channel and outputs a fused image through two convolutional layers. The last layer uses a 1×1 convolutional kernel to integrate multi-channel features. The generator's loss function is composed of adversarial loss. With content loss The adversarial loss is optimized for realism through adversarial training against the discriminator, and the content loss preserves key information by constraining the similarity between the fused image and the source image.

5. The method according to claim 1, characterized in that, The first discriminator, the second discriminator, and the third discriminator have the same network structure; the network structure includes a three-layer convolutional layer and a fully connected layer structure, and feature extraction and classification are achieved through strided convolution, VALID padding, and linear fully connected layers, and a customized loss function is combined to drive the generator to optimize the fusion effect.

6. The method according to claim 5, characterized in that, In the network structure, the first convolutional layer uses a 3×3 convolutional kernel to extract primary features of the input image through a large receptive field and perform channel upscaling; the second and third convolutional layers also use 3×3 convolutional kernels, stacked with small receptive fields to achieve multi-scale feature extraction, while reducing the number of parameters and enhancing nonlinear expressive power; all convolutional layers use the ReLU activation function, and batch normalization (BN) layers are added except for the first layer to accelerate model convergence and stabilize the training process; The network structure replaces the traditional pooling layer with a stride of 2 convolution operation, which reduces information loss while decreasing the resolution of the feature map layer by layer. The stride convolution controls the decrease in feature map size, and finally outputs a compact feature vector for processing by the fully connected layer. All convolutional layers use VALID padding to avoid introducing extra pixel padding, thus preventing noise interference and ensuring the accuracy of feature extraction. The feature map output from the last convolutional layer is unfolded into a linear sequence, and then weighted and fused through a fully connected layer to generate a discriminator that scores the authenticity of the input image. The weights of the fully connected layer are matched with the number of feature points, completing a non-linear mapping from the feature space to the discrimination result, and outputting a binary classification result. The first discriminator loss consists of the sum of the authenticity score of the visible light color image and the false score of the fused color image, driving the fused image to retain visible light texture details; the second discriminator loss consists of the sum of the authenticity score of the infrared image and the false score of the fused brightness image, ensuring that the fused image contains infrared thermal radiation information; the loss function is jointly optimized with the generator loss by minimizing the discrimination score of the source image and maximizing the discrimination score of the fused image, achieving adversarial balance.

7. The method according to claim 1, characterized in that, The generator's loss function is used to deceive the discriminator and maintain content similarity with the source image; the generator's loss function Losses due to confrontation and content loss The structure is as follows, where the weight λ is used to control the balance between the two, and the function expression is: ； The definition of adversarial loss is: ； in, The first discriminator is the one that judges the fused color image. The identification results The second discriminator is the fused brightness image The identification results The third discriminator is used to analyze the fused brightness image. The identification results; The content loss function is: ； in, This represents the content loss weights when comparing the generated color image with the real color image; This indicates the content loss weights when comparing the generated brightness image with the real infrared image; This indicates the content loss weights when comparing the generated brightness image with the Y channel of the real visible light color image; This represents the generated brightness image; Represents an infrared image; y represents the Y channel of the photoluminescent image; This represents the generated color image; Represents a true color image; The loss function of the first discriminator is: ； The loss function of the second discriminator is: ； The loss function of the third discriminator is: ； The generator's loss function is used to confuse the discriminator, while the discriminator's loss function is used to improve the discriminator's performance. The generator and discriminator undergo adversarial training.

8. The method according to claim 1, characterized in that, The generator and discriminator are trained in the following manner: With fixed generator network parameters, batch training data is input into the generator to generate fused images. The discriminator distinguishes the fused images from the source images and calculates the discriminator loss function value. If the discriminator loss value is greater than the first threshold or the current training round has not reached the set value, the gradient descent method is used to optimize the discriminator network parameters. When the discriminant loss value is less than the second threshold, the generator loss value is greater than the third threshold, or the total number of training iterations reaches the preset threshold, the current training phase is deemed to have met the target, and the process is switched to the generator training phase; otherwise, the discriminant network continues to be iteratively optimized. The discriminator network parameters are fixed, and the generator parameters are updated through backpropagation using the generator loss function. If the generator loss value does not reach the convergence threshold or the training rounds are not completed, the alternating training process is repeated. During training, the loss function value and total number of training iterations are monitored in real time, and the switching conditions for training phases are dynamically adjusted until the network converges to the optimal state.

9. A multi-source image fusion device based on a three-discriminator generative adversarial network, characterized in that, The device includes: The combination module is used to perform color space conversion processing on the visible light color image to obtain the color space YCbCr corresponding to the visible light color image, and to cascade the Y channel with the single-channel infrared image in the channel dimension to obtain the combined image; wherein, the visible light color image is an RGB three-channel color image, and the combined image is a two-channel image; A generation module is used to input the combined image as an input image into a generator to obtain a fused brightness image; wherein the fused brightness image is a single-channel image and the spatial size of the fused brightness image is the same as the spatial size of the input image; A fusion module is used to perform color space conversion processing on the fused brightness image as the Y channel and the CbCr channel to obtain a fused color image; wherein, the fused color image is an RGB image with the same color space as the visible light color image; the generator is trained based on a discriminator before performing image fusion, the discriminator includes a first discriminator, a second discriminator and a third discriminator, the first discriminator is used to identify the authenticity of the fused color image and the visible light color image, the second discriminator is used to identify the authenticity of the fused brightness image and the infrared image, and the third discriminator is used to identify the authenticity of the fused brightness image and the Y channel.

10. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the multi-source image fusion method based on a three-discriminator generative adversarial network as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the multi-source image fusion method based on a three-discriminator generative adversarial network as described in any one of claims 1 to 8.

Images