Method for establishing multispectral image fusion model based on generative adversarial network and fusion method

Abstract

The application discloses a kind of based on the establishment method of multispectral image fusion model of generative adversarial network, it includes: establishing the training set and verification set comprising infrared light picture, visible light picture;Respectively, the picture in training set is input into a generator with residual module, and generates fusion image;Fusion image is respectively input into two discriminators with DROPOUT layer and is discriminated processing;According to the loss function value of calculating discriminator and the loss function value of generator, generator parameter and discriminator parameter are updated until loss converges, and trained generator and discriminator are obtained;Using verification set, trained generator and discriminator are verified, after passing through verification, multispectral image fusion model is obtained according to generator.The application also discloses corresponding image fusion method.Implementation of the present application can avoid overfitting problem in model training process, and reduce the training time of training resource, improve the fusion effect of fusion model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image fusion data processing, and in particular to a method for establishing a multispectral image fusion model based on a generative adversarial network and a fusion method. BACKGROUND

[0002] In the power scenario, as the power equipment ages or rusts, poor contact causes excessive current and voltage abnormalities, and the equipment will produce thermal abnormalities. If the maintenance personnel fail to timely discover these thermal abnormalities, serious safety accidents and property losses may be caused. Therefore, it is very important to timely discover thermal abnormalities of power equipment. Common thermal abnormality detection methods include an infrared image sensor-based power equipment thermal abnormality detection method. An infrared thermal imager has the advantages of non-contact and uninterrupted power detection of power equipment and is widely used in various power units. With the continuous expansion of the power grid scale in China, the infrared inspection pressure of power equipment is also increasing, and more and more infrared thermal image data collected by inspection robots, fixed-point monitoring track robots and other equipment are used, and a fusion method using visible light image sensors and infrared image sensors.

[0003] Image fusion refers to combining images or video information of a specific scene acquired by multiple sensors at the same time by using a selected algorithm to generate a new image related to the scene, which contains relevant information of images acquired by multiple sensors. Compared with a single spectral image, multispectral image fusion has many advantages: multispectral images have redundancy and also have some information content that cannot be captured, so they can complement each other, and multispectral images can also improve the accuracy and clarity of observation information.

[0004] Some methods for training image fusion models using deep learning models (such as generative adversarial networks) have appeared. For example, Figure 1 As shown in the figure, a schematic diagram of a generative adversarial network is shown. However, in the existing training process of the discriminator of the generative adversarial network, there are some deficiencies, such as the network being prone to overfitting. Overfitting is manifested in that the model performs better and better on the training set, but the error on the cross-validation set becomes larger and larger, and the generalization ability and robustness become worse and worse. Overfitting of the discriminator in the generative adversarial network model will cause the generator, i.e., the image fusion generator, to learn incorrect gradient information, resulting in a certain deviation in the generated fusion image. In order to solve the problem of overfitting of the discriminator of the generative adversarial network, multiple discriminators are often trained, but this will bring new problems of too long training time and too large model memory occupation. SUMMARY

[0005] The technical problem solved by the present application is to provide a method for establishing a multispectral image fusion model based on a generative adversarial network and a fusion method, which can avoid overfitting in the model training process, reduce the training time of the training resources, and improve the fusion effect of the fusion model.

[0006] To solve the technical problem, in one aspect of the present application, a method for establishing a multispectral image fusion model based on a generative adversarial network is provided, which comprises at least the following steps:

[0007] Step S10: establishing a training set and a validation set, wherein each of the training set and the validation set comprises a plurality of groups of picture pairs, and each group of picture pairs comprises an infrared light picture and a gray value picture of a visible light picture taken at the same position;

[0008] Step S11: inputting each group of picture pairs in the training set into a generator with a residual module respectively to generate a fusion image;

[0009] Step S12: inputting the picture pairs and the fusion image into two discriminators with a DROPOUT layer respectively for discrimination processing to obtain a judgment value of the two images in the picture pairs, and calculating a discriminator loss function value and a generator loss function value;

[0010] Step S13: performing back propagation calculation according to the discriminator loss function value and the generator loss function value, updating the generator parameters and the discriminator parameters, until the generator and the discriminator loss converge, and obtaining a trained generator and a trained discriminator;

[0011] Step S14: verifying the trained generator and the trained discriminator using the validation set, and obtaining a multispectral image fusion model from the generator after passing the verification.

[0012] Preferably, the step S10 further comprises:

[0013] acquiring a first number of initial picture pairs, each initial picture pair comprising a visible light picture and an infrared light picture;

[0014] converting each initial picture pair into a gray scale image with an intensity value of 0-1;

[0015] randomly cropping a plurality of groups of pictures with a predetermined size in each initial picture pair to form a plurality of groups of new picture pairs, and forming a training set;

[0016] obtaining a picture pair with a predetermined size in each initial picture pair using a center cropping method to form a validation set.

[0017] Preferably, the step S11 further comprises:

[0018] Step S110, obtaining a generator written by a model.py file, the generator comprising an input layer, a residual module group, and an output layer;

[0019] The input layer comprises a convolution, a PRELU activation function processing layer; the residual module group comprises a plurality of residual modules, each residual module comprising a convolution, a batch normalization, a PRELU activation function, a convolution, a batch normalization, and an element-wise multiplication processing layer; and the output layer comprises a convolution, a PRELU activation function, and a convolution processing layer.

[0020] Step S111, randomly inputting a picture pair in a training set into the generator with the residual module to generate a fusion image.

[0021] Preferably, the step S12 further comprises:

[0022] Step S120, obtaining two discriminators written by a model.py file, each discriminator comprising six layers, the first five layers each comprising a convolution, a batch normalization, a Leaky activation function, a convolution, a batch normalization, and a pooling processing layer; and the sixth layer comprising a DROPOUT layer, a convolution, and a pooling processing layer.

[0023] Step S121, inputting the fusion image into the two discriminators, wherein one discriminator discriminates the fusion image and an infrared gray value picture in the image pair; and the other discriminator discriminates the fusion image and a visible light gray value picture in the image pair.

[0024] Step S122, obtaining a loss function written in a loss.py file, and calculating a loss function value of the discriminators.

[0025] Preferably, the step S122 further comprises:

[0026] The loss function value of the discriminators is calculated by the following formula:

[0027] loss G =L image +αL gradient +βL detail +δL tee +γL adversarial

[0028] L image and L gradient are content losses, L detail is a detail loss, L tee is a target edge enhancement loss, and L adversarial is an adversarial loss.

[0029] L image:

[0030]

[0031] I r(x，y) denotes the value of the infrared light image at (x, y), I f(x，y) denotes the value of the fused image at (x, y), i.e. the square of the difference of the pixel values at the same position of each pixel is calculated;

[0032] L is calculated using the following formula gradient :

[0033]

[0034] D v(x，y) denotes the gradient value of the visible light image at (x, y), D f(x，y) denotes the gradient value of the fused image at (x, y), i.e. the square of the difference of the pixel values at the same position of each pixel is calculated;

[0035] L is calculated using the following formula detail :

[0036]

[0037] φ v(i，j) denotes the value of the visible light feature map at (i, j), φ f(i，j) denotes the value of the fused image feature map at (i, j);

[0038] L is calculated using the following formula tee :

[0039]

[0040] The weight G is used to make the fused image pay more attention to the edge information, and G is defined as follows:

[0041] G(x, y) = N k＝3 (D r(x，y) )+ N k＝5 (D r(x，y) )+ N k＝7 (D r(x，y) )

[0042] D v(x，y) denotes the gradient value of the visible light image at (x, y), N is a Gaussian kernel function, and k = 3, 5, 7 are empirical values;

[0043] L is calculated using the following formula adversarial :

[0044]

[0045] Imix a mixed input representing a visible light and an infrared light image, represents the probability that the discriminator determines the fused image to be real, and N is the batch size.

[0046] Preferably, the step S13 further comprises:

[0047] In each training, in each discriminator, first make the discriminator in the training mode, make the DROPOUT layer activate neurons with a first probability, calculate the discriminator loss according to the output value obtained after inputting the discriminator image data, perform back propagation calculation, and update the discriminator parameters;

[0048] Then make the discriminator in the test mode, make the DROPOUT layer activate neurons with a second probability, calculate the generator loss according to the input generator image data and the obtained fused image, perform back propagation calculation, and update the generator parameters;

[0049] Continue training until the generator and discriminator loss converge, and then terminate the training, so as to obtain the trained generator and discriminator.

[0050] Correspondingly, another aspect of the present application also provides a multispectral image fusion method based on a generative adversarial network, which comprises the following steps:

[0051] Obtain a plurality of groups of picture pairs of the same position of the power equipment to be tested to form a test set, and each group of picture pairs comprises corresponding gray value pictures of infrared light pictures and visible light pictures;

[0052] Import the picture pairs in the test set into the multispectral image fusion model trained by the foregoing method to obtain a fused image.

[0053] The embodiment of the present application has the following beneficial effects:

[0054] The present application provides a multispectral image fusion model based on a generative adversarial network and a method for establishing the same, and by using a generative adversarial network model and adding a DROPOUT layer in the discriminator, the present application solves the problem of overfitting of the discriminator of the generative adversarial network and the problem of increasing training time and memory occupation caused by training multiple discriminators, and is a relatively reasonable solution. By implementing the present application, the problem of overfitting in the model training process can be avoided, the training time of the training resources can be reduced, and the fusion effect of the fusion model is improved.

[0055] The multispectral image fusion method provided by the present application should be applied to fault detection of power equipment, for example, the present application is deployed on power equipment monitoring and other visual sensing devices, and artificial intelligence is used for fault detection, which can greatly improve the probability of problem discovery and reduce labor costs. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a schematic diagram of a general generative adversarial network structure;

[0058] Figure 2 This is a schematic diagram of the main flow of an embodiment of the method for establishing a multispectral image fusion model based on generative adversarial networks provided by the present invention;

[0059] Figure 3 This invention relates to a schematic diagram of the generator structure;

[0060] Figure 4 This is a schematic diagram of the discriminator involved in the present invention;

[0061] Figure 5 This is a schematic diagram of the DROPOUT principle involved in this invention;

[0062] Figure 6 The present invention relates to schematic diagrams of images before and after fusion;

[0063] Figure 7 This is a schematic diagram illustrating the fitting effect of the generative adversarial network involved in this invention. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0065] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0066] like Figure 2 The diagram illustrates the main flowchart of an embodiment of a method for establishing a multispectral image fusion model based on generative adversarial networks provided by the present invention; combined with... Figures 2 to 7As shown, in the embodiment, the method at least includes the following steps:

[0067] Step S10, establishing a training set and a validation set, the training set and the validation set both include a plurality of groups of picture pairs, each group of picture pairs includes an infrared light picture, a gray value picture of a visible light picture taken at the same position;

[0068] Specifically, in one example, the step S10 further includes:

[0069] Obtaining a first number of initial picture pairs, each initial picture pair containing a visible light picture and an infrared light picture; in one example of the present application, the training data set includes 33 groups of initial picture pairs, i.e. contains 33 visible light pictures and 33 infrared light images, all of which are 640*480 size images; through observation of the data set, it is found that the infrared light picture has excellent edge contour information, and the visible light picture contains more texture information; because the devices for collecting the pictures are different, the visible light picture and the infrared light picture cannot maintain the same angle and position, so there may be slight deviations between the visible light picture and the infrared light picture. Save the visible light picture and the infrared light picture to the local end, wrap the reading function in the package such as torch and torchvision, and uniformly write in the image data processing tool package data_utils.py file in this experiment;

[0070] Convert each initial picture pair (RGB picture) into a gray image with intensity value of 0-1;

[0071] Randomly crop a plurality of groups of pictures of a predetermined size in each initial picture pair to form a plurality of new picture pairs to constitute a training set; because the training data in this experiment is less, the pictures of 88*88 size are randomly cropped in the same position from the two original pictures in each initial picture pair, and are converted into gray images as the training set, so as to expand the number of training set samples, so that the original 33 picture data training set can be expanded into many similar but not completely identical visible light and infrared light image data sets, which can greatly enhance the generalization ability of the model, better learn the essence of image fusion, and prevent the generator model from overfitting on the training set images. The training set reading class is TrainDatasetFromFolder; the random cropping method is written in the imagecrop.py file, which is called through the data_utils.py file mentioned in the above text;

[0072] In each initial picture pair, a center crop is used to obtain a picture pair of a predetermined size, forming a verification set. The verification set is constructed by center cropping 88*88 size pictures and converting them into grayscale images to verify the effectiveness of the network. The verification set reading class is ValDatasetFromFolder.

[0073] Step S11, respectively input each group of picture pairs in the training set into a generator with a residual module to generate a fusion image.

[0074] It can be understood that the generative adversarial network used in the present application is similar to Figure 1 , except that the discriminator is two.

[0075] Specifically, in one example, the step S11 further comprises:

[0076] Step S110, obtaining a generator written by a model.py file, the generator comprising an input layer, a residual module group and an output layer.

[0077] As shown in Figure 3 , the input layer comprises a convolution (9*9Conv), a PRELU activation function processing layer; the residual module group comprises a plurality of residual modules, each residual module comprising a convolution (3*3Conv), a batch normalization (BN), a PRELU activation function, a convolution (3*3Conv), a batch normalization (BN) and an element-wise processing layer; and the output layer comprises a convolution (3*3Conv), a PRELU activation function and a convolution (1*1Conv) processing layer.

[0078] Figure 3 For example, "k3n64s1" represents a convolution layer with a convolution kernel size of 3*3, an output channel number of 64 and a convolution kernel step of 1. In the structural diagram, if not specially specified, the default padding value of all convolution layers is 1.

[0079] Step S111, randomly inputting a picture pair (one visible light picture and one infrared light picture) in the training set into the generator with the residual module to generate a fusion image.

[0080] Further, for the residual module, the output needs to be added to the input of the module, which is expressed by the following formula:

[0081] y = ResidualBlock (x) + x

[0082] x represents the input of the residual block, and y represents the output of the residual block, such a residual structure has been proved to have the effect of deepening the network while enhancing the fitting effect of the network in the neural network, therefore, the network structure of the present application also adopts the residual structure as an important component of the generator.

[0083] In step S12, the image pair combined fusion image is input into two discriminators with DROPOUT layers for discrimination processing, to obtain the judgment value of the two images in the image pair, and to calculate the discriminator loss function value and the generator loss function value.

[0084] Specifically, in one example, the step S12 further comprises:

[0085] In step S120, two discriminators written by using a model.py file are obtained, as shown in the following table. Figure 4 Each discriminator contains 6 layers, the first five layers contain convolution (3*3Conv), batch normalization (BN), Leaky activation function, convolution (3*3Conv), batch normalization (BN) and pooling (Pool) processing layers; the sixth layer contains a DROPOUT layer, a convolution (3*3Conv) and a pooling (Pool) processing layer; for example, "2k3n64s1" represents two convolution layers with a kernel size of 3*3, an output channel number of 64 and a kernel step of 1, and the default padding value of the convolution layer is 1.

[0086] In step S121, the fusion image is input into the two discriminators, one of which discriminates the fusion image and the infrared gray value image in the image pair, and the other discriminates the fusion image (I f ) and the visible light gray value image (I v ) in the image pair, and outputs a discrimination value.

[0087] In step S122, a loss function written in a loss.py file is obtained to calculate the loss function value of the discriminator. The discriminator loss function mainly uses a Wasserstein loss function, and the specific form is as follows:

[0088] loss D =D(G(x))-D(x)

[0089] Where D(x) represents the discriminator output, and G(x) represents the generator output. The generator loss function is divided into four parts: content loss, detail loss, target edge enhancement loss and adversarial loss.

[0090] Specifically, the step S122 further comprises:

[0091] The loss function value of the discriminator is calculated by using the following formula:

[0092] loss G =L image +αL gradient +βL detail +δL tee +γL adversarial

[0093] Wherein, L image and L gradient are content losses, L detail is a detail loss, L tee is a target edge enhancement loss, and L adversarial is an adversarial loss.

[0094] L image is calculated by using the following formula:

[0095]

[0096] I r(x，y) represents the value of the infrared light image at (x, y), and I f(x，y) represents the value of the fusion image at (x, y), i.e., the square of the difference between the pixel value at the same position of each pixel of the fusion image and the infrared light image.

[0097] L gradient is calculated by using the following formula:

[0098]

[0099] D v(x，y) represents the gradient value of the visible light image at (x, y), and D f(x，y) represents the gradient value of the fusion image at (x, y), i.e., the square of the difference between the pixel value at the same position of each pixel of the fusion image and the visible light image.

[0100] L detail is calculated by using the following formula:

[0101]

[0102] φ v(i，j) represents the value of the visible light feature map at (i, j), and φ f(i，j) represents the value of the fusion image feature map at (i, j).

[0103] L tee is calculated by using the following formula:

[0104]

[0105] The weight G is used to make the fusion image pay more attention to edge information, and G is defined as follows:

[0106] G(x, y) = N k＝3 (D r(x，y) )+N k＝5 (D r(x，y) )+N k＝7 (D r(x，y) )

[0107] D v(x，y) represents the gradient value of the visible light image at (x, y), N is a Gaussian kernel function, k = 3, 5, 7 is an empirical value;

[0108] L is calculated using the following formula adversarial :

[0109]

[0110] I mix represents the mixed input of the visible light and infrared light images, represents the probability that the discriminator determines that the fusion image is real, and N is the batch size.

[0111] Step S13, according to the discriminator loss function value and the generator loss function value, back propagation calculation is performed to update the generator parameters and the discriminator parameters, until the generator and the discriminator loss converge, and a trained generator and discriminator are obtained.

[0112] Specifically, in one example, the step S13 further includes:

[0113] In each training, in each discriminator, first, the discriminator is in a training mode, the DROPOUT layer activates the neurons at a first probability (such as P = 0.5), the discriminator loss is calculated according to the output value obtained after inputting the discriminator image data, back propagation calculation is performed, and the discriminator parameters are updated.

[0114] Then, the discriminator is in a test mode, the DROPOUT layer activates the neurons at a second probability (such as P = 1), the generator loss is calculated according to the input generator image data and the obtained fusion image, back propagation calculation is performed, and the generator parameters are updated.

[0115] The training is continued until the generator and the discriminator loss converge, and the training is terminated, thereby obtaining a trained generator and discriminator.

[0116] More specifically, in the embodiment of the present application, the code of the training model is written in the train.py file, first the data set reading class encapsulated in the data_utils.py file is called to read the data set into the memory and convert it into a tensor format that can be calculated using the pytorch framework, then the generator and discriminator classes in the model.py file are called to instantiate the generator and discriminator objects, and through the loop, the picture data in the training set is taken out in batches of 16 to input the model for training. Finally, the generator and discriminator parameters are saved as.PTH files.

[0117] It can be understood that for a neural network with N nodes, after DROPOUT, it can be regarded as a set of 2n models, but the number of parameters to be trained is unchanged. In the training of the discriminator stage, the probability process is added after the output of the previous layer with the DROPOUT layer, which not only solves the problem of overfitting by using ensemble learning, but also solves the problem of long training time.

[0118] The use of the DROPOUT layer when it is proposed is to apply it to the neural network, enable the training mode of the DROPOUT layer in the training stage of the neural network, so as to stop a part of the neurons from working, and the stop probability usually uses p=0.5, which is equivalent to training a plurality of neural networks with different parameters at the same time. Then enable the test mode of the DROPOUT layer in the training stage of the neural network, so that all neurons restore to working state, which is equivalent to integrating learning of the trained multiple networks, thereby preventing the overfitting problem caused by single network training.

[0119] At the same time, the DROPOUT layer can not only be applied to the fully connected layer to make a part of the neuron output zero, but also be applied to the convolution layer to make a part of the feature map zero. In the field of deep learning, especially in the research of computer vision, because the 1x1 convolution operation has good optimization and acceleration effect in the code bottom layer, so usually a 1x1 convolution layer is used instead of a fully connected layer at the end of the model. In the subsequent experiments of the present application, a 2D DROPOUT operation on the convolution layer is also used, which makes a part of the feature map output by the upper convolution layer zero before passing it to the next 1x1 convolution. The actual effect is the same as the above-mentioned convolution effect of the fully connected layer. Similarly, the training mode of the 2D DROPOUT layer is enabled in the training stage of the neural network, so as to stop a part of the feature map from participating in the discrimination. The stop probability usually uses p=0.5, which is equivalent to training a plurality of networks with different parameters at the same time. Then enable the test mode of the 2D DROPOUT layer in the training stage of the neural network.

[0120] Specifically, as Figure 5As shown, the neurons connected by the dashed line represent the units that are zeroed in the first step.

[0121] In step S14, the trained generator and discriminator are verified using the verification set, and after passing the verification, a multispectral image fusion model is obtained according to the generator.

[0122] Specifically, by calling the data set reading class encapsulated in the data_utils.py file, the verification data set is instantiated, and then the generator and discriminator models in the model.py file are instantiated and given the parameters in the aforementioned saved.PTH file, thereby obtaining the trained generator and discriminator, and finally the fusion effect of the infrared and visible light images of the generator on the verification set is observed as shown in Figure 6 , and evaluated.

[0123] It can be understood that the present application adopts a generation verification network, which can be mainly divided into two parts of a generator and a discriminator, the generator is used to generate new images, and the discriminator is used to judge the authenticity of the images, whether the images are generated by the generator, and the fitting process is as shown in Figure 7 , the thick dotted line represents the distribution of real data, the solid line represents the distribution of data generated by the generator, and the fine dotted line represents the probability value determined by the discriminator.

[0124] In order to better verify the effect of the method provided by the present application, the following verification work can be carried out.

[0125] The effect of the fusion image obtained by the present application is compared with the fusion image obtained by the fusion network model without adding the DROPOUT layer, and it can be obviously found that the fusion image effect after adding the DROPOUT layer has been obviously improved, the image edge lines are clearer, the detail texture information is more abundant, and even the texture details of the sky and clouds are better learned, more useful information is retained than the original model, and the image fusion effect is obviously improved.

[0126] Meanwhile, in order to objectively evaluate the fusion effect of the method provided by the present application, the present application is compared with the existing visible light and infrared image fusion methods from the qualitative and quantitative aspects. The qualitative evaluation is a subjective evaluation, and the main influencing factor is the visual perception of human beings. For the fused image, we expect that it has both considerable thermal radiation information and rich texture details. The quantitative evaluation is to evaluate the fused image objectively through some statistical indexes. The present application selects six evaluation indexes commonly used in the field, such as information entropy (Entropy, EN), spatial frequency (Spatial Frequency, SF), structural similarity (Structural similarity, SSIM), peak signal-to-noise ratio (Peak Signal-to-Noise Ratio, PSNR), Qabf and standard deviation (standard deviation). The information entropy mainly measures how much information the image contains. The greater the information entropy, the more information it contains. The spatial frequency reflects the rate of change of the image gray scale. The greater the spatial frequency, the clearer the image and the better the quality of the fused image. The value range of the structural similarity is [-1, 1]. The closer to 1, the higher the similarity and the better the fusion quality. The peak signal-to-noise ratio is used to measure the ratio between the effective information and the noise of the image, and can reflect whether the image is distorted. The greater the value, the better the quality of the fused image. Qabf uses local measurement to estimate the performance of significant information from the input in the fused image. The higher the value of Qabf, the better the quality of the fused image. The standard deviation is an objective evaluation index for measuring the information richness of the image. The greater the value, the more dispersed the gray scale distribution of the image, the more information the image carries, and the better the quality of the fused image.

[0127] The comparison of the model fused image and other models in various indexes is shown in Table 1.

[0128] Table 1 Model fused image index

[0129]

[0130] It can be seen that the image fusion effect of the method provided by the present application (FusionGAN with DROPOUT) is very good. In the VIF and AG two indexes, the FusionGAN model surpasses other more common models. Although the other three indexes do not reach the optimal value, they are greatly improved compared with the original FusionGAN model.

[0131] By calling the thop package, the parameter quantity of the generative adversarial network model is calculated, and the parameter quantities of the model before and after adding the DROPOUT layer and the GAN model with a double discriminator structure are compared, as shown in Table 2.

[0132] Table 2 Parameter quantity before and after model improvement

[0133] No Dropout model Dropout model Dual discriminator model Parameter amount 6337609 6337609 11027210

[0134] It can be seen that the introduction of the DROPOUT layer does not increase the parameters of the model compared with the original model, and the DROPOUT model reduces 42.5% compared with the double discriminator model, and brings obvious improvement of the fused image effect.

[0135] By calling the thop package, the calculation amount required for generating the fused image is calculated, and the FLOPs of the generator model before and after adding the DROPOUT layer are compared, as shown in Table 3:

[0136] Table 3 FLOPs before and after model improvement

[0137] No Dropout model Dropout model FLOPs 380141568000.0 380141568000.0

[0138] It can be seen that the introduction of the DROPOUT layer on the discriminator does not affect the calculation speed of the generator model, which shows that the generator model finds a better local minimum point of the parameters without increasing the training time and memory occupation of the training, and adding the DROPOUT layer in the discriminator model is an effective improvement method.

[0139] Correspondingly, another aspect of the present application also provides a multispectral image fusion method based on a generative adversarial network, which comprises the following steps:

[0140] A plurality of image pairs of the same position of the power equipment to be tested are obtained to form a test set, and each image pair includes corresponding gray value images of infrared light images and visible light images;

[0141] The image pairs in the test set are imported into the multispectral image fusion model trained by the multispectral image fusion network establishment method described above. Figures 2 to 7 The fused image is obtained.

[0142] It can be understood that the multispectral image fusion method provided by the present application should be used for fault detection of power equipment, for example, the present application is deployed on power equipment monitoring and other visual sensing devices, and artificial intelligence means is used for fault detection, which can greatly improve the problem discovery probability and reduce the labor cost.

[0143] In actual examples, the generator model is deployed to a mobile terminal, and after visible light and infrared light images are collected, image fusion can be performed through the generator model, which is separated from the fault detection module, so that fusion is performed before detection, the power system fault detection process is modularized and stepped, and it is more conducive to the progress iteration of future technology and product update.

[0144] The generator and the discriminator model can also be deployed to a mobile terminal, and after visible light and infrared images are collected, real-time training is adopted on the mobile terminal, so that the complex environment and weather can be better adapted, but at the same time, a longer fusion image generation time is caused.

[0145] At present, image fusion is also a relatively important research direction in the field of remote sensing, because remote sensing needs to combine infrared, visible light and even ultraviolet image information as much as possible, and fuse these image information into the same image, which is equivalent to effectively compressing the information, and for the remote sensing equipment with small memory, it may have broad development prospects.

[0146] The embodiment of the present application has the following beneficial effects:

[0147] The present application provides a kind of based on the establishment method and fusion method of multispectral image fusion model of generative adversarial network, by using generative adversarial network model, and adding DROPOUT layer in discriminator, compared with prior art, the present application solves the problem of overfitting of generative adversarial network discriminator, and solves the problem of increasing training time and memory occupation caused by training multiple discriminators, it is a more reasonable solution. Implement the present application, can avoid overfitting problem in model training process, and reduce the training time of training resource, improve the fusion effect of fusion model.

[0148] The multispectral image fusion method provided by the present application should be applied to fault detection of power equipment, for example, the present application is deployed in power equipment monitoring and other visual sensing devices, and artificial intelligence means is used for fault detection, which will greatly improve the problem discovery probability and reduce labor cost.

[0149] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, device or computer program product. Therefore, the present application can adopt a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of computer program product implemented on one or more computer usable storage media containing computer usable program code (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.).

[0150] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions specified in the flowchart block or blocks. Figure 1 The flowchart and / or block diagram in the Figures illustrate the architecture, functionality, and operation of possible implementations of apparatuses and computer program products according to various embodiments of the present application. In this regard, each flowchart block and / or combination of flowchart blocks can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing device or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart block or blocks. Figure 1 These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks.

[0151] The above description is only preferred embodiments of the present application, and is not intended to limit the scope of the claims of the present application. Therefore, any equivalent changes or modifications made without departing from the spirit of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for establishing a multi-spectral image fusion model based on a generative adversarial network, characterized in that, At least comprising the following steps: Step S10, establishing a training set and a verification set, wherein the training set and the verification set each comprise a plurality of groups of picture pairs, and each group of picture pairs comprises an infrared light picture, a visible light picture, and a gray value picture of the visible light picture taken at the same position; Step S11, inputting each group of picture pairs in the training set into a generator with a residual module respectively to generate a fusion image; Step S12, inputting the picture pairs and the fusion image into two discriminators with a DROPOUT layer respectively to perform discrimination processing, obtaining a judgment value of the two images in the picture pairs, and calculating a discriminator loss function value and a generator loss function value; Step S13, performing back propagation calculation according to the discriminator loss function value and the generator loss function value, updating the generator parameters and the discriminator parameters until the generator and the discriminator loss converge, and obtaining a trained generator and a trained discriminator; Step S14, verifying the trained generator and the trained discriminator by using the verification set, and obtaining a multispectral image fusion model according to the generator after passing the verification; The step S11 further comprises: Step S110, obtaining a generator written by using a model.py file, wherein the generator comprises an input layer, a residual module group, and an output layer; The input layer comprises a convolution, a PRELU activation function processing layer; the residual module group comprises a plurality of residual modules, each residual module comprises a convolution, a batch normalization, a PRELU activation function, a convolution, a batch normalization, and an element-wise multiplication processing layer; and the output layer comprises a convolution, a PRELU activation function, and a convolution processing layer; Step S111, randomly inputting the picture pairs in the training set into the generator with the residual module to generate the fusion image; The step S12 further comprises: Step S120, obtaining two discriminators written by using a model.py file, wherein each discriminator comprises six layers, the first five layers each comprise a convolution, a batch normalization, a Leaky activation function, a convolution, a batch normalization, and a pooling processing layer; and the sixth layer comprises a DROPOUT layer, a convolution, and a pooling processing layer; Step S121, inputting the fusion image into the two discriminators, wherein one discriminator discriminates the fusion image and the infrared light gray value picture in the image pair; and the other discriminator discriminates the fusion image and the visible light gray value picture in the image pair; Step S122, obtaining a loss function written in a loss.py file to calculate the loss function value of the discriminator.

2. The method of claim 1, wherein, The step S10 further comprises: Obtaining a first number of initial picture pairs, each initial picture pair comprising a visible light picture and an infrared light picture; Converting each initial picture pair into a gray value image with an intensity value of 0-1; Randomly cropping a plurality of groups of pictures with a predetermined size in each initial picture pair to form a plurality of groups of new picture pairs, thereby forming the training set; Obtaining a picture pair with a predetermined size in each initial picture pair by using a center cropping method, thereby forming the verification set.

3. The method of claim 2, wherein, The step S122 further comprises: The loss function value of the discriminator is calculated by using the following formula: wherein, and are content losses, is a detail loss, is a target edge enhancement loss, is a adversarial loss; The following formula was used to calculate : denotes the value of the infrared light image at (x, y), denotes the value of the fused image at (x, y), i.e. the square of the difference of the pixel values at the same position for each pixel. The following formula was used to calculate : represents the gradient value of the visible light image at (x, y), represents the gradient value of the fused image at (x, y), i.e. the square of the difference of the pixel values at the same position of each image is calculated. The following formula was used to calculate : denotes the value of the visible light feature map at (i,j), denotes the value of the fused image feature map at (i,j); The following formula was used to calculate : The weight G is used to make the fused image pay more attention to edge information, and G is defined as follows: represents the gradient value of the visible light image at (x, y), N is a Gaussian kernel function, k = 3, 5, 7 is an empirical value; The following formula was used to calculate : a mixed input representing a visible light and infrared light image, represents the probability that the discriminator judges the fused image as real, N is the batch size.

4. The method of claim 3, wherein, The step S13 further includes: In each training, in each discriminator, first make the discriminator in training mode, make the DROPOUT layer activate neurons with a first probability, calculate the loss of the discriminator according to the output value obtained after inputting the discriminator image data, perform back propagation calculation, and update the parameters of the discriminator; Then make the discriminator in test mode, make the DROPOUT layer activate neurons with a second probability, calculate the loss of the generator according to the input generator image data and the obtained fused image, perform back propagation calculation, and update the parameters of the generator; Continue training until the loss of the generator and the discriminator converges, and then terminate the training, thereby obtaining the trained generator and the discriminator.

5. A multi-spectral image fusion method based on a generative adversarial network, characterized in that, The method includes the following steps: A plurality of image pairs of the same position of the power equipment to be tested are obtained to form a test set, and each image pair includes a gray value image corresponding to an infrared light image and a visible light image; The image pairs in the test set are input into the multispectral image fusion model trained by using the method in any one of claims 1 to 4 to obtain a fused image.

Images