Multispectral image fusion network establishment method and fusion method based on self-attention mechanism

Abstract

The present invention discloses a method for establishing a fusion network for multispectral images based on a self-attention mechanism, comprising: establishing a training set comprising multiple groups of image pairs, each group of image pairs comprising an infrared image and a visible light image; inputting the image pairs in the training set into a generator employing a self-attention mechanism to generate a preliminary fused image, and calculating the generator's loss function value; inputting the image pairs, combined with the preliminary fused image, into two discriminators for discrimination processing, and calculating the combined loss function value of the two discriminators; and adjusting the weights of the generator weights to minimize the generator's loss function value corresponding to the image pairs in the training set and the combined loss function value of the discriminator. The present invention also provides a corresponding fusion method. Implementing the present invention can improve the fusion effect of visible light images and infrared images, and can retain more high-frequency and low-frequency details.

Description

Technical Field

[0001] The present invention relates to the field of smart grid information technology, and in particular to a multispectral image fusion network establishment method and a fusion method based on a self-attention mechanism. Background Art

[0002] With social progress and rapid economic development, people's work and daily lives are increasingly dependent on electricity. Providing safe, efficient, and stable power without compromising people's quality of life and work efficiency is a major challenge facing Chinese power workers. Condition-based maintenance primarily uses real-time monitoring to detect potential safety risks and implements power outages for equipment with potential safety risks.

[0003] A variety of electrical equipment faults occur in power grids, most of which are accompanied by overheating. Infrared detection technology categorizes these defects into external and internal defects. External faults primarily involve poor connections in connectors, fixtures, or contacts within power transmission lines, which increase contact resistance and lead to localized overheating. Internal faults primarily involve faults in closed circuits within insulator housings or devices, as well as deterioration or cracking of the insulating medium. These external and internal defects are primarily accompanied by temperature fluctuations and are difficult to detect with the naked eye. Therefore, infrared detection technology can be used to monitor the condition and potential hazards of electrical equipment through infrared image analysis of electrical equipment.

[0004] As we all know, infrared sensors can only capture target infrared radiation information, which reflects the temperature characteristics of equipment, and are insensitive to changes in scene brightness. Visible light images, on the other hand, are only sensitive to changes in scene brightness and offer high resolution. However, they cannot reflect changes in equipment temperature or identify any hidden hazards. Therefore, the fusion of infrared and visible light spectra is often used for fault detection in electrical equipment. The initial information from these two is complementary, reflecting both brightness details and temperature field changes. Therefore, the fusion of visible light and infrared images can yield more detailed and accurate information, pinpointing heat generation locations and accurately identifying hidden problems in electrical equipment. Visible light images offer rich detail and high resolution, but are less resistant to weather interference. Infrared images effectively avoid the weather interference introduced by visible light images. Fusion of the two images preserves all useful information, resulting in a richly informative fused image.

[0005] In traditional image fusion algorithms, image decomposition is the first step in image fusion. Image decomposition segments the source image into several images with distinct characteristics. These images are then synthesized based on image fusion rules to ensure that the resulting image not only exhibits distinct infrared features but also retains visible image details and texture. Among various image fusion methods, the most popular is the one based on multi-scale transformation. First, the source image is decomposed into a series of sub-images of varying scales. Next, a fusion rule based on multi-scale transformation is used to fuse the images of varying scales while preserving their scale characteristics. Finally, the fused image is obtained by inversely transforming the scaled images.

[0006] With the rise of machine learning, more and more excellent algorithms have begun to emerge, such as the CNN convolutional neural network, which has achieved good results in image processing and natural language processing; Resnet, which solves the gradient vanishing or gradient exploding phenomenon that occurs with the increase of network depth in deep learning; and GAN generative adversarial network, whose main feature is that it can generate the desired results without the need for real values, which is crucial in many tasks where real values ​​cannot be provided.

[0007] FusionGAN's infrared and visible light image fusion method: This method describes fusion as an adversarial game between preserving infrared thermal radiation information and maintaining visible appearance texture information. More specifically, it can be viewed as a minimax problem between the generator and the discriminator. This method first pre-trains a discriminator capable of determining basic truth. The image is then fed into the generator. The generator and discriminator then form an adversarial relationship, forming a two-player minimax game, in which the generator is trained to fool the discriminator, while the discriminator attempts to distinguish the generated samples. As the adversarial process progresses, the data generated by the generator becomes increasingly difficult to distinguish from real data.

[0008] CNN: Existing deep learning-based image fusion techniques typically rely on CNN models, which have a key premise that ground truth should be obtained in advance. For multi-focus image fusion and pan-sharpening problems, the ground truth is well defined, for example, a sharp image without blurred areas or a multispectral image with the same resolution as the corresponding panchromatic image. However, in the task of infrared and visible light image fusion, it is unrealistic to define a standard for the fused image, and therefore, establishing ground truth is not considered. Based on this, instead of learning an end-to-end model that requires a ground truth fusion image, existing infrared and visible light image fusion techniques learn a deep model to determine the degree of blur for each patch in the source image, and then calculate a weight map accordingly to generate the final fused image.

[0009] Traditional image fusion methods use an average value for both low-frequency and high-frequency components when selecting a fusion rule. This fails to preserve the full advantage of low-frequency information, resulting in a loss of contrast and a "washed" effect in the fused image. Existing deep learning-based image fusion techniques typically rely on CNN models, which have a critical prerequisite: obtaining ground truth values ​​in advance. However, in the task of fusion of infrared and visible light images, defining a fusion image standard is unrealistic, and therefore determining ground truth values ​​is not an option.

[0010] The generator model in FusionGan is too simple. Subjectively, the hierarchy is not clear enough. Objectively, the evaluation of various indicators is low, and it is impossible to achieve a satisfactory fusion effect. The training images used for training are large in size, and the existing FusionGan network parameters cannot meet the requirements, so the trained power fusion images are not clear enough. Summary of the Invention

[0011] The technical problem to be solved by the present invention is to provide a multispectral image fusion network establishment method and a fusion method based on the self-attention mechanism, which can improve the fusion effect of visible light images and infrared light images and retain more high-frequency and low-frequency details.

[0012] In order to solve the above technical problems, one aspect of the present invention provides a method for establishing a fusion network of multispectral images based on a self-attention mechanism, which comprises at least the following steps:

[0013] Step S10: establishing a training set, wherein the training set includes multiple groups of image pairs, each group of image pairs including infrared light images and visible light images taken at the same location;

[0014] Step S11: Input each image pair in the training set into a generator using a self-attention mechanism to generate a preliminary fused image and calculate the loss function value of the generator;

[0015] Step S12: inputting the image pair and the preliminary fusion image into two discriminators for discrimination processing, and calculating the comprehensive loss function value of the two discriminators;

[0016] Step S13, continuously adjust the weights of the generator weights so that the loss function value of the generator corresponding to the image in the training set and the comprehensive loss function value of the discriminator are both less than the corresponding threshold value, obtain the trained generator, and construct a multispectral image fusion network based on the trained generator.

[0017] Preferably, the step of calculating the loss function value of the generator in step S11 further includes:

[0018] The following formula is used to calculate the loss function value of the generator:

[0019] ,

[0020] ,

[0021]

[0022] in, is the loss function value of the generator; To combat loss value; is the content loss value; is a predetermined constant used to balance the content loss value and the adversarial loss value; is the number of fused images; are the height and width of the image respectively; represents the square of the Frobenius norm of the matrix; represents the gradient operator; To measure the fused image Preserved contrast information; To measure the fused image details retained; is a predetermined constant used to balance the above two terms.

[0023] Preferably, the step of calculating the comprehensive loss function value of the two discriminators for discrimination in step S12 further includes:

[0024] The following formula is used to calculate the loss function value of the generator:

[0025] ,

[0026] ,

[0027]

[0028] in, Represents the comprehensive loss function value when the two discriminators discriminate, and Represent the loss function values ​​of the two discriminators respectively, and is a predetermined constant used to balance the impact of the two discriminators on the fused image; Respectively 、 Tags, 、 、 、 Respectively and about 、 The judgment result of .

[0029] Preferably, the generator includes: a self-attention module, a local detail module and a merging module, wherein:

[0030] The self-attention module is used to receive the image pair, obtain the obvious features therein, and form a feature map;

[0031] The local detail module is used to receive the image pair and obtain detail information therein;

[0032] The merging module is used to merge the feature maps with obvious features and detail information to form a preliminary fused image.

[0033] Preferably, the self-attention module comprises 5 layers, the first two layers each comprising convolution, spectral normalization, LeakyReLU activation function, and maximum pooling layers, with the kernel and stride set to 2; after the first two layers, the height and width of the feature map are both one-eighth of the original input;

[0034] The third layer is the self-attention layer;

[0035] The fourth layer includes upsampling, convolution, spectral normalization, and LeakyReLU activation function layers; wherein the upsampling process is to upsample the output feature map using nearest neighbor interpolation;

[0036] The fifth layer contains convolution, spectral normalization, LeakyReLU activation function and upsampling processing layer, and outputs feature map.

[0037] Preferably, the local detail module comprises five layers, each layer comprising a convolution, a spectrum normalization and a Leaky ReLU activation function processing layer; in each layer, The filter generates 40 feature weight maps; all strides are set to 1, and densely connected blocks are applied to compensate for feature loss, and all previously calculated features are directly connected to subsequent layers to obtain detailed information at the end.

[0038] Preferably, the merging module comprises three layers, the first two layers are convolution, spectrum normalization and Leaky ReLU activation function processing layers, the third layer comprises convolution and tanh activation function processing layers, and outputs a preliminary fused image.

[0039] Preferably, the two discriminators have the same structure, each discriminator comprises five layers, the first four layers comprise convolution, spectral normalization, and LeakyReLU activation function processing layers; the fifth layer is a fully connected layer activated by a tanh function to generate a scalar;

[0040] The input images of one discriminator are visible light images and preliminary fused images, and the input images of the other discriminator are infrared light images and preliminary fused images.

[0041] Preferably, the step S13 further includes:

[0042] Use SGD optimizer to update the parameters in the discriminator , to minimize the comprehensive loss function value in the discriminator ;

[0043] Use Adam optimizer to update the parameters in the generator , to minimize the loss function value of the generator ;

[0044] Set the generator learning rate to 0.0006 with exponential decay, and set the discriminator learning rate to twice the generator learning rate;

[0045] Set the batch size to 19 and the epoch to 2. , .

[0046] Accordingly, another aspect of the present invention further provides a multispectral image fusion method based on a self-attention mechanism, which comprises the following steps:

[0047] Obtain multiple sets of image pairs of the same position of the power equipment to be tested to form a test set, where each set of image pairs includes an infrared image and a visible light image;

[0048] The image pairs in the test set are imported into the trained generator as described above in the form of concat to obtain a fused image.

[0049] The implementation of the embodiments of the present invention has the following beneficial effects:

[0050] This invention provides a method for establishing a multispectral image fusion network and a fusion method based on a self-attention mechanism. This method uses the self-attention mechanism to enhance the details of the fused image and introduces a dual discriminator to improve the quality of the fused image. The resulting fused image can improve the performance of indicator assessments. This improves the fusion of visible light and infrared images, preserving more high- and low-frequency details. It can be widely used to identify the type and faults of power equipment, improving recognition effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 A schematic diagram of the principle of a method for establishing a multispectral image fusion network based on a self-attention mechanism provided by the present invention;

[0053] Figure 2 A schematic diagram of a training network for a multispectral image fusion network according to the present invention;

[0054] Figure 3 A schematic structural diagram of a generator according to the present invention;

[0055] Figure 4 It is a structural diagram of a discriminator involved in the present invention;

[0056] Figure 5 Schematic diagram of the images before and after fusion involved in the present invention; wherein (a) is a visible light image; (b) is an infrared light image; (c) is the fusion result;

[0057] Figure 6 Schematic diagram comparing the fusion images formed by the method provided by the present invention and the existing methods; wherein, (a) is the test result of DenseFuse on the power dataset; (b) is the test result of FusionGAN on the power dataset; (c) is the test result of NestFuse on the power dataset; (d) is the test result of SDNet on the power dataset; (e) is the test result of U2Fusion on the power dataset; and (f) is the test result of this method on the power dataset. DETAILED DESCRIPTION

[0058] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0059] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, the accompanying drawings only show structures and / or processing steps closely related to the solutions according to the present invention, while other details that are not closely related to the present invention are omitted.

[0060] like Figure 1 As shown, a schematic diagram of the main process of an embodiment of a method for establishing a fusion network of multispectral images based on a self-attention mechanism provided by the present invention is shown. Figures 2 to 4 As shown, in this embodiment, the method includes at least the following steps:

[0061] Step S10: establishing a training set, wherein the training set includes multiple groups of image pairs, each group of image pairs including infrared light images and visible light images taken at the same location;

[0062] In step S11, each image pair in the training set is input into a generator using a self-attention mechanism to generate a preliminary fused image and calculate the loss function value of the generator; specifically, Figure 2 As shown in the figure, the pre-trained images (visible light and infrared light image pairs) are fed into the generator using the concat (direct accumulation on the channel) method to generate a preliminary fused image.

[0063] In the present invention, the structure of the generator is as follows Figure 3 As shown, it includes: self-attention module, local detail module and merging module, where:

[0064] The self-attention module is used to receive the image pair, obtain the obvious features therein, and form a feature map. In one example, the self-attention module includes 5 layers, the first two layers each include convolution, spectral normalization, LeakyReLU activation function, and maximum pooling processing layer, and the kernel and stride are set to 2. In order to accelerate training and avoid gradient explosion / vanishing, spectral normalization (sn) is added after the convolution layer. Introducing spectral normalization in the training of the generator can prevent the increase of parameter amplitude and avoid abnormal gradients, which is beneficial to the adjustment of the generator in GAN performance. After the first two layers, the height and width of the feature map are both one-eighth of the original input.

[0065] The third layer is the self-attention layer; the fourth layer includes upsampling, convolution, spectral normalization, and LeakyReLU activation function layers; wherein, the upsampling process is to upsample the output feature map using nearest neighbor interpolation; the fifth layer includes convolution, spectral normalization, LeakyReLU activation function and upsampling processing layer, and outputs the feature map.

[0066] More specifically, the details in the visible infrared image are mainly concentrated on the circuit breakers, while the details in the visible light image are mainly concentrated in the wires in the background. In this case, feature extraction and fusion should be accompanied by attention-driven. Therefore, it is necessary to associate the different locations between the fusion result and the original image. Due to the limited kernel size, it is a difficult task for ordinary convolutional layers and requires deeper layers. Convolution obtains two feature spaces and These two feature spaces form a query and a set of key-value pairs. Processed with softmax and transpose The matrix multiplication result of β is obtained. Similarly, x is obtained by another Convolution (i.e. ), and multiplied by β to generate the self-attention feature map o. Using a learning parameter γ, the output feature map can be expressed as The attention map β is obtained by flattening and size Multiply, and flatten and transpose and Therefore, the obtained attention map has a larger size If the values ​​of H and W are about a few hundred, the size of the attention map β will be ,and For multiple batches, this will pose a challenge in terms of storage space and computing power.

[0067] Furthermore, during the training phase, images in the training dataset are cropped into multiple image patches and fed into the generator. The height and width of these patches are much smaller than the original infrared or visible light images. During the testing phase, if the test source images are cropped into patches in the same manner and then stitched together to obtain a fused image, the introduction of the self-attention layer will lead to significant differences in brightness between patches. This will severely impact the visual quality of the fused image. Considering both the training and testing processes, as well as performance and computational cost, a max pooling layer and a longer stride in the convolutional layers are used before the self-attention layer. To compensate for the loss of detail caused by these operations, another module consisting of only a few simple layers is added. This module is called the local detail module.

[0068] The local detail module is used to receive the image pair and obtain the detail information therein; in one example, the local detail module includes five layers, each layer includes a convolution, a spectrum normalization and a Leaky ReLU activation function processing layer; in each layer, no matter how many channels the layer inputs, the layer always passes The filter generates 40 feature weight maps; all strides are set to 1. Furthermore, to reduce gradient vanishing and reuse previously computed features, densely connected blocks are applied in the local detail module to compensate for feature loss. All previously computed features are directly connected to subsequent layers to obtain detailed information. Spectral normalization is also applied in layers 2 through 4 to prevent parameter amplitude increases and avoid abnormal gradients.

[0069] The merging module is used to merge the feature maps with distinct features and detail information to form a preliminary fused image. In one example, the merging module includes three layers, the first two of which are convolution, spectral normalization, and LeakyReLU activation function processing layers, and the third layer includes convolution and tanh activation function processing layers, and outputs the preliminary fused image.

[0070] Understandably, the generator network employs a self-attention architecture. The self-attention module extracts features by introducing attention-driven and long-range dependencies, considering weights from all feature locations rather than a limited receptive field. However, because the max pooling layer in this module reduces the scale of the feature map and causes distortion, we add a local detail module to preserve some details that might otherwise be lost. The merging module combines the features obtained by the self-attention block and the local detail module to generate the final fused image.

[0071] More specifically, the step of calculating the loss function value of the generator in step S11 further includes:

[0072] The following formula is used to calculate the loss function value of the generator:

[0073] ,

[0074] ,

[0075]

[0076] in, is the loss function value of the generator; To combat loss value; is the content loss value; is a predetermined constant used to balance the content loss value and the adversarial loss value; is the number of fused images; are the height and width of the image respectively; represents the square of the Frobenius norm of the matrix; represents the gradient operator; To measure the fused image Preserved contrast information; To measure the fused image details retained; is a predetermined constant used to balance the above two terms.

[0077] Step S12: inputting the image pair and the preliminary fusion image into two discriminators for discrimination processing, and calculating the comprehensive loss function value of the two discriminators;

[0078] like Figure 4As shown in the figure, the two discriminators have the same structure. Each discriminator consists of five layers. The first four layers include convolution, spectral normalization, and LeakyReLU activation function processing layers. The fifth layer is a fully connected layer and is activated by the tanh function to generate a scalar.

[0079] The input images of one discriminator are visible light images and preliminary fused images, and the input images of the other discriminator are infrared light images and preliminary fused images.

[0080] More specifically, the step of calculating the comprehensive loss function value of the two discriminators for discrimination in step S12 further includes:

[0081] The following formula is used to calculate the loss function value of the generator:

[0082] ,

[0083] ,

[0084]

[0085] in, Represents the comprehensive loss function value when the two discriminators discriminate, and Represent the loss function values ​​of the two discriminators respectively, and is a predetermined constant used to balance the impact of the two discriminators on the fused image; Respectively 、 Tags, 、 、 、 Respectively and about 、 The judgment result of .

[0086] Step S13, continuously adjust the weights of the generator weights so that the loss function value of the generator corresponding to the image in the training set and the comprehensive loss function value of the discriminator are both less than the corresponding threshold value, obtain the trained generator, and construct a multispectral image fusion network based on the trained generator.

[0087] It's understandable that changing the generator's weights makes the generated images more capable of "deceiving" the discriminator. This means continuously reducing the generator's loss to minimize both the content loss and the adversarial loss. Reducing the content loss allows it to retain more contrast and detail information, while reducing the adversarial loss allows the generated images to increasingly meet the discriminator's requirements.

[0088] Continuously reduce the loss value of the discriminator, that is, make the discriminator's judgment result closer to the label value (predetermined threshold), until the image generated by the generator can "perfectly" deceive the discriminator.

[0089] In a specific example, the step S13 further includes:

[0090] Use SGD optimizer to update the parameters in the discriminator , to minimize the comprehensive loss function value in the discriminator ;

[0091] Use Adam optimizer to update the parameters in the generator , to minimize the loss function value of the generator ;

[0092] For the generator and discriminator, since TTUR has been proven to be effective, different learning rates can be implemented. The generator learning rate is set to 0.0006 with exponential decay, and the discriminator learning rate is set to twice the generator learning rate;

[0093] Set the batch size to 19 and the epoch to 2. , .

[0094] It is understood that another aspect of the present invention further provides a multispectral image fusion method based on a self-attention mechanism, which includes the following steps:

[0095] Obtain multiple sets of image pairs of the same position of the power equipment to be tested to form a test set, where each set of image pairs includes an infrared image and a visible light image;

[0096] Import the image pairs in the test set into the form of concat Figures 1 to 4 The trained generator generated by the described method obtains the fused image.

[0097] like Figure 6As shown, a schematic diagram comparing the fused images formed by the method provided by the present invention and existing methods is shown; almost all methods can effectively achieve the fusion of infrared light and visible light, but the main part of FusionGAN is slightly blurred, and the image of SDNet is darker. The method of the present invention has higher background brightness than other algorithms and better contrast than other methods. This is mainly due to the addition of a local detail layer in the generator and the use of two discriminators, which increases the influence of visible light on the fused image. On the circuit breaker in the figure, the patented method is slightly better than the other five methods, while the background, especially the wires, is more blurred. This is because the attention mechanism is used, which makes the details of the main part obvious and puts more "attention" on the circuit breaker.

[0098] To further verify the feasibility and effectiveness of this invention, 21 pairs of infrared and visible light images were selected for fusion testing and quantitatively evaluated and compared with five other methods. Quantitative evaluation objectively assesses fusion performance through statistical metrics. This example selected six widely used evaluation metrics in the field of image fusion, including information entropy (EN), spatial frequency (SF), mutual information (MI), average gradient (AG), and visual fidelity (VIF). EN measures the information content of the fused image; a higher EN value indicates more information in the fused image. SF measures the overall detail richness of the fused image; a higher SF indicates richer texture in the fused image. MI measures the amount of information from the source images contained in the fused image; a higher MI indicates more information from the source images in the fused image. VIF is a new criterion based on the natural scene statistics model (NSS), image distortion, and human visual distortion modeling. A higher AG value indicates better image quality. AG characterizes image clarity; a higher calculated value indicates a clearer fused image.

[0099] Table 1 below gives the objective evaluation indicators of the experimental results of 33 pairs of infrared and visible light images selected under different fusion methods.

[0100] Table 1 Evaluation indicators of various fusion methods

[0101]

[0102] The bold and underlined data represent the optimal and suboptimal values ​​of the evaluation indicators, respectively. As can be seen from the data in Table 1, the infrared and visible light image fusion method based on multimodal features given in this embodiment achieves the best scores in the EN, MI, SF, and AG indicators, but is slightly inferior in VIF. These results show that the method of the present invention transfers the most information from the source image to the fused image during the fusion process and can better preserve edges. The fused image has the highest contrast, contains the most information, retains more of the global structure and edge features of the source image, and also has better visual effects.

[0103] The implementation of the embodiments of the present invention has the following beneficial effects:

[0104] This invention provides a method for establishing a multispectral image fusion network and a fusion method based on a self-attention mechanism. This method uses the self-attention mechanism to enhance the details of the fused image and introduces a dual discriminator to improve the quality of the fused image. The resulting fused image can improve the performance of indicator assessments. This improves the fusion of visible light and infrared images, preserving more high- and low-frequency details. It can be widely used to identify the type and faults of power equipment, improving recognition effectiveness.

[0105] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatus, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0106] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0107] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of the claims of the present invention. Therefore, any other equivalent changes or modifications that do not depart from the spirit disclosed by the present invention should be included in the scope of the claims of the present invention.

Claims

1. A method for establishing a fusion network for multispectral images based on a self-attention mechanism, characterized in that: At least the following steps are included: Step S10: establishing a training set, wherein the training set includes multiple groups of image pairs, each group of image pairs including infrared light images and visible light images taken at the same location; Step S11: Input each image pair in the training set into a generator using a self-attention mechanism to generate a preliminary fused image and calculate the loss function value of the generator; Step S12: inputting the image pair and the preliminary fusion image into two discriminators for discrimination processing, and calculating the comprehensive loss function value of the two discriminators; Step S13, continuously adjust the weights of the generator weights so that the loss function value of the generator corresponding to the image in the training set and the comprehensive loss function value of the discriminator are both less than the corresponding threshold value, obtain the trained generator, and construct a multispectral image fusion network based on the trained generator.

2. The method according to claim 1, wherein The step of calculating the loss function value of the generator in step S11 further includes: The following formula is used to calculate the loss function value of the generator: ， ， in, is the loss function value of the generator; To combat loss value; is the content loss value; is a predetermined constant used to balance the content loss value and the adversarial loss value; is the number of fused images; are the height and width of the image respectively; represents the square of the Frobenius norm of the matrix; represents the gradient operator; To measure the fused image Preserved contrast information; To measure the fused image details retained; is a predetermined constant, is the nth fused image, is the reference image, For the discriminator about The judgment result of Tags, To fuse the image The gradient, I v is a visible light image, For visible light images I v gradient.

3. The method according to claim 2, wherein The step of calculating the comprehensive loss function value of the two discriminators in step S12 further includes: The following formula is used to calculate the loss function value of the generator: ， ， in, Represents the comprehensive loss function value when the two discriminators discriminate, and Represent the loss function values ​​of the two discriminators respectively, and is a predetermined constant used to balance the impact of the two discriminators on the fused image; Respectively 、 Tags, 、 、 、 Respectively and about 、 The judgment result of .

4. The method according to any one of claims 1 to 3, wherein The generator includes: a self-attention module, a local detail module and a merging module, wherein: The self-attention module is used to receive the image pair, obtain the obvious features therein, and form a feature map; The local detail module is used to receive the image pair and obtain detail information therein; The merging module is used to merge feature maps with obvious features and detail information to form a preliminary fused image.

5. The method according to claim 4, wherein in: The self-attention module consists of 5 layers. The first two layers each contain convolution, spectral normalization, LeakyReLU activation function, and max pooling layers, with kernel and stride set to 2. After the first two layers, the height and width of the feature map are both one-eighth of the original input. The third layer is the self-attention layer; The fourth layer includes upsampling, convolution, spectral normalization, and LeakyReLU activation function layers; wherein the upsampling process is to upsample the output feature map using nearest neighbor interpolation; The fifth layer contains convolution, spectral normalization, LeakyReLU activation function and upsampling processing layer, and outputs feature map.

6. The method according to claim 5, wherein in: The local detail module consists of five layers, each of which includes convolution, spectrum normalization and Leaky ReLU activation function processing layers; in each layer, The filter generates 40 feature weight maps; All strides are set to 1 and densely connected blocks are applied to compensate for feature loss. All previously computed features are directly connected to subsequent layers to obtain detailed information at the end.

7. The method according to claim 6, wherein in: The merging module includes three layers, the first two layers are convolution, spectrum normalization and Leaky ReLU activation function processing layers, the third layer includes convolution and tanh activation function processing layers, and outputs a preliminary fused image.

8. The method according to claim 7, wherein in, The two discriminators have the same structure. Each discriminator consists of five layers. The first four layers include convolution, spectral normalization, and LeakyReLU activation function processing layers. The fifth layer is a fully connected layer and is activated by the tanh function to generate a scalar. The input images of one discriminator are visible light images and preliminary fused images, and the input images of the other discriminator are infrared light images and preliminary fused images.

9. The method according to claim 8, wherein The step S13 further comprises: Use SGD optimizer to update the parameters in the discriminator , to minimize the comprehensive loss function value in the discriminator ; Use Adam optimizer to update the parameters in the generator , to minimize the loss function value of the generator ; Set the generator learning rate to 0.0006 with exponential decay, and set the discriminator learning rate to twice the generator learning rate; Set the batch size to 19 and the training epoch to 2. , .

10. A multispectral image fusion method based on self-attention mechanism, characterized in that: The steps include: Obtain multiple sets of image pairs of the same position of the power equipment to be tested to form a test set, where each set of image pairs includes an infrared image and a visible light image; The image pairs in the test set are imported into the generator trained as claimed in any one of claims 1 to 9 in a concat form to obtain a fused image.

Images