Dual-light image fusion method for light power equipment

Abstract

The invention relates to a dual-light image fusion method for light power equipment, which corrects and enhances an image by adjusting color distortion and detail errors, and restrains whether the enhanced image is properly exposed or not through closed-loop brightness optimization so as to obtain a visible light image with balanced brightness, rich details and natural colors. The encoder adopts depth and multi-scale feature extraction, and a two-stage attention mechanism is introduced in a fusion stage, so that the retention capability of heat source features in an infrared image and detail features in a visible light image is enhanced, and the influence on subsequent substation detection tasks is reduced. The decoder uses depth separable convolution to reduce the complexity of the model, so that the size and parameters of the model are remarkably reduced, and the method can be operated efficiently and is suitable for deployment of mobile equipment in a substation inspection scene. Finally, a high-quality fusion image is generated through fusion, and meanwhile, the efficiency and precision of substation equipment image fusion are improved.

Description

TECHNICAL FIELD

[0001] The application relates to a light power equipment dual-light image fusion method. BACKGROUND

[0002] The substation is the core of the power system, and its safe and stable operation is directly related to the operation of the power grid, so equipment inspection is crucial. With the popularization of unattended substations, traditional manual inspection has been difficult to meet the real-time monitoring demand, and intelligent automatic inspection has become the key. Infrared sensors play an important role in fault detection, but have low resolution and lack detailed information, while visible light sensors provide rich texture information. Infrared and visible light image fusion can improve detection robustness and is widely used in semantic segmentation, target detection and tracking.

[0003] In recent years, there are existing technologies that propose various fusion methods, mainly divided into traditional methods and deep learning methods. Traditional methods such as multi-scale transformation, sparse representation and subspace fusion have certain effects, but are insufficient in robustness, generalization ability and detail preservation. Fusion methods based on deep learning, some based on CNN, solve the problem of illumination degradation in low light environments by disentangling scene brightness; some based on AE, alleviate information loss and edge blur problems through double encoders; some based on GAN, DANT-GAN solves the problem of detail loss. Although these methods have certain effects, they still face problems such as generated images being too dark or high model complexity and large computational overhead in low light environments, which cannot meet the needs of resource-constrained scenarios such as substation inspection. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a light power equipment dual-light image fusion method, which is based on closed-loop brightness optimization and can improve image clarity and information integrity.

[0005] The technical solution adopted by the present application is:

[0006] A light power equipment dual-light image fusion method, comprising the following steps:

[0007] S01: enhancing the input low-light visible light image through a closed-loop brightness optimization module;

[0008] S02: processing the enhanced visible light image and infrared image through an encoder, the encoder comprising a deep feature extraction module using deep separable convolution for feature extraction and a multi-scale global feature extraction module using a multi-layer branch structure to extract multi-scale features;

[0009] S03: fusing features through a two-stage attention mechanism, including an entropy attention mechanism and a spatial attention mechanism;

[0010] S04: The final fused image is obtained by reconstructing the fused feature through a decoder adopting deep separable convolution;

[0011] S05: The network is trained through a joint loss function, including image enhancement loss, feedback loss and texture loss.

[0012] Further, in S01, the specific method includes:

[0013] The input image is decomposed into an N-layer Laplacian pyramid;

[0014] The layers of the pyramid are optimized layer by layer through the residual layer;

[0015] The spatial information is compressed to extract the illumination feature, and the low-light and high-light probability distribution is calculated to form the light intensity feedback.

[0016] Further, the closed-loop brightness optimization module in S01 further includes an EFAN network, which includes four U-Net automatic encoders, wherein the feature extraction part includes three layers of convolution, each layer including convolution and Leaky ReLU activation function, the reconstruction part includes three layers of convolution, the first two layers adopt Leaky ReLU activation function, and the last layer adopts Tanh activation function, the feedback part extracts brightness information through two layers of convolution, integrates the features through average pooling, and calculates the brightness probability distribution through a fully connected layer and adopts Mish activation function.

[0017] Further, in S02, the deep feature extraction module adopts three layers of deep separable convolution, and the parameters are set as follows:

[0018] The first layer: output channel number 16, convolution kernel size 1x1, activation function Leaky ReLU;

[0019] The second layer: output channel number 32, convolution kernel size 3x3, activation function Leaky ReLU;

[0020] The third layer: output channel number 64, convolution kernel size 3x3, activation function Leaky ReLU.

[0021] Further, in S02, the multi-scale global feature extraction module includes one straight connection layer, three dilated convolution layers and one pooling layer, and the features are fused through 1x1 convolution, wherein the dilated rate of the dilated convolution is set to [1, 2, 3].

[0022] Further, in S03,

[0023] Fused feature is defined as follows:

[0024] (1)

[0025] wherein and respectively represent the spatial attention mechanism and the entropy attention mechanism;

[0026] The entropy attention mechanism optimizes feature selection and weight distribution by measuring the information richness or uncertainty of the features, and its calculation formula is:

[0027] (2)

[0028] (3)

[0029] (4)

[0030] (5)

[0031] wherein, represents the 1-norm, represents the local entropy, represents the multi-modal feature , ), = , = obtained by the encoder, R represents the calculated neighborhood radius, m is the multi-scale, (x, y) represents the coordinates on the space, and the entropy value is for the coordinates of the pixel, represents the normalized probability of the pixel in the i-th row and the j-th column, and the logarithmic base is the default value 2;

[0032] The local entropy is calculated by a circular sliding window with a radius of R, and the entropy value is used as the feature of the corresponding pixel to generate a feature map and extract the local structure information of the image. Finally, the entropy attention feature map is multiplied by the input feature map to enhance the feature expression.

[0033] The calculation formula of the spatial attention mechanism is as follows:

[0034] (6)

[0035] wherein, represents the 1-norm, ,..., and = 2, represents the weight map calculated by the 1-norm and the soft-max operator;

[0036] The generated weight and are used to generate the depth feature The calculation formula is as follows:

[0037] (7)

[0038] The last spatial attention fusion image is added to obtain:

[0039] (8)

[0040] Further, the decoder of S04 includes:

[0041] Five layers of 3 3 kernel depth separable convolution layer, using Leaky ReLU as activation function; and

[0042] One 1 1 convolution layer, using the Tanh function as the activation function to compress the output features to the range of (-1, 1).

[0043] Further, the joint loss function of S05 is calculated as:

[0044] (12)

[0045] Wherein, is the joint loss function, is the image enhancement loss, is is the feedback loss, is, and is used to balance the loss;

[0046] The image enhancement loss calculation formula is:

[0047] (13)

[0048] Wherein and are the width and height of the two times of the l-th level in the Laplacian pyramid of the training image, respectively, and p is the index of each pixel in the enhanced image Y(l) at the l-th level and the normal image T(l) at the same level of the present application;

[0049] (14)

[0050] (15)

[0051] H and W represent the height and width of the training image, wherein p is the pixel index of the normal image T and the enhanced image Y, S is the sigmoid function, and D is the DNN trained by the main network;

[0052] The feedback loss calculation formula is:

[0053] (16)

[0054] (17)

[0055] where h denotes the illumination mark of the input image, = denotes the predicted probability, ranging from 0 to 1, for providing the exposure feedback of the fused image, is the cross-entropy loss, is a hyperparameter, is 0, and the maximum is 10, denotes the 1-norm;

[0056] The texture loss calculation formula is:

[0057] (18)

[0058] (19)

[0059] (20)

[0060] (21)

[0061] In the above formula, , and are hyperparameters for balancing each loss function, is the gradient loss, denotes the soble gradient, is the structural similarity loss, is the texture loss, and W and H respectively denote the width and height of the image.

[0062] The positive effects of the present application are:

[0063] The present application first designs a closed-loop brightness optimization module to adaptively adjust the visible light image brightness. Secondly, the encoder innovatively extracts depth and multi-scale global features, fully mining key information. Subsequently, a two-stage attention mechanism is designed in the fusion part to enhance the clarity and information integrity of the fused image and highlight the significant target. Finally, depth separable convolution is used instead of standard convolution for feature reconstruction, making the model lightweight and suitable for deployment on edge devices such as inspection robots. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 is the network structure diagram of the present application;

[0065] Figure 2 is the closed-loop brightness optimization module diagram of the present application;

[0066] Figure 3 This is the overall structure diagram of the encoder of the present invention;

[0067] Figure 4 This is a multi-layer branch structure diagram of the present invention;

[0068] Figure 5 This is the ESA two-stage attention fusion strategy diagram of the present invention;

[0069] Figure 6 This is a comparison diagram of the convolution process of the present invention;

[0070] Figure 7 This is a diagram of the encoder network structure of the present invention;

[0071] Figure 8 This is an objective evaluation diagram of power equipment in a substation according to the present invention;

[0072] Figure 9 This is the objective evaluation diagram of the LLVIP and TNO datasets of the present invention;

[0073] Figure 10 This is a quantitative comparison result diagram of different algorithms of the present invention on the substation dataset;

[0074] Figure 11 This is a qualitative comparison result diagram of the ablation experiment of the present invention. DETAILED DESCRIPTION

[0075] The present invention discloses a dual-light image fusion method for lightweight power equipment based on closed-loop brightness optimization. The method is based on an automatic encoder and enhances visibility in low-light environments while ensuring lightweight, which is helpful for the implementation of all-weather substation inspection tasks. Figure 1 As shown in the figure, the model consists of four parts: image pre-enhancement, feature extraction, feature fusion, and feature reconstruction. and For low-light visible and infrared images, represents the enhanced visible light image, The resulting fused image is shown in Figure 1. By adjusting the brightness of the visible light image, extracting high- and low-frequency information and fusing them in the entropy and spatial dimensions, image clarity and information integrity are improved. Finally, lightweight convolution is used for feature reconstruction, optimizing deployment on mobile devices. Table 1 shows a preliminary comparison with other algorithms in terms of model size and parameter count, visually demonstrating the advantages of this method.

[0076]

[0077] Each step of the present invention is explained in detail below.

[0078] Closed-loop brightness optimization module

[0079] In low light conditions, the texture details of power equipment are difficult to identify, affecting feature extraction and power equipment fault point positioning. In the existing method, color distortion and detail loss are easily caused when fusing the luminance channel in the YCbCr color space. To solve this problem, the invention designs a closed-loop brightness optimization module, which decomposes the image into an N-layer Laplacian pyramid, optimizes layer by layer, extracts global color at the bottom layer, restores details at the high-frequency layer, and realizes fine correction through the residual layer. Finally, the spatial information is compressed to extract the lighting features, calculate the low light and high light probability, form the light intensity feedback, and ensure reasonable brightness adjustment.

[0080] As shown in Figure 2 The EFAN part is composed of four U-Net autoencoders, including feature extraction and reconstruction networks. The feature extraction part is composed of three convolutional layers, each containing convolution and Leaky ReLU activation function; the reconstruction part is also composed of three convolutional layers, except that the last layer uses Tanh, the rest use Leaky ReLU. The feedback part extracts brightness information through two convolutional layers, integrates brightness features through average pooling, and calculates brightness probability distribution through two fully connected layers, uses Mish activation function to enhance expression ability and optimize gradient liquidity.

[0081] Encoder

[0082] The encoder extracts infrared and enhanced visible light images First, deep feature extraction is performed to obtain high-level abstract features, then multi-scale feature extraction is used to capture image information at different scales, and dense connection is used to alleviate the information flow sparsity problem of deep network. The network structure is as shown in Figure 3 .

[0083] Deep feature extraction is performed through depth separable convolution, which improves the feature extraction ability while maintaining the computing efficiency, and the specific parameters are as shown in Table 2.

[0084]

[0085] Due to the difficulty of convolution operation to capture global features, and the dependence on local pixels that easily leads to noise amplification, affecting the fusion quality. The invention designs a multi-layer branch structure, uses different dilated convolution to extract features layer by layer, captures global information in a multi-scale manner, and improves the fusion effect, as shown in Figure 4 .

[0086] The features extracted by the deep feature extraction are taken as the input of the multi-scale global feature extraction module, key information is extracted through 1*1 convolution for dimension reduction and dimension increase, and then rich non-linear features are obtained through dilated convolution. The dilated convolution expands the receptive field without increasing the parameters and computational complexity, captures local context information, reduces information loss, and enhances the model representation ability. The structure of the application includes five layers: one layer is directly connected, three layers combine standard convolution and dilated convolution, and the last layer is a pooling operation. The feature fusion is optimized from channel splicing to 1*1 convolution fusion, improving the information integration efficiency.

[0087] Two-stage attention fusion

[0088] The fusion strategy is a key link of image fusion. Traditional methods such as direct splicing or weighted sum are difficult to capture deep associations of different modalities in autoencoder networks, which easily leads to loss of details. Spatial attention is widely used in fusion tasks, but it is limited to local information. To make up for the above shortcomings, the present application proposes to fuse infrared and visible light images in two stages of attention (ESA), enhance infrared heat source features and visible light detail retention, and highlight significant targets, as shown in the structure of Figure 5

[0089] Fusion features are defined as follows:

[0090] (1)

[0091] wherein and represent spatial attention mechanism and entropy attention mechanism respectively

[0092] 1) Entropy attention model

[0093] Entropy attention optimizes feature selection and weight distribution by measuring the information richness or uncertainty of features. This mechanism can exploit the complementary information of infrared and visible light, enhance the attention of key areas, highlight significant targets, and improve the clarity and detail retention ability of the fused image. Its weight is determined by the entropy calculation formula:

[0094] (2)

[0095] (3)

[0096] (4)

[0097] (5)

[0098] wherein, represents the 1-norm, represents the local entropy, represents the multi-modal feature​ , , =, , =, is obtained by the encoder, R represents the calculated neighborhood radius, and m is the multi-scale. (x, y) represents the coordinates on the space, indicating that the entropy value is for the coordinates of the pixel, represents the normalized probability of the pixel point in the i-th row and the j-th column, and the logarithm base is the default value 2.

[0099] The local entropy is calculated by a circular sliding window with a radius of R, and the entropy value is used as the feature of the corresponding pixel to generate a feature map and extract the local structure information of the image. Finally, the entropy attention feature map is multiplied by the input feature map to enhance the feature expression.

[0100] 2) Spatial attention model

[0101] Spatial attention can be used to focus on the salient regions of the image to improve the expression ability of key features and suppress redundant information. The application can help locate the salient region when applied to the substation fusion image. The specific formula is as follows:

[0102] (6)

[0103] wherein, represents the 1-norm, , and = 2, represents the weight map calculated by the 1-norm and the soft-max operator.

[0104] The generated weight is weighted with to generate a deep feature The calculation formula is as follows:

[0105] (7)

[0106] Finally, the spatial attention fusion image is obtained by adding these deep features:

[0107] (8)

[0108] In the substation infrared and visible light fusion, the infrared image provides device heat source information, and the visible light image presents clear texture details. The spatial attention locates the salient region, and combines with the entropy attention to capture the high-entropy region, highlights the salient region, realizes the feature modeling from global to local and from saliency to information complexity, and improves the fusion quality.

[0109] Decoder

[0110] In use 3 The calculation cost of traditional convolution is about 9 times of depthwise separable convolution with 3 Figure 6 The traditional convolution simultaneously performs spatial filtering and channel fusion, while the depthwise separable convolution decomposes the traditional convolution into depthwise convolution for processing spatial filtering of each channel and pointwise convolution for channel fusion through 1x1 convolution.

[0111] Suppose the size of the input feature map is , and the size of the convolution kernel is If the size of the output feature map is , then the corresponding number of convolution kernels is needed The parameter amount of traditional convolution is:

[0112] (9)

[0113] For depthwise separable convolution, the total parameter amount is:

[0114] (10)

[0115] The calculation cost ratio of depthwise separable convolution to traditional convolution is:

[0116] (11)

[0117] It can be seen from the above formula that the calculation cost of depthwise separable convolution is much lower than that of traditional convolution.

[0118] The feature reconstruction network is shown in Figure 7 It is composed of five 3 3 kernel depthwise separable convolution layers and one 1 1 convolution layer, which significantly reduces the calculation cost of the model. Leaky ReLU is used as the activation function of the 3 3 convolution layer to gradually extract and fuse the details and overall information of infrared and visible light, and the 1 1 convolution layer adopts Tanh as the activation function to compress the output features to the range of (-1, 1), balancing the dynamic range of infrared and visible light.

[0119] The loss function

[0120] The present application designs a loss function combined with feedback loss to train the network together to ensure reasonable iteration of the network in the optimization process, and the specific formula is:

[0121] (12)

[0122] wherein and for balancing loss

[0123] The image enhancement network loss is guided by the pyramid loss, reconstruction and adversarial loss, so that each subnetwork can follow the reconstruction process of the Laplacian pyramid to balance the reality and fidelity of the content of the image.

[0124] (13)

[0125] wherein and are the width and height of the two times of the l-th level in the Laplacian pyramid of the training image, respectively, and p is the index of each pixel in our enhanced image at the l-th level Y(l) and the normal image at the same level T(l).

[0126] (14)

[0127] (15)

[0128] H and W represent the height and width of the training image, wherein p is the pixel index of the normal image T and the enhanced image Y, S is a sigmoid function, and D is a DNN trained by the main network.

[0129] The application adopts a cross-entropy loss matrix to constrain a binary classification task, effectively guides the network optimization process by measuring the difference between the probability distribution predicted by the model and the real distribution, and the formula is:

[0130] (16)

[0131] (17)

[0132] wherein h represents the illumination label of the input image, = represents the predicted probability, ranging from 0 to 1, and is used to provide exposure feedback of the fused image, is the cross-entropy loss, is a hyperparameter, is 0, and the maximum is 10, represents the 1-norm.

[0133] In order to retain more texture details of the source image and improve the visual effect, a texture loss function is adopted. The structural similarity loss is used to comprehensively measure the brightness, contrast and structural information for global constraint, and the gradient loss is introduced to enhance the edge retention, so as to ensure that the outline of the fused image is clear and the fusion effect is improved.

[0134] (18)

[0135] (19)

[0136] (20)

[0137] (21)

[0138] in the above formula , and are hyperparameters for balancing each loss function, is the gradient loss, represents the soble gradient, is the structural similarity loss. is the texture loss, and W and H represent the width and height of the image, respectively.

[0139] The method of the present application is experimentally verified below, including evaluation indicators, qualitative and quantitative comparison with six advanced fusion algorithms on the substation and LLVIP data sets, and verification of the generalization ability on the TNO data set. Ablation experiments are also conducted to prove the effectiveness of the modules and settings, and the computational efficiency is further verified by comparing FLOPs, model parameters and size.

[0140] Evaluation indicators and configurations

[0141] The present application adopts qualitative and quantitative evaluation of image quality. The quantitative indicators include: average gradient (AG) to measure the degree of change in pixel value, standard deviation (SD) to evaluate the dispersion of pixels, information entropy (EN) to measure the richness of information, visual fidelity (VIF) to measure the similarity between the fused image and the reference image, and spatial frequency (SF) to represent the spatial variation frequency.

[0142] 50 daytime and nighttime 1024x1280 images are selected from the LLVIP data set and cropped to 64x64 for training to learn the internal relationship of the image and extract features. The substation data set contains 30 pairs of infrared and visible light images collected specifically for power equipment, taken by an inspection robot equipped with a Jetson Xavier NX. The Adam optimizer is used to adaptively adjust the network parameters, and the experimental environment is NVIDIA GTX 1050Ti, Intel i7-8750H, 32GB memory. The network hyperparameters are shown in Table 3.

[0143]

[0144] Comparative experiment

[0145] The proposed fusion algorithm was applied to infrared and visible light images of substation power equipment. Six different methods were compared on the substation dataset and the LLVIP dataset. These methods included: FusionGAN, a GAN-based method; NestFuse, an autoencoder-based method; and two CNN-based methods, PIAFusion and U2 Fusion, as well as DIVFusion and LiMFusion. Generalization experiments were conducted on the TNO dataset to verify the proposed method's performance.

[0146] Subjective evaluation

[0147] A set of representative images were selected from the substation dataset to conduct quantitative comparisons of various methods. Figure 8 , where the last row is the image processed by the method of the present invention. In the qualitative analysis, the red frame marks the texture detail area, and the green frame marks the thermal radiation target area.

[0148] Depend on Figure 8 As can be seen, the fused images generated by FusionGAN and NestFuse are darker, with poor clarity and texture detail. U2 Fusion, DIVFusion, and LiMFusion can preserve cloud texture to a certain extent in overcast, low-light environments, but all suffer from edge information loss and artifacts. In contrast, the method of the present invention generates bright scenes in low-light environments, clearly depicting the outlines of power equipment, preserving cloud detail while effectively highlighting the thermal characteristics of electrical equipment.

[0149] On the LLVIP and TNO datasets, we zoomed in on some local areas of the image to improve the visualization of key details and enhance the visual contrast. The results are as follows: Figure 9 As shown, the bottom row in the figure shows the effect after being processed by the method of the present invention.

[0150] The visualization results of each algorithm on the LLVIP dataset are shown in Figure 9 The results for the TNO dataset are shown in the first four columns. FusionGAN suffers from artifacts and severe contrast loss, performing poorly in low-light environments. NestFuse and U2 Fusion exhibit blurry areas and uneven edge transitions. DIVFusion, PIAFusion, and LiMFusion exhibit issues with brightness balance, with some dark areas lost, impacting overall visual quality. Existing algorithms fail to effectively preserve details in visible light images, such as the ground, utility poles, and fonts. This results in a loss of brightness and texture in the fused image, reflecting the imbalance in the fusion of infrared and visible light information.

[0151] In contrast, the last row shows that the method of the present application maintains balanced brightness and contrast under various lighting conditions, reduces artifacts, and makes the fused image more natural, which is further verified by the visualization results of the TNO dataset.

[0152] Objective evaluation

[0153] The fusion results were quantitatively analyzed by objective evaluation indicators, and the results are shown in Table 3 and Table 4. Figure 10 Figure 10 Quantitative comparison of different algorithms on the substation dataset is shown, which verifies the superiority of the algorithm of the present application. The excellent performance of the algorithm in average gradient, information entropy and spatial frequency shows that it can effectively enhance the image details and texture, especially in low light environment. Higher standard deviation and visual fidelity show that the fused image has good contrast and retains the perception of the original image by the human eye to the greatest extent, which is of great significance especially in the subsequent tasks of power equipment or substation equipment.

[0154]

[0155] As can be seen from Table 4, in the LLVIP dataset, the EN of the algorithm of the present application reaches the optimal value, and the evaluation index values of the relatively suboptimal algorithm are improved by 1.74%, 12.01%, 3.07% and 5.29% respectively. In the TNO dataset, although the EN index is slightly lower than that of other algorithms, the VIF is improved by 1.61%, the SD is improved by 7.64%, the SF is improved by 2.81%, and the AG is improved by 2.44%.

[0156] In summary, through comparison on two datasets, the algorithm of the present application shows excellent performance. The improvement of each index shows that the generated image has improved in details, edge definition and contrast, while effectively retaining key information and maintaining high visual consistency with the original image.

[0157] Ablation experiment

[0158] Four groups of ablation experiments were conducted to verify the effectiveness of the used modules. In experiment 1, the image enhancement module removes the pyramid and reconstruction adversarial loss and feedback loss; in experiment 2, the multi-scale feature extraction module removes the multi-layer branch structure; in experiment 3, the fusion module removes the two-stage attention mechanism, and directly splices the output features on the channel as the input of the decoder; and in experiment 4, it is the complete fusion network. The thermal radiation target is marked by a red box, and the texture detail background part is marked by a green box, and the results are shown in Table 5. Figure 11

[0159] ​​In order to verify the effectiveness of the lightweight technology adopted in the method of the present application, ablation comparison is carried out by using traditional convolution and deep separable convolution respectively, as shown in Figure 6, the influence of both on the model parameter quantity is analyzed to verify the balance ability of the method of the present application between the calculation efficiency and the performance.

[0160] Through observation, it is found that in experiment 1, the brightness and contrast of the fusion image are poor due to no image enhancement; in experiment 2, part of the image information is lost and the clarity of the fusion image is reduced due to the failure to perform multi-scale extraction; in experiment 3, the contrast of the image is reduced and the expression of texture details is reduced, and the significant target cannot be highlighted. In contrast, the image generated by the complete fusion network has good brightness and clarity, the target is highlighted, the texture information and edge information are rich, and the fusion effect is excellent.

[0161] In addition, Table 5 is the objective evaluation index result of the ablation experiment image, the bold font is the optimal result in several experiments, and the underlined result is the sub-optimal. It is obvious that the removal of any module will cause the fluctuation of the evaluation index to a certain extent. Although the complete model performs best in the key indicators such as SD, SF and AG, after removing part of the module, individual indicators have increased. In experiment one, visible light is not enhanced, which is closer to the original image, resulting in unexpected increase of VIF. In experiment three, since the information is not filtered, all the information is superimposed together, resulting in more pixels in the image, irrelevant details, and increase of EN. It can be seen that each module of the model has a synergistic complementary effect on the overall performance, and the complete structure helps to achieve a better balance in multiple indicators.

[0162]

[0163] AEROFusion-Conv is used to replace the deep separable convolution of the method of the present application with traditional convolution. In summary, it can be seen from the graph that after adopting the edge deployment technology, the size parameter quantity and other parameters of the model are significantly reduced, and the size parameter quantity of the model accounts for only 20.89% of that using traditional convolution.

[0164] The application adjusts and enhances the image by adjusting color distortion and detail error, and at the same time, through closed-loop brightness optimization, whether the enhanced image is properly exposed is constrained to obtain a visible light image with balanced brightness, rich details and natural colors, thereby alleviating the information degradation of power equipment in low light environment. The encoder adopts deep and multi-scale feature extraction, and a two-stage attention mechanism is introduced in the fusion stage to enhance the heat source feature in the infrared image and the detail feature in the visible light image, thereby reducing the influence on the subsequent substation detection task. The decoder uses deep separable convolution to reduce the model complexity, thereby significantly reducing the model size and parameters, so that the application can be efficiently run and applied to mobile device deployment in the substation inspection scene. Finally, a high-quality fusion image is generated through fusion, and at the same time, the efficiency and accuracy of substation equipment image fusion are improved. Through qualitative and quantitative evaluation and generalization experiment of the substation dataset, the superiority of the method of the application is verified, and in the future, the post-fusion fault detection task will be further studied.

[0165] The above examples are only used to illustrate the technical solutions of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A dual-light image fusion method for light power equipment, characterized in that It includes the following steps: S01: Enhance the input low-light visible light image through the closed-loop brightness optimization module; S02: processing the enhanced visible light image and infrared image respectively through an encoder, wherein the encoder includes a deep feature extraction module that uses depthwise separable convolution to extract features and a multi-scale global feature extraction module that uses a multi-layer branch structure to extract multi-scale features; S03: Feature fusion via a two-stage attention mechanism, including entropy attention mechanism and spatial attention mechanism; S04: Reconstruct the fused features by using a decoder with depthwise separable convolution to obtain the final fused image; S05: Train the network through a joint loss function, including image enhancement loss, feedback loss, and texture loss.

2. A dual-light image fusion method for light power equipment according to claim 1, characterized in that In S01, the specific method includes: Decompose the input image into an N-layer Laplacian pyramid; Optimize each layer of the pyramid layer by layer through the residual layer; Compress spatial information to extract lighting features, calculate the probability distribution of low light and high light to form light intensity feedback.

3. A dual-light image fusion method for light power equipment according to claim 2, characterized in that The closed-loop brightness optimization module in S01 also includes a FEAN network, which contains four U-Net autoencoders. The feature extraction part includes three layers of convolution, each layer contains convolution and Leaky ReLU activation function. The reconstruction part includes three layers of convolution, the first two layers use Leaky ReLU activation function, and the last layer uses Tanh activation function. The feedback part extracts brightness information through two layers of convolution, average pooling integrates features, and the fully connected layer calculates the brightness probability distribution and uses Mish activation function.

4. A dual-light image fusion method for light power equipment according to claim 1, characterized in that In S02, the deep feature extraction module uses three layers of depth-wise separable convolution, and the parameters are set as follows: First layer: 16 output channels, 1×1 convolution kernel, and Leaky ReLU activation function. Second layer: 32 output channels, 3×3 convolution kernel size, and Leaky ReLU activation function. The third layer has 64 output channels, a convolution kernel size of 3×3, and an activation function called Leaky ReLU.

5. The dual-light image fusion method for light power equipment according to claim 1 is characterized in that In S02, the multi-scale global feature extraction module consists of a direct connection layer, three dilated convolution layers, and a pooling layer. Feature fusion is performed through 1×1 convolution, where the dilation rate of the dilated convolution is set to [1, 2, 3].

6. A dual-light image fusion method for light power equipment according to claim 1, characterized in that In S03, Fusion Features The definition is as follows: (1) in and Represent the spatial attention mechanism and entropy attention mechanism respectively; The entropy attention mechanism optimizes feature selection and weight distribution by measuring the information richness or uncertainty of the features. Its calculation formula is: (2) (3) (4) (5) in, represents the 1 norm, represents the local entropy, Represents multimodal features ( , ), = , = Obtained by the encoder, R represents the calculated neighborhood radius, m is multi-scale, (x, y) represents the coordinates in space, which means that the entropy value is for the coordinates of the pixel point. Represents the normalized probability of the pixel at row i and column j, with the default value of logarithm base 2; The local entropy is calculated through a circular sliding window with a radius of R, and the entropy value is used as the feature of the corresponding pixel to generate a feature map to extract the local structural information of the image. Finally, the entropy attention feature map is multiplied with the input feature map to enhance the feature expression; The calculation formula of the spatial attention mechanism is as follows: (6) in, represents the 1-norm, ,..., and =2, Represents the weight graph calculated by 1-norm and soft-max operator; The weights generated by and , weighted generation of deep features The calculation formula is as follows: (7) Final spatial attention fusion image It is obtained by adding these deep features: (8)。 7. The method for dual-light image fusion of light power equipment according to claim 1 is characterized in that The S04 decoder includes: Five-layer 3 A 3-kernel depthwise separable convolutional layer using leaky ReLU as the activation function; and One 1 1 convolution layer, using the hyperbolic tangent function Tanh as the activation function to compress the output features to the range of (-1,1).

8. The method for dual-light image fusion of light power equipment according to claim 1 is characterized in that The calculation formula of the joint loss function of S05 is: （12） in, is the joint loss function, is the image enhancement loss, To combat losses, To compensate for losses, To rebuild the losses, and Used to balance losses; The image enhancement loss calculation formula is: （13） in and are respectively twice the width and height of the lth level in the Laplacian pyramid of the training image, and p is the index of each pixel in the enhanced image of the present invention at the lth level Y(l) and the normal image at the same level T(l); （14） （15） H and W represent the height and width of the training image, where p is the pixel index of the normal image T and the enhanced image Y, S is the sigmoid function, and D is the DNN trained by the main network; The feedback loss calculation formula is: （16） （17） where h represents the lighting signature of the input image, = Represents the prediction probability, ranging from 0 to 1, used to provide exposure feedback for the fused image. is the cross entropy loss, is a hyperparameter, When it is 0, the maximum value is 10. represents the 1-norm; The texture loss calculation formula is: （18） （19） （20） （21） In the above formula 、 and is a hyperparameter used to balance the loss functions. is the gradient loss, represents the soble gradient, is the structural similarity loss, is the texture loss, W and H represent the width and height of the image respectively.

Images