Infrared and visible light image fusion security identification method based on target perception

Abstract

The invention relates to a safe identification method for infrared and visible light image fusion based on target perception, and belongs to the technical field of computer image processing. The method comprises the following steps: acquiring an infrared light image and a visible light image of an original image, and preprocessing the images; a feature extraction network, a feature correction and fusion network and a dual-task reconstruction network are constructed and trained, and modal consistency loss, modal complementary information loss, target detection loss and the like are introduced; and finally, based on a target detection result graph and a fusion image output by the dual-task reconstruction network, carrying out identification and early warning on security risks in an electric power operation scene. According to the method, the problem of error accumulation caused by step-by-step processing of unregistered image fusion and target detection in a traditional method is solved, and the accuracy and robustness of power operation scene safety monitoring are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of computer image processing technology and relates to a safety identification method based on the fusion of infrared and visible light images with target perception. Specifically, it relates to a safety risk identification method for power operation scenarios based on the fusion of infrared and visible light images with target perception. Background Technology

[0002] In power industry operations, safety risk monitoring is a crucial aspect of ensuring production safety. Infrared sensors can detect abnormal temperatures and potential faults in equipment, but they lack detail in describing texture details. Visible light sensors can clearly depict the appearance of equipment and details of personnel operations, but they are significantly affected by lighting conditions. Traditional methods typically perform image registration, fusion, and target detection in steps, leading to error accumulation and making it difficult to meet the high-precision requirements of power operation scenarios. Furthermore, existing methods often assume that the input image is already registered, but in actual power operation scenarios, infrared and visible light images often have misregistration issues due to differences in viewing angle or equipment.

[0003] Image information acquired by a single image sensor often cannot provide a complete description of a scene. Integrating information from infrared and visible light images into a more informative fused image will enhance the image's ability to represent the scene. Therefore, infrared and visible light image fusion, as a low-level processing task upstream of downstream advanced vision tasks, has been widely applied in various fields, such as object detection, tracking, pedestrian re-identification, and semantic segmentation.

[0004] Over the past few decades, numerous image fusion techniques have been proposed, including traditional methods and deep learning-based approaches. Traditional methods are generally categorized into those based on multi-scale transformations and those based on sparse representations. Deep learning-based methods typically include those based on autoencoders (AEs), convolutional neural networks (CNNs), and generative adversarial networks (GANs). Although recent deep learning-based image fusion algorithms have produced satisfactory fused images, several challenging problems remain to be solved in the field of image fusion. On the one hand, existing infrared and visible light image fusion algorithms tend to pursue better visual quality and superior evaluation metrics, but rarely systematically consider whether the fusion result benefits the performance of downstream tasks. On the other hand, even recent infrared and visible light image fusion methods driven by advanced vision tasks, such as TarDAL's two-layer optimization model that uses a target-aware dual adversarial learning network to simultaneously solve the problems of infrared and visible light image fusion and target detection; SeAFusion using semantic loss to constrain the fused image result, making it contain richer semantic information; and DetFusion utilizing target-related information learned in a target detection network to guide multimodal image fusion. However, these methods all assume that the input source images to be fused are perfectly aligned in spatial location, which does not match the application scenarios in real-world scenarios.

[0005] Therefore, overcoming the shortcomings of existing technologies is an urgent problem to be solved in the field of computer image processing technology. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a security identification method based on the fusion of infrared and visible light images for target perception.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A security identification method based on target perception and fusion of infrared and visible light images includes the following steps: Step 1: Obtain the infrared and visible light images of the original image and preprocess the images; Step 2: Construct a feature extraction network; then use the feature extraction network to obtain the unique and common features of the original image; The feature extraction network includes an infrared-specific feature extraction module, a visible light-specific feature extraction module, an infrared common feature extraction module, and a visible light common feature extraction module; The infrared unique feature extraction module is used to extract unique features from infrared images; The visible light-specific feature extraction module is used to extract the specific features of visible light images; The infrared common feature extraction module is used to extract common features from infrared images; The visible light common feature extraction module is used to extract common features of visible light images; Step 3: Construct a feature correction and fusion network; then use the feature correction and fusion network to predict the deformation field and perform spatial registration of the common and unique features of the infrared image, and then fuse it with the common and unique features of the visible light image. The feature correction and fusion network includes a registration network, a registration loss module, and a connection module; The registration network is used to register the infrared common features with deformation and the infrared specific features with deformation, so as to obtain the registered infrared common features and the registered infrared specific features. The registration loss module is used to calculate the loss after registration; The connection module is used to connect the registered infrared-specific features, the registered infrared common features, the visible light common features, and the visible light-specific features according to channels to obtain fused features; Step 4: Construct a dual-task reconstruction network; then, based on the fusion features, use the dual-task reconstruction network to obtain the fused image and its target detection results. The dual-task reconstruction network includes an object detection task head, an image restoration task head, an object detection loss module, and a fusion loss module; The object detection task head is used to perform object detection based on the fusion features obtained in Step 3, thereby obtaining the object detection result map; The image restoration task head is used to restore the image based on the fusion features obtained in Step 3, thereby obtaining the fused image; The target detection loss module is used to calculate the target detection loss; The fusion loss module is used to obtain the fusion loss; Step 5: Real-time acquisition of infrared and visible light images in the power operation scenario, preprocessing according to the method in Step 1, and then sequentially passing through a feature extraction network, a feature correction and fusion network, and a dual-task reconstruction network to obtain a fused image and its target detection result map; finally, safety risk identification and early warning are performed based on the fused image and its target detection result map.

[0008] Furthermore, in Step 1, the preprocessing involves randomly cropping the infrared and visible light images to the same size, then performing a normalization operation, followed by data augmentation.

[0009] Furthermore, the infrared and visible light images are randomly cropped into sizes of [size missing]. image.

[0010] Furthermore, data augmentation methods include horizontal flipping and contrast enhancement.

[0011] Furthermore, the specific method for Step 2 is as follows: Suppose that each set of original images in the training dataset used to train the feature extraction network contains three images: pixel-by-pixel strictly registered infrared images. Visible light images And infrared images with deformation ; infrared images And infrared images with deformation The corresponding infrared common features are then input into the infrared common feature extraction module to obtain the infrared common features. Common infrared features with deformation ; infrared images And infrared images with deformation The corresponding infrared-specific features are obtained by inputting them into the infrared-specific feature extraction module. Infrared features with deformation ; Visible light images Input the visible light common feature extraction module to obtain the visible light common features. ; Visible light images Input the visible light-specific feature extraction module to obtain visible light-specific features. ; Calculate modal consistency loss and modal complementary information loss :

[0012]

[0013] in, H The height of the feature maps corresponding to the infrared common features and visible light common features. W The width of the corresponding feature map. C This represents the number of channels in the corresponding feature map. express l 1 Norm; The infrared-specific feature extraction module consists of a 64-channel 7×7 convolutional layer and two 128-channel residual blocks connected in sequence. The visible light-specific feature extraction module consists of a sequentially connected 64-channel 5×5 convolutional layer and a densely connected block; The infrared common feature extraction module includes an encoder, which is composed of four 3×3 convolutional layers connected in sequence. The visible light common feature extraction module includes an encoder, which is composed of four 3×3 convolutional layers connected in sequence.

[0014] Furthermore, the specific method for Step 3 is as follows: The registration network consists of sequentially connected channel and spatial augmentation blocks and multi-scale registration blocks; The channel and spatial enhancement block consists of six sequentially connected feature extraction layers and a global average pooling layer; each feature extraction layer is composed of a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function connected in sequence; the number of channels in the convolutional layers of the six feature extraction layers are 64, 128, 256, 512, 1024, and 2048 respectively, following the principle of gradual expansion; The multi-scale registration block consists of three 3×3 convolutional layers and one 1×1 convolutional layer. The multi-scale registration block adopts a three-layer structure design. First, multi-scale features are extracted in parallel through convolutions with different dilation rates: the first layer uses a 3×3 convolutional layer with dilation rate 1, the second layer uses a 3×3 convolutional layer with dilation rate 2, and the third layer uses a 3×3 convolutional layer with dilation rate 3. Then, the outputs of the three convolutional layers are connected to the input of the 1×1 convolutional layer. In the multi-scale registration block, the features obtained from the output of three convolutional layers are concatenated, and finally the number of channels is adjusted from 1024 to 64 by a 1×1 convolutional layer before output.

[0015] Furthermore, there are two channel and space augmentation blocks; the parameters of the two channel and space augmentation blocks are not shared. Infrared common features with deformation The input is fed into the first channel and the spatial enhancement block to obtain the enhanced infrared common features. ; Common features of visible light The input is fed into the second channel and the spatial enhancement block to obtain the enhanced visible light common features. ; Will and The data is stitched together along the channel dimension, and the results are input into a multi-scale registration block to predict the deformation field. ; The registration loss module is used to calculate the registration loss. :

[0016] in, H For the height of the feature map, W The width of the feature map, For deformation field Tags; express l 1Norm; use Common infrared features with deformation and infrared-specific features with deformation Registration is performed to obtain the registered infrared common features. and infrared-specific characteristics The registration calculation formula is as follows:

[0017] in, Indicates a Warp operation; Registered infrared common features Infrared characteristics after registration Common characteristics of visible light and the unique characteristics of visible light By concatenating along the channel dimension, the fused features are obtained. :

[0018] in, This indicates a splicing operation along the channel dimension.

[0019] Furthermore, the infrared common features with deformation will be... The input is fed into the first channel and the spatial enhancement block to obtain the enhanced infrared common features. The specific method is as follows: Infrared common features with deformation The input is fed into the first channel and the spatial augmentation block, and then passed through the first three feature extraction layers to obtain the spatial augmentation weights. ; Will and Perform element-wise multiplication:

[0020] in, This indicates the enhanced features at the spatial level. This represents the element-wise multiplication operation; Will As input to the fourth feature extraction layer, the features are sequentially passed through the following three feature extraction layers and global average pooling to obtain features used for channel-level enhancement. ; Will and Element-wise multiplication yields features enhanced at both the spatial and channel levels. : .

[0021] Furthermore, the common features of visible light The input is fed into the second channel and the spatial enhancement block to obtain the enhanced visible light common features. The specific method is as follows: Common features of visible light The input is fed into the second channel and the spatial enhancement block, and then passed through the first three feature extraction layers to obtain the spatial enhancement weights. ; Will and Perform element-wise multiplication:

[0022] in, This indicates the enhanced features at the spatial level. This represents the element-wise multiplication operation; Will As input to the fourth feature extraction layer, the features are sequentially passed through the following three feature extraction layers and global average pooling to obtain features used for channel-level enhancement. ; Will and Element-wise multiplication yields features enhanced at both the spatial and channel levels. : .

[0023] Furthermore, the specific method for Step 4 is as follows: YOLOv5 was used as the header for the object detection task; The image restoration head consists of three sequentially connected feature extraction layers. The first two feature extraction layers are each composed of a 3×3 convolutional layer, a batch normalization layer, and a Leaky ReLU activation function connected in sequence. The last feature extraction layer is composed of a 3×3 convolutional layer, a batch normalization layer, and a Tanh activation function connected in sequence. The fusion features obtained after Step 3 Input into the object detection task header to obtain the object detection result image. ; The fusion features obtained after Step 3 Input into the image reconstruction task head to obtain the fused image. ;

[0024] Calculate target detection loss Brightness loss gradient loss and fusion loss The calculation formula is as follows:

[0025] in, This is the loss function used during YOLOv5 training, which includes the sum of three parts: localization loss, classification loss, and confidence loss. Represents the label value of the detected object;

[0026] in, Indicates the equilibrium hyperparameters;

[0027]

[0028] Among them, its H For the height of the feature map, W The width of the feature map, Represents the Sobel operator; Infrared image; Visible light image; To merge images, express l 1 Norm; Finally, calculate the total loss. The feature extraction network, feature correction and fusion network, and dual-task reconstruction network are trained with the goal of minimizing the total loss. The calculation formula is as follows:

[0029] in:

[0030]

[0031] in, , all represent the equilibrium hyperparameters; For modal consistency loss, For modal complementary information loss, C The number of channels in the feature map. These are unique characteristics of infrared radiation. These are common characteristics of visible light. These are unique characteristics of infrared radiation. This is a characteristic unique to visible light.

[0032] In Step 1 of this invention, the Z-score standardization method is preferably used for the normalization operation.

[0033] In Step 2 of this invention, the infrared-specific feature extraction module uses a 7×7 convolutional layer and two 128-channel residual blocks. First, shallow features are extracted through a 64-channel 7×7 convolution, and then deep features are extracted through two 128-channel residual blocks. The infrared-specific feature extraction module is used to extract thermal radiation features in infrared images, such as abnormal heating areas of equipment. The visible light-specific feature extraction module adopts a structure of 5×5 convolutional layers and densely connected blocks, which preserves detailed features through 64-channel 5×5 convolution; the visible light-specific feature extraction module is used to extract texture details in visible light images, such as security signs and personnel equipment. The infrared common feature extraction module adopts a four-layer 3×3 convolution encoder structure; the infrared common feature extraction module is used to extract structural features in infrared images that are shared with visible light, such as device outlines and scene layout.

[0034] The visible light shared feature extraction module uses a four-layer 3×3 convolutional encoder that is symmetrical to the infrared shared feature module to extract structural features shared with the infrared image in the visible light image, such as the working area and background environment.

[0035] In Step 3 of this invention, the channel and spatial enhancement blocks adopt a cascaded structure design, consisting of six feature extraction layers and a global average pooling layer connected sequentially. The six feature extraction layers adopt the same structural design, each consisting of a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function connected sequentially. The first three convolutional layers keep the number of channels constant for spatial feature enhancement, while the last three convolutional layers gradually increase the number of channels to enhance the channel dimension representation capability. The number of channels in the six layers are 64, 128, 256, 512, 1024, and 2048, respectively. The six feature extraction layers are connected in a serial manner, with the output of the previous layer serving as the input of the next layer for progressive feature transformation. The output of the last feature extraction layer is connected to the global average pooling layer, which aggregates global contextual information by reducing the spatial dimension to form the final enhanced feature representation. The multi-scale registration block adopts a three-layer structure design. First, multi-scale features are extracted in parallel through convolutions with different dilation rates: the first layer uses a 3×3 convolutional layer with dilation rate 1 to capture detailed features, the second layer uses a 3×3 convolutional layer with dilation rate 2 to obtain medium-range features, and the third layer uses a 3×3 convolutional layer with dilation rate 3 to extract global features. Then, the features from the three layers are concatenated, and the number of channels is adjusted by a 1×1 convolutional layer before output. This design significantly reduces computational complexity while ensuring registration accuracy, making it particularly suitable for scenarios requiring real-time processing, such as power equipment monitoring. In this invention, For deformation field The labels, i.e., the reference deformation fields obtained from the training set, are usually generated by existing image registration methods; relevant implementations can be found at https: / / paperswithcode.com / dataset / eth-laser-registration-datasets.

[0036] In Step 4 of this invention, the fusion features obtained in Step 3 are input into the target detection task head to obtain the target detection result image; the fusion features obtained in Step 3 are input into the image reconstruction task head to obtain the fused image.

[0037] The detection objectives of this invention include the comprehensive identification and localization of multiple key targets. For example, the detection targets include construction workers themselves, their safety helmets, and those not wearing helmets. By identifying and classifying each target in the image, structured detection results can be output, including target category (e.g., "personnel," "helmet," "person not wearing a helmet"), location bounding box (used to select the target area in the image), and confidence score (indicating the reliability of the identification). This information can not only be used for real-time alarms of on-site violations but also provides a solid data foundation for subsequent construction safety supervision and data analysis. Furthermore, it should be noted that the target detection content of this invention goes far beyond "whether a helmet is worn"; it encompasses a comprehensive perception and assessment of the safety status of personnel throughout the entire construction site.

[0038] In this invention, the YOLOv5 training loss function consists of three parts: localization loss, classification loss, and confidence loss. The implementation can be found at https: / / github.com / ultralytics / yolov5. The labels used for training are provided by the dataset and contain information such as the target category and bounding box location, used to supervise model learning.

[0039] In Step 5 of this invention, the infrared and visible light images in the power operation scenario can be derived from pictures or videos in the power operation scenario, and this invention does not impose any restrictions on this.

[0040] In Step 5 of this invention, the infrared and visible light images in the power operation scenario are preprocessed according to the method in Step 1, and then sequentially passed through the feature extraction network, the feature correction and fusion network, and the dual-task reconstruction network to obtain a clear image (i.e., the fused image obtained by the image restoration task head through image restoration) and obtain the target detection result (e.g., the target detection task head detects situations such as not wearing a safety helmet or standing in a dangerous area). In this invention, This represents the balancing hyperparameter, which is randomly selected in the grid through a grid search. , where , represent the balancing hyperparameters, which are also randomly selected in the grid through grid search.

[0041] This invention integrates multimodal features from infrared and visible light images, along with prior knowledge of power operation scenarios (such as safety distance thresholds, abnormal equipment temperature thresholds, and protective equipment detection rules). Based on the target detection result image and the fused image, it identifies typical safety risks in real time, such as operator violations (e.g., not wearing insulated gloves), equipment overheating hazards, and foreign object intrusion into dangerous areas. Ultimately, it can output risk levels and warning signals, achieving intelligent safety monitoring of power operation scenarios.

[0042] This invention proposes a security recognition method based on target perception and fusion of infrared and visible light images. The key to this method is achieving superior performance simultaneously in both unregistered image fusion tasks and advanced vision tasks. Specifically, the method involves first connecting a target detection task head after the image fusion stage, and constructing a target detection loss function using its output detection results. Then, the detection loss is propagated back to the front-end fusion network via a backpropagation algorithm, thereby constraining the fusion process to retain key feature information for the target detection task. A specific total loss function is designed to address the multimodal image registration problem. This function effectively solves the problem of feature misalignment between modes caused by different imaging mechanisms by minimizing the distribution difference between infrared and visible light features in the latent space.

[0043] This invention is the first to unify unregistered image fusion and object detection tasks into a single framework, breaking down the barriers to real-world object detection applications and applying it to safety risk identification in power operation scenarios. The proposed method demonstrates superior performance in image alignment, fusion, and object detection across multiple datasets.

[0044] Compared with the prior art, the beneficial effects of this invention are as follows: 1. This invention is the first to integrate unregistered image fusion and object detection into a unified framework, solving the error accumulation problem caused by the step-by-step processing in traditional methods. This framework can not only process unregistered infrared and visible light images, but also preserve the key features of the target during the fusion process, thereby significantly improving the applicability and robustness of object detection in complex real-world scenarios.

[0045] 2. To address the domain difference issue between infrared and visible light images, a modality consistency loss function is proposed. This loss function effectively reduces feature offset between modalities by constraining the consistency of cross-modal feature distribution in the semantic space. Furthermore, combining this with a multi-scale feature alignment strategy further improves the accuracy of image registration.

[0046] 3. The method of this invention demonstrates superior performance on multiple publicly available datasets. In image fusion tasks, it can preserve more detailed information and target features; in target detection tasks, it can achieve higher detection accuracy in complex backgrounds and harsh environments. This multi-task collaborative optimization design significantly improves the overall system performance. Attached Figure Description

[0047] Figure 1 This is a general framework diagram of the feature extraction network, feature correction and fusion network, and dual-task reconstruction network of the present invention; Figure 2 This is a structural diagram of the registration network of the present invention; Figure 3 This is a qualitative comparison chart of the registration + fusion + target detection method and the fusion results of the method of this invention; the top and bottom rows are images of two scenes respectively; the first and second columns are the misaligned original input images, the first column is the infrared image and the second column is the visible light image; columns 3 to 7 are the results obtained by different fusion methods; Figure 4 This is a qualitative comparison chart of the fusion results of the joint registration and fusion + target detection method and the method of this invention; the top and bottom rows are images of two scenes respectively; the first and second columns are the misaligned original input images, the first column is the infrared image and the second column is the visible light image; columns 3 to 7 are the results obtained by different fusion methods; Figure 5 This is a graph showing the results of the ablation experiment. Detailed Implementation

[0048] The present invention will now be described in further detail with reference to the embodiments.

[0049] Those skilled in the art will understand that the following embodiments are for illustrative purposes only and should not be construed as limiting the scope of the invention. Where specific techniques or conditions are not specified in the embodiments, they are performed in accordance with the techniques or conditions described in the literature in the field or according to the product instructions. Materials or equipment whose manufacturers are not specified are all conventional products that can be obtained by purchase.

[0050] Example 1 A security identification method based on target perception and fusion of infrared and visible light images includes the following steps: Step 1: Obtain the infrared and visible light images of the original image and preprocess the images; Step 2: Construct a feature extraction network; then use the feature extraction network to obtain the unique and common features of the original image; The feature extraction network includes an infrared-specific feature extraction module, a visible light-specific feature extraction module, an infrared common feature extraction module, and a visible light common feature extraction module; The infrared unique feature extraction module is used to extract unique features from infrared images; The visible light-specific feature extraction module is used to extract the specific features of visible light images; The infrared common feature extraction module is used to extract common features from infrared images; The visible light common feature extraction module is used to extract common features of visible light images; Step 3: Construct a feature correction and fusion network; then use the feature correction and fusion network to predict the deformation field and perform spatial registration of the common and unique features of the infrared image, and then fuse it with the common and unique features of the visible light image. The feature correction and fusion network includes a registration network, a registration loss module, and a connection module; The registration network is used to register the infrared common features with deformation and the infrared specific features with deformation, so as to obtain the registered infrared common features and the registered infrared specific features. The registration loss module is used to calculate the loss after registration; The connection module is used to connect the registered infrared-specific features, the registered infrared common features, the visible light common features, and the visible light-specific features according to channels to obtain fused features; Step 4: Construct a dual-task reconstruction network; then, based on the fusion features, use the dual-task reconstruction network to obtain the fused image and its target detection results. The dual-task reconstruction network includes an object detection task head, an image restoration task head, an object detection loss module, and a fusion loss module; The object detection task head is used to perform object detection based on the fusion features obtained in Step 3, thereby obtaining the object detection result map; The image restoration task head is used to restore the image based on the fusion features obtained in Step 3, thereby obtaining the fused image; The target detection loss module is used to calculate the target detection loss; The fusion loss module is used to obtain the fusion loss; Step 5: Real-time acquisition of infrared and visible light images in the power operation scenario, preprocessing according to the method in Step 1, and then sequentially passing through a feature extraction network, a feature correction and fusion network, and a dual-task reconstruction network to obtain a fused image and its target detection result map; finally, safety risk identification and early warning are performed based on the fused image and its target detection result map.

[0051] The overall framework diagram of the feature extraction network, feature correction and fusion network, and dual-task reconstruction network is as follows: Figure 1 As shown.

[0052] Example 2 like Figures 1-2 As shown, the security identification method based on target perception and fusion of infrared and visible light images includes the following steps: Step 1: Obtain the infrared and visible light images of the original image and preprocess the images; Step 2: Construct a feature extraction network; then use the feature extraction network to obtain the unique and common features of the original image; The feature extraction network includes an infrared-specific feature extraction module, a visible light-specific feature extraction module, an infrared common feature extraction module, and a visible light common feature extraction module; The infrared unique feature extraction module is used to extract unique features from infrared images; The visible light-specific feature extraction module is used to extract the specific features of visible light images; The infrared common feature extraction module is used to extract common features from infrared images; The visible light common feature extraction module is used to extract common features of visible light images; Step 3: Construct a feature correction and fusion network; then use the feature correction and fusion network to predict the deformation field and perform spatial registration of the common and unique features of the infrared image, and then fuse it with the common and unique features of the visible light image. The feature correction and fusion network includes a registration network, a registration loss module, and a connection module; The registration network is used to register the infrared common features with deformation and the infrared specific features with deformation, so as to obtain the registered infrared common features and the registered infrared specific features. The registration loss module is used to calculate the loss after registration; The connection module is used to connect the registered infrared-specific features, the registered infrared common features, the visible light common features, and the visible light-specific features according to channels to obtain fused features; Step 4: Construct a dual-task reconstruction network; then, based on the fusion features, use the dual-task reconstruction network to obtain the fused image and its target detection results. The dual-task reconstruction network includes an object detection task head, an image restoration task head, an object detection loss module, and a fusion loss module; The object detection task head is used to perform object detection based on the fusion features obtained in Step 3, thereby obtaining the object detection result map; The image restoration task head is used to restore the image based on the fusion features obtained in Step 3, thereby obtaining the fused image; The target detection loss module is used to calculate the target detection loss; The fusion loss module is used to obtain the fusion loss; Step 5: Real-time acquisition of infrared and visible light images in the power operation scenario, preprocessing according to the method in Step 1, and then sequentially passing through a feature extraction network, a feature correction and fusion network, and a dual-task reconstruction network to obtain a fused image and its target detection result map; finally, safety risk identification and early warning are performed based on the fused image and its target detection result map.

[0053] The overall framework diagram of the feature extraction network, feature correction and fusion network, and dual-task reconstruction network is as follows: Figure 1 As shown.

[0054] In Step 1, the preprocessing involves randomly cropping the infrared and visible light images to the same size, then performing a normalization operation, followed by data augmentation.

[0055] The infrared and visible light images were randomly cropped to a size of [size missing]. image.

[0056] Data augmentation methods include horizontal flipping and contrast enhancement.

[0057] The specific method for Step 2 is as follows: Suppose that each set of original images in the training dataset used to train the feature extraction network contains three images: pixel-by-pixel strictly registered infrared images. Visible light images And infrared images with deformation ; infrared images And infrared images with deformation The corresponding infrared common features are then input into the infrared common feature extraction module to obtain the infrared common features. Common infrared features with deformation ; infrared images And infrared images with deformation The corresponding infrared-specific features are obtained by inputting them into the infrared-specific feature extraction module. Infrared features with deformation ; Visible light images Input the visible light common feature extraction module to obtain the visible light common features. ; Visible light images Input the visible light-specific feature extraction module to obtain visible light-specific features. ; Calculate modal consistency loss and modal complementary information loss :

[0058]

[0059] in, H The height of the feature maps corresponding to the infrared common features and visible light common features. W The width of the corresponding feature map. C This represents the number of channels in the corresponding feature map. express l 1 Norm; The infrared-specific feature extraction module consists of a 64-channel 7×7 convolutional layer and two 128-channel residual blocks connected in sequence. The visible light-specific feature extraction module consists of a sequentially connected 64-channel 5×5 convolutional layer and a densely connected block; The infrared common feature extraction module includes an encoder, which is composed of four 3×3 convolutional layers connected in sequence. The visible light common feature extraction module includes an encoder, which is composed of four 3×3 convolutional layers connected in sequence.

[0060] The specific method for Step 3 is as follows: The registration network consists of sequentially connected channel and spatial enhancement blocks and multi-scale registration blocks, such as Figure 2 As shown; The channel and spatial enhancement block consists of six sequentially connected feature extraction layers and a global average pooling layer; each feature extraction layer is composed of a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function connected in sequence; the number of channels in the convolutional layers of the six feature extraction layers are 64, 128, 256, 512, 1024, and 2048 respectively, following the principle of gradual expansion; The multi-scale registration block consists of three 3×3 convolutional layers and one 1×1 convolutional layer. The multi-scale registration block adopts a three-layer structure design. First, multi-scale features are extracted in parallel through convolutions with different dilation rates: the first layer uses a 3×3 convolutional layer with dilation rate 1, the second layer uses a 3×3 convolutional layer with dilation rate 2, and the third layer uses a 3×3 convolutional layer with dilation rate 3. Then, the outputs of the three convolutional layers are connected to the input of the 1×1 convolutional layer. In the multi-scale registration block, the features obtained from the output of three convolutional layers are concatenated, and finally the number of channels is adjusted from 1024 to 64 by a 1×1 convolutional layer before output.

[0061] There are two channel and space augmentation blocks; the parameters of the two channel and space augmentation blocks are not shared. Infrared common features with deformation The input is fed into the first channel and the spatial enhancement block to obtain the enhanced infrared common features. ; Common features of visible light The input is fed into the second channel and the spatial enhancement block to obtain the enhanced visible light common features. ; Will and The data is stitched together along the channel dimension, and the results are input into a multi-scale registration block to predict the deformation field. ; The registration loss module is used to calculate the registration loss. :

[0062] in, H For the height of the feature map, W The width of the feature map, For deformation field Tags; express l 1 Norm; use Common infrared features with deformation and infrared-specific features with deformation Registration is performed to obtain the registered infrared common features. and infrared-specific characteristics The registration calculation formula is as follows:

[0063] in, Indicates a Warp operation; Registered infrared common features Infrared characteristics after registration Common characteristics of visible light and the unique characteristics of visible light By concatenating along the channel dimension, the fused features are obtained. :

[0064] in, This indicates a splicing operation along the channel dimension.

[0065] Infrared common features with deformation The input is fed into the first channel and the spatial enhancement block to obtain the enhanced infrared common features. The specific method is as follows: Infrared common features with deformation The input is fed into the first channel and the spatial augmentation block, and then passed through the first three feature extraction layers to obtain the spatial augmentation weights. ; Will and Perform element-wise multiplication:

[0066] in, This indicates the enhanced features at the spatial level. This represents the element-wise multiplication operation; Will As input to the fourth feature extraction layer, the features are sequentially passed through the following three feature extraction layers and global average pooling to obtain features used for channel-level enhancement. ; Will and Element-wise multiplication yields features enhanced at both the spatial and channel levels. : .

[0067] Common features of visible light The input is fed into the second channel and the spatial enhancement block to obtain the enhanced visible light common features. The specific method is as follows: Common features of visible light The input is fed into the second channel and the spatial enhancement block, and then passed through the first three feature extraction layers to obtain the spatial enhancement weights. ; Will and Perform element-wise multiplication:

[0068] in, This indicates the enhanced features at the spatial level. This represents the element-wise multiplication operation; Will As input to the fourth feature extraction layer, the features are sequentially passed through the following three feature extraction layers and global average pooling to obtain features used for channel-level enhancement. ; Will and Element-wise multiplication yields features enhanced at both the spatial and channel levels. : .

[0069] The specific method for Step 4 is as follows: YOLOv5 was used as the header for the object detection task; The image restoration head consists of three sequentially connected feature extraction layers. The first two feature extraction layers are each composed of a 3×3 convolutional layer, a batch normalization layer, and a Leaky ReLU activation function connected in sequence. The last feature extraction layer is composed of a 3×3 convolutional layer, a batch normalization layer, and a Tanh activation function connected in sequence. The fusion features obtained after Step 3 Input into the object detection task header to obtain the object detection result image. ; The fusion features obtained after Step 3 Input into the image reconstruction task head to obtain the fused image. ; Calculate target detection loss Brightness loss gradient loss and fusion loss The calculation formula is as follows:

[0070] in, This is the loss function used during YOLOv5 training, which includes the sum of three parts: localization loss, classification loss, and confidence loss. Represents the label value of the detected object;

[0071] in, Indicates the equilibrium hyperparameters;

[0072]

[0073] Among them, its H For the height of the feature map, W The width of the feature map, Represents the Sobel operator; Infrared image; Visible light image; To merge images, express l 1 Norm; Finally, calculate the total loss. The feature extraction network, feature correction and fusion network, and dual-task reconstruction network are trained with the goal of minimizing the total loss. The calculation formula is as follows:

[0074] in:

[0075]

[0076] in, , all represent the equilibrium hyperparameters; For modal consistency loss, For modal complementary information loss, C The number of channels in the feature map. These are unique characteristics of infrared radiation. These are common characteristics of visible light. These are unique characteristics of infrared radiation. This is a characteristic unique to visible light.

[0077] Example 3 A security identification method based on target perception and fusion of infrared and visible light images includes the following steps: Step 1: Obtain the infrared and visible light images of the original image and preprocess the images; In Step 1, the preprocessing involves randomly cropping the infrared and visible light images to the same size, then normalizing them before inputting them into the feature extraction network, feature correction and fusion network, and dual-task reconstruction network, followed by data augmentation.

[0078] The infrared and visible light images were randomly cropped to a size of [size missing]. image.

[0079] Data augmentation methods include horizontal flipping and contrast enhancement, which increase the diversity of data samples to improve the robustness and generalization ability of the network.

[0080] The feature extraction network, feature correction and fusion network, and dual-task reconstruction network are sequentially connected to form the model of this invention, as follows: Figure 1 As shown; Step 2: Construct a feature extraction network; then use the feature extraction network to obtain the unique and common features of the original image; The feature extraction network includes an infrared-specific feature extraction module IR-SFE, a visible light-specific feature extraction module VI-SFE, an infrared common feature extraction module IR-CFE, and a visible light common feature extraction module VI-CFE. The infrared unique feature extraction module is used to extract unique features from infrared images; The visible light-specific feature extraction module is used to extract the specific features of visible light images; The infrared common feature extraction module is used to extract common features from infrared images; The visible light common feature extraction module is used to extract common features of visible light images; The specific method is as follows: Suppose that each set of original images in the training dataset used to train the feature extraction network contains three images: pixel-by-pixel strictly registered infrared images. Visible light images And infrared images with deformation ; infrared images And infrared images with deformation The corresponding infrared common features are then input into the infrared common feature extraction module to obtain the infrared common features. Common infrared features with deformation ; infrared images And infrared images with deformation The corresponding infrared-specific features are obtained by inputting them into the infrared-specific feature extraction module. Infrared features with deformation ; Visible light images Input the visible light common feature extraction module to obtain the visible light common features. ; Visible light images Input the visible light-specific feature extraction module to obtain visible light-specific features. ; To reduce the modal differences between infrared and visible light images and thus establish a more accurate pixel-by-pixel correspondence, we introduce modal consistency loss. Furthermore, considering that the goal of image fusion is to integrate as much complementary information as possible from cross-modal source images into a single image, we introduce modal complementarity information loss. This is to further enrich the complementary information from the source images in the fused image.

[0081] Calculate modal consistency loss and modal complementary information loss :

[0082]

[0083] in, H The height of the corresponding visible light common features and visible light common features feature maps. W The width of the feature map, C The number of channels in the feature map. express l 1 Norm; The infrared-specific feature extraction module consists of a 64-channel 7×7 convolutional layer and two 128-channel residual blocks connected in sequence. The visible light-specific feature extraction module consists of a sequentially connected 64-channel 5×5 convolutional layer and a densely connected block; The infrared common feature extraction module includes an encoder, which is composed of four 3×3 convolutional layers connected in sequence. The visible light common feature extraction module includes an encoder, which is composed of four 3×3 convolutional layers connected in sequence.

[0084] Step 3: Construct a feature correction and fusion network; then use the feature correction and fusion network to predict the deformation field and correct the common and unique features of the infrared image in spatial position, and then fuse it with the common and unique features of the visible light image. The feature correction and fusion network includes a registration network, a registration loss module, and a connection module; The registration network is used to register the deformed infrared common features and deformed infrared specific features to obtain the registered infrared common features and the registered infrared specific features; the registration loss module is used to calculate the registration loss. The connection module is used to connect the registered infrared-specific features, the registered infrared common features, the visible light common features, and the visible light-specific features according to channels to obtain fused features; The specific method is as follows: The registration network includes a Channel and Spatial Enhancement Block (CSEB) and a Multi-Scale Registration Block (MSRB), such as... Figure 2 As shown, CSEB is mainly used to enhance information in the input features that is conducive to registration at the channel level and spatial level, thereby improving the accuracy of the predicted deformation field.

[0085] The channel and spatial enhancement block consists of six sequentially connected feature extraction layers and a global average pooling (GAP) layer. Each feature extraction layer is composed of a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function connected in sequence. The number of channels in the convolutional layers of the six feature extraction layers are 64, 128, 256, 512, 512, and 1024, respectively. The multi-scale registration block consists of three 3×3 convolutional layers and one 1×1 convolutional layer. The multi-scale registration block adopts a three-layer structure design. First, multi-scale features are extracted in parallel through convolutions with different dilation rates: the first layer uses a 3×3 convolutional layer with dilation rate 1, the second layer uses a 3×3 convolutional layer with dilation rate 2, and the third layer uses a 3×3 convolutional layer with dilation rate 3. Then, the outputs of the three convolutional layers are connected to the input of the 1×1 convolutional layer. In the multi-scale registration block, the features obtained from the output of three convolutional layers are concatenated, and finally the number of channels is adjusted from 1024 to 64 by a 1×1 convolutional layer before output.

[0086] There are two channel and space augmentation blocks; the parameters of the two channel and space augmentation blocks are not shared. Infrared common features with deformation The input is fed into the first channel and the spatial enhancement block to obtain the enhanced infrared common features. The specific method is as follows: Infrared common features with deformation The input is fed into the first channel and the spatial augmentation block, and then passed through the first three feature extraction layers to obtain the spatial augmentation weights. ; Will and Perform element-wise multiplication:

[0087] in, This indicates the enhanced features at the spatial level. This represents the element-wise multiplication operation; Will As input to the fourth feature extraction layer, the features are sequentially passed through the following three feature extraction layers and global average pooling to obtain features used for channel-level enhancement. ; Will and Element-wise multiplication yields features enhanced at both the spatial and channel levels. : .

[0088] Common features of visible light The input is fed into the second channel and the spatial enhancement block to obtain the enhanced visible light common features. The specific method is as follows: Common features of visible light The input is fed into the second channel and the spatial enhancement block, and then passed through the first three feature extraction layers to obtain the spatial enhancement weights. ; Will and Perform element-wise multiplication:

[0089] in, This indicates the enhanced features at the spatial level. This represents the element-wise multiplication operation; Will As input to the fourth feature extraction layer, the features are sequentially passed through the following three feature extraction layers and global average pooling to obtain features used for channel-level enhancement. ; Will and Element-wise multiplication yields features enhanced at both the spatial and channel levels. : .

[0090] Will and The data is stitched together along the channel dimension, and the results are input into a multi-scale registration block to predict the deformation field. ; The registration loss module is used to calculate the registration loss. :

[0091] in, H For the height of the feature map, W The width of the feature map, For deformation field The labels, i.e., the reference deformation fields obtained from the training set, are usually generated by existing image registration methods. For relevant implementations, please refer to https: / / paperswithcode.com / dataset / eth-laser-registration-datasets. express l 1 Norm; use Common infrared features with deformation and infrared-specific features with deformation Registration is performed to obtain the registered infrared common features. and infrared-specific characteristics The registration calculation formula is as follows:

[0092] in, This indicates a Warp operation, which can be based on... The deformed feature map is resampled to correct the deformation in the feature map.

[0093] Finally, during the fusion process, in order to minimize information loss, the registered infrared shared features are... Infrared characteristics after registration Common characteristics of visible light and the unique characteristics of visible light By concatenating along the channel dimension, the fused features are obtained. :

[0094] in, This indicates a splicing operation along the channel dimension.

[0095] Step 4: Construct a dual-task reconstruction network; then, based on the fusion features, use the dual-task reconstruction network to obtain the fused image and its target detection results. The dual-task reconstruction network includes an object detection task head, an image restoration task head, an object detection loss module, and a fusion loss module; The target detection task head is used to perform target detection based on the fusion features obtained in Step 3, thereby obtaining the target detection result image; the target detection result is the people on the construction site who are not wearing helmets, and they are selected with a rectangle. The image restoration task head is used to restore the image based on the fusion features obtained in Step 3, thereby obtaining the fused image; The target detection loss module is used to calculate the target detection loss. ; The fusion loss module is used to obtain the fusion loss. ; Step 5: Real-time acquisition of infrared and visible light images in the power operation scene, preprocessing according to the method in Step 1, and then sequentially passing through a feature extraction network, a feature correction and fusion network, and a dual-task reconstruction network to obtain a fused image and its target detection result map, thereby obtaining the target detection result as the safety risk identification result.

[0096] The specific method is as follows: YOLOv5 was used as the header for the object detection task; The image restoration head consists of three sequentially connected feature extraction layers. The first two feature extraction layers are each composed of a 3×3 convolutional layer, a batch normalization layer, and a Leaky ReLU activation function connected in sequence. The last feature extraction layer is composed of a 3×3 convolutional layer, a batch normalization layer, and a Tanh activation function connected in sequence. The fusion features obtained after Step 3 Input into the object detection task header to obtain the object detection result image. ; The fusion features obtained after Step 3 Input into the image reconstruction task head to obtain the fused image. ; To obtain high-quality object detection result maps, we introduce object detection loss. To constrain the network; furthermore, to encourage the fused image to include as much shared and complementary information as possible from the infrared and visible light images, we introduce brightness losses respectively. and gradient loss And constructed fusion loss ;Specifically: Calculate target detection loss Brightness loss gradient loss and fusion loss The calculation formula is as follows:

[0097] in, This is the loss function used during YOLOv5 training. This represents the label value for object detection. The YOLOv5 training loss function consists of three parts: localization loss, classification loss, and confidence loss. For implementation details, please refer to https: / / github.com / ultralytics / yolov5. The labels used for training are provided by the dataset and contain information such as the object's category and bounding box location, used to supervise model learning.

[0098]

[0099] in, Indicates the equilibrium hyperparameters;

[0100]

[0101] Among them, its H For the height of the feature map, W The width of the feature map, Represents the Sobel operator; Infrared image; Visible light image; To merge images, express l 1 Norm; Finally, calculate the total loss. The calculation formula is as follows:

[0102] in:

[0103]

[0104] , all represent the equilibrium hyperparameters; For modal consistency loss, For modal complementary information loss, C The number of channels in the feature map. These are unique characteristics of infrared radiation. These are common characteristics of visible light. These are unique characteristics of infrared radiation. This is a characteristic unique to visible light.

[0105] Application Examples To verify the effectiveness of the method of this invention, we used 15200 pairs of samples with a size of... The model was trained on infrared and visible light images from RoadScene, using 18 pairs of images from RoadScene and 17 pairs from M. 3 Model testing was performed on FD images. The deformed infrared images were obtained by randomly applying a hybrid rigid and non-rigid deformation to the original precisely registered infrared images. This hybrid deformation was randomly applied to the original precisely registered images in each epoch to obtain more training data. The proposed method was implemented using the PyTorch framework and trained on a single NVIDIA GeForce RTX 3090 GPU. We trained on the training set for 150 epochs with a batch size of 8 and a learning rate of 1×10⁻⁶. -3 The Adam Optimizer is used to update the model parameters. The four hyperparameters set in the loss function are as follows: , , , .

[0106] We selected four commonly used image quality evaluation metrics to objectively evaluate the quality of the fusion results, including the correlation coefficient (CQC). ), gradient-based fusion performance ( Chen-Varshney metric ) and structural similarity ( ) . The degree of linear correlation between the fused image and the source image was evaluated, and their similarity was measured. The amount of edge information transferred from the source image to the fused image is evaluated. This approach simultaneously considers edge information in the fused image and human visual perception. By comparing the fused image and the source image, the information loss and distortion in the fused image are quantified. Among these evaluation metrics, A lower value for the metric indicates better quality of the fused image, while higher values ​​for the other metrics indicate better fusion quality. Furthermore, we also selected...

[40] As an evaluation metric for object detection tasks, higher This indicates better target detection performance.

[0107] We first compare the proposed method with the fusion results of two types of misaligned infrared-visible image fusion methods, and then compare the subsequent target detection results of the two types of misaligned infrared-visible image fusion methods. The first type of method first performs registration processing on the images to be fused to obtain aligned images, then fuses the aligned images, and finally performs target detection. We name this type of method "Registration + Fusion + Target Detection". The second type of method uses a joint training approach for registration and fusion to achieve the fusion of misaligned images, and then performs target detection. We name this type of method "Joint Registration and Fusion + Target Detection".

[0108] In the registration + fusion + target detection method, we follow the processing mode of existing methods and select the high-performance registration method CrossRAFT to align the images to be fused. Then, we use four advanced infrared and visible light image fusion methods to fuse these aligned results. These methods include DAFTUUSE (adaptively fuses infrared and visible light features through a deep attention mechanism, and enhances the fusion effect of important regions by combining channel attention and spatial attention), TarDAL (a domain-adaptive learning method based on target perception, which reduces the modal differences between infrared and visible light images through adversarial training and improves the alignment ability of cross-modal features), YDTR (adopts a lightweight Transformer architecture, jointly optimizes feature fusion in the spatial and frequency domains, and balances computational efficiency and fusion quality), and EMMA (based on multi-scale feature enhancement and adaptive weighting strategy, which optimizes detail preservation and thermal radiation information transmission through a multi-stage fusion network). Figure 3The visual effects of the results obtained by different methods are shown. These fusion results demonstrate that our proposed method not only possesses stronger structure and texture preservation capabilities but also minimizes distortion and artifacts caused by feature misalignment. Furthermore, we objectively evaluated the results obtained by different methods, as shown in Table 1. Our proposed method exhibits the best performance across all four evaluation metrics, where upward arrows indicate higher performance for larger values, and downward arrows indicate higher performance for smaller values.

[0109] Table 1

[0110] Similarly, we conducted experiments on a joint registration and fusion + object detection method. To verify the superiority of our method over these methods, we compared its performance with four joint registration and fusion methods. These four methods are: IMF (which achieves multimodal image alignment through an iterative mutual information registration strategy and enhances target region feature representation through adaptive weighted fusion), IVFWSR (which combines sparse representation and feature weighting mechanism, and optimizes cross-modal feature fusion through dictionary learning after registration), MURF (which optimizes spatial alignment and feature integration processes simultaneously based on a multi-scale registration network and a unified feature fusion architecture), and SuperFusion (which uses a supervised registration network to generate a deformation field to guide fusion, and achieves collaborative optimization of registration and fusion through end-to-end training). Figure 4 The qualitative results of different fusion methods are compared, demonstrating that the method of this invention exhibits significant advantages in feature alignment, contrast preservation, and detail protection. Furthermore, quantitative experiments were conducted to visually compare the performance differences between them. As shown in Table 2, the method of this invention achieved the best performance across all four indicators, where upward arrows indicate higher performance for larger values, and downward arrows indicate higher performance for smaller values.

[0111] Table 2

[0112] The core of the method proposed in this invention lies in eliminating the loss of modal differences—modal consistency loss. and modal complementary information loss As shown in Table 3 below, we conducted ablation experiments on these core components, such as the registration network and the fusion network, to verify their effectiveness. These experiments were all conducted on M... 3 The ablation experiment was performed on the FD dataset. The results show that the removal... and Subsequently, the model's ability to correct local deformations decreased, such as... Figure 5As shown in Table 3, although the visualization quality of the fused image does not decrease significantly when only complementary information is used for stitching without incorporating shared information during the fusion process, the performance does decrease in the objective evaluation results. The upward arrows indicate that the larger the value, the higher the performance, and the downward arrows indicate that the smaller the value, the higher the performance.

[0113] Table 3

[0114] By fusing multimodal features from infrared and visible light images and combining prior knowledge of power operation scenarios (such as safe distance thresholds, abnormal equipment temperature thresholds, and protective equipment detection rules), the system can identify typical safety risks in real time, such as operator violations (e.g., not wearing insulated gloves), equipment overheating hazards, and foreign objects entering dangerous areas, based on the target detection result image and the fused image, thereby achieving intelligent safety monitoring of power operation scenarios.

[0115] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A security identification method based on target perception and fusion of infrared and visible light images, characterized in that, Includes the following steps: Step 1: Obtain the infrared and visible light images of the original image and preprocess the images; Step 2: Construct a feature extraction network; then use the feature extraction network to obtain the unique and common features of the original image; The feature extraction network includes an infrared-specific feature extraction module, a visible light-specific feature extraction module, an infrared common feature extraction module, and a visible light common feature extraction module; The infrared unique feature extraction module is used to extract unique features from infrared images; The visible light-specific feature extraction module is used to extract the specific features of visible light images; The infrared common feature extraction module is used to extract common features from infrared images; The visible light common feature extraction module is used to extract common features of visible light images; Step 3: Construct a feature correction and fusion network; then use the feature correction and fusion network to predict the deformation field and perform spatial registration of the common and unique features of the infrared image, and then fuse it with the common and unique features of the visible light image. The feature correction and fusion network includes a registration network, a registration loss module, and a connection module; The registration network is used to register the infrared common features with deformation and the infrared specific features with deformation, so as to obtain the registered infrared common features and the registered infrared specific features. The registration loss module is used to calculate the loss after registration; The connection module is used to connect the registered infrared-specific features, the registered infrared common features, the visible light common features, and the visible light-specific features according to channels to obtain fused features; Step 4: Construct a dual-task reconstruction network; then, based on the fusion features, use the dual-task reconstruction network to obtain the fused image and its target detection results. The dual-task reconstruction network includes an object detection task head, an image restoration task head, an object detection loss module, and a fusion loss module; The object detection task head is used to perform object detection based on the fusion features obtained in Step 3, thereby obtaining the object detection result map; The image restoration task head is used to restore the image based on the fusion features obtained in Step 3, thereby obtaining the fused image; The target detection loss module is used to calculate the target detection loss; The fusion loss module is used to obtain the fusion loss; Step 5: Real-time acquisition of infrared and visible light images in the power operation scenario, preprocessing according to the method in Step 1, and then sequentially passing through a feature extraction network, a feature correction and fusion network, and a dual-task reconstruction network to obtain a fused image and its target detection result map; finally, safety risk identification and early warning are performed based on the fused image and its target detection result map.

2. The security identification method based on target perception and infrared and visible light image fusion according to claim 1, characterized in that, In Step 1, the preprocessing involves randomly cropping the infrared and visible light images to the same size, then performing a normalization operation, followed by data augmentation.

3. The security identification method based on target perception and infrared and visible light image fusion according to claim 2, characterized in that, The infrared and visible light images were randomly cropped to a size of [size missing]. image.

4. The security identification method based on target perception and infrared and visible light image fusion according to claim 2, characterized in that, Data augmentation methods include horizontal flipping and contrast enhancement.

5. The security identification method based on target perception and infrared and visible light image fusion according to claim 1, characterized in that, The specific method for Step 2 is as follows: Suppose that each set of original images in the training dataset used to train the feature extraction network contains three images: pixel-by-pixel strictly registered infrared images. Visible light images And infrared images with deformation ; infrared images And infrared images with deformation The corresponding infrared common features are then input into the infrared common feature extraction module to obtain the infrared common features. Common infrared features with deformation ; infrared images And infrared images with deformation The corresponding infrared-specific features are obtained by inputting them into the infrared-specific feature extraction module. Infrared features with deformation ; Visible light images Input the visible light common feature extraction module to obtain the visible light common features. ; Visible light images Input the visible light-specific feature extraction module to obtain visible light-specific features. ; Calculate modal consistency loss and modal complementary information loss : ； ； in, H The height of the feature maps corresponding to the infrared common features and visible light common features. W The width of the corresponding feature map. C This represents the number of channels in the corresponding feature map. express l 1 Norm; The infrared-specific feature extraction module consists of a 64-channel 7×7 convolutional layer and two 128-channel residual blocks connected in sequence. The visible light-specific feature extraction module consists of a sequentially connected 64-channel 5×5 convolutional layer and a densely connected block; The infrared common feature extraction module includes an encoder, which is composed of four 3×3 convolutional layers connected in sequence. The visible light common feature extraction module includes an encoder, which is composed of four 3×3 convolutional layers connected in sequence.

6. The security identification method based on target perception and infrared and visible light image fusion according to claim 1, characterized in that, The specific method for Step 3 is as follows: The registration network consists of sequentially connected channel and spatial augmentation blocks and multi-scale registration blocks; The channel and spatial enhancement block consists of six sequentially connected feature extraction layers and a global average pooling layer; each feature extraction layer is composed of a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function connected in sequence; the number of channels in the convolutional layers of the six feature extraction layers are 64, 128, 256, 512, 1024, and 2048 respectively, following the principle of gradual expansion; The multi-scale registration block consists of three 3×3 convolutional layers and one 1×1 convolutional layer. The multi-scale registration block adopts a three-layer structure design. First, multi-scale features are extracted in parallel through convolutions with different dilation rates: the first layer uses a 3×3 convolutional layer with dilation rate 1, the second layer uses a 3×3 convolutional layer with dilation rate 2, and the third layer uses a 3×3 convolutional layer with dilation rate 3. Then, the outputs of the three convolutional layers are connected to the input of the 1×1 convolutional layer. In the multi-scale registration block, the features obtained from the output of three convolutional layers are concatenated, and finally the number of channels is adjusted from 1024 to 64 by a 1×1 convolutional layer before output.

7. The security identification method based on target perception and infrared and visible light image fusion according to claim 6, characterized in that, There are two channel and space augmentation blocks; the parameters of the two channel and space augmentation blocks are not shared. Infrared common features with deformation The input is fed into the first channel and the spatial enhancement block to obtain the enhanced infrared common features. ; Common features of visible light The input is fed into the second channel and the spatial enhancement block to obtain the enhanced visible light common features. ; Will and The data is stitched together along the channel dimension, and the results are input into a multi-scale registration block to predict the deformation field. ; The registration loss module is used to calculate the registration loss. : ； Where H is the height of the feature map, W The width of the feature map, For deformation field Tags; express l 1 Norm; use Common infrared features with deformation and infrared-specific features with deformation Registration is performed to obtain the registered infrared common features. and infrared-specific characteristics The registration calculation formula is as follows: ； in, Indicates a Warp operation; Registered infrared common features Infrared characteristics after registration Common characteristics of visible light and the unique characteristics of visible light By concatenating along the channel dimension, the fused features are obtained. : ； in, This indicates a splicing operation along the channel dimension.

8. The security identification method based on target perception and infrared and visible light image fusion according to claim 7, characterized in that, Infrared common features with deformation The input is fed into the first channel and the spatial enhancement block to obtain the enhanced infrared common features. The specific method is as follows: Infrared common features with deformation The input is fed into the first channel and the spatial augmentation block, and then passed through the first three feature extraction layers to obtain the spatial augmentation weights. ; Will and Perform element-wise multiplication: ； in, This indicates the enhanced features at the spatial level. This represents the element-wise multiplication operation; Will As input to the fourth feature extraction layer, the features are sequentially passed through the following three feature extraction layers and global average pooling to obtain features used for channel-level enhancement. ; Will and Element-wise multiplication yields features enhanced at both the spatial and channel levels. : 。 9. The security identification method based on target perception and infrared and visible light image fusion according to claim 7, characterized in that, Common features of visible light The input is fed into the second channel and the spatial enhancement block to obtain the enhanced visible light common features. The specific method is as follows: Common features of visible light The input is fed into the second channel and the spatial enhancement block, and then passed through the first three feature extraction layers to obtain the spatial enhancement weights. ; Will and Perform element-wise multiplication: ； in, This indicates the enhanced features at the spatial level. This represents the element-wise multiplication operation. Will As input to the fourth feature extraction layer, the features are sequentially passed through the following three feature extraction layers and global average pooling to obtain features used for channel-level enhancement. ; Will and Element-wise multiplication yields features enhanced at both the spatial and channel levels. : 。 10. The security identification method based on target perception and infrared and visible light image fusion according to claim 1, characterized in that, The specific method for Step 4 is as follows: YOLOv5 was used as the header for the object detection task; The image restoration head consists of three sequentially connected feature extraction layers. The first two feature extraction layers are each composed of a 3×3 convolutional layer, a batch normalization layer, and a Leaky ReLU activation function connected in sequence. The last feature extraction layer is composed of a 3×3 convolutional layer, a batch normalization layer, and a Tanh activation function connected in sequence. The fusion features obtained after Step 3 Input into the object detection task header to obtain the object detection result image. ; The fusion features obtained after Step 3 Input into the image reconstruction task head to obtain the fused image. ; Calculate target detection loss Brightness loss gradient loss and fusion loss The calculation formula is as follows: ； in, This is the loss function used during YOLOv5 training, which includes the sum of three parts: localization loss, classification loss, and confidence loss. Represents the label value of the detected object; ； in, Indicates the equilibrium hyperparameters; ； ； Among them, its H Let W be the height of the feature map, and W be the width of the feature map. Represents the Sobel operator; Infrared image; Visible light image; To merge images, Represents the l1 norm; Finally, calculate the total loss. The feature extraction network, feature correction and fusion network, and dual-task reconstruction network are trained with the goal of minimizing the total loss. The calculation formula is as follows: ； in: ； ； in, , all represent the equilibrium hyperparameters; For modal consistency loss, For modal complementary information loss, C The number of channels in the feature map. These are unique characteristics of infrared radiation. These are common characteristics of visible light. These are unique characteristics of infrared radiation. This is a characteristic unique to visible light.

Images