Infrared visible light fusion target detection method and related device

Abstract

The invention discloses an infrared and visible light fusion target detection method and a related device, and relates to the technical field of distribution network equipment inspection, and the method comprises the steps: carrying out the space-time alignment and calibration of an infrared image and a visible light image, obtaining an aligned image pair, and building a comparable temperature baseline; extracting multi-scale features from the infrared image and the visible light image in the aligned image pair by using a double-flow feature extraction network, and performing fusion processing on the multi-scale features by using a cross-modal attention enhancement mechanism to obtain enhanced multi-scale features; performing global dependency fusion and multi-scale feature reconstruction on the enhanced multi-scale features to obtain a fused multi-scale feature map; performing target detection on the fused multi-scale feature map through an improved YOLO detection head to obtain a preliminary detection result; and carrying out post-processing and decision fusion on the detection result to obtain structured alarm information. According to the invention, the problems of low detection accuracy and poor real-time performance under complex working conditions in the prior art are solved.

Description

Technical Field

[0001] This invention relates to the field of power distribution network equipment inspection technology, and in particular to a target detection method and related device using infrared and visible light fusion. Background Technology

[0002] Distribution network equipment is a core component of the power distribution system, and its operating status directly determines the reliability and power supply quality of the power grid. Under conditions of long-term high-load operation, environmental aging, and insufficient maintenance, equipment is prone to defects such as overheating and damage. If these defects are not identified in time, they may cause local power outages or even system failures, seriously affecting power grid safety. Therefore, developing efficient automatic detection technology is crucial to achieve early warning and accurate location of equipment anomalies.

[0003] In recent years, multispectral imaging technology has been widely used in the inspection of power distribution equipment because it can combine visible light imagery with infrared thermal imaging temperature information. Visible light images can clearly show the structural features of the equipment (such as insulator cracks), while infrared images can reveal abnormal heat points (such as overheating caused by poor contact). The two complement each other to improve the robustness of the detection.

[0004] However, the efficient utilization of multispectral data faces significant challenges: images of different modalities exhibit differences in resolution and features, and direct fusion can easily lead to information loss or false detections. Traditional target detection methods are mostly based on single modalities or simple fusion strategies, making it difficult to adapt to the complex scenarios of power distribution network equipment. For example, while convolutional neural networks (CNNs) perform excellently in visible light detection, they cannot fully capture the long-range dependencies between infrared and visible light, and their adaptability to multi-scale targets (such as large transformers and small terminals) is insufficient. Furthermore, images of power distribution network equipment are often affected by occlusion and changes in illumination, requiring the detection model to have strong generalization capabilities. Current multispectral target detection methods can be broadly classified into three categories: methods based on traditional image processing, CNN-based fusion methods, and attention-based fusion methods. Traditional image processing methods (such as feature stitching or weighted averaging) are simple to implement, but rely on manually designed fusion rules, are sensitive to modal differences, and are prone to missed detections in distribution network equipment images due to temperature and texture mismatches. CNN-based fusion methods (such as dual-stream networks) automatically learn features through deep learning, but the local receptive field of CNNs limits the integration of global contextual information, making it difficult to capture the correlation between overall equipment defects and local defects. For example, in dense equipment areas, CNNs may ignore cross-modal complementary cues, leading to misjudgments of overheated equipment. Attention-based fusion methods (such as Transformers) can model long-distance dependencies through self-attention mechanisms, but have high computational complexity and are not optimized for the multi-scale characteristics of distribution network equipment. Cross-modal fusion transformers (CFTs) improve fusion results, but the generation of correlation matrices requires processing a large amount of intermediate data, resulting in poor real-time performance and making it difficult to directly apply to high-resolution distribution network images. Summary of the Invention

[0005] This invention provides a target detection method and related apparatus based on infrared and visible light fusion, which addresses the problems of low detection accuracy and poor real-time performance in existing technologies under complex working conditions.

[0006] In view of this, the first aspect of the present invention provides a target detection method by infrared-visible light fusion, the method comprising:

[0007] The infrared and visible light images acquired simultaneously are spatiotemporally aligned and calibrated to obtain aligned image pairs, and a comparable temperature baseline is established.

[0008] Using a dual-stream feature extraction network, multi-scale features are extracted from the infrared and visible light images of the aligned image pair, respectively. A cross-modal attention enhancement mechanism is then used to fuse the multi-scale features to obtain enhanced multi-scale features.

[0009] The enhanced multi-scale features are subjected to global dependency fusion and multi-scale feature reconstruction to obtain a fused multi-scale feature map.

[0010] The improved YOLO detection head is used to perform target detection and anomaly determination on the fused multi-scale feature map to obtain preliminary detection results;

[0011] Based on the comparable temperature baseline, the preliminary detection results are post-processed and fused with decision-making data to generate and output structured alarm information.

[0012] Optionally, the step of performing spatiotemporal alignment and calibration on the synchronously acquired infrared and visible light images to obtain aligned image pairs and establishing a comparable temperature baseline includes:

[0013] The infrared and visible light images acquired simultaneously are aligned at the pixel level to obtain aligned image pairs.

[0014] The infrared grayscale of the infrared image is converted into temperature, the mean and fluctuation values ​​of the background temperature are obtained, and the aligned image pairs are combined to construct a comparable temperature baseline.

[0015] Optionally, the step of using a dual-stream feature extraction network to extract multi-scale features from the infrared and visible light images of the aligned image pair, respectively, further includes:

[0016] Asymmetric data augmentation is performed on the aligned image pairs to obtain augmented image pairs, which are used to train the dual-stream feature extraction network and the cross-modal attention enhancement mechanism.

[0017] The asymmetric data enhancement includes: dividing the infrared image and the visible light image in the aligned image pair into several identical regions, performing complementary occlusion processing on each region in the infrared image and the visible light image based on a preset probability, and superimposing noise on each region; the complementary occlusion processing includes: when occlusion is applied to a region of either the infrared image or the visible light image, no occlusion is applied to the corresponding region of the other image.

[0018] Optionally, the fusion process includes:

[0019] In each channel of the dual-stream feature extraction network used to output the multi-scale features, attention evaluation is first performed on the channel to highlight the feature channels related to device anomalies, and then spatial attention processing is performed on the multi-scale features to focus on the high-risk areas of the device, resulting in enhanced multi-scale features.

[0020] Optionally, the step of performing global dependency fusion and multi-scale feature reconstruction on the enhanced multi-scale features to obtain a fused multi-scale feature map includes:

[0021] A shift-window self-attention mechanism is used to perform global dependency fusion on the enhanced multi-scale features;

[0022] The feature pyramid network is used to perform multi-scale processing on the features after global dependency fusion, and output the fused multi-scale feature map. The multi-scale processing includes fusing feature information of different scales through bidirectional paths of top-down and bottom-up.

[0023] The shifted window self-attention mechanism involves dividing the multi-scale features into several windows and alternately using normal windows and shifted windows. Multi-head self-attention computation is performed within each window to associate the long-distance dependency between temperature gradient and structural semantics.

[0024] Optionally, the step of performing target detection and anomaly determination on the fused multi-scale features using an improved YOLO detection head to obtain preliminary detection results includes:

[0025] The improved YOLO detection head adopts a decoupled structure and includes classification and regression branches;

[0026] The classification branch, based on the fused multi-scale features, simultaneously predicts the type confidence and appearance anomaly confidence of the target device;

[0027] The regression branch adopts an anchor-free regression method, which regresses the position offset and size of the target bounding box relative to the point on the fused multi-scale feature map, and outputs the position information of the bounding box.

[0028] By combining the type confidence, the appearance anomaly confidence, and the bounding box position information, a preliminary detection result containing several candidate bounding boxes is generated.

[0029] Optionally, the post-processing and decision fusion of the preliminary detection results based on the comparable temperature baseline to generate and output structured alarm information includes:

[0030] Non-maximum suppression is applied to each of the candidate bounding boxes to remove those that overlap and whose type confidence and appearance anomaly confidence are lower than the corresponding preset thresholds.

[0031] For the candidate bounding boxes retained after non-maximum suppression processing, their corresponding regions in the infrared image are obtained, and the average temperature and peak temperature of the region are calculated and compared with the comparable temperature baseline to obtain the relative temperature rise.

[0032] The calculated average temperature, peak temperature, and relative temperature rise are compared with an adaptive absolute threshold based on device type and ambient temperature, and a relative threshold relative to the comparable temperature baseline, to generate an infrared temperature anomaly determination result.

[0033] The appearance anomaly confidence scores of the retained candidate bounding boxes are weighted and fused with the infrared temperature anomaly determination results to generate a comprehensive anomaly confidence score and an anomaly determination conclusion. The generation rules for the comprehensive anomaly confidence score include: when both are determined to be anomalies, the comprehensive anomaly confidence score is determined to be high; when only one is determined to be anomalies, the comprehensive anomaly confidence score is determined to be low, or an independent pending verification status flag is output.

[0034] The output structured alarm information includes at least: device type, the reserved candidate bounding box, the anomaly determination conclusion, the appearance anomaly confidence level, the average temperature, the peak temperature, and the relative temperature rise.

[0035] A second aspect of the present invention provides an infrared-visible light fusion target detection system, the system comprising:

[0036] The construction unit is used to perform spatiotemporal alignment and calibration on synchronously acquired infrared and visible light images to obtain aligned image pairs and establish comparable temperature baselines.

[0037] The enhancement unit is used to extract multi-scale features from the infrared image and the visible light image in the aligned image pair using a dual-stream feature extraction network, and to fuse the multi-scale features using a cross-modal attention enhancement mechanism to obtain enhanced multi-scale features.

[0038] The fusion unit is used to perform global dependency fusion and multi-scale feature reconstruction on the enhanced multi-scale features to obtain a fused multi-scale feature map.

[0039] The first detection unit is used to perform target detection and anomaly determination on the fused multi-scale feature map using an improved YOLO detection head, and obtain preliminary detection results.

[0040] The second detection unit is used to perform post-processing and decision fusion on the preliminary detection results based on the comparable temperature baseline, and generate and output structured alarm information.

[0041] A third aspect of the present invention provides an infrared-visible light fusion target detection device, the device comprising a processor and a memory:

[0042] The memory is used to store program code and transmit the program code to the processor;

[0043] The processor is configured to execute the steps of the infrared-visible light fusion target detection method as described in the first aspect above, according to the instructions in the program code.

[0044] A fourth aspect of the present invention provides a computer-readable storage medium for storing program code for executing the infrared-visible light fusion target detection method described in the first aspect above.

[0045] As can be seen from the above technical solutions, the present invention has the following advantages:

[0046] This invention provides an infrared-visible light fusion target detection method. First, a comparable temperature baseline is established. During training, asymmetric cross-modal enhancement is introduced. Then, a two-level cross-modal attention mechanism ("channel → space") is combined with a backbone network and a feature pyramid network for global and multi-scale fusion, achieving robust focusing and anti-spoofing of small components such as connectors / terminals. Finally, an improved YOLO detection head outputs an integrated result of "device + location + anomaly," and temperature factors and visual scores (appearance anomaly confidence) are combined using a dual-track thresholding and concise fusion (considering device type, ambient temperature, and relative background fluctuations), effectively reducing false alarms caused by seasonal temperature increases and metallic reflections. This target detection method can operate in real-time at the edge, forming a closed loop of "discovery—grading—repair—review," improving the accuracy, real-time performance, and reliability of target detection, thus solving the problems of low accuracy and poor real-time performance in existing technologies under complex working conditions. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A schematic flowchart of an infrared-visible light fusion target detection method provided in an embodiment of the present invention;

[0049] Figure 2 A schematic diagram of infrared and visible light images provided in an embodiment of the present invention;

[0050] Figure 3 This is the cross-modal fusion backbone framework provided in the embodiments of the present invention;

[0051] Figure 4 This is a schematic diagram of the structure of an infrared-visible light fusion target detection system provided in an embodiment of the present invention. Detailed Implementation

[0052] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0053] Please see Figure 1 The present invention provides a target detection method using infrared-visible light fusion, comprising:

[0054] Step 101: Perform spatiotemporal alignment and calibration on the synchronously acquired infrared and visible light images to obtain aligned image pairs and establish comparable temperature baselines.

[0055] In one embodiment, step 101 includes:

[0056] Step 1011: Align the infrared and visible light images acquired synchronously at the pixel level to obtain aligned image pairs.

[0057] It should be noted that, firstly, infrared and visible light images of the environment where the target to be detected is located are simultaneously acquired (see [link to relevant documentation]). Figure 2 Then, the infrared image and the visible light image are aligned at the pixel level to obtain an aligned image pair, which can be understood as containing both an infrared image and a visible light image.

[0058] The expression for pixel-level image alignment is as follows:

[0059] (1.1)

[0060] In the formula, / An infrared image representing infrared coordinates projected onto visible light coordinates; Represents geometric mapping, This represents the pixel coordinates of any pixel in the image. The calibration formula (1.1) maps the infrared image to the visible light image coordinate system through geometric mapping, aiming to adjust and match the temperature information of the two images. Specifically, the transformation matrix is ​​used to transform the coordinates of the infrared image so that it is aligned with the visible light image in the same space, thereby achieving temperature calibration. This helps in the precise adjustment of temperature data.

[0061] Step 1012: Convert the infrared grayscale of the infrared image into temperature, obtain the mean and fluctuation values ​​of the background temperature, and combine them with the aligned image pairs to construct a comparable temperature baseline.

[0062] It should be noted that, specifically, the infrared grayscale of the infrared image can be converted into temperature using the radiometric calibration coefficient of the infrared camera; and the mean temperature of the unequipped safe background area (mean background temperature) and the temperature fluctuation of the safe background area (temperature fluctuation of the background temperature) can be estimated.

[0063] Further combining aligned image pairs to construct a comparable temperature baseline can be understood as follows: 1) Pixel-level spatial alignment: Geometrically aligning infrared and visible light images to ensure the spatial position of the same device is consistent in both modalities, providing a spatial basis for cross-modal information fusion. 2) Infrared radiometric calibration conversion: Using the radiometric calibration coefficient of the infrared camera, converting the grayscale values ​​of the infrared image into true temperature values ​​to achieve quantitative characterization of temperature. 3) Background temperature benchmark statistics: Selecting a safe background area without devices in the image and calculating the mean and standard deviation (fluctuation value) of the background temperature in this area as a benchmark for subsequent anomaly temperature determination. 4) Integrated baseline output: Integrating the spatial alignment results, true temperature values, mean and fluctuation values ​​of the background temperature to form a multimodal input baseline of "comparable temperature + comparable semantics," providing a stable reference for anomaly determination in subsequent steps.

[0064] The infrared radiation calibration conversion uses the following formula:

[0065] (1.2)

[0066] The expressions for the mean and standard deviation (fluctuation value) of background temperature are as follows:

[0067] (1.3)

[0068] (1.4)

[0069] In the formula, Infrared pixel values; For temperature; All are temperature calibration coefficients; For a safe background area; These represent the mean and standard deviation (fluctuation value) of the background temperature, respectively.

[0070] In one embodiment, step 102 further includes:

[0071] Asymmetric data augmentation is performed on the aligned image pairs to obtain augmented image pairs, which are then used to train a two-stream feature extraction network and a cross-modal attention enhancement mechanism.

[0072] The asymmetric data augmentation includes: dividing the infrared image and the visible light image in the aligned image pair into several identical blocks, performing complementary occlusion processing on each block of the infrared image and the visible light image based on a preset probability, and superimposing noise on each block; the complementary occlusion processing includes: when occlusion is applied to a certain area of ​​either the infrared image or the visible light image, the corresponding area of ​​the other image is not occluded.

[0073] It should be noted that asymmetric data augmentation is only used during training, meaning the dual-stream feature extraction network and cross-modal attention enhancement mechanism are trained in advance. In other words, it is only enabled during model training and not used during inference (actual detection) to avoid occlusion affecting real-time performance and result integrity.

[0074] Specifically, the infrared and visible light images in the aligned image pair are divided into several identical regions, such as 10×10 small regions. Then, based on a preset probability, complementary occlusion is randomly applied to the infrared (IR) and visible light (RGB) images (i.e., regions occluded in RGB are not occluded in IR, and vice versa), where the preset probability can be p=0.3. It can be understood that during training, each input RGB-IR image pair has a 30% probability of triggering this complementary occlusion operation, forcing the model to learn cross-modal complementary information (when local information in one modality is missing, recognition is completed by relying on the corresponding region of another modality); thereby improving the model's robustness in complex power distribution network scenarios such as rain, fog, backlight, and partial occlusion, and reducing false positives / false negatives caused by missing single-modal information. Furthermore, noise can be superimposed on each area, such as mild fogging, brightness disturbances, or random corner occlusion. The enhancement only applies to pixels and does not modify the original bounding boxes and labels to ensure image integrity and real-time stability. In addition, during operation, asymmetric data augmentation strategies can be written into the data pipeline (DataLoader) to perform layered sampling according to device type or weather labels to improve coverage.

[0075] The expression for complementary occlusion is as follows:

[0076] (2.1)

[0077] (2.2)

[0078] In the formula, For the original input, For complementary masks, For the complementary overall image, This is a transformation matrix specifically used to add RGB information supplementary masks. This is a transformation matrix specifically used to add infrared information supplementary masks. Overall, this process aims to effectively combine information from RGB and infrared images and adjust them into a unified output image for use in subsequent calculations or analyses.

[0079] Step 102: Using a dual-stream feature extraction network, multi-scale features are extracted from the infrared and visible light images in the aligned image pairs, respectively. A cross-modal attention enhancement mechanism is then used to fuse the multi-scale features to obtain enhanced multi-scale features.

[0080] Please see Figure 3 For each frame of the image pair aligned in step 101, containing both the infrared and visible light images, a dual-stream feature extraction network is used. Specifically, two lightweight but expressive backbone networks (such as CSPDarknet53) are run simultaneously to extract multi-scale features from the visible light and infrared modalities, respectively. Let the visible light image (RGB) backbone network and the infrared image (IR) backbone network be denoted as... and They output feature maps at three pyramid scales. and ,satisfy .in This corresponds to three scales with resolutions from high to low (e.g., P3 / P4 / P5). Next, a cross-modal attention enhancement mechanism is used to fuse the multi-scale features to obtain enhanced multi-scale features. Specific fusion processing is described in the following examples.

[0081] In one embodiment, step 102, the fusion process includes:

[0082] In the dual-stream feature extraction network, each channel used to output multi-scale features is first evaluated for attention to highlight the feature channels related to device anomalies. Then, spatial attention processing is applied to the multi-scale features to focus on high-risk areas of the device, resulting in enhanced multi-scale features.

[0083] It is understandable that channel splicing is performed on features of the same scale. To ensure information alignment, first perform an "importance assessment" at the channel level, highlighting channels related to heat / damage textures and suppressing artifact channels such as reflections and nameplates. Then perform spatial attention, focusing on small, high-risk areas such as connectors, terminals, sleeve roots, and fixtures. Perform this two-stage enhancement on P3 / P4 / P5 to obtain enhanced multi-scale features.

[0084] Step 103: Perform global dependency fusion and multi-scale feature reconstruction on the enhanced multi-scale features to obtain the fused multi-scale feature map.

[0085] In one embodiment, step 103 includes:

[0086] A shift-window self-attention mechanism is used to perform global dependency fusion on the enhanced multi-scale features;

[0087] The feature pyramid network is used to perform multi-scale processing on the features fused by global dependency, and output the fused multi-scale feature map. The multi-scale processing includes fusing feature information of different scales through bidirectional paths from top to bottom and bottom to top.

[0088] The shifted window self-attention mechanism involves dividing multi-scale features into several windows and alternating between normal windows and shifted windows. Multi-head self-attention computation is performed within each window to associate the long-distance dependency between temperature gradients and structural semantics.

[0089] It should be noted that the multi-scale features are divided into several windows, specifically 8×8 windows. A shifted window self-attention mechanism is used alternately in two layers: "normal window" and "shifted window," covering cross-window connections. Then, global dependency fusion, i.e., self-attention fusion, is performed: multi-head self-attention is applied within each window to correlate temperature gradients with structural semantics over long distances. The output from the previous step is reorganized back into the spatial structure to form a feature map containing global dependencies. Finally, a feature pyramid network is used to process the globally dependent fused features at multiple scales, outputting a fused multi-scale feature map. This multi-scale processing includes bidirectional paths from top to bottom and bottom to top, complementing the information from large and small targets. Ensure that P3 / P4 / P5 remain compatible with the input branches of the detection head after fusion.

[0090] Understandably, by using a shift window to obtain the global correlation of "band-like / point-like" heating through self-attention, and then performing multi-scale neck bidirectional fusion, it is possible to take into account both large-volume devices and small components, and reduce missed detections caused by occlusion.

[0091] Step 104: Perform target detection and anomaly determination on the fused multi-scale feature map using the improved YOLO detection head to obtain preliminary detection results.

[0092] In one embodiment, step 104: The improved YOLO detection head adopts a decoupled structure and includes classification and regression branches.

[0093] It should be noted that the improved YOLO detection head adopts a decoupled structure and includes classification and regression branches. The classification branch is only responsible for "likeness + whether it is abnormal," while the regression branch is only responsible for "where to place the bounding box and how large to place it." In essence, the improved YOLO detection head is a decision-maker that performs three tasks simultaneously: first, it identifies the equipment type (such as transformers, circuit breakers, cable joints, etc.); second, it determines whether it is abnormal (merging temperature abnormalities and appearance abnormalities into two categories, "abnormal / normal," without further subdivision); and third, it provides a precise location bounding box. See the explanation of the classification and regression branches below for details.

[0094] Step 1041: The classification branch, based on the fused multi-scale features, simultaneously predicts the confidence level of the target device's type and the confidence level of its appearance anomaly.

[0095] Specifically, the classification branch introduces a multi-label classification mechanism to extract the correlation features between equipment type and appearance anomalies from the fused features. For equipment type identification, the classification branch uses an improved Softmax activation function to calculate the type confidence score for each preset equipment category, such as transformers, circuit breakers, and cable joints, ensuring the distinguishability of features for different equipment types. For appearance anomaly judgment, the branch maps various appearance anomalies such as rust, damage, and deformation to a unified "anomaly" category, and outputs the appearance anomaly confidence score through a Sigmoid activation function, achieving synchronous prediction with equipment type. Both confidence scores are initially filtered through a dynamic threshold screening mechanism. When the type confidence score is higher than the first threshold (e.g., 0.7) and the appearance anomaly confidence score is higher than the second threshold (e.g., 0.65), the target equipment is judged to have an anomaly suspicion, providing a basis for subsequent comprehensive anomaly judgment.

[0096] Step 1042: The regression branch adopts an anchor-free regression method, which regresses the position offset and size of the target bounding box relative to the point on the fused multi-scale feature map, and outputs the position information of the bounding box.

[0097] It should be noted that the regression uses an anchor-free representation, allowing each point on the feature map to directly regress to the distance to the four sides of the target using its own coordinates as a reference, and then decode it into a bounding box. This method does not rely on pre-set template boxes, making it simpler and more efficient, and is especially suitable for small parts (such as terminals and connectors) and devices with diverse shapes in this field.

[0098] Step 1043: Combine type confidence, appearance anomaly confidence and bounding box location information to generate preliminary detection results containing several candidate bounding boxes.

[0099] It should be noted that candidate bounding boxes refer to rectangular regions in an image that may contain the target device, located by the model after collaborative prediction through the classification and regression branches. Each candidate bounding box is associated with a corresponding device type confidence score and appearance anomaly confidence score. These initially selected candidate boxes will serve as the basis for subsequent processing, providing selection targets for further improving detection accuracy and reducing false positive rates.

[0100] Understandably, step 104 employs an improved YOLO detector head with a decoupled structure, while using anchor-free distance regression to achieve fine localization. This makes the localization of small parts more accurate, the training process more stable, and the inference speed faster.

[0101] Step 105: Based on the comparable temperature baseline, perform post-processing and decision fusion on the preliminary detection results to generate and output structured alarm information.

[0102] In one embodiment, step 105 includes:

[0103] Step 1051: Perform non-maximum suppression processing on each candidate bounding box to remove candidate bounding boxes that overlap with each other and whose type confidence and appearance anomaly confidence are lower than the corresponding preset thresholds.

[0104] Understandably, only the highest score is retained for overlapping candidate bounding boxes (i.e., non-maximum suppression is performed to remove candidate bounding boxes that overlap with each other and whose type confidence and appearance anomaly confidence are lower than the corresponding preset thresholds) to avoid multiple alarms from a single device.

[0105] Step 1052: For the candidate bounding boxes retained after non-maximum suppression processing, obtain their corresponding regions in the infrared image, calculate the average temperature and peak temperature of the region, and compare them with the comparable temperature baseline to obtain the relative temperature rise.

[0106] It should be noted that the relative temperature rise is obtained by comparing it with the comparable temperature baseline obtained in step 101. This involves subtracting the standard average temperature and standard peak temperature of the corresponding equipment type under the same operating conditions from the comparable temperature baseline, thus obtaining a quantified relative temperature rise value. For example, if the average temperature of an electrical joint area is 58℃ and the peak temperature is 65℃, while the standard average temperature in its comparable temperature baseline is 42℃ and the standard peak temperature is 48℃, then the calculated relative temperature rise for this area is an average temperature rise of 16℃ and a peak temperature rise of 17℃.

[0107] Furthermore, those skilled in the art can also use statistical methods that resist extreme noise interference (such as replacing the mean with the median, removing outliers using the quartile method, etc.) to remove anomalous temperature values ​​that jump randomly, ensuring that the temperature statistics truly reflect the actual state of the equipment. They can also use visible light images to identify and mark interference areas such as reflections and glare that are not equipment anomalies (these areas are easily misjudged as anomalies). In subsequent temperature analysis or evidence fusion, the information in these areas can be ignored to avoid false alarms caused by artifacts. Combining the two methods ensures the reliability of temperature data and eliminates visual artifact interference, ultimately improving the accuracy of anomaly detection in distribution network equipment.

[0108] Step 1053: Compare the calculated average temperature, peak temperature, and relative temperature rise with the adaptive absolute threshold based on the equipment type and ambient temperature, and the relative threshold relative to the comparable temperature baseline, to generate an infrared temperature anomaly determination result.

[0109] Specifically, both absolute thresholds (adaptive to equipment type and ambient temperature) and relative thresholds (relative to background temperature fluctuations) are used simultaneously. For three-phase equipment, inter-phase comparisons are supplemented; a single phase with a significantly higher threshold is flagged as suspicious. It is understandable that different types of distribution network equipment (such as transformers, circuit breakers, and disconnectors) have significantly different insulation material temperature tolerances and normal operating temperature ranges. Therefore, differentiated absolute threshold benchmarks need to be preset according to the specific type of equipment. Meanwhile, changes in ambient temperature directly affect the heat dissipation efficiency and normal operating temperature of the equipment. For example, in high-temperature environments in summer, the overall temperature level of the equipment is generally higher than in low-temperature environments in winter. In this case, the absolute threshold needs to be dynamically corrected using data collected in real time by ambient temperature sensors to form an adaptive absolute threshold that matches the current environmental conditions. The relative threshold is determined by comparing the temperature data of the current detection area with comparable temperature baselines collected at the same time (such as the historical normal operating temperature of the equipment during the same period, the temperature of similar equipment in the same environment, and the temperature of non-heat-generating parts of the equipment). A reasonable allowable range for relative temperature rise is set. When the relative temperature rise exceeds this range, it is determined that there is a risk of temperature anomalies. For three-phase power equipment, due to the symmetry of its three-phase structure, the temperature of each phase should remain basically consistent during normal operation. If the temperature of one phase is significantly higher than that of the other two phases (exceeding the preset interphase temperature difference threshold), even if the temperature of that phase does not reach the absolute or relative threshold, it should be marked as a suspected anomaly for further investigation and analysis. Through the collaborative comparison of multi-dimensional thresholds, infrared temperature anomalies in distribution network equipment can be comprehensively and accurately identified.

[0110] Step 1054: The appearance anomaly confidence score of the retained candidate bounding boxes is weighted and fused with the infrared temperature anomaly determination result to generate a comprehensive anomaly confidence score and an anomaly determination conclusion. The generation rules for the comprehensive anomaly confidence score are as follows: when both are determined to be anomalies, the comprehensive anomaly confidence score is determined to be high; when only one is determined to be anomalies, the comprehensive anomaly confidence score is determined to be low, or an independent pending verification status mark is output.

[0111] Understandably, weighted fusion of appearance and temperature features can effectively improve the accuracy and reliability of anomaly detection. When both the appearance anomaly confidence score and the infrared temperature anomaly detection result of the candidate bounding box are abnormal, it indicates that the target exhibits significant abnormalities in both visual morphology and thermal characteristics. In this case, the overall anomaly confidence score is determined to be high, meaning the anomaly is relatively clear and requires priority processing. However, when only one of them is determined to be abnormal, there may be a misjudgment due to a single feature. Therefore, the overall anomaly confidence score is determined to be low, or an independent pending verification status marker is output, prompting relevant personnel to conduct further manual inspection and confirmation to avoid misjudgment or missed judgment due to the limitations of a single feature. Finally, the corresponding anomaly detection conclusion is generated based on the overall anomaly confidence score.

[0112] Step 1055: Output structured alarm information, including at least: device type, retained candidate bounding boxes, anomaly determination conclusion, appearance anomaly confidence level, average temperature, peak temperature and relative temperature rise.

[0113] Understandably, outputting structured alarm information provides standardized and traceable data support for subsequent anomaly handling and analysis. Specifically, the equipment type information clarifies the specific category of the object causing the anomaly, facilitating rapid location of the target device's position and functional attributes within the overall system; the retained candidate bounding boxes precisely mark the spatial coordinates of the anomaly area in the image, allowing technicians to intuitively view the specific location and extent of the anomaly; the anomaly determination directly reflects the results of the comprehensive analysis, providing a basis for decision-making regarding whether to initiate an emergency response or further investigation; and the appearance anomaly confidence level and temperature-related parameters (average temperature, peak temperature, and relative temperature rise) retain key data from the original feature analysis, which can be used to verify the rationality of the anomaly determination and provide quantitative evidence for further in-depth analysis of the anomaly's cause (e.g., whether the temperature anomaly stems from equipment failure or environmental factors). This structured information organization improves the readability and usability of alarm information, enabling anomaly information to be efficiently integrated into the equipment management platform or operation and maintenance system.

[0114] This invention provides an infrared-visible light fusion target detection method. First, a comparable temperature baseline is established. During training, asymmetric cross-modal enhancement is introduced. Then, a two-level cross-modal attention mechanism ("channel → space") combined with a backbone network and a feature pyramid network is used for global and multi-scale fusion, achieving robust focusing and anti-spoofing of small components such as connectors / terminals. Finally, an improved YOLO detection head outputs an integrated result of "device + location + anomaly," and temperature factors and visual scores (appearance anomaly confidence) are combined using a dual-track thresholding and concise fusion (considering device type, ambient temperature, and relative background fluctuations), effectively reducing false alarms caused by seasonal temperature increases and metallic reflections. This invention can operate in real-time at the edge, forming a closed loop of "discovery—grading—repair—review," improving the accuracy, real-time performance, and reliability of target detection, thus solving the problems of low detection accuracy and poor real-time performance in existing technologies under complex conditions.

[0115] The above is a target detection method based on infrared and visible light fusion provided in the embodiments of the present invention. The following is a target detection system based on infrared and visible light fusion provided in the embodiments of the present invention.

[0116] Please see Figure 4 An infrared-visible light fusion target detection system provided in this embodiment of the invention includes:

[0117] The construction unit 201 is used to perform spatiotemporal alignment and calibration on the synchronously acquired infrared and visible light images to obtain aligned image pairs and establish a comparable temperature baseline.

[0118] The enhancement unit 202 is used to extract multi-scale features from the infrared image and the visible light image in the aligned image pair using a dual-stream feature extraction network, and to fuse the multi-scale features using a cross-modal attention enhancement mechanism to obtain the enhanced multi-scale features.

[0119] The fusion unit 203 is used to perform global dependency fusion and multi-scale feature reconstruction on the enhanced multi-scale features to obtain the fused multi-scale feature map.

[0120] The first detection unit 204 is used to perform target detection and anomaly determination on the fused multi-scale feature map using an improved YOLO detection head, and obtain preliminary detection results.

[0121] The second detection unit 205 is used to perform post-processing and decision fusion on the preliminary detection results based on a comparable temperature baseline, and generate and output structured alarm information.

[0122] Furthermore, this embodiment of the invention also provides an infrared-visible light fusion target detection device, the device including a processor and a memory:

[0123] The memory is used to store program code and transmit the program code to the processor;

[0124] The processor is used to execute the steps of the infrared-visible light fusion target detection method as described in the above method embodiments, according to the instructions in the program code.

[0125] Furthermore, this embodiment of the invention also provides a computer-readable storage medium for storing program code for executing the infrared-visible light fusion target detection method described in the above method embodiments.

[0126] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0127] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0128] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0129] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0130] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0131] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for infrared-visible light fusion target detection, characterized in that, The method comprises: spatiotemporal alignment and calibration of the infrared image and the visible light image obtained by synchronous acquisition to obtain an aligned image pair, and establishment of a comparable temperature baseline; extracting multi-scale features from the infrared image and the visible light image in the aligned image pair respectively by using a double-flow feature extraction network, and performing fusion processing on the multi-scale features by using a cross-modal attention enhancement mechanism to obtain enhanced multi-scale features; performing global dependence fusion and multi-scale feature reconstruction on the enhanced multi-scale features to obtain a fused multi-scale feature map; performing target detection and anomaly determination on the fused multi-scale feature map by using an improved YOLO detection head to obtain a preliminary detection result; performing post-processing and decision fusion on the preliminary detection result based on the comparable temperature baseline to generate and output structured alarm information.

2. The infrared-visible light fused object detection method of claim 1, wherein, The spatiotemporal alignment and calibration of the infrared image and the visible light image obtained by synchronous acquisition to obtain an aligned image pair, and establishment of a comparable temperature baseline, comprises: aligning the infrared image and the visible light image obtained by synchronous acquisition at the pixel level to obtain an aligned image pair; converting the infrared grayscale of the infrared image into temperature, obtaining the mean value of background temperature and the fluctuation value of background temperature, and combining the aligned image pair to construct a comparable temperature baseline.

3. The infrared-visible fusion target detection method according to claim 2, wherein, The method further comprises, before the step of extracting multi-scale features from the infrared image and the visible light image in the aligned image pair respectively by using a double-flow feature extraction network: performing asymmetric data enhancement on the aligned image pair to obtain an enhanced image pair, which is used for training the double-flow feature extraction network and the cross-modal attention enhancement mechanism; wherein the asymmetric data enhancement comprises: dividing the infrared image and the visible light image in the aligned image pair into the same number of blocks, performing complementary occlusion processing on each block in the infrared image and the visible light image based on a preset probability, and adding noise to each block; the complementary occlusion processing comprises: when occlusion is applied to a certain region of any one of the infrared image and the visible light image, the corresponding region of the other image is not subjected to occlusion.

4. The infrared-visible fusion target detection method according to claim 3, characterized in that, The fusion processing comprises: in each channel of the double-flow feature extraction network for outputting the multi-scale features, first performing attention evaluation on the channel to highlight the feature channels related to device anomalies, and then performing spatial attention processing on the multi-scale features to focus on high-risk areas of the device to obtain enhanced multi-scale features.

5. The infrared-visible fusion target detection method according to claim 4, characterized in that, The method further comprises, before the step of performing global dependence fusion and multi-scale feature reconstruction on the enhanced multi-scale features to obtain a fused multi-scale feature map: performing global dependence fusion on the enhanced multi-scale features by using a shift window self-attention mechanism; performing multi-scale processing on the features subjected to global dependence fusion by using a feature pyramid network to output a fused multi-scale feature map, wherein the multi-scale processing comprises: fusing feature information of different scales by a bidirectional path from top to bottom and from bottom to top; The shift window self-attention mechanism is: dividing the multi-scale features into several windows, alternately using normal windows and shift windows, and performing multi-head self-attention calculation in each window to associate long-distance dependencies of temperature gradient and structure semantics.

6. The infrared-visible fusion target detection method according to claim 5, wherein, The improved YOLO detection head is used for target detection and abnormality judgment on the fused multi-scale features to obtain a preliminary detection result, including: The improved YOLO detection head adopts a decoupling structure and includes a classification branch and a regression branch; The classification branch synchronously predicts the type confidence and the appearance abnormality confidence of the target device based on the fused multi-scale features; The regression branch adopts an Anchor-free regression mode, and outputs the position information of the bounding box based on the position offset and the size of the bounding box relative to the point on the fused multi-scale feature map; The type confidence, the appearance abnormality confidence and the position information of the bounding box are combined to generate the preliminary detection result including several candidate bounding boxes.

7. The infrared-visible fusion target detection method according to claim 6, wherein, The preliminary detection result is post-processed and decision fusion is performed based on the comparable temperature baseline to generate and output structured alarm information, including: Non-maximum suppression is performed on each candidate bounding box to eliminate the candidate bounding boxes that are mutually overlapped and have a type confidence and an appearance abnormality confidence lower than a corresponding preset threshold; The corresponding region of the candidate bounding box retained after the non-maximum suppression processing in the infrared image is obtained, and the average temperature and the peak temperature of the region are calculated and compared with the comparable temperature baseline to obtain the relative temperature rise; The average temperature, the peak temperature and the relative temperature rise calculated are compared with an adaptive absolute threshold based on the device type and the environment temperature, and a relative threshold relative to the comparable temperature baseline to generate an infrared temperature abnormality judgment result; The appearance abnormality confidence of the retained candidate bounding box and the infrared temperature abnormality judgment result are weighted and fused to generate a comprehensive abnormality confidence and generate an abnormality judgment conclusion, wherein the generation rule of the comprehensive abnormality confidence includes: when both are determined to be abnormal, the comprehensive abnormality confidence is determined to be high-level; when only one of them is determined to be abnormal, the comprehensive abnormality confidence is determined to be low-level, or an independent pending review state mark is outputted; The structured alarm information is outputted, including at least: the device type, the retained candidate bounding box, the abnormality judgment conclusion, the appearance abnormality confidence, the average temperature, the peak temperature and the relative temperature rise.

8. An infrared-visible light fusion target detection system, characterized in that, The construction unit is configured to perform time-space alignment and calibration on the infrared image and the visible light image synchronously acquired to obtain an aligned image pair, and establish a comparable temperature baseline; The enhancement unit is configured to use a double-flow feature extraction network to extract multi-scale features from the infrared image and the visible light image in the aligned image pair respectively, and use a cross-modal attention enhancement mechanism to fuse and process the multi-scale features to obtain enhanced multi-scale features; The construction unit is configured to perform time-space alignment and calibration on the infrared image and the visible light image synchronously acquired to obtain an aligned image pair, and establish a comparable temperature baseline; The enhancement unit is configured to use a double-flow feature extraction network to extract multi-scale features from the infrared image and the visible light image in the aligned image pair respectively, and use a cross-modal attention enhancement mechanism to fuse and process the multi-scale features to obtain enhanced multi-scale features; A fusion unit is configured to perform global dependence fusion and multi-scale feature reconstruction on the enhanced multi-scale features to obtain a fused multi-scale feature map; A first detection unit is configured to perform target detection and anomaly determination on the fused multi-scale feature map by using an improved YOLO detection head to obtain a preliminary detection result; A second detection unit is configured to perform post-processing and decision fusion on the preliminary detection result based on the comparable temperature baseline to generate and output structured alarm information.

9. An infrared-visible light fusion target detection device, characterized by, The device comprises a processor and a memory: The memory is configured to store program code and transmit the program code to the processor; The processor is configured to execute the infrared and visible light fusion target detection method according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium is configured to store program code, and the program code is configured to execute the infrared and visible light fusion target detection method. The computer readable storage medium is configured to store program code, and the program code is configured to execute the infrared and visible light fusion target detection method.

Images