Multi-modal and physically constrained power equipment run-dry detection method, system and medium

Abstract

The application discloses a kind of multi-modal and physical restraint's power equipment run, method, system and medium of leakage detection.It comprises: obtaining the double mode image sequence and working condition data of inspection scene;Space registration and quantification registration reliability;Combining structure prior, through target detection model, extract candidate box;According to candidate box, generate multiple prompt, drive general image segmentation big model to output final segmentation mask;Based on mask, extract morphology, time sequence evolution, heat and gravity diffusion physical consistency features, and dynamically adjust feature fusion weight and classification threshold to output discrimination result;When registration misalignment or infrared quality is poor, trigger single visible light time sequence sliding window discrimination mode fallback.The application eliminates shadow, water disturbance, greatly reduces false alarm 4 rate, and guarantees all-weather inspection continuity.

Description

Technical Field

[0001] This invention relates to the fields of intelligent inspection of power systems, computer vision and artificial intelligence, and in particular to a method, system and medium for detecting leaks in power equipment under multimodal and physical constraints. Background Technology

[0002] During long-term operation, power equipment is prone to leaks due to environmental factors and equipment aging, such as transformer oil leaks, pipe seepage, and equipment seal failures. These problems typically manifest as irregular liquid areas on equipment surfaces or the ground, which may gradually spread over time. If not detected and addressed promptly, they can easily lead to equipment malfunctions or even safety accidents. Therefore, efficient and accurate detection of leaks is of great importance. Currently, power line inspections mainly rely on manual inspections or intelligent analysis methods based on video surveillance. Manual inspections suffer from low efficiency, reliance on experience, and inability to achieve continuous monitoring. With the development of computer vision technology, deep learning-based object detection methods (such as the YOLO model) are increasingly being applied to power line inspection scenarios, enabling automatic identification of equipment status by detecting abnormal areas in images.

[0003] However, existing technologies still have the following shortcomings in detecting leaks: (1) Lack of the ability to express irregular targets in detail Leaking and spilling areas are usually non-rigid targets with unclear boundaries and irregular shapes. Existing target detection methods usually only output rectangular bounding boxes, which cannot accurately describe the true shape and spatial distribution characteristics of the target, making it difficult to make effective judgments based on morphological features.

[0004] (2) Susceptible to environmental disturbance factors In actual power inspection scenarios, there are many interfering factors that are similar in appearance to the leak area, such as: shadow areas created by equipment or structures, water stains formed by rainwater or ground water, and reflections or dark areas caused by changes in lighting. The aforementioned interference areas are highly similar to leaks and spills in terms of color, brightness, and texture, making it difficult for traditional detection methods to effectively distinguish them, resulting in a high false alarm rate.

[0005] (3) Lack of dynamic analysis capabilities in the time dimension Leaks and spills typically exhibit a dynamic characteristic of gradually spreading over time, while shadowed or static stain areas remain relatively stable. However, existing methods mostly rely on single-frame images for judgment, lacking the ability to model the temporal evolution characteristics of the target area and failing to utilize the crucial discriminative information of "diffusion behavior."

[0006] (4) Lack of multimodal information fusion capability Traditional visual inspection methods mainly rely on visible light images, which cannot obtain the temperature or thermal radiation characteristics of the target area. In real-world scenarios, oil leaks may be accompanied by temperature changes, while shadowed areas do not change the heat distribution. Relying solely on single-modal information makes it difficult to distinguish different types of targets from a physical attribute perspective, thus limiting further improvements in detection accuracy.

[0007] (5) The segmentation model has limitations when applied independently. In recent years, general image segmentation models (such as SegmentAnythingModel (SAM)) have demonstrated strong pixel-level segmentation capabilities, enabling fine segmentation of any target in an image. However, these models lack automatic target region localization capabilities, making them difficult to directly apply to large-scale scene analysis in power line inspection. In summary, existing technologies for leak detection suffer from problems such as coarse localization, susceptibility to interference, and a lack of dynamic analysis and multimodal fusion capabilities, making it difficult to meet the demands for high-precision, low-false-alarm intelligent detection in complex power line inspection environments.

[0008] Therefore, this invention proposes a comprehensive detection method that integrates target detection, pixel-level segmentation, time series analysis, and multimodal information to improve the accuracy and robustness of leak detection. Summary of the Invention

[0009] This invention discloses a multimodal and physically constrained method, system, and medium for detecting leaks in power equipment. The method first uses the YOLO model to detect suspected shadows or abnormally dark areas in visible light images, obtaining candidate regions. Then, the candidate regions are cropped and, combined with prompts, input into the SegmentAnythingModel (SAM) model for pixel-level segmentation, generating a binary segmentation mask for the target region. Based on the segmentation mask, morphological features such as area, perimeter, irregularity, and compactness are extracted. Furthermore, time-series features are constructed by combining video sequences to analyze the area changes and diffusion trends of the target region. Simultaneously, infrared images are used to obtain temperature difference information between the target region and the background region. Finally, morphological features, time-series features, and temperature features are fused to construct a comprehensive discrimination model, effectively distinguishing leaks from interference factors such as shadows and rainwater. This invention, by introducing pixel-level segmentation, dynamic diffusion modeling, and a multimodal fusion mechanism, improves the accuracy and robustness of leak detection in complex environments, effectively reducing false alarm rates and possessing significant engineering application value.

[0010] To achieve the above objectives, the present invention proposes the following technical solution: This invention discloses a multimodal and physically constrained method for detecting leaks in power equipment, comprising the following steps: Step A: Multimodal data acquisition; acquire visible light image sequences, infrared image sequences, and optional environmental condition data for the same inspection scenario; The visible light image sequence is used to characterize the appearance texture, color, shadow and liquid trace information of the power equipment and its surrounding area; the infrared image sequence is used to characterize the thermal radiation distribution information of the target area and its surrounding background area; and the environmental condition data includes ambient temperature, humidity, rainfall status, light intensity, wind speed, equipment load and equipment operating status. Step B: Adaptive spatiotemporal registration; Perform time synchronization and spatial geometric registration on the visible light image sequence and the infrared image sequence, and quantize the dual-modal registration reliability. The time synchronization includes performing frame-level correspondence between two types of image sequences based on hardware trigger signals, unified timestamps, or inter-frame time offset estimation. The spatial registration includes obtaining the homography matrix, affine transformation matrix, or nonlinear mapping relationship between the visible light image coordinate system and the infrared image coordinate system based on calibration board feature points, device structural feature points, preset installation parameters, or image matching features. The registration reliability is calculated based on reprojection error, feature matching residual, bimodal edge consistency, and historical registration stability. Step C: Coarse positioning of component structure prior; Based on the prior of the equipment structure, determine the high-risk concern area, and within the high-risk concern area, call the target detection model to perform candidate detection of suspected leakage targets and output the candidate bounding boxes of suspected leakage targets; The prior knowledge of the equipment structure includes transformer oil tanks, flange connections, valves, radiator interfaces, pipe connections, bottom of the tank, sealing edges, bushing roots, oil drains, water drains, equipment support surfaces, and areas along paths where liquids may seep or spread due to gravity. The high-risk areas of concern are generated through manual configuration, equipment structure template mapping, CAD / BIM model projection, identification of key component detection models, or statistical analysis of historical defect locations. Step D: Fine segmentation with multiple prompts; dynamically generate spatial box prompts and point prompts based on the candidate boxes, and generate a temporal propagation mask as a time prompt by combining the segmentation status of historical frames; combine the spatial box prompts, point prompts, time prompts and the current frame image into a prompt-based segmentation model, output the pixel-level initial segmentation mask of the target region, and perform morphological post-processing restoration to obtain the final segmentation mask; The spatial bounding box prompt is formed by expanding the candidate bounding box according to a preset expansion ratio; the point prompt includes the center point of the candidate bounding box, the foreground sampling point of the candidate region, and the sampling point of the boundary uncertain region; the time prompt includes the segmentation mask of the previous frame, the candidate bounding box of the previous frame, the target motion trajectory, the optical flow propagation mask, and the historical stable region. The prompting segmentation model performs pixel-level segmentation of suspected leaking areas based on the spatial frame prompts, point prompts, and time prompts, and performs post-processing operations such as connected component filtering, hole filling, boundary smoothing, morphological opening and closing operations, or maximum connected component preservation on the segmentation results. Step E: Multidimensional physical feature extraction; Based on the final segmentation mask and the infrared image sequence, extract the morphological topological features, temporal evolution features, thermal trace features, structural location features, and physical consistency constraint features that conform to the fluid diffusion law of the target region; The morphological topological features include area, perimeter, compactness, irregularity, convex hull area ratio, rectangularity, circularity, boundary complexity, and skeleton branch density. The temporal evolution features include area change rate, centroid displacement, equivalent radius diffusion rate, duration of frames, diffusion direction stability, and target trajectory continuity. The thermal trace features include the average temperature of the target area, the average temperature difference between the target area and the annular background area, the temperature gradient, the rate of temperature change, and the environmental compensation temperature difference. The structural location features include the distance between the target area and high-risk components, the positional relationship of the target area relative to the bottom of the equipment or the sealing edge, and whether the target area is located on a possible liquid seepage path. The physical consistency constraint features include consistency between the target diffusion direction and the gravity direction, consistency between the target area growth over time, spatial consistency between the target thermal anomaly and the visible light anomaly, consistency between the target location and the location of the suspected leak source, and consistency between the target duration and the medium diffusion law. Step F: Adaptive weighted fusion discrimination; based on registration confidence, modal quality, and the aforementioned features, fusion discrimination is performed to obtain anomaly category, development stage, risk level, target location, segmentation mask, and confidence level; wherein, the modal quality includes visible light image clarity, infrared image clarity, occlusion degree, overexposure degree, underexposure degree, thermal image noise level, and rain / fog interference degree; the fusion discrimination is implemented based on a rule model, logistic regression model, support vector machine model, random forest model, gradient boosting model, lightweight neural network model, or multi-model weighted fusion model, and the fusion weights of each modal feature are dynamically adjusted according to the registration confidence and modal quality; Step G: Dynamic Modal Backoff Control: During the inspection process, the availability of infrared modes and the registration reliability are monitored in real time. When the preset backoff conditions are triggered, the system automatically switches to the single-mode visible light temporal discrimination mode and adjusts the output threshold, risk level or confidence level according to the infrared mode missing situation. The visible light temporal discrimination mode judges at least based on the morphological characteristics, temporal evolution characteristics, structural position characteristics and physical consistency characteristics of the target area, and marks the modal backoff status in the output results for subsequent manual review or inspection work order processing.

[0011] As a further improvement of the present invention, in step B, the specific process of time synchronization is as follows: Let the first The acquisition time for a frame of visible light image is , No. The acquisition time for one frame of infrared image is The optimal synchronization frame pair is determined by retrieving the infrared image sequence using the following formula that minimizes the time difference: When satisfied At that time, the first Frame visible light image and the first The infrared image frame is locked as a synchronized frame pair; among which, The preset maximum time deviation threshold; The time synchronization is achieved by using a hardware trigger signal, a unified timestamp, or a combination of both. If a hardware trigger signal is used, the visible light acquisition unit and the infrared acquisition unit receive the same trigger pulse signal and use the images acquired within the same trigger period as a synchronization frame pair. If a unified timestamp is used, frame-level alignment is completed based on a unified time reference provided by the acquisition terminal, edge computing node, or time synchronization module.

[0012] The spatial registration is performed by obtaining a mapping matrix using calibration board feature points, equipment structural feature points, installation parameters, or a combination thereof. The spatial registration, under the condition of planar approximation, uses the homography matrix H to establish the pixel points in the visible light image coordinate system. The corresponding pixel in the infrared image coordinate system is Mapping relationship between them: Right now: The mapped coordinates in the infrared image are obtained by homogeneous coordinate normalization. When the installation angle change is small or the dual-mode sensor installation structure is stable, an affine transformation matrix can also be used. Perform approximate registration: The mapping matrix is ​​calculated using no fewer than four sets of calibration points, equipment corner points, flange edge points, enclosure edge points, radiator outline points, or other structural feature points.

[0013] The registration reliability is obtained based on feature matching residuals, reprojection errors, or the similarity of corresponding regions in the two modes. Let there be a total of... Group matching feature points, the first Group of visible light feature points The corresponding infrared feature points are The reprojection error is defined as: in, This represents the homogeneous coordinate normalization function. Furthermore, the reprojection error can be normalized to the registration confidence level: The registration reliability The calculation method is as follows: Assume there are N sets of matching feature points, and the normalized reprojection error confidence level is... ,in, For reprojection error, To determine the reprojection error scale parameter, gradient features of the corresponding regions of the two modes are obtained through edge extraction or gradient enhancement, respectively. and And calculate the similarity of the corresponding regions of the two modes. : The final registration confidence level is expressed as: ,in, These are the weighting coefficients. To prevent constants with a denominator of zero, when Below the preset threshold If the current dual-mode registration is deemed unreliable, the weight of infrared features in the fusion discrimination is reduced or the system is switched to visible light temporal discrimination mode.

[0014] As a further improvement of the present invention, in step C, the prior art of the equipment structure includes flange connections, valves, oil tanks, radiator joints, pipe connections, sealing edges, bottom of the housing, sleeve roots, oil drains, water drains, and possible liquid seepage paths, and the high-risk concern areas are generated through template mapping, key component detection, CAD / BIM structural projection, or manual configuration.

[0015] The set of high-risk areas of concern is represented as follows: ,in, The number of high-risk components. For the i-th candidate region output by the YOLO series target detection model. Its structural prior score Defined as: in, Indicates candidate region High-risk areas The intersection and union ratio, This represents the minimum distance between the center of the candidate region and the high-risk component region. For distance normalization parameters, , These are the weighting coefficients; If the target area seeps downwards along the direction of gravity, let the suspected leak source be... target area centroid When in the image coordinate system When the axial direction is consistent with the direction of gravity, the consistency score of the infiltration path is increased. Defined as: in, For indicator functions, This is a parameter for normalizing the lateral offset. If the target area is located below the suspected leak source and has a small lateral offset, the consistency score of the infiltration path is high; if the target area is located above a high-risk component or far from the leak source path, its anomaly discrimination weight is reduced.

[0016] As a further improvement of the present invention, in step D, the time prompt includes the previous frame segmentation mask, the previous frame candidate box, the target trajectory box, or the optical flow propagation mask; and after obtaining the segmentation mask, connected component filtering, hole filling, boundary smoothing, morphological opening and closing operations, or maximum connected region retention post-processing operations are also performed.

[0017] For video scenes, the timing cue includes an optical flow propagation mask. Its final segmentation mask is based on the previous frame. And the optical flow field from the previous frame to the current frame is Pixel position propagation operation based on optical flow field The calculation yielded: The input prompt set of the prompt-based segmentation model can be represented as: in, For a set of dot hints, Provide target trajectory hints or historical area hints.

[0018] The segmentation probability map of the prompt-based general image segmentation model is as follows: ,in For the current frame inspection image, This is a tooltip that has been expanded by a preset expansion ratio. Thresholding the segmentation probability map yields an initial binary mask. ; in, The segmentation threshold is used; the morphological post-processing involves applying the initial binary mask... Perform morphological opening and closing operations to obtain the post-processing mask. ,in, , As a structuring element, the final set of connected components in the post-processing mask. Perform area filtering. in, For the first Area of ​​each connected region This is the minimum area threshold.

[0019] Preferably, when multiple connected components exist, the connected component with a size greater than the minimum area threshold that is closest to the center of the candidate box, has the largest area, or has the highest overlap with the propagation mask of the previous frame is retained to obtain the final segmentation mask. This is to improve the continuity of segmentation results in video scenarios.

[0020] As a further improvement of the present invention, in step E, the temporal evolution characteristics and physical consistency characteristics include at least two of the following: area growth rate, centroid displacement, equivalent radius diffusion rate, consistency between the infiltration direction and the gravity direction, continuous frame duration, distance from the suspected leakage source, and average temperature difference between the target area and the annular background area.

[0021] The specific process for extracting the temporal evolution features is as follows: based on the final segmentation mask... Calculate the target area in frame t: in, This is the pixel area conversion factor. Define the image region; and calculate the target centroid. ;in: The extracted temporal evolution features include the target region area growth rate. Displacement of the center of mass Equivalent radius diffusion rate and continuous frame duration : The target area growth rate is: The target centroid displacement is: The equivalent radius of the target region is: The equivalent radius diffusion rate is: in, This is the time interval between two adjacent frames. If and If the condition is true over multiple consecutive frames, it indicates that the target region has a continuous diffusion trend.

[0022] The duration of consecutive frames is defined as: in, To prevent constants with a denominator of zero, The time interval between two adjacent frames. This represents the number of frames in which the target region is stably detected and segmented within a continuous analysis window.

[0023] As a further improvement of the present invention, the specific process of extracting the thermodynamic features and physical consistency features in step E is as follows: For infrared temperature features, the current segmentation mask will be used. Mapping to infrared images to determine infrared target areas And construct a ring-shaped background area on its outer side. .

[0024] The average temperature of the target area is: The average temperature of the background area is: Calculate the average temperature of the target area Average temperature of the annular background region The difference is used to obtain the average temperature difference. ; and in combination with ambient temperature Introducing environmental compensation for temperature difference: ,in, For reference to ambient temperature, This is the compensation coefficient; The consistency between the infiltration direction and the gravity direction is calculated by the angle between the target center of mass movement direction and the preset gravity direction.

[0025] The physical consistency feature is based on the consistency score between the infiltration direction and the gravity direction. Distance Consistency Score Jointly construct a comprehensive physical consistency score : The directional consistency score is: when The closer the value is to 1, the closer the target's direction of motion is to the direction of gravitational infiltration; when... When the direction of the target movement is inconsistent with the infiltration pattern, the score for detecting leakage can be reduced.

[0026] Let the coordinates of the suspected leak source be... The centroid of the target region is The distance between the target area and the suspected leak source is: Its distance consistency score is: The overall physical consistency score is: in, Let the target centroid displacement vector be... Let s be the preset gravity direction vector, and s be the coordinates of the suspected leak source location. For distance scale parameters, to These are the weighting coefficients. This is a normalization function. The comprehensive physical consistency score is used to characterize whether candidate targets simultaneously satisfy the physical laws of leakage, namely, "continuous diffusion, reasonable infiltration direction, proximity to high-risk leakage sources, thermal differences, and continuous existence".

[0027] As a further improvement of the present invention, in step F, the environmental condition data includes ambient temperature, humidity, rainfall status, light intensity, and equipment load; the fusion discrimination constructs a feature vector based on each feature and dynamically adjusts the feature weights. Let the environmental condition vector be: in, For ambient temperature, For humidity, It is currently experiencing rainfall. For illuminance, For equipment load.

[0028] Let the fusion discriminative feature vector be: in, For morphological characteristics, As a temporal evolution feature, Thermal characteristics, For structural location features, This is a physical consistency characteristic.

[0029] Based on registration reliability Visible light quality Infrared quality and environmental conditions The fusion weight vector of each modality feature is dynamically generated through a neural network or preset rules: in, and For trainable parameters or preset rule parameters, the fusion discrimination score can be expressed as: When the infrared image quality is low, the registration reliability is low, or the environmental reflection interference is strong, reduce the weight of thermal features: When the rainfall is considered effective or the humidity exceeds a preset threshold, the threshold for identifying water stains and condensation should be increased or their alarm confidence level reduced. When the equipment load is high, resulting in a significant overall temperature rise, environmental compensation for temperature difference should be implemented. Replace the original temperature difference .

[0030] According to rainfall status and humidity Adaptively adjust the final classification threshold: in The baseline classification threshold , , For adjustment coefficients; The final judgment result is: in, This indicates an abnormality caused by leaks or spills. This indicates that the target is either not a leaking or disruptive target.

[0031] As a further improvement of the present invention, in step F, the anomaly category includes at least oil leakage, seepage, water stains, condensation, shadows, reflections, and suspected ventilation; the development stage includes at least initial, developing, ongoing, and severe; and the risk level includes at least low risk, medium risk, and high risk.

[0032] Let the probability of an anomaly classification be: The exception category output is: in, This indicates categories such as oil leaks, seepage, water stains, condensation, shadows, reflections, or suspected escaping.

[0033] The final judgment result also includes: Based on duration Area growth rate Equivalent radius diffusion rate Environmental compensation for temperature difference And the overall physical consistency score Score for development stages , And according to the above The size of the abnormality divides its development into four stages: initial stage, development stage, ongoing stage, and severe stage. Risk level can be determined comprehensively based on anomaly category, development stage, equipment importance, and confidence level. A score is assigned based on the development stage. Anomaly confidence level Equipment importance coefficient And the overall physical consistency score Calculate risk score : Where b1 to b4 are weighting coefficients; and based on the aforementioned risk score Based on the relationship between the risk level and the preset risk level threshold, the risk level is divided into low risk, medium risk and high risk.

[0034] in, , This represents the risk level threshold. For targets at a high risk level, the system outputs alarm information, target location, segmentation mask, suspected leak source, development stage, and review suggestions; for medium-risk targets, the system outputs instructions for continuous observation or manual review; for low-risk targets, the system can record information but will not trigger a strong alarm.

[0035] Based on the above method, this invention also proposes a multimodal and physically constrained power equipment leakage detection system, comprising: A dual-modal acquisition module is used to acquire visible light image sequences and infrared image sequences under the same inspection scenario; The synchronization registration module is used to complete time synchronization, spatial registration, and registration reliability evaluation. Specifically, the synchronization registration module generates registration reliability based on synchronization time deviation, reprojection error, feature matching residual, and the similarity of corresponding regions in both modes. ; The structural prior module generates high-risk areas of concern based on flange connections, valves, oil conservators, radiator joints, sealing edges, the bottom of the tank, and potential liquid seepage paths, and calculates the structural prior score between candidate targets and high-risk areas of concern. ; The candidate detection module is used to call the YOLO series of object detection models to obtain candidate boxes within the high-risk interest area; The cue segmentation module is used to generate a pixel-level segmentation mask for the target region based on box cue, point cue, and time cue, and to perform morphological post-processing; wherein, the time cue includes the segmentation mask of the previous frame, the candidate box of the previous frame, the target trajectory box, or the optical flow propagation mask; The feature extraction module is used to extract morphological features, temporal evolution features, thermal features, structural location features, and physical consistency features from the post-processed segmentation mask; wherein, the physical consistency features include at least two of the following: area growth rate, equivalent radius diffusion rate, infiltration direction consistency, distance from the leakage source, and annular background average temperature difference; The fusion discrimination module is used to calculate the fusion score based on registration confidence, modal quality, environmental condition data, and various features. It adaptively adjusts feature weights and classification thresholds, and outputs anomaly category, development stage, risk level, target location, segmentation mask, and confidence level. The rollback and alarm module is used to automatically trigger the mode rollback state to switch to visible light timing discrimination mode when the infrared mode is unavailable, the infrared image quality is lower than a preset threshold, or the registration confidence is lower than a preset confidence threshold, and output inspection alarm records containing review suggestions.

[0036] Furthermore, the present invention also proposes a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the multimodal and physically constrained power equipment leakage detection method described in any of the above claims.

[0037] Beneficial effects: (1) Traditional detection methods (such as conventional YOLO series networks) can only output rectangular candidate boxes. Since liquid spills and leaks often exhibit non-rigid, blurred boundaries, and irregular topological structures, the rectangular boxes contain a large amount of complex background noise. This invention adopts a cascaded architecture of "coarse localization + fine segmentation" to quickly lock candidate boxes through the YOLO network, and creatively combines spatial box hints, point hints, and time hints (optical flow propagation mask M) generated based on dense optical flow fields. t Together, they drive a general image segmentation model. This not only accurately extracts the pixel-level true contours of the leaked medium, but also uses temporal history priors to guide the segmentation of the current frame, effectively solving the problems of single-frame mask breakage and poor edge fitting caused by wind shaking and transient reflections.

[0038] (2) In complex power scenarios such as actual substations, building shadows, rainwater accumulation, metal reflections, and old stains on equipment often cause serious "illusion" false alarms in purely data-driven neural networks due to similar colors or infrared temperature rise characteristics. This invention does not rely on single image pixel features, but for the first time introduces fluid dynamics and thermodynamic physical laws into the algorithm constraints: by calculating the cosine of the angle between the centroid displacement vector and the gravity vector, it verifies whether the leaked material seeps down with gravity; by measuring the relative spatial distance from the geometric centroid of the target to the inherent leakage source such as equipment flanges and valves, it determines the rationality of the leakage logic; by automatically constructing an "outer annular background area" in the infrared image, calculating the "target-background temperature difference" and introducing real-time ambient temperature compensation, it accurately eliminates background radiation temperature rise interference. The above multiple physical quantities are jointly constructed into a comprehensive physical consistency score S. phy This ensures that the algorithm's judgments conform to objective physical laws, reduces the false alarm rate to an extremely low level in the industry, and has excellent engineering interpretability.

[0039] (3) Conventional multimodal fusion inspection systems suffer from rigid registration dependence. In cases of strong winds, lens fogging, or extremely poor infrared signal quality at night, registration failure directly leads to system paralysis or incorrect report output. This invention quantifies and calculates the reliability C of dual-modal registration, which includes reprojection error and gradient mutual information similarity. reg Once registration misalignment or infrared lens obstruction is detected, the system can adaptively, seamlessly, and proactively switch to a single-modal sliding window fallback state in the "visible light temporal discrimination mode," relying solely on the consistency of morphology, temporal propagation, and infiltration direction under visible light to perform independent judgments. This adaptive fault-tolerant control technology maximizes the continuity of equipment inspections under complex outdoor conditions.

[0040] (4) Power equipment inspection faces highly dynamic environmental interferences such as rainfall, high humidity, strong sunlight, and high load current of equipment. This invention constructs a decision feature vector by combining these operating condition data with registration reliability. It not only dynamically solves the optimal fusion weights of morphological, temporal, thermal, structural, and physical consistency features through Softmax, but also adaptively slides the classification threshold. For example, in rainy and foggy weather with high humidity, the classification threshold is automatically raised to suppress rain noise interference; when the registration reliability is high, the threshold is reasonably lowered to improve the sensitivity of capturing weak leaks. The system achieves adaptive optimal working state under different weather and load conditions such as "sunny, rainy, daytime, and nighttime".

[0041] (5) This invention not only provides a binary classification of "leakage / non-leakage" determination, but also deeply integrates the abnormal duration T dur Area growth rate G A Diffusion rate V r Thermal compensation temperature difference ΔTc and physical consistency score S phy A comprehensive stage-based scoring system is established, finely dividing leakage anomalies into four stages: initial, development, ongoing, and severe. Furthermore, by combining this with equipment importance coefficients, a system risk assessment classification is calculated. This mechanism allows the smart grid management system to output strong alarms, conduct continuous monitoring, or directly dispatch manual review work orders for alarms of different risk levels, achieving a refined upgrade of closed-loop management throughout the entire lifecycle of power equipment. Attached Figure Description

[0042] Figure 1 This is a flowchart of the present invention; Figure 2 SAM3 segmentation results Figure 1 ; Figure 3 SAM3 segmentation results Figure 2 ; Figure 4 For infrared registration results Figure 1 ; Figure 5 For infrared registration results Figure 2 ; Figure 6 This is an example of the result of the present invention; Figure 7 This is example two of the results of the present invention. Detailed Implementation

[0043] The present invention will now be described in detail with reference to the accompanying drawings and specific implementation examples.

[0044] This invention discloses a multimodal and physically constrained method for detecting leaks in power equipment, comprising the following steps: Step A: Multimodal data acquisition; acquire visible light image sequences, infrared image sequences, and optional environmental condition data for the same inspection scenario; The visible light image sequence is used to characterize the appearance texture, color, shadow and liquid trace information of the power equipment and its surrounding area; the infrared image sequence is used to characterize the thermal radiation distribution information of the target area and its surrounding background area; and the environmental condition data includes ambient temperature, humidity, rainfall status, light intensity, wind speed, equipment load and equipment operating status. Step B: Adaptive spatiotemporal registration; Perform time synchronization and spatial geometric registration on the visible light image sequence and the infrared image sequence, and quantize the dual-modal registration reliability. The time synchronization includes performing frame-level correspondence between two types of image sequences based on hardware trigger signals, unified timestamps, or inter-frame time offset estimation. The spatial registration includes obtaining the homography matrix, affine transformation matrix, or nonlinear mapping relationship between the visible light image coordinate system and the infrared image coordinate system based on calibration board feature points, device structural feature points, preset installation parameters, or image matching features. The registration reliability is calculated based on reprojection error, feature matching residual, bimodal edge consistency, and historical registration stability. Step C: Coarse positioning of component structure prior; Based on the prior of the equipment structure, determine the high-risk concern area, and within the high-risk concern area, call the target detection model to perform candidate detection of suspected leakage targets and output the candidate bounding boxes of suspected leakage targets; The prior knowledge of the equipment structure includes transformer oil tanks, flange connections, valves, radiator interfaces, pipe connections, bottom of the tank, sealing edges, bushing roots, oil drains, water drains, equipment support surfaces, and areas along paths where liquids may seep or spread due to gravity. The high-risk areas of concern are generated through manual configuration, equipment structure template mapping, CAD / BIM model projection, identification of key component detection models, or statistical analysis of historical defect locations. Step D: Fine segmentation with multiple prompts; dynamically generate spatial box prompts and point prompts based on the candidate boxes, and generate a temporal propagation mask as a time prompt by combining the segmentation status of historical frames; combine the spatial box prompts, point prompts, time prompts and the current frame image into a prompt-based segmentation model, output the pixel-level initial segmentation mask of the target region, and perform morphological post-processing restoration to obtain the final segmentation mask; The spatial bounding box prompt is formed by expanding the candidate bounding box according to a preset expansion ratio; the point prompt includes the center point of the candidate bounding box, the foreground sampling point of the candidate region, and the sampling point of the boundary uncertain region; the time prompt includes the segmentation mask of the previous frame, the candidate bounding box of the previous frame, the target motion trajectory, the optical flow propagation mask, and the historical stable region. The prompting segmentation model performs pixel-level segmentation of suspected leaking areas based on the spatial frame prompts, point prompts, and time prompts, and performs post-processing operations such as connected component filtering, hole filling, boundary smoothing, morphological opening and closing operations, or maximum connected component preservation on the segmentation results. Step E: Multidimensional physical feature extraction; Based on the final segmentation mask and the infrared image sequence, extract the morphological topological features, temporal evolution features, thermal trace features, structural location features, and physical consistency constraint features that conform to the fluid diffusion law of the target region; The morphological topological features include area, perimeter, compactness, irregularity, convex hull area ratio, rectangularity, circularity, boundary complexity, and skeleton branch density. The temporal evolution features include area change rate, centroid displacement, equivalent radius diffusion rate, duration of frames, diffusion direction stability, and target trajectory continuity. The thermal trace features include the average temperature of the target area, the average temperature difference between the target area and the annular background area, the temperature gradient, the rate of temperature change, and the environmental compensation temperature difference. The structural location features include the distance between the target area and high-risk components, the positional relationship of the target area relative to the bottom of the equipment or the sealing edge, and whether the target area is located on a possible liquid seepage path. The physical consistency constraint features include consistency between the target diffusion direction and the gravity direction, consistency between the target area growth over time, spatial consistency between the target thermal anomaly and the visible light anomaly, consistency between the target location and the location of the suspected leak source, and consistency between the target duration and the medium diffusion law. Step F: Adaptive weighted fusion discrimination; based on registration confidence, modal quality, and the aforementioned features, fusion discrimination is performed to obtain anomaly category, development stage, risk level, target location, segmentation mask, and confidence level; wherein, the modal quality includes visible light image clarity, infrared image clarity, occlusion degree, overexposure degree, underexposure degree, thermal image noise level, and rain / fog interference degree; the fusion discrimination is implemented based on a rule model, logistic regression model, support vector machine model, random forest model, gradient boosting model, lightweight neural network model, or multi-model weighted fusion model, and the fusion weights of each modal feature are dynamically adjusted according to the registration confidence and modal quality; Step G: Dynamic Modal Backoff Control: During the inspection process, the availability of infrared modes and the registration reliability are monitored in real time. When the preset backoff conditions are triggered, the system automatically switches to the single-mode visible light temporal discrimination mode and adjusts the output threshold, risk level or confidence level according to the infrared mode missing situation. The visible light temporal discrimination mode judges at least based on the morphological characteristics, temporal evolution characteristics, structural position characteristics and physical consistency characteristics of the target area, and marks the modal backoff status in the output results for subsequent manual review or inspection work order processing.

[0045] As a further improvement of the present invention, in step B, the specific process of time synchronization is as follows: Let the first The acquisition time for a frame of visible light image is , No. The acquisition time for one frame of infrared image is The optimal synchronization frame pair is determined by retrieving the infrared image sequence using the following formula that minimizes the time difference: When satisfied At that time, the first Frame visible light image and the first The infrared image frame is locked as a synchronized frame pair; among which, The preset maximum time deviation threshold; The time synchronization is achieved by using a hardware trigger signal, a unified timestamp, or a combination of both. If a hardware trigger signal is used, the visible light acquisition unit and the infrared acquisition unit receive the same trigger pulse signal and use the images acquired within the same trigger period as a synchronization frame pair. If a unified timestamp is used, frame-level alignment is completed based on a unified time reference provided by the acquisition terminal, edge computing node, or time synchronization module.

[0046] The spatial registration is performed by obtaining a mapping matrix using calibration board feature points, equipment structural feature points, installation parameters, or a combination thereof. The spatial registration, under the condition of planar approximation, uses the homography matrix H to establish the pixel points in the visible light image coordinate system. The corresponding pixel in the infrared image coordinate system is Mapping relationship between them: Right now: The mapped coordinates in the infrared image are obtained by homogeneous coordinate normalization. When the installation angle change is small or the dual-mode sensor installation structure is stable, an affine transformation matrix can also be used. Perform approximate registration: The mapping matrix is ​​calculated using no fewer than four sets of calibration points, equipment corner points, flange edge points, enclosure edge points, radiator outline points, or other structural feature points.

[0047] The registration reliability is obtained based on feature matching residuals, reprojection errors, or the similarity of corresponding regions in the two modes. Let there be a total of... Group matching feature points, the first Group of visible light feature points The corresponding infrared feature points are The reprojection error is defined as: in, This represents the homogeneous coordinate normalization function. Furthermore, the reprojection error can be normalized to the registration confidence level: The registration reliability The calculation method is as follows: Assume there are N sets of matching feature points, and the normalized reprojection error confidence level is... ,in, For reprojection error, To determine the reprojection error scale parameter, gradient features of the corresponding regions of the two modes are obtained through edge extraction or gradient enhancement, respectively. and And calculate the similarity of the corresponding regions of the two modes. : The final registration confidence level is expressed as: ,in, These are the weighting coefficients. To prevent constants with a denominator of zero, when Below the preset threshold If the current dual-mode registration is deemed unreliable, the weight of infrared features in the fusion discrimination is reduced or the system is switched to visible light temporal discrimination mode.

[0048] As a further improvement of the present invention, in step C, the prior art of the equipment structure includes flange connections, valves, oil tanks, radiator joints, pipe connections, sealing edges, bottom of the housing, sleeve roots, oil drains, water drains, and possible liquid seepage paths, and the high-risk concern areas are generated through template mapping, key component detection, CAD / BIM structural projection, or manual configuration.

[0049] The set of high-risk areas of concern is represented as follows: ,in, The number of high-risk components. For the i-th candidate region output by the YOLO series target detection model. Its structural prior score Defined as: in, Indicates candidate region High-risk areas The intersection and union ratio, This represents the minimum distance between the center of the candidate region and the high-risk component region. For distance normalization parameters, , These are the weighting coefficients; If the target area seeps downwards along the direction of gravity, let the suspected leak source be... target area centroid When in the image coordinate system When the axial direction is consistent with the direction of gravity, the consistency score of the infiltration path is increased. Defined as: in, For indicator functions, This is a parameter for normalizing the lateral offset. If the target area is located below the suspected leak source and has a small lateral offset, the consistency score of the infiltration path is high; if the target area is located above a high-risk component or far from the leak source path, its anomaly discrimination weight is reduced.

[0050] As a further improvement of the present invention, in step D, the time prompt includes the previous frame segmentation mask, the previous frame candidate box, the target trajectory box, or the optical flow propagation mask; and after obtaining the segmentation mask, connected component filtering, hole filling, boundary smoothing, morphological opening and closing operations, or maximum connected region retention post-processing operations are also performed.

[0051] For video scenes, the timing cue includes an optical flow propagation mask. Its final segmentation mask is based on the previous frame. And the optical flow field from the previous frame to the current frame is Pixel position propagation operation based on optical flow field The calculation yielded: The input prompt set of the prompt-based segmentation model can be represented as: in, For a set of dot hints, Provide target trajectory hints or historical area hints.

[0052] The segmentation probability map of the prompt-based general image segmentation model is as follows: ,in For the current frame inspection image, This is a tooltip that has been expanded by a preset expansion ratio. Thresholding the segmentation probability map yields an initial binary mask. ; in, The segmentation threshold is used; the morphological post-processing involves applying the initial binary mask... Perform morphological opening and closing operations to obtain the post-processing mask. ,in, , As a structuring element, the final set of connected components in the post-processing mask. Perform area filtering. in, For the first Area of ​​each connected region This is the minimum area threshold.

[0053] Preferably, when multiple connected components exist, the connected component with a size greater than the minimum area threshold that is closest to the center of the candidate box, has the largest area, or has the highest overlap with the propagation mask of the previous frame is retained to obtain the final segmentation mask. This is to improve the continuity of segmentation results in video scenarios.

[0054] As a further improvement of the present invention, in step E, the temporal evolution characteristics and physical consistency characteristics include at least two of the following: area growth rate, centroid displacement, equivalent radius diffusion rate, consistency between the infiltration direction and the gravity direction, continuous frame duration, distance from the suspected leakage source, and average temperature difference between the target area and the annular background area.

[0055] The specific process for extracting the temporal evolution features is as follows: based on the final segmentation mask... Calculate the target area in frame t: in, This is the pixel area conversion factor. Define the image region; and calculate the target centroid. ;in: The extracted temporal evolution features include the target region area growth rate. Displacement of the center of mass Equivalent radius diffusion rate and continuous frame duration : The target area growth rate is: The target centroid displacement is: The equivalent radius of the target region is: The equivalent radius diffusion rate is: in, This is the time interval between two adjacent frames. If and If the condition is true over multiple consecutive frames, it indicates that the target region has a continuous diffusion trend.

[0056] The duration of consecutive frames is defined as: in, To prevent constants with a denominator of zero, The time interval between two adjacent frames. This represents the number of frames in which the target region is stably detected and segmented within a continuous analysis window.

[0057] As a further improvement of the present invention, the specific process of extracting the thermodynamic features and physical consistency features in step E is as follows: For infrared temperature features, the current segmentation mask will be used. Mapping to infrared images to determine infrared target areas And construct a ring-shaped background area on its outer side. .

[0058] The average temperature of the target area is: The average temperature of the background area is: Calculate the average temperature of the target area Average temperature of the annular background region The difference is used to obtain the average temperature difference. ; and in combination with ambient temperature Introducing environmental compensation for temperature difference: ,in, For reference to ambient temperature, The compensation coefficient; The consistency between the infiltration direction and the gravity direction is calculated by the angle between the target center of mass movement direction and the preset gravity direction.

[0059] The physical consistency feature is based on the consistency score between the infiltration direction and the gravity direction. Distance Consistency Score Jointly construct a comprehensive physical consistency score : The directional consistency score is: when The closer the value is to 1, the closer the target's direction of motion is to the direction of gravitational infiltration; when... When the direction of the target movement is inconsistent with the infiltration pattern, the score for detecting leakage can be reduced.

[0060] Let the coordinates of the suspected leak source be... The centroid of the target region is The distance between the target area and the suspected leak source is: Its distance consistency score is: The overall physical consistency score is: in, Let the target centroid displacement vector be... Let s be the preset gravity direction vector, and s be the coordinates of the suspected leak source location. For distance scale parameters, to These are the weighting coefficients. This is a normalization function. The comprehensive physical consistency score is used to characterize whether candidate targets simultaneously satisfy the physical laws of leakage, namely, "continuous diffusion, reasonable infiltration direction, proximity to high-risk leakage sources, thermal differences, and continuous existence".

[0061] As a further improvement of the present invention, in step F, the environmental condition data includes ambient temperature, humidity, rainfall status, light intensity, and equipment load; the fusion discrimination constructs a feature vector based on each feature and dynamically adjusts the feature weights. Let the environmental condition vector be: in, For ambient temperature, For humidity, It is currently experiencing rainfall. For illuminance, For equipment load.

[0062] Let the fusion discriminative feature vector be: in, For morphological characteristics, As a temporal evolution feature, Thermal characteristics, For structural location features, This is a physical consistency characteristic.

[0063] Based on registration reliability Visible light quality Infrared quality and environmental conditions The fusion weight vector of each modality feature is dynamically generated through a neural network or preset rules: in, and For trainable parameters or preset rule parameters, the fusion discrimination score can be expressed as: When the infrared image quality is low, the registration reliability is low, or the environmental reflection interference is strong, reduce the weight of thermal features: When the rainfall is considered effective or the humidity exceeds a preset threshold, the threshold for identifying water stains and condensation should be increased or their alarm confidence level reduced. When the equipment load is high, resulting in a significant overall temperature rise, environmental compensation for temperature difference should be implemented. Replace the original temperature difference .

[0064] According to rainfall status and humidity Adaptively adjust the final classification threshold: in The baseline classification threshold , , For adjustment coefficients; The final judgment result is: in, This indicates an abnormality caused by leaks or spills. This indicates that the target is either not a leaking or disruptive target.

[0065] As a further improvement of the present invention, in step F, the anomaly category includes at least oil leakage, seepage, water stains, condensation, shadows, reflections, and suspected ventilation; the development stage includes at least initial, developing, ongoing, and severe; and the risk level includes at least low risk, medium risk, and high risk.

[0066] Let the probability of an anomaly classification be: The exception category output is: in, This indicates categories such as oil leaks, seepage, water stains, condensation, shadows, reflections, or suspected escaping.

[0067] The final judgment result also includes: Based on duration Area growth rate Equivalent radius diffusion rate Environmental compensation for temperature difference And the overall physical consistency score Score for development stages , And according to the above The size of the abnormality divides its development into four stages: initial stage, development stage, ongoing stage, and severe stage. Risk level can be determined comprehensively based on anomaly category, development stage, equipment importance, and confidence level. A score is assigned based on the development stage. Anomaly confidence level Equipment importance coefficient And the overall physical consistency score Calculate risk score : Where b1 to b4 are weighting coefficients; and based on the aforementioned risk score Based on the relationship between the risk level and the preset risk level threshold, the risk level is divided into low risk, medium risk and high risk.

[0068] in, , This represents the risk level threshold. For targets at a high risk level, the system outputs alarm information, target location, segmentation mask, suspected leak source, development stage, and review suggestions; for medium-risk targets, the system outputs instructions for continuous observation or manual review; for low-risk targets, the system can record information but will not trigger a strong alarm.

[0069] Based on the above method, this invention also proposes a multimodal and physically constrained power equipment leakage detection system, comprising: A dual-modal acquisition module is used to acquire visible light image sequences and infrared image sequences under the same inspection scenario; The synchronization registration module is used to complete time synchronization, spatial registration, and registration reliability evaluation. Specifically, the synchronization registration module generates registration reliability based on synchronization time deviation, reprojection error, feature matching residual, and the similarity of corresponding regions in both modes. ; The structural prior module generates high-risk areas of concern based on flange connections, valves, oil conservators, radiator joints, sealing edges, the bottom of the tank, and potential liquid seepage paths, and calculates the structural prior score between candidate targets and high-risk areas of concern. ; The candidate detection module is used to call the YOLO series of object detection models to obtain candidate boxes within the high-risk interest area; The cue segmentation module is used to generate a pixel-level segmentation mask for the target region based on box cue, point cue, and time cue, and to perform morphological post-processing; wherein, the time cue includes the segmentation mask of the previous frame, the candidate box of the previous frame, the target trajectory box, or the optical flow propagation mask; The feature extraction module is used to extract morphological features, temporal evolution features, thermal features, structural location features, and physical consistency features from the post-processed segmentation mask; wherein, the physical consistency features include at least two of the following: area growth rate, equivalent radius diffusion rate, infiltration direction consistency, distance from the leakage source, and annular background average temperature difference; The fusion discrimination module is used to calculate the fusion score based on registration confidence, modal quality, environmental condition data, and various features. It adaptively adjusts feature weights and classification thresholds, and outputs anomaly category, development stage, risk level, target location, segmentation mask, and confidence level. The rollback and alarm module is used to automatically trigger the mode rollback state to switch to visible light timing discrimination mode when the infrared mode is unavailable, the infrared image quality is lower than a preset threshold, or the registration confidence is lower than a preset confidence threshold, and output inspection alarm records containing review suggestions.

[0070] Furthermore, the present invention also proposes a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the multimodal and physically constrained power equipment leakage detection method and system described in any of the above claims.

[0071] Specifically, such as Figure 1-7 As shown, the method mainly includes the following steps: (1) Image acquisition and preprocessing; First, visible light images or video streams of power equipment are acquired using fixed cameras, inspection robots, drones, handheld inspection terminals, or edge intelligent devices. For video stream data, consecutive video frames are extracted according to a preset frame rate to form an inspection image sequence: in, Indicates the first Frame inspection images, Indicates the number of consecutive frames.

[0072] To reduce the impact of lighting variations, image noise, complex equipment backgrounds, shooting angle changes, and low-light conditions at night on the detection results, the inspection images are preprocessed before being input into the model. This preprocessing includes size normalization, brightness equalization, contrast enhancement, noise reduction filtering, distortion correction, region of interest cropping, and pixel normalization.

[0073] In one specific embodiment, the input image is uniformly scaled to the input size required by the YOLO26 model, for example... or The pixel values ​​are then normalized. For low-light, backlit, or nighttime inspection images, histogram equalization, adaptive brightness enhancement, or Retinex enhancement methods are used to improve the visibility of oil stains, water stains, wet marks, and abnormally dark areas.

[0074] (2) Detection of suspected leaks and spills based on YOLO26: The pre-processed inspection image is input into the pre-trained YOLO26 target detection model. The YOLO26 model performs preliminary localization of suspected leaks and spills on the surface of the power equipment and outputs one or more candidate detection boxes. Each candidate detection box is represented as: in, , Indicates the coordinates of the center point of the candidate box. , These represent the width and height of the candidate box, respectively. Indicates the detection confidence level. Indicates the candidate category.

[0075] The candidate categories output by the YOLO26 model include oil leaks, seepage, water stains, wet marks, oil stains, abnormally dark areas, suspected escaping gas, and other suspected leaks. To reduce interference from background shadows, equipment coating reflections, and residual water after rain, the YOLO26 model is primarily used for coarse localization of suspected abnormal areas, followed by secondary discrimination using SAM3 pixel-level segmentation and morphological, temporal, and structural location features.

[0076] During the training phase, leaks, shadows, reflections, water stains, condensation, and normal equipment backgrounds in power inspection images can be labeled as different categories or difficult examples, enabling the YOLO26 model to learn the appearance differences between leaks and common interference targets.

[0077] (3) Equipment structure prior information and high-risk concern area generation; In order to further improve the targeting of the detection, this embodiment introduces equipment structure prior information. The equipment structure prior information includes flange connections, valves, oil tanks, radiator joints, pipe connections, sealing edges, bottom of the housing, sleeve roots, oil drains, water drains, and possible liquid seepage paths.

[0078] Pre-build a set of high-risk concern areas based on equipment type: in, Indicates the first High-risk areas of concern This indicates the number of high-risk areas.

[0079] High-risk areas of concern can be generated through manual configuration, template mapping, key component inspection, CAD / BIM structural projection, or historical defect statistics. (This refers to candidate areas output by YOLO26.) Calculate its structural prior score: in, This represents the intersection-union ratio of candidate regions and high-risk regions. This represents the minimum distance between the center of the candidate region and the high-risk region. For distance normalization parameters, , These are the weighting coefficients.

[0080] For areas where liquids may seep downwards along the direction of gravity, let the suspected leak source be... The centroid of the target region is When in the image coordinate system When the axial direction is consistent with the direction of gravity, the consistency of the infiltration path is represented as follows: in, For indicator functions, This is the normalized parameter for lateral offset. When the candidate region is located below the suspected leak source and has a small lateral offset, it is considered to be more consistent with the law of liquid infiltration.

[0081] (4) SAM3 cue generation based on YOLO26 detection boxes; After obtaining the candidate boxes output by YOLO26, cue information required for the SAM3 segmentation model is generated based on the candidate boxes. The cue information includes at least one of box cue, point cue, and mask cue. Among them, the box cue is determined by the candidate detection boxes output by YOLO26; the point cue is determined by the center point of the candidate box, high-response points within the candidate region, foreground sampling points, or sampling points near the boundary; the mask cue is determined by the target mask of the previous frame, historical tracking regions, or coarse segmentation results.

[0082] In a preferred embodiment, to prevent the YOLO26 candidate box from being too tight and causing areas with oil stains, water stains, or wet marks to be missed, the candidate box is expanded outwards. Let the original candidate box be: The expanded candidate box is as follows: in, The candidate box expansion coefficient is preferred, with a value range of [value range missing]. .

[0083] The set of dot hints can be represented as: The input set for the SAM3 segmentation model is represented as follows: in, This indicates an optional history mask hint or a priori mask hint.

[0084] For video inspection scenarios, time prompts can also be introduced. Let the segmentation mask of the previous frame be... The optical flow field from the previous frame to the current frame is The optical flow propagation mask is then represented as: in, This represents pixel propagation operations based on optical flow. The cue set for the current frame SAM3 can be represented as: in, Candidate bounding boxes for the current frame. For a set of dot hints, For time propagation mask, Provide target trajectory hints or historical area hints.

[0085] (5) Pixel-level segmentation and post-processing based on SAM3; The inspection image to be detected and the above prompt information are input into the SAM3 segmentation model to perform pixel-level segmentation of the suspected leak area and obtain the target area segmentation probability map: in, This represents the SAM3 segmentation model. Represents pixels The probability of belonging to the target region. This indicates the image to be inspected.

[0086] Thresholding is performed on the segmentation probability map to obtain the initial binary segmentation mask: in, This is the segmentation threshold. This indicates that the pixel belongs to the suspected leak / spillage target area. This indicates that the pixel belongs to the background area.

[0087] To improve the stability of the segmentation results, after obtaining the initial binary segmentation mask, post-processing operations such as connected component filtering, hole filling, boundary smoothing, morphological opening, morphological closing, or preservation of the maximum connected component are performed to obtain a target mask with complete boundaries and less noise. Morphological post-processing can be expressed as: in, , As a structural element, This indicates the opening operation. This represents the closing operation. It applies to the set of connected components. By performing area filtering, the final segmentation mask is obtained: in, Indicates the first Area of ​​each connected region This represents the minimum area threshold. Preferably, when multiple connected components exist, the connected component that is closest to the center of the candidate box, has the largest area, or has the highest overlap with the propagation mask of the previous frame is retained.

[0088] (6) Morphological feature extraction: Based on the binary segmentation mask output by SAM3, the morphological features of the target region are extracted. The morphological features include area, perimeter, aspect ratio, compactness, irregularity, convex hull ratio, circularity, boundary complexity, and skeleton branch density.

[0089] The area of ​​the target region is expressed as: in, This is the pixel area conversion factor. This represents the image region. The perimeter of the target region is represented as: in, This represents the target boundary. The compactness of the target region is expressed as: The irregularity of the target region is represented as: The proportion of the convex hull of the target region is expressed as: in, This represents the area of ​​the convex hull of the target region. To prevent constants with a denominator of zero. Under normal circumstances, the actual leakage area is affected by the surface texture of the equipment, the direction of liquid diffusion, the gravity seepage path and the adsorption characteristics of the material. Its shape often shows characteristics such as irregular boundaries, edge diffusion, low compactness and irregular patchy shape; while ordinary shadows, regular reflections or equipment projection areas usually have more regular boundaries and less shape variation.

[0090] Based on the morphological characteristics, the suspected leakage areas are judged to be regular; when the target area has irregular boundaries, low compactness, low convexity ratio, and the area shows a diffusion trend or the shape is irregular patchy, it is determined that it is more consistent with the characteristics of leakage; when the target area has regular boundaries, stable shape, clear edges and is consistent with the projection direction of the equipment, it is determined that it is more likely to be a shadow, reflection or rain interference.

[0091] The rule-based score can be represented as: in, For weighting coefficients. When When the value exceeds the preset shape threshold, the target area is determined to have an obvious irregular shape.

[0092] (7) Temporal evolution feature extraction: For continuous video frames, the target segmentation mask in adjacent frames or multiple frames is temporally correlated to extract temporal evolution features such as area change rate, centroid displacement, equivalent radius diffusion speed and continuous frame duration, so as to determine whether the suspected target has a continuous diffusion or infiltration trend.

[0093] No. The target area of ​​the frame is: The area change rate is: The target centroid is represented as: The area growth rate is expressed as: The displacement of the center of mass is expressed as: The equivalent radius of the target region is expressed as: The equivalent radius diffusion rate is expressed as: in, This is the time interval between two adjacent frames. If the target region satisfies [condition] in multiple consecutive frames... and This indicates that the target has a continuous diffusion trend, which is consistent with the temporal characteristics of the gradual diffusion of oil, water, or wet marks.

[0094] The duration of consecutive frames is expressed as: in, This indicates the number of frames in which the target is stably detected and segmented within the continuous analysis window.

[0095] (8) Physical Consistency Constraints; To distinguish between actual leaks and interfering targets such as shadows, reflections, and rainwater residue, this embodiment further constructs physical consistency constraints. These constraints include at least the consistency between the infiltration direction and the gravity direction, the consistency of the distance between the target area and the suspected leak source, the consistency of the area diffusion trend, and the consistency of the duration. The physical consistency features are based on the score for the consistency between the infiltration direction and the gravity direction. Distance Consistency Score Jointly construct a comprehensive physical consistency score : Let the target centroid displacement vector be: The preset gravity direction vector is: The score for consistency between the infiltration direction and the gravity direction is expressed as: when The closer the value is to 1, the closer the target's direction of motion is to the direction of gravitational infiltration; when... When the direction of the target movement is inconsistent with the infiltration pattern, the score for detecting leakage can be reduced.

[0096] Based on prior knowledge of the electrical equipment structure, determine the positional relationship between the target area and the flange connection, valve, oil tank, radiator joint, sealing edge, bottom of the enclosure, bushing root, or drain port. Let the coordinates of the suspected leak source be... The centroid of the target region is The distance between the target area and the suspected leak source is expressed as: Distance consistency score is represented as: in, This is a distance scale parameter. The closer the distance, the more likely the target area is to be associated with high-risk areas such as flanges, valves, sealing edges, and radiator joints.

[0097] The overall physical consistency score is expressed as follows: in, to These are the weighting coefficients. This is a normalization function. This score is used to characterize whether candidate targets simultaneously satisfy the physical laws of leakage, such as continuous diffusion, reasonable infiltration direction, proximity to high-risk leakage sources, and persistent existence.

[0098] (9) Comprehensive judgment and alarm output; when the target area is determined to be an abnormal area of ​​running, leaking and dripping, output the abnormality category, target location, segmentation mask, confidence level and alarm information; generate inspection alarm record based on the output result, the inspection alarm record includes inspection time, equipment number, abnormality type, abnormality location, candidate box coordinates, segmentation mask, abnormal area, development trend, confidence level and review suggestion.

[0099] Let the environmental condition vector be: in, For ambient temperature, For humidity, It is currently experiencing rainfall. For illuminance, For equipment load.

[0100] Morphological features, temporal evolution features, structural location features, and physical consistency features are combined to form a fused discriminative feature vector: in, Indicate morphological features, Represents temporal evolution characteristics, Indicates structural location characteristics. Indicates physical consistency characteristics.

[0101] The overall discrimination score is expressed as follows: in, , , , These represent the weights of morphological, temporal, structural, and physical consistency characteristics, respectively.

[0102] The final judgment result is expressed as follows: in, This indicates an abnormality caused by leaks or spills. This indicates that the target is identified as being affected by shadows, reflections, residual rainwater, or other non-leaking or dripping interference. This is the classification threshold.

[0103] When an anomaly is identified as oil spill, seepage, water stains, condensation, shadows, reflections, and suspected gas leakage, the system outputs the anomaly category, target location, candidate box coordinates, pixel-level segmentation mask, anomaly area, development trend, confidence level, risk level, and review recommendations. Anomaly categories include oil spill, seepage, water stains, condensation, shadows, reflections, and suspected gas leakage; development stages include initial, developing, ongoing, and severe; and risk levels include low, medium, and high risk.

[0104] Let the probability of an anomaly classification be: The exception category output is: in, This indicates categories such as oil leaks, seepage, water stains, condensation, shadows, reflections, or suspected escaping.

[0105] The final judgment result also includes: Based on duration Area growth rate Equivalent radius diffusion rate Environmental compensation for temperature difference And the overall physical consistency score Score for development stages , And according to the above The size of the abnormality divides its development into four stages: initial stage, development stage, ongoing stage, and severe stage. Risk level can be determined comprehensively based on anomaly category, development stage, equipment importance, and confidence level. A score is assigned based on the development stage. Anomaly confidence level Equipment importance coefficient And the overall physical consistency score Calculate risk score : Where b1 to b4 are weighting coefficients; and based on the aforementioned risk score Based on the relationship between the risk level and the preset risk level threshold, the risk level is divided into low risk, medium risk and high risk.

[0106] in, , This represents the risk level threshold. For targets at a high risk level, the system outputs alarm information, target location, segmentation mask, suspected leak source, development stage, and review suggestions; for medium-risk targets, the system outputs instructions for continuous observation or manual review; for low-risk targets, the system can record information but will not trigger a strong alarm.

[0107] This invention can also be further applied to valve assemblies, oil tanks, flange connections, cooler piping, GIS accessories, and other electrical equipment that may experience liquid or volatile medium leakage. For those skilled in the art, adjustments can be made to specific model dimensions, thresholds, window lengths, temperature difference values, fusion classifier forms, and output protocols without departing from the concept of this invention; all such adjustments should fall within the scope of protection of this invention. Those skilled in the art should understand that the embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0108] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0109] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0110] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0111] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal and physically constrained method for detecting leaks in power equipment, characterized in that, Includes the following steps: Step A: Multimodal data acquisition; acquire visible light image sequences, infrared image sequences, and optional environmental condition data for the same inspection scenario; Step B: Adaptive spatiotemporal registration; The visible light image sequence and the infrared image sequence are time-synchronized and spatially geometrically registered, and the dual-modal registration reliability is obtained by quantification calculation; Step C: Preliminary rough positioning of component structure; Based on the prior knowledge of the equipment structure, high-risk areas of concern are identified. Within these high-risk areas of concern, a neural network-based target detection model is invoked to perform candidate detection of suspected leaks and spills, and candidate bounding boxes of the suspected leaks and spills are output. Step D: Fine segmentation using multiple prompts; dynamically generate spatial box prompts and point prompts based on the candidate boxes, and simultaneously generate a time propagation mask as a time prompt by combining the segmentation status of historical frames; The spatial bounding box cue, point cue, temporal cue, and the current frame image are input together into the cue-based segmentation model to drive the cue-based segmentation model to output a pixel-level initial segmentation mask of the target region, and morphological post-processing is performed to repair it to obtain the final segmentation mask; Step E: Multidimensional physical feature extraction; Based on the final segmentation mask and the infrared image sequence, extract the morphological topological features, temporal evolution features, thermal features, structural location features, and physical consistency constraint features that conform to the fluid diffusion law of the target region; Step F: Adaptive weighted fusion discrimination; based on registration confidence, modal quality, and the aforementioned morphological topology features, temporal evolution features, thermal features, structural location features, and physical consistency constraint features, fusion discrimination is performed to obtain anomaly category, development stage, risk level, target location, segmentation mask, and confidence level; wherein, the modal quality includes visible light image clarity, infrared image clarity, occlusion degree, overexposure degree, underexposure degree, thermal image noise level, and rain and fog interference degree; the fusion discrimination is implemented based on rule model, logistic regression model, support vector machine model, random forest model, gradient boosting model, lightweight neural network model, or multi-model weighted fusion model, and the fusion weights of each modal feature are dynamically adjusted according to the registration confidence and modal quality; Step G: Dynamic Modal Backoff Control: During the inspection process, the availability of infrared modes and the registration reliability are monitored in real time. When the preset backoff conditions are triggered, the system automatically switches to the single-mode visible light temporal discrimination mode and adjusts the output threshold, risk level or confidence level according to the infrared mode missing situation. The visible light temporal discrimination mode judges at least based on the morphological characteristics, temporal evolution characteristics, structural position characteristics and physical consistency characteristics of the target area, and marks the modal backoff status in the output result for subsequent manual review or inspection work order processing. The preset fallback conditions include at least one of the following: infrared mode unavailable, infrared image quality lower than a preset threshold, or registration confidence lower than a preset confidence threshold.

2. The method for detecting leaks in power equipment based on multimodal and physical constraints according to claim 1, characterized in that, In step B, the specific process of time synchronization is as follows: Let the first The acquisition time for a frame of visible light image is , No. The acquisition time for one frame of infrared image is The optimal synchronization frame pair is determined by retrieving the infrared image sequence using the following formula that minimizes the time difference: When satisfied At that time, the first Frame visible light image and the first The infrared image frame is locked as a synchronized frame pair; among which, The preset maximum time deviation threshold; The spatial registration, under the condition of planar approximation, uses the homography matrix H to establish the pixel points in the visible light image coordinate system. The corresponding pixel in the infrared image coordinate system is Mapping relationship between them: The mapped coordinates in the infrared image are obtained by homogeneous coordinate normalization. The registration reliability The calculation method is as follows: Assume there are N sets of matching feature points, and the normalized reprojection error confidence level is... ,in, For reprojection error, To determine the reprojection error scale parameter, gradient features of the corresponding regions of the two modes are obtained through edge extraction or gradient enhancement, respectively. and And calculate the similarity of the corresponding regions of the two modes. : The final registration confidence level is expressed as: ,in, These are the weighting coefficients. To prevent constants with a denominator of zero.

3. The method for detecting leaks in power equipment based on multimodal and physical constraints according to claim 1, characterized in that, In step C, the set of high-risk areas of concern is represented as follows: ,in, The number of high-risk components; for the i-th candidate region output by the neural network-based target detection model. Its structural prior score Defined as: in, Indicates candidate region High-risk areas The intersection and union ratio, This represents the minimum distance between the center of the candidate region and the high-risk component region. For distance normalization parameters, , These are the weighting coefficients; If the target area seeps downwards along the direction of gravity, let the suspected leak source be... target area centroid When in the image coordinate system When the axial direction is consistent with the direction of gravity, the consistency score of the infiltration path is increased. Defined as: in, For indicator functions, This is the horizontal offset normalization parameter.

4. The method for detecting leaks in power equipment based on multimodal and physical constraints according to claim 1, characterized in that, In step D, for a video scene, the time cue includes an optical flow propagation mask. Its final segmentation mask is based on the previous frame. And the optical flow field from the previous frame to the current frame is Pixel position propagation operation based on optical flow field The calculation yielded: The segmentation probability map of the prompt-based general image segmentation model is as follows: ,in For the current frame inspection image, This is a tooltip that has been expanded according to a preset expansion ratio. For a set of dot hints; Thresholding the segmentation probability map yields an initial binary mask. The morphological post-processing is to process the initial binary mask. Perform morphological opening and closing operations to obtain the post-processing mask. ,in, , Using the structural element as an example, the connected component set in the post-processing mask is finally filtered by area. Connected components with an area greater than the minimum area threshold, the closest distance to the candidate box center, the largest area, or the highest overlap with the propagation mask of the previous frame are retained to obtain the final segmentation mask. .

5. The method for detecting leaks in power equipment based on multimodal and physical constraints according to claim 1, characterized in that, In step E, the specific process of extracting the temporal evolution features is as follows: based on the final segmentation mask... Calculate the target area in frame t: in, This is the pixel area conversion factor. Define the image region; and calculate the target centroid. The extracted temporal evolution features include the target region area growth rate. Displacement of the center of mass Equivalent radius diffusion rate and continuous frame duration : The target area growth rate is: The target centroid displacement is: The equivalent radius of the target region is: The equivalent radius diffusion rate is: The duration of consecutive frames is defined as: in, To prevent constants with a denominator of zero, The time interval between two adjacent frames. This represents the number of frames in which the target region is stably detected and segmented within a continuous analysis window.

6. The method for detecting leaks in power equipment based on multimodal and physical constraints according to claim 1, characterized in that, In step E, the specific process of extracting the thermodynamic features and physical consistency features is as follows: Use the current segmentation mask The infrared image is mapped to determine the infrared target region, and a ring-shaped background region is constructed outside it; Calculate the average temperature of the target area Average temperature of the annular background region The difference is used to obtain the average temperature difference. ; and in combination with ambient temperature Introducing environmental compensation for temperature difference: ,in, For reference to ambient temperature, This is the compensation coefficient; The physical consistency feature is based on the consistency score between the infiltration direction and the gravity direction. Distance Consistency Score Jointly construct a comprehensive physical consistency score : The directional consistency score is: Distance consistency score: The overall physical consistency score is: in, Let the target centroid displacement vector be... Let s be the preset gravity direction vector, and s be the coordinates of the suspected leak source location. For distance scale parameters, to These are the weighting coefficients. This is the normalization function.

7. The method for detecting leaks in power equipment based on multimodal and physical constraints according to claim 1, characterized in that, In step F, the environmental condition data includes ambient temperature, humidity, rainfall, light intensity, and equipment load. The fusion and discrimination are based on constructing feature vectors and dynamically adjusting feature weights. Based on registration reliability Visible light quality Infrared quality and environmental conditions The fusion weight vector of each modality feature is dynamically generated through a neural network or preset rules: in, and For trainable parameters or preset rule parameters, the fusion discrimination score can be expressed as: in, For morphological characteristics, As a temporal evolution feature, Thermal characteristics, For structural location features, It is a physical consistency feature; According to rainfall status and humidity Adaptively adjust the final classification threshold: in The baseline classification threshold , , For adjustment coefficients; if If so, it is determined to be an abnormality of leakage.

8. The method for detecting leaks in power equipment based on multimodal and physical constraints according to claim 7, characterized in that, In step F, the fusion discrimination further includes: Based on duration Area growth rate Equivalent radius diffusion rate Environmental compensation for temperature difference And the overall physical consistency score Score for development stages and according to the The size of the abnormality divides its development into four stages: initial stage, development stage, ongoing stage, and severe stage. Scores based on the aforementioned development stages Anomaly confidence level Equipment importance coefficient And the overall physical consistency score Calculate risk score : Where b1 to b4 are weighting coefficients; and risk levels are classified according to risk scores: in, , The system outputs alarm information, target location, segmentation mask, suspected leakage source, development stage, and review suggestions for high-risk targets, based on risk level thresholds. For medium-risk targets, the system outputs instructions for continuous observation or manual review. For low-risk targets, the system can record information but will not trigger a strong alarm.

9. A multimodal and physically constrained power equipment leakage detection system, characterized in that, include: A dual-modal acquisition module is used to acquire visible light image sequences and infrared image sequences under the same inspection scenario; The synchronization registration module is used to complete time synchronization, spatial registration, and registration reliability evaluation. Specifically, the synchronization registration module generates registration reliability based on synchronization time deviation, reprojection error, feature matching residual, and the similarity of corresponding regions in both modes. ; The structural prior module generates high-risk areas of concern based on flange connections, valves, oil conservators, radiator joints, sealing edges, the bottom of the tank, and potential liquid seepage paths, and calculates the structural prior score between candidate targets and high-risk areas of concern. ; The candidate detection module is used to call a neural network-based object detection model to obtain candidate boxes within the high-risk interest region. The cue segmentation module is used to generate a pixel-level segmentation mask for the target region based on box cue, point cue, and time cue, and to perform morphological post-processing; wherein, the time cue includes the segmentation mask of the previous frame, the candidate box of the previous frame, the target trajectory box, or the optical flow propagation mask; The feature extraction module is used to extract morphological features, temporal evolution features, thermal features, structural location features, and physical consistency features from the post-processed segmentation mask; wherein, the physical consistency features include at least two of the following: area growth rate, equivalent radius diffusion rate, infiltration direction consistency, distance from the leakage source, and annular background average temperature difference; The fusion discrimination module is used to calculate the fusion score based on registration confidence, modal quality, environmental condition data, and various features. It adaptively adjusts feature weights and classification thresholds, and outputs anomaly category, development stage, risk level, target location, segmentation mask, and confidence level. The rollback and alarm module is used to automatically trigger the mode rollback state to switch to visible light timing discrimination mode when the infrared mode is unavailable, the infrared image quality is lower than a preset threshold, or the registration confidence is lower than a preset confidence threshold, and output inspection alarm records containing review suggestions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multimodal and physically constrained method for detecting leaks in power equipment as described in any one of claims 1 to 8.

Images