Power grid inspection method and system with synergistic semantic filtering and specialized defect detectors

Abstract

The application provides a power grid inspection method and system of synergistic semantic screening and special defect detector, comprising: acquiring an inspection picture of a current round, identifying the inspection picture based on a full picture precision inspection mode and a local precision inspection mode corresponding to the current round, and outputting corresponding global defect detection results and local defect detection results; comparing the global defect detection results with the original inspection picture, and comparing the local defect detection results with the output of a semantic coarse positioning model in the local precision inspection mode, to generate defect fusion feedback representing the detection performance of the current round; and updating the picture identification mode of the next round according to the defect fusion feedback of the current round, the effective computing power of the next round, and the preset time delay of the inspection task. The application can fully exert the semantic advantages of the semantic screening large model in complex scene understanding and candidate screening, and the precision advantages of the special detector in small target defect precision inspection, and can balance the precision and real-time performance in different complex detection scenes.

Description

Technical Field

[0001] This invention relates to the field of power grid unmanned aerial vehicle (UAV) inspection technology, specifically to a power grid inspection method and system using collaborative semantic filtering and a dedicated defect detector. Background Technology

[0002] Power grid inspection drones are highly efficient in inspecting distribution lines, towers, insulators, clamps, and crossarms. However, actual inspection images often contain trees, buildings, conductors, shadows, reflections, and complex perspectives (related to the drone's inspection route). Defects such as improper insulator binding, missing protective shells, crossarm corrosion, and tower head damage often appear in inspection images or videos with low proportions, low contrast, and significant morphological variations. Furthermore, the frequency and severity of these problems are random. If the selected recognition model is inaccurate, the effective information in the inspection images can easily be obscured by the complex background, affecting the accuracy of the inspection results.

[0003] Multimodal semantic models possess the capabilities of scene understanding, target semantic representation, and candidate target selection, making them suitable for semantic screening and verification of difficult samples in inspection images. Dedicated detectors feature fast inference speed, fine-grained localization, and low deployment cost, making them suitable for fine-grained detection of small target defects in power grids. Inspection images identified using semantic models, if containing only small targets, are easily limited by inference latency and local localization capabilities. Inspection images identified using dedicated detectors, however, contain several complex targets and lack an understanding of the semantics of the inspection task and the complex scene context.

[0004] Existing multi-model fusion schemes mostly rely on splicing identification results, averaging confidence levels, or simple cascading, lacking a structured collaborative mechanism. Therefore, a collaborative identification method for power grid inspection defects that can leverage the complementary advantages of semantic models and dedicated detectors is needed. Summary of the Invention

[0005] In view of this, the problem to be solved by the present invention is to provide a power grid inspection method and system with collaborative semantic filtering and dedicated defect detector, which can dynamically switch the identification model according to the current computing power, communication status, latency and the previous round of identification results, thereby improving the accuracy of defect identification by inspection drones in complex backgrounds.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A power grid inspection method based on collaborative semantic filtering and dedicated defect detectors includes generating defect detection results, acquiring inspection images for the current round, identifying inspection images based on the full-image fine inspection mode and local fine inspection mode corresponding to the current round, and outputting the corresponding global defect detection results and local defect detection results. Generate defect fusion feedback by comparing the global defect detection results with the original inspection images and comparing the local defect detection results with the output of the semantic coarse localization model in the local fine inspection mode, so as to generate defect fusion feedback representing the detection performance of the current round. Recognition mode update: The image recognition mode for the next round is updated based on the defect fusion feedback of the current round, the effective computing power of the next round, and the preset time delay of the inspection task.

[0007] Furthermore, the full-image fine inspection mode includes a first dedicated detector for fine identification of defects in the full image, and the global defect detection results include the global defect category, global fine inspection defect box, and global fine inspection confidence of each defect in the inspection image; The local fine inspection mode includes a semantic coarse localization model that uses candidate defect boxes to initially select defect regions, and a second dedicated detector that finely identifies defects within the candidate defect boxes. The local defect detection results include the local defect category of each defect in the inspection image, the local fine inspection defect box, and the local fine inspection confidence level.

[0008] Furthermore, the local fine inspection mode includes a triggerable defect supplementation strategy: identify whether there are supplementary defects outside the candidate defect box of the inspection image; if yes, the supplementary defects are added to the defect detection results; otherwise, the defect detection results are directly output.

[0009] Furthermore, the defect fusion feedback of the full-image fine inspection mode includes category legality indicating whether the global defect category conforms to the preset category range, label mapping consistency verification based on the reverse mapping of the global defect category and whether it is consistent with the corresponding global fine inspection defect box, and global fine inspection confidence. The defect fusion feedback of the local fine inspection mode includes the defect fusion degree, which represents the consistency between the defect detection result and the output of the semantic coarse localization model; the degree of conflict between the local defect type and the defect in the corresponding candidate defect box; and the local fine inspection confidence.

[0010] Furthermore, the formula for calculating the defect fusion degree is as follows: , Where F represents the degree of defect fusion, This indicates the coarse location reliability of the semantic coarse localization model. Indicates the confidence level of the local precision test. This represents the intersection-union ratio (IoU) of the local precision defect bounding box and the candidate defect bounding box. This indicates the consistency index between local defect types and candidate defect types.

[0011] Furthermore, both the full-image fine inspection mode and the local fine inspection mode are equipped with a fusion verification strategy based on defect fusion feedback triggering to re-verify and update the defect fusion feedback; The fusion verification strategy includes: receiving relevant data corresponding to the defect fusion feedback that triggers the fusion verification strategy, and reviewing, calculating and updating the corresponding defect fusion feedback based on the relevant data.

[0012] Furthermore, the fusion verification strategy includes cloud-based fusion verification running in the cloud and edge-based fusion verification running at the edge. "Identification mode update" includes: obtaining the connection status with the edge and cloud, matching the computing power required to run the fusion verification strategy with the remaining computing power of the corresponding edge device and cloud, so as to determine whether to perform cloud fusion verification or edge fusion verification.

[0013] Furthermore, the local fine inspection mode includes a candidate box optimization strategy: determining whether the defect fusion degree of the local fine inspection mode in the current round is lower than the corresponding threshold; if not, continuing to obtain the defect fusion degree generated by the local fine inspection mode in subsequent rounds; if yes, based on the intersection-union ratio of the local fine inspection defect box and the candidate defect box. Update the size of the candidate defect box.

[0014] The power grid inspection system with collaborative semantic filtering and dedicated defect detectors includes a data acquisition module for acquiring power grid inspection images, a semantic coarse localization module for running a semantic coarse localization model, a dedicated fine inspection module for performing fine-grained detection of the entire image or candidate defect boxes, a fusion verification module for generating defect fusion feedback, a verification module for controlling the operation of fusion verification strategies and updating defect fusion feedback, a scheduling feedback module for updating the image recognition mode of the next round, a result output module for outputting global defect detection results or local defect detection results, and an edge-cloud communication module for data transmission between the edge and the cloud.

[0015] The beneficial effects of this invention are: Setting up full-image fine inspection mode and local fine inspection mode, the defect fusion feedback of global defect detection results and local defect detection results, and the effective computing power and preset time delay of the next round of inspection tasks to update the image recognition mode of the next round can give full play to the semantic advantages of the semantic screening large model in complex scene understanding and candidate selection and the accuracy advantages of the dedicated detector in small target defect fine inspection. It can reduce the accuracy loss caused by using a single model in unsuitable scenarios for power grid inspection and can balance accuracy and real-time performance in training and inspection scenarios with different complexities.

[0016] The defect fusion feedback includes category validity, label mapping consistency, global precision detection confidence, defect fusion degree, conflict degree, and local precision detection confidence. It can perform multi-faceted cross-validation of candidate targets and precision detection results in terms of space, category, and confidence, avoiding simple post-processing averaging of individual identification results and reducing the probability of false detection and duplicate detection.

[0017] By setting up a fusion verification strategy, and based on the verification modules pre-deployed at the edge and cloud, the samples of conflict defect fusion feedback are verified a second time. This improves the reliability of defect identification in inspection images, while making full use of external computing power and reducing inspection feedback delay. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of the power grid inspection method of collaborative semantic filtering and dedicated defect detector of the present invention. Detailed Implementation

[0019] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed terms.

[0021] This invention provides a power grid inspection method based on collaborative semantic filtering and a dedicated defect detector, such as... Figure 1 As shown, the process includes generating defect detection results, acquiring inspection images for the current round, identifying inspection images based on the full-image fine inspection mode and local fine inspection mode corresponding to the current round, and outputting the corresponding global defect detection results and local defect detection results.

[0022] Specifically, when the full-image fine inspection mode is running, the first dedicated detector used to identify defects performs a full-image fine inspection of the entire inspection image and outputs a global defect detection result that includes the global defect category, global fine inspection defect box, and global fine inspection confidence of each defect in the inspection image.

[0023] When the local fine inspection mode is running, the semantic coarse localization model performs scene understanding on the entire inspection image and outputs a set of candidate defect boxes. Each candidate defect box corresponds to a candidate defect type. The second dedicated detector used to identify defects performs fine inspection on the image within the candidate defect boxes and outputs the local defect detection results, which include the local defect category, local fine inspection defect box, and local fine inspection confidence of each defect in the inspection image.

[0024] When the defects are densely distributed in the inspection image, the local fine inspection mode is used, which utilizes the semantic coarse localization model's semantic advantages in understanding complex scenes and candidate selection to reduce the probability of missed detections; when the defects are sparsely distributed in the inspection image, the full-image fine inspection mode is used to improve detection efficiency and the accuracy of identifying fine defects.

[0025] One embodiment of this application is: the dedicated detector can be a single-stage detector, a two-stage detector, a segmentation model, or other lightweight detection models adapted to power grid defect data.

[0026] The local fine inspection mode includes a defect supplementation strategy: identifying the area outside the candidate defect box in the inspection image; if a high-confidence supplementary defect is identified, it is added to the defect detection result and marked; if no high-confidence supplementary defect is identified outside the candidate defect box, the defect detection result is directly output. One embodiment of this application is that the defect supplementation strategy is triggered at a fixed frequency, or the triggering frequency is dynamically adjusted based on the number of supplementary defects.

[0027] One embodiment of this application states that power grid defect categories include at least improper insulator binding, missing protective shells for parallel clamps, missing protective shells for tension clamps, crossbar corrosion, and tower head damage. A dedicated detector maintains the set of currently identifiable defect categories and the label-index mapping relationship in the form of a label list. The semantic coarse localization model aligns with this label list through a category semantic description configuration table. Dynamic expansion of defect categories can be achieved by replacing or expanding the label list and the category semantic description configuration table. Dynamic expansion of defect categories is achieved through replaceable dedicated detector label sets and configurable category semantic description configuration tables, eliminating the need to retrain the semantic coarse localization model.

[0028] Generate defect fusion feedback by comparing the global defect detection results with the original inspection images and comparing the local defect detection results with the output of the semantic coarse localization model in the local fine inspection mode, so as to generate defect fusion feedback representing the detection performance of the current round.

[0029] The defect fusion feedback in the full-image fine inspection mode includes the category legality indicating whether the global defect category conforms to the preset category range, the label mapping consistency verification based on the reverse mapping of the global defect category and whether it is consistent with the corresponding global fine inspection defect box, and the global fine inspection confidence.

[0030] When an invalid global defect category is detected, it indicates an error in global defect category identification or the presence of an unknown defect in the inspection image. Inconsistent label mapping may indicate an error in global defect category identification, the presence of other defects within the global detailed inspection defect box, or an incomplete defect.

[0031] The defect fusion feedback of the local fine inspection mode includes the defect fusion degree, which represents the consistency between the defect detection results and the localization results output by the semantic coarse localization model; the degree of conflict between the local defect type and the corresponding defect in the candidate defect box; and the local fine inspection confidence.

[0032] The higher the defect fusion degree, the higher the consistency between the output of the semantic coarse localization model and the corresponding defect detection result. The greater the conflict between defect types, the lower the consistency between the candidate defect category output by the semantic coarse localization model and the local defect type.

[0033] The formula for calculating the defect fusion degree F is: , in, This indicates the coarse location reliability of the semantic coarse localization model. Indicates the confidence level of the local precision test. This represents the intersection-union ratio (IoU) of the local precision defect bounding box and the candidate defect bounding box. The consistency index between the local defect type and the candidate defect type is 1 when they are consistent and 0 when they conflict. β1~β4 are preset fusion weight coefficients.

[0034] Intersection and Union The calculation formula is: IoU(Bci, Bfi) = |Bci ∩ Bfi| / |Bci ∪ Bfi| , Where Bci represents the set of pixels of the i-th candidate defect box in the inspection image, and Bfi represents the set of pixels of the local fine inspection defect box corresponding to the i-th candidate defect box in the inspection image.

[0035] Both the full-map fine inspection mode and the local fine inspection mode have a fusion verification strategy that re-verifies and updates the defect fusion feedback. The fusion verification strategy is triggered based on the degree of category conflict in the defect fusion feedback of the local fine inspection mode, whether there are supplementary defects, and whether the local fine inspection confidence level is overloaded with the corresponding threshold. The fusion verification strategy is triggered based on the category legality, label mapping consistency, and whether the global fine inspection confidence level meets the corresponding threshold in the defect fusion feedback of the global fine inspection mode.

[0036] Specifically, when a drone wirelessly connects to cloud or edge devices and executes a fusion verification strategy, it sends defect fusion feedback and corresponding data that meet the verification conditions to the connected edge device or cloud for secondary verification. For example, if the drone uses cloud verification and the label mapping consistency does not meet the requirements, it sends the global defect category and global detailed inspection defect box corresponding to the label mapping consistency to the cloud. The cloud then verifies and updates the label mapping consistency result before feeding it back to the drone.

[0037] Recognition mode update: The image recognition mode for the next round is updated based on the defect fusion feedback of the current round, the effective computing power of the next round, and the preset time delay of the inspection task.

[0038] Specifically, the computation time for the local fine inspection mode and the full-image fine inspection mode are determined based on the effective computing power of the drones in the next round. The primary screening criterion is that the computation time meets the preset delay of the inspection task and is shorter, while the secondary screening criterion is that the confidence level of the local fine inspection or the confidence level of the full-image fine inspection meets the requirements.

[0039] For example: If the computation time of neither image recognition mode meets the preset latency requirement in the next round, the image recognition mode with the shorter computation time will be run in the next round. If only one of the two image recognition modes meets the preset latency requirement in the next round, the image recognition mode with the faster computation time will be run in the next round. If the computation time of both image recognition modes in the next round meets the preset latency requirement, it is determined whether the local precision detection confidence level included in the defect fusion feedback of the current round meets the requirements. If yes, the image recognition mode of the current round is retained; otherwise, the image recognition mode is changed.

[0040] The local fine inspection mode includes a candidate box optimization strategy: It obtains the defect fusion degree from the defect fusion feedback of the local fine inspection mode, determines whether the defect fusion degree of the local fine inspection mode is lower than a corresponding threshold; if not, it continues to obtain the defect fusion degree of subsequent generated local fine inspection modes; otherwise, it is based on the intersection-union ratio (IUU) of the local fine inspection defect boxes and candidate defect boxes. The size of the candidate defect box is updated. This allows the candidate defect box size to increase as the drone approaches the faulty equipment and decrease as the drone moves away from the faulty equipment, improving the environmental adaptability of the local precision inspection mode. One embodiment of this application is: the fusion verification strategy includes cloud-based fusion verification running in the cloud and edge-based fusion verification running at the edge.

[0041] "Identification pattern update" includes: determining the connection status with the edge and cloud based on the communication status of the drone; obtaining the remaining computing power of the edge device and the cloud computing power; and matching the computing power required for the fusion verification strategy with the remaining edge computing power and cloud computing power to determine whether to perform cloud fusion verification or edge fusion verification. One embodiment of this application is as follows: edge fusion verification is used for low-computing-power scenarios with low conflict levels, low local fine-check confidence, and low global fine-check confidence; cloud fusion verification is used for high-computing-power scenarios where the load communication status allows and there are inconsistent label mappings, invalid categories, and high conflict levels.

[0042] One embodiment of this application involves dynamically adjusting the trigger threshold of the fusion verification strategy based on the computation time of the full-image fine inspection mode and the local fine inspection mode. For example, by adjusting the corresponding thresholds of the global fine inspection confidence or the local fine inspection confidence, the overall detection latency is reduced by adjusting the trigger frequency of the fusion verification strategy.

[0043] A power grid inspection system with collaborative semantic filtering and dedicated defect detectors includes: a data acquisition module, used to acquire power grid inspection images, current processing terminal (drone) computing power status, communication status, preset time delay constraints, and recognition mode parameters of the previous frame or batch; The mode scheduling module is used to determine the recognition mode of the current inspection image based on the current inspection task status, the current processing terminal computing power status, the communication status, the preset time delay corresponding to the inspection task, and the recognition result of the previous round. The semantic coarse localization module is used to run the semantic coarse localization model. In the local fine inspection mode, it performs scene understanding and candidate target screening on the power grid inspection images, outputs a set of candidate defect regions, and performs coordinate normalization, boundary expansion, minimum size constraint, cropping and enlargement and sorting on the set of candidate defect boxes corresponding to the inspection images, and outputs a queue of candidate defect boxes. A dedicated fine inspection module is used to perform fine-grained full-image inspection of power grid inspection images in full-image fine inspection mode and output global defect detection results, or to perform candidate region fine inspection of the candidate defect region queue in local fine inspection mode and output local defect detection results. The fusion verification module is used to generate defect fusion feedback. It generates defect fusion feedback that represents the detection performance of the current round by comparing the global defect detection results with the original inspection images and comparing the local defect detection results with the output of the semantic coarse localization model in the local fine inspection mode. The review module is used to control the operation of the fusion verification strategy and update the defect fusion feedback. It sends the global defect detection results, local defect detection results or conflict samples that meet the review conditions to the corresponding edge devices or cloud for secondary review, and receives the review results to update the global defect detection results and local defect detection results. The scheduling feedback module is used to update the recognition mode, candidate region expansion parameters, review threshold, or caller selection for the next frame or the next batch of inspection images. The results output module is used to output the power grid defect identification results, which include defect category, defect location, fusion confidence level and verification mark; The edge-cloud communication module is used to transmit images, candidate regions, fine inspection results, fusion results, verification results, and mode parameters between the airborne end, the edge end, and the cloud.

[0044] A computer device includes one or more processors and a memory, the memory storing computer program instructions executable by the one or more processors, wherein when the computer program instructions are executed by the one or more processors, a power grid inspection method of cooperative semantic filtering and dedicated defect detector is run.

[0045] A computer-readable storage medium storing computer program instructions that, when executed by a processor, run a power grid inspection method for cooperative semantic filtering and a dedicated defect detector.

[0046] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A power grid inspection method based on collaborative semantic filtering and a dedicated defect detector, characterized in that, This includes generating defect detection results, acquiring inspection images for the current round, recognizing inspection images based on the full-image fine inspection mode and local fine inspection mode corresponding to the current round, and outputting the corresponding global defect detection results and local defect detection results. Generate defect fusion feedback by comparing the global defect detection results with the original inspection images and comparing the local defect detection results with the output of the semantic coarse localization model in the local fine inspection mode, so as to generate defect fusion feedback representing the detection performance of the current round. Recognition mode update: The image recognition mode for the next round is updated based on the defect fusion feedback of the current round, the effective computing power of the next round, and the preset time delay of the inspection task.

2. The power grid inspection method based on collaborative semantic filtering and dedicated defect detector according to claim 1, characterized in that, The full-image fine inspection mode includes a first dedicated detector for fine identification of defects in the full image, and the global defect detection results include the global defect category, global fine inspection defect box, and global fine inspection confidence of each defect in the inspection image; The local fine inspection mode includes a semantic coarse localization model that uses candidate defect boxes to initially select defect regions, and a second dedicated detector that finely identifies defects within the candidate defect boxes. The local defect detection results include the local defect category of each defect in the inspection image, the local fine inspection defect box, and the local fine inspection confidence level.

3. The power grid inspection method based on collaborative semantic filtering and dedicated defect detector according to claim 1, characterized in that, The local fine inspection mode includes a triggerable defect supplementation strategy: detect whether there are supplementary defects outside the candidate defect box of the inspection image. If yes, the supplementary defect is added to the defect detection result; otherwise, the defect detection result is directly output.

4. The power grid inspection method based on collaborative semantic filtering and dedicated defect detector according to claim 2, characterized in that, The defect fusion feedback of the full-image fine inspection mode includes the category legality indicating whether the global defect category conforms to the preset category range, the label mapping consistency verification based on the reverse mapping of the global defect category and whether it is consistent with the corresponding global fine inspection defect box, and the global fine inspection confidence. The defect fusion feedback of the local fine inspection mode includes the defect fusion degree, which represents the consistency between the defect detection result and the output of the semantic coarse localization model; the degree of conflict between the local defect type and the defect in the corresponding candidate defect box; and the local fine inspection confidence.

5. The power grid inspection method based on collaborative semantic filtering and dedicated defect detector according to claim 4, characterized in that, The formula for calculating the defect fusion degree is: ， Where F represents the degree of defect fusion, This indicates the coarse location reliability of the semantic coarse localization model. Indicates the confidence level of the local precision test. This represents the intersection-union ratio (IoU) of the local precision defect bounding box and the candidate defect bounding box. This indicates the consistency index between local defect types and candidate defect types.

6. The power grid inspection method based on collaborative semantic filtering and dedicated defect detector according to claim 4, characterized in that, Both the full-map fine inspection mode and the local fine inspection mode are equipped with a fusion verification strategy based on defect fusion feedback triggering to re-verify and update the defect fusion feedback; The fusion verification strategy includes: receiving relevant data corresponding to the defect fusion feedback that triggers the fusion verification strategy, and reviewing, calculating and updating the corresponding defect fusion feedback based on the relevant data.

7. The power grid inspection method based on collaborative semantic filtering and dedicated defect detector according to claim 6, characterized in that, The fusion verification strategy includes cloud-based fusion verification running in the cloud and edge-based fusion verification running at the edge. "Identification mode update" includes: obtaining the connection status with the edge and cloud, matching the computing power required to run the fusion verification strategy with the remaining computing power of the corresponding edge device and cloud, so as to determine whether to perform cloud fusion verification or edge fusion verification.

8. The power grid inspection method based on collaborative semantic filtering and dedicated defect detector according to claim 4, characterized in that, The local fine inspection mode includes a candidate box optimization strategy: It determines whether the defect fusion degree of the local fine inspection mode in the current round is lower than the corresponding threshold. If not, it continues to obtain the defect fusion degree generated by the local fine inspection mode in subsequent rounds. If yes, it is based on the intersection-union ratio (IUU) of the local fine inspection defect box and the candidate defect box. Update the size of the candidate defect box.

9. A power grid inspection system using collaborative semantic filtering and a dedicated defect detector, comprising the power grid inspection method using collaborative semantic filtering and a dedicated defect detector according to any one of claims 1-8, characterized in that, It includes a data acquisition module for acquiring power grid inspection images, a semantic coarse localization module for running a semantic coarse localization model, a dedicated fine inspection module for performing fine-grained detection of the entire image or candidate defect boxes, a fusion verification module for generating defect fusion feedback, a verification module for controlling the operation of fusion verification strategy and updating defect fusion feedback, a scheduling feedback module for updating the image recognition mode of the next round, a result output module for outputting global defect detection results or local defect detection results, and an edge-cloud communication module for data transmission between the edge and the cloud.

Images