Defect detection method, device, equipment and medium

Abstract

The invention discloses a defect detection method, device and equipment and a medium, and relates to the technical field of power grids, artificial intelligence and the like, and the method comprises the steps: carrying out the multi-view collection of a target object based on an unmanned plane, and obtaining an original image; performing defect detection on the target object based on the original image to obtain a preliminary detection result; under the condition that the preliminary detection result shows that the confidence coefficient of the defect of the target object is smaller than a preset threshold value, triggering a meta-cognitive processing mode, and performing three-dimensional reconstruction based on the original image to obtain a three-dimensional model of the target object and acquisition parameter information of the original image; obtaining a sampling view angle sequence of the target object based on the three-dimensional model, the original image and the acquisition parameter information; based on the sampling view angle sequence, rendering a three-dimensional image corresponding to the original image in the three-dimensional model to obtain an image rendering result; and performing defect detection on the target object based on the image rendering result to obtain a defect detection result.

Description

Technical Field

[0001] The present invention relates to technical fields such as power grids and artificial intelligence, and in particular to a defect detection method, device, equipment and medium. Background Art

[0002] The distribution network is a vital component of the power system, and its safe and stable operation is crucial. Key components on distribution towers, such as insulators and tension clamps, provide electrical insulation and mechanical support. Due to long-term exposure to complex natural environments, these components can develop defects such as cracks, breakage, and spontaneous explosions due to factors such as lightning strikes, material aging, and external damage. If these defects are not detected and addressed promptly, they can compromise line safety and threaten the security of the power grid. Therefore, improving the accuracy and efficiency of defect detection has become a key concern.

[0003] Related technologies use drones for defect detection, and their accuracy is highly dependent on the image quality and viewing angle of a single shot. The visibility of component defects on pole towers is highly directional, resulting in the limitation of a single viewing angle. Another related technology uses drones to perform complex circling flights or multi-point hovering shots on each pole tower to be inspected, which greatly increases the inspection time of a single-base tower and significantly consumes the precious power of the drone, thereby reducing the overall inspection mileage and efficiency of a single flight. Another related technology performs inspections through several predetermined viewing angles, resulting in a large difference in the viewing angles of the captured images, which directly leads to a decrease in detection performance due to factors such as angle and lighting during subsequent inspections. Another related technology addresses the viewing angle problem by optimizing hardware and flight strategies, requiring manual intervention by pilots or background personnel to determine the optimal observation angle. The degree of automation is not high, and it mainly relies on a deep learning-based target detection model to analyze single or multiple frames of images taken by drones. There is a lack of correlation between images, and there is no active verification mechanism when encountering uncertain situations. Summary of the Invention

[0004] The embodiments of the present invention are intended to at least partially address one of the technical problems in the related art. To this end, one object of the present invention is to provide a defect detection method, apparatus, device, and medium that can improve the accuracy and efficiency of defect detection and can actively verify defect detection results.

[0005] An embodiment of the present invention provides a defect detection method, which includes: performing multi-perspective acquisition of a target object based on a drone to obtain an original image; performing defect detection on the target object based on the original image to obtain a preliminary detection result; when the preliminary detection result indicates that the confidence level of the target object's defect is less than a preset threshold, triggering a metacognitive processing mode, performing three-dimensional reconstruction based on the original image to obtain a three-dimensional model of the target object and acquisition parameter information of the original image; obtaining a sampling perspective sequence of the target object based on the three-dimensional model, the original image, and the acquisition parameter information; rendering a three-dimensional image corresponding to the original image in the three-dimensional model based on the sampling perspective sequence to obtain an image rendering result; performing defect detection on the target object based on the image rendering result to obtain a defect detection result.

[0006] Exemplarily, the original image is located in an image group; based on the three-dimensional model, the original image and acquisition parameter information, a sampling perspective sequence of the target object is obtained, including: indexing and determining the original image from the image group based on the preliminary detection results, and performing image segmentation on the original image to obtain mask information of the original image; based on the three-dimensional model, the mask information of the original image and the acquisition parameter information, a sampling perspective sequence of the target object is obtained.

[0007] Exemplarily, the preliminary detection result includes image index information, first defect confidence information, and position information of the target object in the original image; based on the preliminary detection result, the original image is indexed and determined from the image group, and the original image is segmented to obtain the mask information of the original image, including: based on the image index information, the original image is indexed and determined from the image group; based on the position information, the original image is segmented to obtain the mask information of the original image. Exemplarily, the acquisition parameter information of the original image includes intrinsic reference information, extrinsic reference information and depth information, and the sampling view sequence includes target intrinsic reference information and target extrinsic reference information; based on the three-dimensional model, the mask information of the original image and the acquisition parameter information, a sampling view sequence of the target object is obtained, including: determining the target intrinsic reference information as the target intrinsic reference information; based on the intrinsic reference information, extrinsic reference information and depth information of the original image, according to the three-dimensional model, the pixel points of the mask information of the original image are back-projected into the three-dimensional space to obtain three-dimensional point cloud data; based on the center point of the three-dimensional point cloud data, reference extrinsic reference information is obtained; based on the reference extrinsic reference information, geometric transformation and combination are performed to obtain target extrinsic reference information.

[0008] Exemplarily, the geometric transformation includes depth translation, spherical interpolation sampling and tangent plane translation; geometric transformation and combination are performed based on the reference extrinsic parameter information to obtain target extrinsic parameter information, including: taking the reference extrinsic parameter information as the center, performing depth translation along the depth axis direction to obtain the first extrinsic parameter; taking the reference extrinsic parameter information as the center of the sphere, performing coordinate interpolation movement on the spherical surface to obtain the second extrinsic parameter; taking the reference extrinsic parameter information as the center, performing tangent plane translation along the direction perpendicular to the center plane to obtain the third extrinsic parameter; based on a combination of any one or more of the first extrinsic parameter, the second extrinsic parameter, and the third extrinsic parameter, the target extrinsic parameter information is obtained.

[0009] Exemplarily, based on the sampling perspective sequence, the three-dimensional image corresponding to the original image in the three-dimensional model is rendered to obtain an image rendering result, including: taking the center point of the three-dimensional point cloud data as the rendering center, rendering the three-dimensional image corresponding to the original image in the three-dimensional model to obtain an image rendering result.

[0010] Exemplarily, the image rendering result includes image index information, first defect confidence information and location information corresponding to the preliminary detection result; based on the image rendering result, defect detection is performed on the target object to obtain a defect detection result, including: based on the image rendering result, defect detection is performed on the target object to obtain second defect confidence information; when the second defect confidence information indicates that the confidence that there is a defect in the target object is less than a preset threshold, the corresponding preliminary detection result is removed based on the image index information to obtain a defect detection result.

[0011] Exemplarily, the defect detection result includes image index information corresponding to the preliminary detection result, first defect confidence information corresponding to the preliminary detection result or second defect confidence information corresponding to the defect detection result, and location information corresponding to the preliminary detection result; the method also includes: based on the image index information, performing structured organization processing on the defect detection result to obtain a defect report; and displaying the defect report.

[0012] Exemplarily, the target object includes a distribution pole tower; the original image is obtained by multi-perspective acquisition of the target object by a drone, including: the front view, left view, right view, top of the distribution pole tower and the whole view of the distribution pole tower are acquired by the drone. Another embodiment of the present invention provides a defect detection device, which includes: an acquisition module for performing multi-perspective acquisition of a target object based on a drone to obtain an original image; a first detection module for performing defect detection on the target object based on the original image to obtain a preliminary detection result; a reconstruction module for triggering a metacognitive processing method when the preliminary detection result indicates that the confidence level of the target object defect is less than a preset threshold, and performing three-dimensional reconstruction based on the original image to obtain a three-dimensional model of the target object and acquisition parameter information of the original image; an acquisition module for obtaining a sampling perspective sequence of the target object based on the three-dimensional model, the original image and the acquisition parameter information; a rendering module for rendering the three-dimensional image corresponding to the original image in the three-dimensional model based on the sampling perspective sequence to obtain an image rendering result; and a second detection module for performing defect detection on the target object based on the image rendering result to obtain a defect detection result.

[0013] Another embodiment of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method of any of the above embodiments when executing the computer program.

[0014] Another embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method of any one of the above embodiments are implemented.

[0015] In the above embodiment, the defect detection method includes: performing multi-perspective acquisition of the target object based on the drone to obtain the original image; performing defect detection on the target object based on the original image to obtain a preliminary detection result; when the preliminary detection result indicates that the confidence level of the existence of defects in the target object is less than a preset threshold, triggering the metacognitive processing method, performing three-dimensional reconstruction based on the original image to obtain a three-dimensional model of the target object and acquisition parameter information of the original image; obtaining a sampling perspective sequence of the target object based on the three-dimensional model, the original image and the acquisition parameter information; rendering the three-dimensional image corresponding to the original image in the three-dimensional model based on the sampling perspective sequence to obtain an image rendering result; performing defect detection on the target object based on the image rendering result to obtain a defect detection result. The above-mentioned metacognitive processing is triggered when the confidence level of the defect is less than the preset threshold, and three-dimensional reconstruction is performed to obtain a sampling perspective sequence. Active rendering and re-detection are performed from the sampling perspective, which can effectively "see through" occlusions and "bypass" adverse reflections, thereby accurately identifying those hidden and tiny defects that are difficult to find under the original shooting perspective (such as side cracks, obscured self-explosions, etc.), greatly improving the accuracy and recall rate of defect detection; and significantly shortening the operation time of a single tower, increasing the inspection coverage of a single flight, directly reducing the operating cost of power inspection, and achieving the best balance between detection efficiency and accuracy.

[0016] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flow chart of a defect detection method provided by an embodiment of the present invention; Figure 2 An overall flow chart of the defect detection method provided by an embodiment of the present invention; Figure 3 A block diagram of a defect detection device provided in another embodiment of the present invention; Figure 4 A block diagram of an electronic device provided in accordance with another embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0019] The distribution network is a vital component of the power system, and its safe and stable operation is crucial. Key components on distribution towers, such as insulators and tension clamps, provide electrical insulation and mechanical support. Due to long-term exposure to complex natural environments, these components can develop defects such as cracks, breakage, and spontaneous explosions due to factors such as lightning strikes, material aging, and external damage. If these defects are not discovered and addressed promptly, they can compromise line safety and threaten the security of the power grid.

[0020] In recent years, the use of drones equipped with high-definition cameras for power transmission and distribution line inspections has become a mainstream trend. Compared to traditional manual tower inspections, drone inspections offer significant advantages, including high efficiency, low cost, improved safety, and unrestricted terrain. During the inspection process, the drones fly along a predetermined route, capturing images of the towers and key components. Intelligent recognition algorithms then automatically detect defects.

[0021] However, the accuracy of related drone-based vision inspection methods is highly dependent on the image quality and viewing angle of a single shot, and the visibility of component defects on towers is highly directionally sensitive. For example, if an insulator explodes (shed damage), the defect will be clearly visible if the drone's camera is facing directly or facing away from the damaged surface. However, if the camera is taken from the side (i.e., along the insulator's radial direction), the damaged feature may be obscured by the intact shed structure.

[0022] To overcome the limitations of a single viewing angle, current industry practice typically requires drones to perform complex circling flights or multi-point hovering photography of each tower under inspection to capture as many observation angles as possible. While this approach can improve detection rates to a certain extent, it significantly increases the inspection time for a single tower and consumes valuable drone power, thereby reducing the overall inspection range and efficiency of a single flight. Furthermore, some inspections are performed using a few predefined viewing angles (typically five: front view, left view, right view, tower top (or overhead view), and full view). While this method offers faster capture speed, the resulting discrepancies between the captured images can lead to performance degradation during subsequent inspections due to factors such as angle and lighting. Therefore, improving the efficiency of drone inspections while maintaining high detection rates and accuracy remains a pressing technical challenge in this field.

[0023] Related technologies mainly deal with the perspective problem by optimizing hardware and flight strategies. For example, a drone equipped with a high-magnification zoom gimbal can be used to zoom in and observe suspicious points after they are found. However, this still requires manual intervention by the pilot or backstage personnel to determine the best observation angle, and the degree of automation is not high. At the algorithm level, it mainly relies on the target detection model based on deep learning to analyze single or multiple frames of images taken by the drone. There is a lack of correlation between images, and there is no active verification mechanism when encountering uncertain situations. The problems are as follows: (1) The contradiction between inspection efficiency and detection accuracy: In order to ensure high accuracy and low missed detection rate, the drone needs to perform time-consuming and power-consuming circling flights, which leads to low inspection efficiency; if a fast flyby method is used, the detection accuracy will be sacrificed due to the single shooting angle. (2) Limited intelligence level and lack of active verification ability: The existing algorithm is passive in reasoning. For a target that "looks like but is uncertain", the model cannot actively seek more information to support or overturn its initial judgment, that is, it lacks a "metacognitive" reflection ability similar to that of human experts.

[0024] To this end, embodiments of the present invention provide a defect detection method that no longer treats a set of drone-captured images (hereinafter referred to as raw images) as independent inspection tasks. Instead, it treats them as a sparse, multi-view scene input to construct a high-precision local 3D digital twin. When preliminary analysis of the raw images indicates the presence of an uncertain or suspected defect, the system triggers a metacognitive process: It uses the constructed 3D model to find the optimal sampling perspective based on the current uncertain defect, then renders a new image for further inspection. This "virtual close inspection" eliminates uncertainty, ultimately forming a highly reliable inspection conclusion (defect detection result) that is verified from multiple perspectives.

[0025] Figure 1 A flow chart of a defect detection method provided in an embodiment of the present invention.

[0026] like Figure 1 As shown, the defect detection method 100 includes steps S110 to S160.

[0027] Step S110 , acquiring original images by performing multi-view capture of the target object using the drone. For example, the target object includes a distribution tower, and the multi-view acquisition can be a front view, a left view, a right view, a top view, and a full view of the distribution tower, etc. The original image is an image sequence I composed of multi-view images, including multiple images, such as the image sequence I={ } consists of N pictures.

[0028] Step S120 : performing defect detection on the target object based on the original image to obtain a preliminary detection result. Exemplarily, defect detection uses a defect detection model M1, which can be a trained yolov5 component defect detection model M1. The specific architecture of the defect detection model M1 is not specifically limited. The preliminary detection result R may include image index information (used to correspond the preliminary detection result R to the original image), confidence information (used to indicate the reliability of the preliminary detection result R), and position information (used to indicate the specific position of the preliminary detection result R). For the input image sequence I, the corresponding preliminary detection result R may include multiple, such as preliminary detection result R={ },in , R1 represents the detection result of I1 in the original image sequence, R2 represents the detection result of I2 in the original image sequence, and so on, Indicates the image index (i.e., which original image it corresponds to). Indicates the confidence level of M1 in determining that the target is a defect. Indicates that M1 determines the position of the target.

[0029] In step S130, when the confidence level of the preliminary detection result indicating that the target object has defects is less than a preset threshold, a metacognitive processing method is triggered to perform three-dimensional reconstruction based on the original image to obtain a three-dimensional model of the target object and acquisition parameter information of the original image.

[0030] For example, the acquisition parameter information of the original image may include internal reference information, external reference information and depth information. The preliminary detection result includes a confirmed result and a result to be verified (distinguished based on the confidence information in the preliminary detection result). When the defect confidence information of the preliminary detection result is higher than the preset threshold T (confirmed result), the preliminary detection result is used as the final defect detection result, and the confirmed result is marked with r_certain={ ,..., }; When the defect confidence information of the preliminary test result is lower than the preset threshold T (result to be verified), the preliminary test result needs to be further confirmed, and the result to be verified is recorded as r_uncertain= { ,... }, it will actively trigger the metacognitive processing mode, obtain the original image corresponding to the preliminary detection result according to the index information corresponding to the preliminary detection result, input the original image into the forward 3D reconstruction algorithm, and obtain the reconstructed 3D model M2 of the distribution tower and the camera intrinsic parameter K, extrinsic parameter P and depth map D corresponding to the image in I, where the intrinsic parameter information is shared and the extrinsic parameter information P={ } and depth information D={ Each original image has a 3D reconstruction algorithm, which is a public algorithm. It can be AnySplat, which supports 3D scene reconstruction from sparse multi-view input and predicts camera intrinsic parameters, extrinsic parameters, and depth maps.

[0031] Step S140 : obtaining a sampling view sequence of the target object based on the three-dimensional model, the original image, and the acquisition parameter information.

[0032] Exemplarily, the sampling view sequence includes intrinsic reference information and extrinsic reference information. According to the original image corresponding to the preliminary detection result, the public Segment Everything (SAM) model is used to obtain the mask of the corresponding target. The pixel points on the mask are back-projected based on the acquisition parameter information to obtain the reference extrinsic reference information. The three-dimensional model is geometrically transformed based on the reference extrinsic reference information to obtain the sampling view sequence, wherein the sampling view sequence may include multiple.

[0033] Step S150 : Rendering the three-dimensional image corresponding to the original image in the three-dimensional model based on the sampling view sequence to obtain an image rendering result.

[0034] Exemplarily, based on the obtained multiple sampling perspective sequences, the three-dimensional image corresponding to the original image in the three-dimensional model is converted into a high-quality two-dimensional image under a specific perspective (sampling perspective sequence), and the image rendering result includes the two-dimensional images corresponding to the multiple sampling perspective sequences.

[0035] Step S160 , performing defect detection on the target object based on the image rendering result to obtain a defect detection result.

[0036] Exemplarily, the defect detection result includes image index information, confidence information and location information. The image rendering result is subjected to defect detection using a defect detection model M1. The defect detection model may be a trained yolov5 component defect detection model M1. The specific architecture adopted by the defect detection model M1 is not specifically limited. The image index information (used to correspond the result obtained using the defect detection model with the original image), confidence information (used to indicate the reliability of the result obtained using the defect detection model) and location information (used to indicate the specific location of the result obtained using the defect detection model) are obtained. The confidence information is compared with a preset threshold T, and the results to be verified that are smaller than the preset threshold are eliminated. The results to be verified that are higher than the preset threshold T and the confident results higher than the preset threshold T in step S130 are used as the final defect detection results.

[0037] In the above embodiment, when the confidence level of a defect is less than a preset threshold, metacognitive processing is triggered, and three-dimensional reconstruction is performed to obtain a sampling perspective sequence. Active rendering and re-detection are performed from the sampling perspective, which can effectively "see through" obstructions and "bypass" adverse reflections, thereby accurately identifying those hidden and minor defects (such as side cracks, obscured self-explosions, etc.) that are difficult to detect from the original shooting perspective, greatly improving the accuracy and recall rate of defect detection; and significantly shortening the operation time of a single-base tower, increasing the inspection coverage of a single flight, and directly reducing the operating cost of power inspection. The high-cost three-dimensional analysis and virtual rendering process is only started for low-confidence targets "to be verified", achieving the best balance between detection efficiency and accuracy.

[0038] The original image is located in the image group; based on the three-dimensional model, the original image and the acquisition parameter information, a sampling perspective sequence of the target object is obtained, including: indexing and determining the original image from the image group based on the preliminary detection result, and performing image segmentation on the original image to obtain mask information of the original image; based on the three-dimensional model, the mask information of the original image and the acquisition parameter information, a sampling perspective sequence of the target object is obtained.

[0039] Specifically, image segmentation uses the public Segment Everything (SAM) model to obtain a mask corresponding to the target. Based on the acquisition parameter information, the pixel points on the mask are back-projected to obtain reference extrinsic parameter information. Based on the reference extrinsic parameter information, a geometric transformation is performed on the three-dimensional model to obtain a sampling perspective sequence, where the sampling perspective sequence can include multiple.

[0040] Exemplarily, the preliminary detection result includes image index information, first defect confidence information, and position information of the target object in the original image; based on the preliminary detection result, the original image is indexed and determined from the image group, and the original image is segmented to obtain the mask information of the original image, including: based on the image index information, the original image is indexed and determined from the image group; based on the position information, the original image is segmented to obtain the mask information of the original image. For example, for each element in r_uncertain ,according to Get the original image , input the image and (location information) to SAM to get the corresponding target (Mask information).

[0041] Exemplarily, the acquisition parameter information of the original image includes intrinsic reference information, extrinsic reference information and depth information, and the sampling view sequence includes target intrinsic reference information and target extrinsic reference information; based on the three-dimensional model, the mask information of the original image and the acquisition parameter information, a sampling view sequence of the target object is obtained, including: determining the target intrinsic reference information as the target intrinsic reference information; based on the intrinsic reference information, extrinsic reference information and depth information of the original image, according to the three-dimensional model, the pixel points of the mask information of the original image are back-projected into the three-dimensional space to obtain three-dimensional point cloud data; based on the center point of the three-dimensional point cloud data, reference extrinsic reference information is obtained; based on the reference extrinsic reference information, geometric transformation and combination are performed to obtain target extrinsic reference information.

[0042] Specifically, the intrinsic parameter information of the original image is shared, and all original images correspond to the same intrinsic parameter information. The geometric transformation can include depth translation, spherical interpolation sampling and tangent plane translation. The target extrinsic parameter information includes multiple, which can be the same target intrinsic parameter information K and multiple target extrinsic parameter information. .

[0043] For example, using the camera pose {K, } and depth information , unproject the pixels on the two-dimensional mask mask_2d into the three-dimensional space to obtain a set of three-dimensional point clouds describing the spatial location of the defect ;calculate The center point of the defect is set as the center point of the rendered image, and the camera external parameters facing the defect center can be obtained. (Refer to external reference information); according to , a set of preset geometric transformation strategies (depth translation, spherical interpolation sampling and tangent plane translation) are used to generate diverse sampling perspectives (sampling perspective sequences).

[0044] Exemplarily, the geometric transformation includes depth translation, spherical interpolation sampling and tangent plane translation; geometric transformation and combination are performed based on the reference extrinsic parameter information to obtain target extrinsic parameter information, including: taking the reference extrinsic parameter information as the center, performing depth translation along the depth axis direction to obtain the first extrinsic parameter; taking the reference extrinsic parameter information as the center of the sphere, performing coordinate interpolation movement on the spherical surface to obtain the second extrinsic parameter; taking the reference extrinsic parameter information as the center, performing tangent plane translation along the direction perpendicular to the center plane to obtain the third extrinsic parameter; based on a combination of any one or more of the first extrinsic parameter, the second extrinsic parameter, and the third extrinsic parameter, the target extrinsic parameter information is obtained.

[0045] Specifically, the target extrinsic parameter information can be the first extrinsic parameter, the second extrinsic parameter or the third extrinsic parameter, or a combination of the first extrinsic parameter and the second extrinsic parameter, the second extrinsic parameter and the third extrinsic parameter, or a combination of the first extrinsic parameter, the second extrinsic parameter and the third extrinsic parameter, wherein the first extrinsic parameter is a plurality of extrinsic parameter information obtained by moving with the reference extrinsic parameter information as the center, the second extrinsic parameter is a plurality of extrinsic parameter information obtained by moving with the reference extrinsic parameter information as the center of the sphere, and the third extrinsic parameter is a plurality of extrinsic parameter information obtained by translating along a tangent plane perpendicular to the tangent plane centered on the reference extrinsic parameter information. The geometric transformation is not limited to depth translation, spherical interpolation sampling and tangent plane translation, but may also include other forms of geometric transformation.

[0046] For example, according to (Refer to external reference information) A set of preset geometric transformation strategies are used to generate diverse sampling perspectives. These strategies include but are not limited to: a) Translation along the depth axis: Move the camera forward and backward a certain distance along its principal optical axis (i.e., toward the center of the defect) to simulate the observation effects of “zooming in” and “zooming out”.

[0047] b) Spherical interpolation sampling: Position the camera at the defect center point Spherical coordinate interpolation is performed on a virtual sphere at the center of the sphere, while keeping the camera always facing the center of the sphere to simulate the effect of surround observation.

[0048] c) Tangential plane translation: The camera position is translated up, down, left, and right on a plane perpendicular to its principal optical axis to observe the morphology of defects under different grazing angles of illumination.

[0049] By combining these strategies, we finally get u_A new camera extrinsics, which constitute the complete sampling perspective set of the defect to be verified. (target external reference information) and (target external reference information), where ={ }, Directly obtain the intrinsic reference information K of the original image, and then there is a group A of sampling angles (an intrinsic reference and an extrinsic reference together determine a shooting angle and field of view, that is, the shooting scene is determined).

[0050] In the above embodiment, 2D segmentation mask, camera pose and 3D model geometric information are used to jointly determine one or a group of virtual observation perspectives that can maximize defect information gain, which greatly improves the accuracy and recall rate of defect detection.

[0051] Based on the sampling perspective sequence, the three-dimensional image corresponding to the original image in the three-dimensional model is rendered to obtain an image rendering result, including: taking the center point of the three-dimensional point cloud data as the rendering center, rendering the three-dimensional image corresponding to the original image in the three-dimensional model to obtain an image rendering result.

[0052] Specifically, based on the obtained multiple sampling view sequences, the three-dimensional image corresponding to the original image in the three-dimensional model is converted into Converted into a high-quality two-dimensional image under a specific perspective (sampling perspective sequence). The image rendering result includes two-dimensional images corresponding to multiple sampling perspective sequences.

[0053] For example, For the center to render center, according to the 3D model M2 and the sampling view sequence , , render the corresponding image sequence , and get the image rendering result.

[0054] The image rendering result includes image index information, first defect confidence information and position information corresponding to the preliminary detection result; based on the image rendering result, defect detection is performed on the target object to obtain a defect detection result, including: based on the image rendering result, defect detection is performed on the target object to obtain second defect confidence information; when the second defect confidence information indicates that the confidence level of the existence of a defect in the target object is less than a preset threshold, the corresponding preliminary detection result is removed based on the image index information to obtain a defect detection result.

[0055] Specifically, the first defect confidence information includes confidence information corresponding to the preliminary detection result. The defect detection uses a defect detection model M1 for defect detection. The defect detection model can be a trained yolov5 component defect detection model M1. The specific architecture adopted by the defect detection model M1 is not specifically limited. The image index information (used to correspond the result obtained using the defect detection model with the original image), confidence information (used to indicate the reliability of the result obtained using the defect detection model) and location information (used to indicate the specific location of the result obtained using the defect detection model) are obtained. Based on the confidence information and the preset threshold T, the results to be verified that are less than the preset threshold are eliminated, and the results to be verified that are higher than the preset threshold T and the confident results higher than the preset threshold T in step S130 are used as the final defect detection results.

[0056] For example, using the M1 model to render Perform the test and output the corresponding test results to obtain the highest confidence level. If the highest confidence level is still lower than the preset threshold T, it is considered not a defect.

[0057] Filter from r_uncertain (results to be verified); finally, obtain the target set r_uncertain_update (results to be verified after removal) after removing all non-defects; integrate the "confident" detection result r_certain and the result r_uncertain_update after the metacognitive verification process to output the final result.

[0058] The defect detection results include image index information corresponding to the preliminary detection results, first defect confidence information corresponding to the preliminary detection results or second defect confidence information corresponding to the defect detection results, and location information corresponding to the preliminary detection results; the method also includes: based on the image index information, performing structured organization processing on the defect detection results to obtain a defect report; and displaying the defect report.

[0059] For example, the "certain" defect set r_certain is integrated with the verified defect set r_uncertain_update to form a final, highly reliable defect report. This integration is done as follows: First, all defect records in r_certain are directly incorporated into the final report. Second, each verified defect record in r_uncertain_update is also incorporated into the final report. The final report R_new can be structured according to the original image index img_index. For example, for each original image, all "certain" defects and verified defects contained therein are listed, along with their final, high-confidence scores and precise locations.

[0060] In the above embodiment, by effectively integrating the "confident" result from the original image and the "verified" result from the virtual image to form a final high-reliability report, the defect detection results can be observed more intuitively.

[0061] The target object includes a distribution pole tower; the original image is obtained by performing multi-view acquisition of the target object based on the drone, including: collecting the front view, left view, right view, top view and overall view of the distribution pole tower based on the drone to obtain the original image.

[0062] For example, the multi-view capture can be a front view, a left view, a right view, a top view of a distribution tower, and a full view of a distribution tower, etc., which is not a limitation of the present application. The original image is an image sequence I composed of multi-view images, which includes multiple images, such as a picture sequence I={ } consists of N pictures.

[0063] In the above-mentioned embodiment, by constructing a high-fidelity 3D digital twin and proactively rendering and re-inspecting it from the "optimal" virtual perspective, the system can effectively "see through" obstructions and "circumvent" adverse reflections, accurately identifying hidden and subtle defects (such as side cracks and obscured explosions) that are difficult to detect from the original shooting perspective. This significantly improves the accuracy and recall rate of defect detection. Furthermore, through a purely algorithmic approach of "virtual close-up inspection," multi-angle analysis of suspicious points can be performed on the server side. This allows the drone to perform efficient and rapid standard flight path photography at the front end, eliminating the need to repeatedly adjust its posture and position for individual suspicious points. This significantly shortens the operation time of a single tower, increases the inspection coverage of a single flight, and directly reduces the operating costs of power inspections. Furthermore, through a "metacognitive" working paradigm, the system can autonomously plan "where to look" (optimal perspective) and "how to look" (geometric transformation strategy). This represents an intelligent upgrade from passive response to active exploration, bringing the inspection system closer to the thinking mode of human experts.

[0064] Figure 2 This is an overall flow chart of the defect detection method provided in an embodiment of the present invention.

[0065] like Figure 2 As shown, the defect detection method 200 includes steps S201 to S208.

[0066] Step S201: a sequence of pictures taken at different points on the same mast.

[0067] For example, for the image group I taken of the same tower, the image group I of a tower in this embodiment is composed of five predetermined perspectives (front view, left view, right view, tower top, and full view of the tower). The image group can also be composed in other ways.

[0068] Step S202: Defect detection model M1 obtains detection result R.

[0069] Exemplarily, the images in the image group I are input one by one into the first-order component defect detection model M1 to obtain a preliminary detection result R.

[0070] Step S203A: obtaining the to-be-verified set r_certain according to the threshold.

[0071] Step S203B: obtaining the to-be-verified set r_uncertain according to the threshold.

[0072] Exemplarily, the detection results are analyzed, and the results with a confidence level lower than a preset threshold T are marked as "to be verified", and subsequent steps S204-S207 are performed, and the remaining detection results are marked as "confident".

[0073] In step S204 , a 3D reconstruction algorithm obtains corresponding internal parameters, external parameters, a depth map, and a 3D model.

[0074] Step S205, determine whether the results in r_uncertain are traversed to completion, if not, go to S206, if yes, integrate the results.

[0075] Step S206: Calculate the optimal sampling view sequence and render the image, and use M1 detection to obtain the detection result.

[0076] Step S207, determine whether the highest confidence level of the detection result is greater than a threshold; if not, update r_uncertain; if so, go to step S205.

[0077] Step S208: integrating the results.

[0078] Exemplarily, the “certain” detection result r_certain and the result r_uncertain_update after the metacognitive verification process are integrated to output the final result.

[0079] In the above embodiment, the optimal balance between detection efficiency and accuracy is achieved through the workflow of multi-view initial inspection → confidence-based uncertainty recognition and metacognitive triggering → three-dimensional scene reconstruction based on multi-view input → active optimal virtual perspective planning based on 2D / 3D information linkage → rendering, re-inspection and final judgment based on virtual perspective → result integration and output closed loop.

[0080] Figure 3 This is a block diagram of a defect detection device according to another embodiment of the present invention.

[0081] The embodiment of the present invention provides a defect detection device 300, see Figure 3The defect detection apparatus 300 comprises: an acquisition module 310, a first detection module 320, a reconstruction module 330, an obtaining module 340, a rendering module 350, and a second detection module 360.

[0082] The acquisition module 310 is configured to acquire the original image based on multi-view acquisition of the target object by the UAV.

[0083] The first detection module 320 is configured to perform defect detection on the target object based on the original image to obtain a preliminary detection result.

[0084] The reconstruction module 330 is configured to, in a case where a confidence level of the preliminary detection result indicating that the target object has a defect is less than a preset threshold, trigger a meta-cognition processing mode, perform three-dimensional reconstruction based on the original image to obtain a three-dimensional model of the target object and acquisition parameter information of the original image.

[0085] The obtaining module 340 is configured to obtain a sampling view sequence of the target object based on the three-dimensional model, the original image, and the acquisition parameter information.

[0086] The rendering module 350 is configured to perform rendering on a three-dimensional image corresponding to the original image in the three-dimensional model based on the sampling view sequence to obtain an image rendering result.

[0087] The second detection module 360 is configured to perform defect detection on the target object based on the image rendering result to obtain a defect detection result.

[0088] It can be understood that the specific description of the defect detection apparatus 300 can refer to the description of the defect detection method in the foregoing, and will not be repeated here.

[0089] The original image is located in an image group; the obtaining module 340 is further configured to determine the original image from the image group based on the preliminary detection result, perform image segmentation on the original image to obtain mask information of the original image, and obtain the sampling view sequence of the target object based on the three-dimensional model, the mask information of the original image, and the acquisition parameter information. The preliminary detection result comprises image index information, first defect confidence level information, and position information of the target object in the original image; the obtaining module 340 is further configured to determine the original image from the image group based on the image index information, perform image segmentation on the original image based on the position information to obtain mask information of the original image.

[0090] Exemplarily, the acquisition parameter information of the original image includes intrinsic reference information, extrinsic reference information and depth information, and the sampling perspective sequence includes target intrinsic reference information and target extrinsic reference information; the acquisition module 340 is also used to determine the target intrinsic reference information with the intrinsic reference information of the original image; based on the intrinsic reference information, extrinsic reference information and depth information of the original image, the pixel points of the mask information of the original image are back-projected into the three-dimensional space according to the three-dimensional model to obtain three-dimensional point cloud data; based on the center point of the three-dimensional point cloud data, reference external reference information is obtained; based on the reference external reference information, geometric transformation and combination are performed to obtain target external reference information. Exemplarily, the geometric transformation includes depth translation, spherical interpolation sampling and tangent plane translation; the acquisition module 340 is also used to perform depth translation along the depth axis with the reference extrinsic parameter information as the center to obtain the first extrinsic parameter; with the reference extrinsic parameter information as the center of the sphere, coordinate interpolation movement is performed on the spherical surface to obtain the second extrinsic parameter; with the reference extrinsic parameter information as the center, tangent plane translation is performed in a direction perpendicular to the center plane to obtain the third extrinsic parameter; based on a combination of any one or more of the first extrinsic parameter, the second extrinsic parameter and the third extrinsic parameter, the target extrinsic parameter information is obtained.

[0091] Exemplarily, the rendering module 350 is further configured to render a three-dimensional image corresponding to the original image in the three-dimensional model with the center point of the three-dimensional point cloud data as the rendering center to obtain an image rendering result. Exemplarily, the image rendering result includes image index information, first defect confidence information and location information corresponding to the preliminary detection result; the second detection module 360 ​​is also used to perform defect detection on the target object based on the image rendering result to obtain second defect confidence information; when the second defect confidence information indicates that the confidence that there is a defect in the target object is less than a preset threshold, the corresponding preliminary detection result is removed based on the image index information to obtain a defect detection result. Exemplarily, the defect detection result includes image index information corresponding to the preliminary detection result, first defect confidence information corresponding to the preliminary detection result or second defect confidence information corresponding to the defect detection result, and location information corresponding to the preliminary detection result; the defect detection device 300 also includes: based on the image index information, performing structured organization processing on the defect detection result to obtain a defect report; and displaying the defect report.

[0092] Exemplarily, the target object includes a distribution pole tower; the acquisition module 310 is further used to acquire the front view, left view, right view, top view and overall view of the distribution pole tower based on the drone to obtain the original image. Figure 4 A block diagram of an electronic device provided in accordance with another embodiment of the present invention.

[0093] An embodiment of the present application provides an electronic device including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0094] like Figure 4 As shown, for ease of understanding, the embodiment of the present application shows a specific electronic device 400.

[0095] The electronic device 400 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are intended to be examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0096] like Figure 4 As shown, electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. Various programs and data required for the operation of electronic device 400 may also be stored in RAM 403. Computing unit 401, ROM 402, and RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to bus 404.

[0097] Multiple components in electronic device 400 are connected to input / output (I / O) interface 405, including an input unit 406, such as a keyboard and mouse; an output unit 407, such as various types of displays and speakers; a storage unit 408, such as a magnetic disk and optical disk; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows electronic device 400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0098] Computing unit 401 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Computing unit 401 performs the various methods described above. For example, in some embodiments, any one or more of the various methods described above may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed onto electronic device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by computing unit 401, one or more steps of any one or more of the various methods described above may be performed. Alternatively, in other embodiments, computing unit 401 may be configured to perform any one or more of the various methods described above via any other suitable means (e.g., via firmware).

[0099] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method of any one of the above embodiments are implemented.

[0100] It should be noted that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of the present invention, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0101] It should be understood that various components of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof may be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0102] In the description of the present invention, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In the present invention, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0103] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0104] In addition, the terms "first" and "second" used in the embodiments of the present invention are only used for descriptive purposes and should not be understood as indicating or implying relative importance, or implicitly indicating the number of technical features indicated in this embodiment. Therefore, the features defined by the terms "first" and "second" in the embodiments of the present invention can explicitly or implicitly indicate that the embodiment includes at least one of such features. In the description of the present invention, the word "plurality" means at least two or two or more, such as two, three, four, etc., unless otherwise clearly and specifically defined in the embodiments.

[0105] In the present invention, unless otherwise clearly specified or limited in the embodiments, the terms "installed," "connected," "connect," and "fixed" appearing in the embodiments should be understood in a broad sense. For example, the connection may be a fixed connection, a detachable connection, or an integral connection. It can also be a mechanical connection, an electrical connection, etc.; of course, it can also be a direct connection, or an indirect connection through an intermediate medium, or it can be internal communication between two elements, or an interaction between two elements. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood based on the specific implementation.

[0106] In the present invention, unless otherwise expressly specified or limited, when a first feature is "above" or "below" a second feature, it may mean that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediary. Furthermore, when a first feature is "above," "above," or "above" a second feature, it may mean that the first feature is directly above or diagonally above the second feature, or simply means that the first feature is at a higher level than the second feature. When a first feature is "below," "below," or "below" a second feature, it may mean that the first feature is directly below or diagonally below the second feature, or simply means that the first feature is at a lower level than the second feature.

[0107] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A defect detection method, characterized in that: The method comprises: The original image is obtained by collecting multi-viewpoint images of the target object based on the UAV; Based on the original image, defect detection is performed on the target object to obtain a preliminary detection result; When the confidence level of the preliminary detection result indicating that the target object has a defect is less than a preset threshold, a metacognitive processing mode is triggered to perform three-dimensional reconstruction based on the original image to obtain a three-dimensional model of the target object and acquisition parameter information of the original image; Obtaining a sampling view sequence of the target object based on the three-dimensional model, the original image, and the acquisition parameter information; Rendering a three-dimensional image corresponding to the original image in the three-dimensional model based on the sampling view sequence to obtain an image rendering result; Based on the image rendering result, defect detection is performed on the target object to obtain a defect detection result.

2. The defect detection method according to claim 1, characterized in that: The original image is located in the image group; The obtaining of a sampling view sequence of the target object based on the three-dimensional model, the original image, and the acquisition parameter information includes: Determining the original image by indexing from the image group based on the preliminary detection result, and performing image segmentation on the original image to obtain mask information of the original image; A sampling view sequence of the target object is obtained based on the three-dimensional model, the mask information of the original image and the acquisition parameter information.

3. The defect detection method according to claim 2, characterized in that: The preliminary detection result includes image index information, first defect confidence information, and position information of the target object in the original image; Determining the original image from the image group based on the preliminary detection result and performing image segmentation on the original image to obtain mask information of the original image includes: Based on the image index information, index and determine the original image from the image group; Based on the position information, the original image is segmented to obtain mask information of the original image.

4. The method according to claim 2, characterized in that The acquisition parameter information of the original image includes internal reference information, external reference information and depth information, and the sampling view sequence includes target internal reference information and target external reference information; The obtaining of a sampling view sequence of the target object based on the three-dimensional model, the mask information of the original image, and the acquisition parameter information includes: Determining the internal reference information of the original image as target internal reference information; Based on the intrinsic parameter information of the original image, the extrinsic parameter information and the depth information, back-projecting the pixel points of the mask information of the original image into a three-dimensional space according to the three-dimensional model to obtain three-dimensional point cloud data; Obtaining reference extrinsic parameter information based on the center point of the three-dimensional point cloud data; The target extrinsic parameter information is obtained by performing geometric transformation and combination based on the reference extrinsic parameter information.

5. The defect detection method according to claim 4, characterized in that: The geometric transformation includes depth translation, spherical interpolation sampling, and tangent plane translation; the geometric transformation and combination based on the reference extrinsic parameter information to obtain the target extrinsic parameter information includes: Taking the reference extrinsic parameter information as the center, depth translation is performed along the depth axis to obtain a first extrinsic parameter; Taking the reference extrinsic parameter information as the sphere center, coordinate interpolation movement is performed on the spherical surface to obtain a second extrinsic parameter; Taking the reference extrinsic reference information as the center, performing a tangent plane translation along a direction perpendicular to the center plane to obtain a third extrinsic reference; The target external parameter information is obtained based on a combination of any one or more of the first external parameter, the second external parameter, and the third external parameter.

6. The defect detection method according to claim 4, characterized in that: The rendering of the three-dimensional image corresponding to the original image in the three-dimensional model based on the sampling view sequence to obtain an image rendering result includes: The three-dimensional image corresponding to the original image in the three-dimensional model is rendered with the center point of the three-dimensional point cloud data as the rendering center to obtain an image rendering result.

7. The defect detection method according to claim 3, characterized in that: The image rendering result includes image index information, first defect confidence information, and position information corresponding to the preliminary detection result; The performing defect detection on the target object based on the image rendering result to obtain a defect detection result includes: Based on the image rendering result, performing defect detection on the target object to obtain second defect confidence information; When the second defect confidence information indicates that the confidence level of the defect in the target object is less than the preset threshold, the corresponding preliminary detection result is removed based on the image index information to obtain a defect detection result.

8. The defect detection method according to claim 7, characterized in that: The defect detection result includes image index information corresponding to the preliminary detection result, first defect confidence information corresponding to the preliminary detection result or second defect confidence information corresponding to the defect detection result, and position information corresponding to the preliminary detection result; The method further comprises: Based on the image index information, the defect detection results are structured and organized to obtain a defect report; Present the defect report.

9. The defect detection method according to any one of claims 1 to 8, characterized in that: The target object includes a distribution tower; and the method of acquiring an original image by performing multi-view acquisition of the target object based on a drone includes: The original image is obtained based on the drone collecting the front view, left view, right view, top of the distribution pole tower and the whole view of the distribution pole tower.

10. A defect detection device, characterized in that: The device comprises: An acquisition module is used to acquire original images from multiple perspectives of the target object based on the drone; A first detection module is used to perform defect detection on the target object based on the original image to obtain a preliminary detection result; a reconstruction module configured to trigger a metacognitive processing mode when the confidence level of the preliminary detection result indicating that the target object has a defect is less than a preset threshold, and to perform a three-dimensional reconstruction based on the original image to obtain a three-dimensional model of the target object and acquisition parameter information of the original image; an acquisition module, configured to obtain a sampling view sequence of the target object based on the three-dimensional model, the original image, and the acquisition parameter information; a rendering module, configured to render a three-dimensional image corresponding to the original image in the three-dimensional model based on the sampling view sequence to obtain an image rendering result; The second detection module is used to perform defect detection on the target object based on the image rendering result to obtain a defect detection result.

11. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the defect detection method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the defect detection method according to any one of claims 1 to 9 are implemented.

Images