Defect detection method, device and equipment and storage medium

Abstract

The invention discloses a defect detection method, device and equipment and a storage medium, and belongs to the technical field of unmanned aerial vehicle intelligent inspection. The method comprises the following steps: acquiring inspection multi-modal data collected by an unmanned aerial vehicle in an inspection target area; the inspection multi-modal data are input into a defect detection model, and defect preliminary detection information is obtained and comprises defect preliminary type information, defect position information and defect confidence degree information; the to-be-rechecked data and the rechecking type information are input into a defect rechecking model, defect rechecking information is obtained, the to-be-rechecked data are inspection multi-modal data with defect confidence coefficient information smaller than or equal to a first preset confidence coefficient and larger than or equal to a second preset confidence coefficient, and the rechecking type information is defect preliminary type information of the to-be-rechecked data; and generating a defect detection report based on the defect analysis model in combination with the defect preliminary detection information and the defect reinspection information. According to the technical scheme provided by the invention, the accuracy and reliability of defect detection of unmanned aerial vehicle inspection can be improved.

Description

Technical Field

[0001] The present invention relates to the field of drone intelligent inspection technology, and in particular to a defect detection method, device, equipment and storage medium. Background Art

[0002] With the rapid development of drone technology, its application in fields such as power inspection, photovoltaic testing, and bridge monitoring is becoming increasingly widespread. Traditional inspection methods rely primarily on manual labor or simple automated equipment, which are subject to low efficiency, high costs, and high risks. Drone inspection technology, on the other hand, can efficiently cover large areas and is equipped with high-resolution cameras, infrared sensors, and other equipment for rapid data collection. However, accurately identifying defects (such as cracks, rust, and contamination) from massive amounts of inspection data remains a challenge for current technology.

[0003] Existing drone inspection systems typically identify defects based on deep learning-based target detection algorithms (such as Faster RCNN). However, these methods can only identify a limited number of defect types. Detecting defects in only one environment cannot meet the requirements of other environments, increasing the cost of defect detection for different application environments. Furthermore, the defect type detection accuracy is not very good, and there is a disadvantage of a high false positive rate.

[0004] Therefore, a more reliable solution needs to be provided. Summary of the Invention

[0005] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a defect detection method, apparatus, equipment and storage medium that can improve the accuracy and reliability of defect detection during drone inspections.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] In one aspect, the present invention provides a defect detection method, comprising:

[0008] Obtain multimodal inspection data collected by drones in target inspection areas;

[0009] Inputting the inspection multimodal data into a defect detection model to obtain preliminary defect detection information; the preliminary defect detection information includes preliminary defect type information, defect location information, and defect confidence information;

[0010] Inputting the data to be re-inspected and the re-inspection type information into the defect re-inspection model to obtain defect re-inspection information; wherein the data to be re-inspected is the inspection multimodal data corresponding to the defect confidence information being less than or equal to the first preset confidence and greater than or equal to the second preset confidence, and the re-inspection type information is the preliminary defect type information corresponding to the data to be re-inspected;

[0011] Based on the defect analysis model, combined with the preliminary defect detection information and the defect re-inspection information, a defect detection report is generated.

[0012] In some possible implementations, the defect review model includes an image feature extraction model, a text feature extraction model, a multimodal feature fusion model, and a review model;

[0013] The step of inputting the data to be reviewed and the review type information into the defect review model to obtain defect review information includes:

[0014] Inputting the data to be re-inspected into the image feature extraction model to perform image feature extraction processing to obtain re-inspection image features;

[0015] Inputting the re-inspection type information into the text feature extraction model to perform text feature extraction processing to obtain re-inspection text features;

[0016] Inputting the re-inspection image features and the re-inspection text features into a multimodal feature fusion model for feature fusion processing to obtain re-inspection fusion features;

[0017] The re-inspection fusion features are input into the re-inspection model for defect detection processing to obtain the defect re-inspection model.

[0018] In some possible implementations, generating a defect detection report based on the defect analysis model and in combination with the preliminary defect detection information and the defect re-inspection information includes:

[0019] Determining the inspection multimodal data having defect confidence information greater than a first preset confidence level in the defect detection information as first data to be analyzed;

[0020] When the defect re-inspection type information in the defect re-inspection information is consistent with the re-inspection type information, determining that the data to be re-inspected is the second data to be analyzed;

[0021] The first data to be analyzed and preliminary defect detection information corresponding to the first data to be analyzed, as well as the second data to be analyzed and preliminary defect detection information corresponding to the second data to be analyzed are input into the defect analysis model to generate the defect detection report.

[0022] In some possible implementations, the defect analysis model includes a feature recognition model, an analysis model, and a report generation model;

[0023] The step of inputting the first data to be analyzed and the preliminary defect detection information corresponding to the first data to be analyzed, and the second data to be analyzed and the preliminary defect detection information corresponding to the second data to be analyzed into the defect analysis model to generate the defect detection report includes:

[0024] Inputting the first data to be analyzed and the preliminary defect detection information corresponding to the first data to be analyzed, as well as the second data to be analyzed and the preliminary defect detection information corresponding to the second data to be analyzed into the feature recognition model for feature recognition processing to obtain defect feature recognition information, wherein the defect feature recognition information includes defect feature information and defect degree information;

[0025] Inputting the defect feature identification information into the analysis model for analysis and processing to obtain defect analysis information;

[0026] The defect analysis information is input into the report generation model, and the defect detection report is generated by combining a preset report template and text generation technology.

[0027] In some possible implementations, the defect detection model includes a feature extraction model, a feature fusion model, and a detection model;

[0028] Inputting the inspection multimodal data into the defect detection model to obtain preliminary defect detection information includes:

[0029] Inputting the inspection multimodal data into the feature extraction model for hierarchical feature extraction processing to obtain inspection multimodal features of different levels;

[0030] Inputting the inspection multimodal features of different levels into the feature fusion model for feature fusion processing to obtain inspection multimodal fusion features;

[0031] The inspection multimodal fusion features are input into the detection model for defect detection processing to obtain preliminary defect detection information.

[0032] In some possible implementations, the defect detection model is trained in the following manner:

[0033] Acquire sample inspection multimodal data collected by the drone inspection sample area, wherein the sample inspection multimodal data includes a sample appearance detection image and a sample thermal state detection image;

[0034] Performing data enhancement processing on the sample appearance detection image to obtain an expanded appearance detection image;

[0035] performing thermal diffusion enhancement processing on the sample thermal state detection image to obtain an expanded thermal state detection image;

[0036] Determining the expanded appearance inspection image and the expanded thermal state inspection image as sample expanded inspection multimodal data, and determining preset defect detection information corresponding to the sample expanded inspection multimodal data;

[0037] Inputting the sample-expanded inspection multimodal data into the defect detection model to be trained for defect detection processing to obtain target defect detection information;

[0038] The defect detection model to be trained is trained according to the preset defect detection information and the target defect detection information to obtain a defect detection model.

[0039] In some possible implementations, the inspection multimodal data includes appearance detection images and thermal state detection images; and obtaining the inspection multimodal data collected by the drone inspection target area includes:

[0040] Acquire original inspection multimodal data collected by the drone during inspection of the target area, wherein the original inspection multimodal data includes an original appearance detection image and an original thermal state detection image;

[0041] Performing sliding block processing on the original appearance detection image according to a preset window size and a preset overlap rate to obtain an appearance detection block image;

[0042] Performing illumination equalization processing on the appearance detection block image to obtain the appearance detection image;

[0043] Performing color enhancement processing on the original thermal state detection image to obtain the thermal state detection image;

[0044] Image alignment processing is performed on the appearance detection image and the thermal state detection image to obtain the inspection multimodal data.

[0045] Another aspect provides a defect detection device, the device comprising:

[0046] The data acquisition module is used to obtain the multimodal inspection data collected by the drone in the inspection target area;

[0047] A defect detection module is used to input the inspection multimodal data into a defect detection model to obtain preliminary defect detection information; the preliminary defect detection information includes preliminary defect type information, defect location information and defect confidence information;

[0048] a defect re-inspection module, configured to input the data to be re-inspected and the re-inspection type information into the defect re-inspection model to obtain defect re-inspection information; wherein the data to be re-inspected is the inspection multimodal data corresponding to the defect confidence information being less than or equal to the first preset confidence and greater than or equal to the second preset confidence, and the re-inspection type information is the preliminary defect type information corresponding to the data to be re-inspected;

[0049] A report generation module is used to generate a defect detection report based on the defect analysis model and in combination with the preliminary defect detection information and the defect re-inspection information.

[0050] On the other hand, an electronic device is provided, which includes a processor and a memory, wherein the memory stores at least one instruction and at least one program, and the at least one instruction and the at least one program are loaded and executed by the processor to implement the defect detection method as described above.

[0051] On the other hand, a computer-readable storage medium is provided, in which at least one instruction and at least one program are stored. The at least one instruction and the at least one program are loaded and executed by a processor to implement the defect detection method described above.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] In the present invention, by acquiring the inspection multimodal data collected by the drone in the inspection target area, the inspection multimodal data is input into the defect detection model to obtain preliminary defect detection information, which includes preliminary defect type, defect location information and defect confidence information; then the data to be re-inspected and the re-inspection type information are input into the defect re-inspection model to obtain defect re-inspection information, the data to be re-inspected is the inspection multimodal data corresponding to the defect confidence information less than or equal to the first preset confidence and greater than or equal to the second preset confidence, and the re-inspection type information is the preliminary defect type information corresponding to the data to be re-inspected, and the defect detection model is combined with the defect re-inspection model to apply Used in defect detection during drone inspections, it can improve the accuracy and efficiency of defect detection, reduce false detections and missed detections, identify different types of defects, broaden the scope of defect detection, and improve the comprehensiveness of defect detection, thereby ensuring the safe operation of infrastructure; then, based on the defect analysis model, combined with the preliminary defect detection information and the defect re-inspection information, a defect detection report is generated, and a structured defect detection report is automatically generated, which improves the intelligence and automation of drone inspection defect detection, further improves the accuracy and efficiency of drone inspection defect detection, and improves the timeliness of information feedback, so that corresponding response measures can be taken in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 1 is a flow chart of a defect detection method provided by an embodiment of the present invention;

[0056] Figure 2 This is a flow chart of inputting the data to be re-inspected and the re-inspection type information into the defect re-inspection model to obtain defect re-inspection information, provided by the present invention;

[0057] Figure 3 It is a structural schematic diagram of a defect detection device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0059] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0060] In the embodiments of the present invention, the term "module" or "unit" refers to a computer program or portion of a computer program that has a predetermined function and works together with other related components to achieve a predetermined goal. The term "module" or "unit" refers to a computer program or portion of a computer program that has a predetermined function and works together with other related components to achieve a predetermined goal. The term "module" or "unit" may be implemented in whole or in part using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a single processor (or multiple processors or memories) may be used to implement one or more modules or units. Furthermore, each module or unit may be part of an overall module or unit that incorporates the functionality of that module or unit.

[0061] Various exemplary embodiments, features, and aspects of the present invention will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0062] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0063] The term "and / or" herein simply describes an association relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent the existence of three situations: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0064] In addition, numerous specific details are provided in the following detailed description to better illustrate the present invention. Those skilled in the art will appreciate that the present invention may be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of the present invention.

[0065] Figure 1 It is a flowchart of a defect detection method provided by an embodiment of the present invention. This specification provides method operation steps such as the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative work. The order of steps listed in the embodiment is only one way of executing the steps among many orders, and does not represent the only order of execution. When the actual system or server product is executed, it can be executed in the order shown in the embodiment or the figure or in parallel (for example, in a parallel processor or multi-threaded processing environment). Specifically, Figure 1 As shown, the above method may include:

[0066] S101: Acquire multimodal inspection data collected by the drone in the target inspection area;

[0067] In a specific embodiment, the target area may be an area to be inspected for defects by a drone. Alternatively, the target area may be an insulator or photovoltaic module used for power transmission. The multimodal inspection data may be data collected by the drone using different acquisition devices carried by the drone. Optionally, the different acquisition devices may include a visible light camera and an infrared thermal imager.

[0068] In an optional embodiment, the above-mentioned inspection multimodal data may include appearance detection images and thermal state detection images; the above-mentioned inspection multimodal data collected by the drone inspection target area may include:

[0069] Obtaining original inspection multimodal data collected by the drone in the target inspection area. The original inspection multimodal data includes original appearance inspection images and original thermal state inspection images.

[0070] Perform sliding block processing on the original appearance detection image according to a preset window size and a preset overlap rate to obtain an appearance detection block image;

[0071] Performing illumination equalization processing on the appearance detection block image to obtain the appearance detection image;

[0072] Performing color enhancement processing on the original thermal state detection image to obtain a thermal state detection image;

[0073] Image alignment processing is performed on the appearance inspection image and the thermal state inspection image to obtain inspection multimodal data.

[0074] In one specific embodiment, a drone can fly over a target area using a preset or dynamically planned inspection route. It can use its onboard visible light camera to capture visual inspection images of the target area, and its onboard infrared thermal imager to capture thermal inspection images of the target area. Optionally, the visual inspection images can be visible light image data used to characterize the target area's appearance, with a resolution of no less than 2K. The thermal inspection images can be infrared thermal image data used to detect hot spots, localized overheating, and early signs of corrosion in equipment within the target area.

[0075] In a specific embodiment, the preset window size can be set based on actual application requirements, optionally 640×640. The preset overlap ratio can also be set based on actual application requirements, optionally 10% to 20%. Setting the preset overlap ratio can prevent detected defects from being cut off by window edges, thereby preventing loss of defect information. Sliding block processing of the original appearance inspection image according to the preset window size and overlap ratio can avoid the problem of missed detection of small objects in high-resolution visible light images. Optionally, histogram equalization can be used to perform illumination equalization on the appearance inspection block images to obtain an appearance inspection image. This can eliminate the effects of uneven illumination and enhance defect features in low-contrast areas, making the defect features clearer. Optionally, pseudo-colorization and color enhancement can be performed on the original thermal state inspection image, rendering the originally grayscale infrared image in color to highlight defect features. Furthermore, morphological opening and closing operations can be used to process the original thermal state inspection image to remove small areas of noise, making the thermal state inspection image more reliable. Optionally, the appearance inspection image and the thermal state inspection image of the same scene can use the same image number, and then use different suffixes to distinguish the modalities to ensure that the appearance inspection image and the thermal state inspection image are aligned, providing a basis for subsequent defect detection using multimodal data. Specifically, during the image alignment and storage process, the appearance inspection image can end with W, and the thermal state inspection image can end with H. For example, the appearance inspection image file name can be "001W", and the thermal state inspection image file name can be "001H".

[0076] In the above embodiment, by processing the original appearance detection image and the original thermal state detection image, the data quality of the appearance detection image and the thermal state detection image can be improved, thereby improving the subsequent defect detection accuracy.

[0077] S102: Inputting the inspection multimodal data into the defect detection model to obtain preliminary defect detection information;

[0078] In a specific embodiment, the defect detection model can be a model used for defect detection; the preliminary defect detection information can include preliminary defect type information, defect location information and defect confidence information. The preliminary defect detection information can represent the defect identification result determined by the model based on the inspection multimodal data; the preliminary defect type information can represent the physical property category of the defect, for example, the preliminary defect type information can be insulator damage, wire breakage, hot spot, etc.; the defect location information can represent the bounding box information corresponding to the defect; the defect confidence information can represent the probability of the existence of a defect in the bounding box.

[0079] In an optional embodiment, the above-mentioned defect detection model may include a feature extraction model, a feature fusion model and a detection model;

[0080] The above-mentioned input of the inspection multimodal data into the defect detection model to obtain preliminary defect detection information may include:

[0081] The inspection multimodal data is input into the feature extraction model for hierarchical feature extraction processing to obtain inspection multimodal features at different levels;

[0082] The inspection multimodal features of different levels are input into the feature fusion model for feature fusion processing to obtain the inspection multimodal fusion features;

[0083] The multimodal fusion features of the inspection are input into the detection model for defect detection processing to obtain preliminary defect detection information.

[0084] In a specific embodiment, the model structure of the defect detection model can be set in combination with actual application requirements. Optionally, the defect detection model can be an improved YOLOv8 lightweight model, which has the characteristics of high detection accuracy, fast detection speed and strong deployment flexibility.

[0085] The feature extraction model can be used for feature processing. Its structure can be tailored to the specific application requirements. Optionally, the model can employ an improved CSPNet (Cross Stage Partial Network) structure, which can serve as the backbone of the YOLOv8 lightweight model. By using cross-stage partial connections, a portion of the multimodal inspection data can be directly passed to the next stage, while the remaining portion is processed by convolutional layers at different levels and then concatenated with the directly passed portion. This effectively extracts both shallow and deep features from the multimodal inspection data, reducing computational complexity while enhancing feature reusability.

[0086] A feature fusion model can be used for feature fusion. Its architecture can be customized based on actual application requirements. Optionally, the model can adopt a PANet (Path Aggregation Network) structure, which can be the core network of the YOLOv8 lightweight model. Through a top-down and bottom-up bidirectional feature fusion process, the multimodal inspection features extracted by the feature extraction model at different levels can be fully integrated. Optionally, the feature fusion model can fuse the multimodal inspection features at different levels based on their corresponding feature weights to generate fused multimodal inspection features. The feature weights at different levels can represent their contributions to the fusion process. The feature fusion model can simultaneously consider both semantic and location information of defects during feature fusion, thereby improving defect detection accuracy.

[0087] The detection model can be used for defect detection. Its structure can be tailored to actual application requirements. The model can be the head network of a YOLOv8 lightweight model. It can detect defects and output preliminary defect detection information. For each frame of multimodal inspection data input, the detection model generates preliminary defect type information, defect location information, and defect confidence information. Multiple defect location information can be represented by multiple bounding box information. Each bounding box information can include four parameters (x, y, w, h), representing the center coordinates (x, y), width (w), and height (h) of the bounding box, respectively. Each bounding box information corresponds to a defect confidence information.

[0088] In the above embodiment, defect detection is performed using the above defect detection model, which can accurately identify defects. For images containing multiple defects, defects can be accurately located and classified, thereby improving detection accuracy and reducing false detections and missed detections. It can also be based on a lightweight design. While ensuring detection accuracy, the number of model parameters and the amount of calculation can be reduced, and a large amount of image data can be processed in a shorter time, thereby improving detection efficiency. The model can support the deployment of multiple hardware platforms. At the same time, it also provides a wealth of development interfaces and tools to facilitate developers to integrate the model into different application systems. Whether it is a large-scale industrial inspection system or a small mobile device application, the model can be flexibly adapted, providing convenience for actual defect detection applications.

[0089] In an optional embodiment, the above-mentioned defect detection model can be trained in the following manner:

[0090] Acquire sample inspection multimodal data collected by the drone inspection sample area, the sample inspection multimodal data including sample appearance detection images and sample thermal state detection images;

[0091] Perform data enhancement processing on the sample appearance detection image to obtain an expanded appearance detection image;

[0092] Performing thermal diffusion enhancement processing on the sample thermal state detection image to obtain an expanded thermal state detection image;

[0093] Determining the expanded appearance inspection image and the expanded thermal state inspection image as sample expanded inspection multimodal data, and determining preset defect detection information corresponding to the sample expanded inspection multimodal data;

[0094] Input the sample-expanded inspection multimodal data into the defect detection model to be trained for defect detection processing to obtain target defect detection information;

[0095] The defect detection model to be trained is trained according to the preset defect detection information and the target defect detection information to obtain a defect detection model.

[0096] In one specific embodiment, data enhancement technology is used to enhance the sample appearance inspection image to obtain an expanded appearance inspection image. Hot spot data is synthesized using a heat diffusion equation, and heat diffusion enhancement is performed on the sample thermal state inspection image to obtain an expanded thermal state inspection image, thereby expanding the dataset used to train the model. Optionally, the preset defect detection information can be the defect detection information already annotated in the sample expanded inspection multimodal data; and the target defect detection information can be the defect detection information obtained by performing defect detection processing on the sample expanded inspection multimodal data using the defect detection model to be trained.

[0097] In a specific embodiment, the above-mentioned training of the defect detection information to be trained based on the preset defect detection information and the target defect detection information to obtain the defect detection model may include: determining the model loss based on the preset defect detection information and the target defect detection information; and training the defect detection model to be trained based on the model loss to obtain the defect detection model. Optionally, the model loss can be calculated in combination with a preset loss function, and the preset loss function can be set in combination with actual application requirements. For example, the model loss can be determined based on the Focal-EIOU (Focal-Efficient Intersection over Union Loss) loss function. The above-mentioned model loss can characterize the defect detection accuracy of the current defect detection model to be trained. Optionally, the above-mentioned Focal-EIOU Loss is as follows:

[0098] ;

[0099] in, represents Focal-EIOU loss; represents IOU loss; Represents the target bounding box in the target defect detection information The center point of the preset defect detection information and the preset bounding box The Euclidean distance between the center points; Indicates the width of the minimum enclosing rectangle; Indicates the height of the minimum enclosing rectangle; Indicates the width of the target bounding box; Indicates the height of the target bounding box; , Represents the parameter for balancing difficult and easy samples; Represents the Focal modulation factor; It represents the intersection-over-union ratio between the target bounding box information and the preset bounding box information. By introducing the Focal modulation factor, the gradient weight of small defect samples can be increased, thereby improving the recall rate of small targets smaller than 20 pixels.

[0100] In a specific embodiment, the above-mentioned training of the defect detection model to be trained based on the model loss to obtain the defect detection model may include: updating the model parameters of the defect detection model to be trained based on the model loss, and repeating the above-mentioned training iteration steps of inputting the sample expansion inspection multimodal data into the defect detection model to be trained for defect detection processing based on the updated defect detection model to be trained, obtaining target defect detection information, and updating the model parameters of the defect detection model to be trained based on the model loss, until the preset training convergence condition is met. Optionally, the above-mentioned preset training convergence condition can be that the model loss is less than or equal to a preset loss threshold, or the number of training iteration steps reaches a preset number, etc. Specifically, the preset loss threshold and the preset number of times can be set in combination with the model accuracy and training speed requirements in actual applications.

[0101] In the above embodiment, the process of training the defect detection model can improve the accuracy of defect detection and improve the training efficiency of the defect detection model.

[0102] In a specific embodiment, the inspection multimodal data corresponding to defect confidence information greater than a first preset confidence level can be marked as a high-confidence defect; the inspection multimodal data corresponding to defect confidence information less than or equal to the first preset confidence level and greater than or equal to a second preset confidence level can be marked as a suspicious defect; and the inspection multimodal data corresponding to defect confidence information less than the second preset confidence level can be marked as a low-confidence defect. Optionally, the first preset confidence level can be set based on actual application requirements, and the first preset confidence level can be 0.6; the second preset confidence level can be set based on actual application requirements, and the second preset confidence level can be 0.4. Optionally, the inspection multimodal data corresponding to high-confidence defects and preliminary defect detection information can be uploaded to the cloud as input to the defect analysis model; the inspection multimodal data corresponding to suspicious defects and preliminary defect detection information can be uploaded to the cloud as input to the defect re-inspection model, awaiting review and detection by the defect re-inspection model; and the inspection multimodal data corresponding to low-confidence defects can be discarded and no longer processed. Optionally, the information of high-confidence defects and suspected defects uploaded to the cloud may include: preliminary defect type information, defect location information (defect bounding box information), defect confidence information, and positioning information and timestamp of the defect image frame.

[0103] S103: Input the data to be re-inspected and the re-inspection type information into the defect re-inspection model to obtain defect re-inspection information;

[0104] In a specific embodiment, the data to be reviewed can be patrol multimodal data corresponding to defect confidence information less than or equal to a first preset confidence and greater than or equal to a second preset confidence; the review type information can be preliminary defect type information corresponding to the data to be reviewed. Specifically, the data to be reviewed is patrol multimodal data corresponding to defect confidence information of 0.4-0.6. The defect review model can be a model for performing defect review and detection. The model structure of the defect review model can be set in combination with actual application requirements. Optionally, the defect review model can be a Qwen-VL visual language model. The defect review information can represent the defect type information after the review inspection.

[0105] In an optional embodiment, the defect review model may include an image feature extraction model, a text feature extraction model, a multimodal feature fusion model and a review model. Figure 2 The present invention provides a flow chart of inputting the data to be re-inspected and the re-inspection type information into the defect re-inspection model to obtain defect re-inspection information; Figure 2 As shown, the above-mentioned input of the data to be reviewed and the review type information into the defect review model to obtain defect review information may include:

[0106] S201: Inputting the data to be re-inspected into the image feature extraction model for image feature extraction processing to obtain re-inspection image features;

[0107] S202: Inputting the re-inspection type information into a text feature extraction model to perform text feature extraction processing to obtain re-inspection text features;

[0108] S203: Inputting the re-inspection image features and the re-inspection text features into a multimodal feature fusion model for feature fusion processing to obtain re-inspection fusion features;

[0109] S204: Input the re-inspection fusion features into the re-inspection model to perform defect detection processing to obtain a defect re-inspection model.

[0110] In a specific embodiment, the image feature extraction model can be a model for performing image feature extraction. The model structure of the image feature model can be set in combination with actual application requirements. Optionally, the image feature extraction model can include a visual encoder and an attention enhancement module. Optionally, the above-mentioned inputting the data to be reviewed into the image feature extraction model for image feature extraction processing to obtain the reviewed image features can include: performing preliminary feature extraction processing on the data to be reviewed to obtain preliminary features of the reviewed image; and performing advanced feature extraction processing on the preliminary features of the reviewed image to obtain features of the reviewed image. Optionally, the visual encoder can adopt a multi-layer convolutional neural network (CNN) structure and incorporate a residual connection mechanism. When the input image enters the visual encoder, it first passes through a series of convolutional layers. The convolution kernel in the convolutional layer can slide on the image and perform convolution operations to gradually extract local features of the image, such as basic information such as edges and textures. For example, when processing images containing defects in power facilities, convolution kernels can capture the edge features of subtle cracks on the surface of insulators or the contour features of hot spots on photovoltaic panels. As the number of network layers increases, local features are downsampled through pooling layers to reduce their dimensionality, which can reduce the amount of computation while enhancing the translation invariance of features. The residual connection mechanism allows gradients to propagate directly around certain layers, effectively solving the gradient vanishing problem during model training and enabling the model to learn more complex and advanced semantic features, such as the shape and structure of objects. After processing through multiple layers of convolution, pooling layers, and residual connections, the shallow features of the image can be initially converted into more representative high-level semantic features, obtaining preliminary features of the re-inspected image. To make the model focus more on important areas and key features in the image, the defect review model introduces an attention enhancement module. This module generates an attention weight for each position in the preliminary features of the review image output by the visual encoder. The size of the weight reflects the importance of the feature at that position. For example, when detecting insulator damage, the attention enhancement module will make the model pay more attention to the area where the insulator is located, assigning it a higher weight, while relatively reducing the feature weights of irrelevant areas such as the background. By multiplying the attention weight with the corresponding position of the preliminary features of the review image, the weighted preliminary features of the review image, namely the review image features, are obtained. This enables the model to more accurately identify various detailed information in the image, such as small damage on the insulator, hot spots on photovoltaic panels, etc. Optionally, the preliminary features of the review image can reflect the high-level features of the data to be reviewed; the review image features can reflect the detailed information and key features of the data to be reviewed.

[0111] The text feature extraction model can be a model for extracting text features. The model structure of the text feature extraction model can be set based on actual application requirements. Optionally, the text feature extraction model can include a text editor. Optionally, the re-inspected text features can represent the type and characteristics of the defect. Specifically, defect types can include insulator damage, hot spots, etc., and defect characteristics can include the size and shape of the damage.

[0112] The multimodal feature fusion model can be a model for fusing image features and text features, and the model structure of the multimodal feature fusion model can be set in combination with actual application requirements. Optionally, the model adopts an interactive method based on the attention mechanism, allowing image features and text features to pay attention to and influence each other; the above-mentioned input of the re-inspected image features and re-inspected text features into the multimodal feature fusion model for feature fusion processing, and obtaining the re-inspected fusion features can include: determining the similarity between the re-inspected image features and the re-inspected text features; based on the similarity, generating multimodal attention weights, the multimodal attention weights include image weight information and text weight information, the re-inspected image features correspond to the image weight information, and the re-inspected text features correspond to the text weight information; based on the image weight information and the text weight information, performing weighted summation processing on the re-inspected image features and the re-inspected text features to obtain the re-inspected fusion features. Feature fusion processing can enable image features and text features to fully interact, and closely combine the defect information described in the text with the visual features in the image.

[0113] The re-inspection model can be used for defect detection. Its structure can be tailored to actual application requirements. Optionally, it can include an MLP (Multilayer Perceptron). The MLP analyzes and judges the re-inspection fusion features through nonlinear transformations. It analyzes defects from multiple dimensions (such as visual features and textual descriptions) to determine whether defects exist in the data to be re-inspected, as well as their type and severity.

[0114] In the above embodiment, the above defect review model can utilize its visual perception and reasoning capabilities to conduct in-depth correlation analysis on the image visual features and text features of the data to be reviewed, determine the defect review information, further determine the defect type, and improve the accuracy and reliability of defect detection.

[0115] In a specific embodiment, if the defect review information indicates that a defect exists and the defect review type information in the defect review information is consistent with the review type information, the data to be reviewed is determined to be the second data to be analyzed for report generation. If the defect review information indicates that a defect exists and the defect review type information in the defect review information is inconsistent with the review type information, the data to be reviewed is determined to be manual review data for manual review. If the defect review information indicates that a defect does not exist, the corresponding test result is directly discarded.

[0116] Optionally, during the manual re-inspection process, the defect results of the manual re-inspection can be promptly fed back to optimize and train the defect detection model and the defect re-inspection model, thereby further improving the accuracy of the model detection.

[0117] In a specific embodiment, during a drone inspection mission for power facilities, the drone inspects the transmission lines in the target inspection area, captures visible light image data of the transmission line through the visible light camera carried by the drone, and collects infrared thermal image data of the transmission line through the infrared thermal imager carried by the drone, and determines that the visible light image data and infrared thermal image data of the transmission line are original inspection multimodal data; the original inspection multimodal data are preprocessed to obtain inspection multimodal data; the inspection multimodal data are input into a defect detection model to obtain preliminary defect detection information, where the defect confidence information of a transmission line insulator is 0.52, and the preliminary defect type information is that the insulator has a surface contamination defect. An image with a defect confidence score of 0.52 is used as the data for re-inspection. The data and re-inspection type information are then fed into the defect re-inspection model. The model extracts image features using an image feature extraction model, identifying details such as the shape, color, and texture of the insulators in the image. For example, it identifies irregular dark areas on the insulator surface. Simultaneously, the text feature extraction model combines the input text information about "insulator surface contamination" with the text. The multimodal feature fusion model and the re-inspection model fuse the image and text features, performing a comprehensive analysis from both image and text perspectives. The model considers knowledge about the appearance of normal insulators and common manifestations of surface contamination to determine whether the image meets the characteristics of surface contamination. If the defect re-inspection model determines that the insulator in the image does have surface contamination, which is consistent with the defect type determined by the defect detection model, the result can be marked as a reliable detection result, and the report generation process will then begin. The report will detail the location of the insulator, the surface contamination, and other information. The defect re-inspection model analysis shows some anomalies on the insulator surface, but these anomalies are more likely to be residual water stains on the insulator surface rather than surface contamination. In this case, the defect type determined by the defect re-inspection model is inconsistent with the defect type determined by the defect detection model. This situation is marked as requiring manual re-inspection and enters the manual re-inspection process. If the defect re-inspection model analysis finds that the dark area on the insulator surface is actually a light and shadow effect during the shooting, not a real defect, the model will determine that there are no defects in the image and directly discard the detection result.

[0118] S104: Generate a defect detection report based on the defect analysis model and in combination with the preliminary defect detection information and the defect re-inspection information.

[0119] In a specific embodiment, the defect analysis model may be a model that analyzes defect detection information and generates a corresponding defect detection report. The defect detection report may be a structured report summarizing the defect detection information, which is convenient for staff to review, so as to promptly understand the defects of the corresponding facilities and perform timely maintenance and repairs.

[0120] In an optional embodiment, the defect analysis model is used to generate a defect detection report in combination with the preliminary defect detection information and the defect re-inspection information, which may include:

[0121] Determining inspection multimodal data having defect confidence information greater than a first preset confidence level in the defect detection information as first data to be analyzed;

[0122] When the defect re-inspection type information in the defect re-inspection information is consistent with the re-inspection type information, determining the data to be re-inspected as the second data to be analyzed;

[0123] The first data to be analyzed and the preliminary defect detection information corresponding to the first data to be analyzed, as well as the second data to be analyzed and the preliminary defect detection information corresponding to the second data to be analyzed are input into the defect analysis model to generate a defect detection report.

[0124] In an optional embodiment, the defect analysis model may include a feature recognition model, an analysis model, and a report generation model;

[0125] The above-mentioned inputting the first data to be analyzed and the preliminary defect detection information corresponding to the first data to be analyzed, as well as the second data to be analyzed and the preliminary defect detection information corresponding to the second data to be analyzed into the defect analysis model to generate a defect detection report may include:

[0126] Inputting the first data to be analyzed and the preliminary defect detection information corresponding to the first data to be analyzed, as well as the second data to be analyzed and the preliminary defect detection information corresponding to the second data to be analyzed into a feature recognition model for feature recognition processing to obtain defect feature recognition information, where the defect feature recognition information includes defect feature information and defect degree information;

[0127] Inputting defect feature identification information into the analysis model for analysis and processing to obtain defect analysis information;

[0128] The defect analysis information is input into the report generation model, and the defect detection report is generated by combining the preset report template and text generation technology.

[0129] In a specific embodiment, the defect detection report may include: an inspection task overview, including the inspection task number, inspection time, and inspection weather; a defect summary table, including the inspection task number, defect type information, defect coordinate information, defect severity, and defect handling suggestions; appearance inspection images and infrared thermal images; and reference disposal opinions. Optionally, by performing feature recognition on the first data to be analyzed and the preliminary defect detection information corresponding to the first data to be analyzed, as well as the second data to be analyzed and the preliminary defect detection information corresponding to the second data to be analyzed, the defect feature information existing in the inspection multimodal data and the defect degree information corresponding to the different defect feature information can be determined, and the defect degree information can represent the severity of the defect; the defect feature recognition information is analyzed by the analysis model of the defect analysis model to extract key information and generate defect analysis information; the defect analysis information is summarized by combining the defect analysis model with text generation technology according to preset logic and language specifications to generate summary content covering the inspection situation, and the report generation model is combined with the preset report model to generate a defect detection report based on the generated information and summary content.

[0130] Specifically, the defect analysis model conducts in-depth analysis and understanding of input data. This model can accurately identify the key characteristics and severity of different defect types. For example, during a photovoltaic panel inspection, it can distinguish the severity levels of defects such as hot spots, bird droppings, and dust accumulation. The defect analysis model leverages its powerful natural language processing capabilities to analyze and summarize inspection data. It performs semantic parsing on textual information within the input data, extracting key information such as device status and defect type. The model then uses text generation technology to generate an overview of the inspection using pre-set logic and language specifications. This overview, for example, includes statistics on the total number of defects found during the inspection, the percentage of each defect type, and a comparative analysis with previous inspection results, providing a comprehensive basis for decision-making. The model then organizes and presents these analysis results in natural and fluent language using pre-set report templates. Optionally, defect detection reports can be pushed to the operation and maintenance platform, mobile devices, or responsible personnel communication channels via 5G / 4G or satellite links. These reports can also be generated with one click, creating a closed-loop inspection-analysis-dispatch mechanism.

[0131] Specifically, the defect analysis model, based on its powerful natural language processing and generation capabilities, can automatically generate standardized defect detection reports in the following format:

[0132] Inspection Task Overview: The defect analysis model extracts key information from system data and generates a task overview that includes the inspection task number, the number of the drone performing the task, the inspection route / area number, and the flight time. This allows report users to quickly understand the basic background of the inspection. For example, "This inspection task number is TS20240701, the drone performing the task is UAV-005, the inspection route / area number is XL-QY008, and the flight time is from 9:00 AM to 11:00 AM on July 1st."

[0133] Defect Summary Table: The defect analysis model integrates preliminary defect detection and re-inspection information to list all confirmed defect entries. Each entry includes the defect number, defect category (such as conductor crack, insulator corrosion, etc.), and precise GPS coordinates. For example, "Defect number QX001, defect category is conductor crack, GPS coordinates are (116.3974, 39.9093)."

[0134] Defect Detail Map: The defect analysis model guides the system to generate a defect detail map that overlays the original visible light image with a pixel-level segmented heat map. Key values ​​such as crack length, corrosion area, hotspot temperature, and vibration amplitude are clearly labeled below the map. For example, for a wire crack defect, the annotation "crack length is 2.5 cm" is used.

[0135] Multimodal Analysis Description: The defect analysis model combines multimodal information, such as infrared images and vibration data, to conduct a comprehensive analysis of each defect. It details the possible causes of the defect, such as "Analysis indicates that the microcracks in the conductor are likely caused by a combination of mechanical fatigue and wind-induced vibration."

[0136] Maintenance recommendations and risk warnings: Based on defect severity, the defect analysis model automatically generates maintenance priorities and processing timelines, and provides corresponding risk warnings. For example, "This insulator corrosion defect is high risk, maintenance priority is level 1, processing timeline is 24 hours, and risk warning: On-site personnel should not approach, high voltage danger."

[0137] Statistics and visualization: The defect analysis model generates a line chart of inspection coverage based on inspection data, visually displaying the actual coverage ratio of this inspection. Inspection data can include the aforementioned inspection task overview and inspection summary table; a pie chart of the proportion of different defect types, clearly showing the proportion of each defect type; a heat map showing the distribution of high-risk defect points on a geographic map; and a bar chart of flight mileage, flight time, and energy consumption, comprehensively presenting flight-related information for this inspection.

[0138] In the above embodiment, the standardized defect detection report generated with the assistance of the defect analysis model is comprehensive and well-organized, which is convenient for operation and maintenance personnel to review, and provides an intuitive and accurate decision-making basis for operation and maintenance personnel, facilitating timely maintenance or repair of facilities and improving the safety of facilities.

[0139] It can be seen from the technical solutions provided by the embodiments of this specification that this specification obtains the inspection multimodal data collected by the drone inspection target area, inputs the inspection multimodal data into the defect detection model, and obtains preliminary defect detection information, which includes preliminary defect type, defect location information and defect confidence information; then inputs the data to be re-inspected and the re-inspection type information into the defect re-inspection model to obtain defect re-inspection information, the data to be re-inspected is the inspection multimodal data corresponding to the defect confidence information less than or equal to the first preset confidence and greater than or equal to the second preset confidence, the re-inspection type information is the preliminary defect type information corresponding to the data to be re-inspected, and the defect detection model is combined with the defect detection model to obtain defect re-inspection information. The combination of the defect detection and re-inspection model and its application in the defect detection of drone inspection can improve the accuracy and efficiency of defect detection, reduce false detection and missed detection, and identify different types of defects, broaden the scope of defect detection, and improve the comprehensiveness of defect detection, thereby ensuring the safe operation of infrastructure; then based on the defect analysis model, combined with the preliminary defect detection information and the defect re-inspection information, a defect detection report is generated, and a structured defect detection report is automatically generated, which improves the intelligence and automation of drone inspection defect detection, further improves the accuracy and efficiency of drone inspection defect detection, and improves the timeliness of information feedback, so that corresponding response measures can be taken in time.

[0140] The embodiment of the present invention further provides a defect detection device, correspondingly, Figure 3 Schematic diagram of a defect detection device provided by an embodiment of the present invention; Figure 3 As shown, the above device includes:

[0141] The data acquisition module 310 is used to acquire the inspection multimodal data collected by the drone during the inspection of the target area;

[0142] The defect detection module 320 is configured to input the inspection multimodal data into a defect detection model to obtain preliminary defect detection information; the preliminary defect detection information includes preliminary defect type information, defect location information, and defect confidence information;

[0143] The defect re-inspection module 330 is configured to input the data to be re-inspected and the re-inspection type information into the defect re-inspection model to obtain defect re-inspection information; wherein the data to be re-inspected is the inspection multimodal data corresponding to the defect confidence information less than or equal to the first preset confidence level and greater than or equal to the second preset confidence level, and the re-inspection type information is the preliminary defect type information corresponding to the data to be re-inspected;

[0144] The report generation module 340 is configured to generate a defect detection report based on the defect analysis model and in combination with the preliminary defect detection information and the defect re-inspection information.

[0145] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0146] An embodiment of the present invention also provides an electronic device, comprising: a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement a defect detection method as described in any one of the method embodiments.

[0147] An embodiment of the present invention also provides a computer storage medium, which can be set in a server to store at least one instruction, at least one program, code set or instruction set for implementing the method embodiment. The at least one instruction, the at least one program, the code set or instruction set is loaded and executed by the processor to implement the defect detection method as described in any one of the method embodiments.

[0148] Optionally, in an embodiment of the present invention, the storage medium may be located in at least one of a plurality of network servers in a computer network. Optionally, in an embodiment of the present invention, the storage medium may include, but is not limited to, a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard drive, a magnetic disk, or an optical disk, among other media capable of storing program code.

[0149] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0150] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple flow charts and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0151] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple flow charts and / or boxes Figure 1 The function specified in one or more boxes.

[0152] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple flow charts and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0153] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the function or action of the specification, or can be implemented by a combination of dedicated hardware and computer instructions.

[0154] Finally, it should be noted that the embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which are all protected by the present invention.

Claims

1. A defect detection method, characterized in that: The method comprises: Obtain multimodal inspection data collected by drones in target inspection areas; Inputting the inspection multimodal data into a defect detection model to obtain preliminary defect detection information; the preliminary defect detection information includes preliminary defect type information, defect location information, and defect confidence information; Inputting the data to be re-inspected and the re-inspection type information into the defect re-inspection model to obtain defect re-inspection information; wherein the data to be re-inspected is the inspection multimodal data corresponding to the defect confidence information being less than or equal to the first preset confidence and greater than or equal to the second preset confidence, and the re-inspection type information is the preliminary defect type information corresponding to the data to be re-inspected; Based on the defect analysis model, combined with the preliminary defect detection information and the defect re-inspection information, a defect detection report is generated.

2. The defect detection method according to claim 1, characterized in that: The defect re-inspection model includes an image feature extraction model, a text feature extraction model, a multimodal feature fusion model and a re-inspection model; The step of inputting the data to be reviewed and the review type information into the defect review model to obtain defect review information includes: Inputting the data to be re-inspected into the image feature extraction model to perform image feature extraction processing to obtain re-inspection image features; Inputting the re-inspection type information into the text feature extraction model to perform text feature extraction processing to obtain re-inspection text features; Inputting the re-inspection image features and the re-inspection text features into a multimodal feature fusion model for feature fusion processing to obtain re-inspection fusion features; The re-inspection fusion features are input into the re-inspection model for defect detection processing to obtain the defect re-inspection model.

3. The defect detection method according to claim 1, wherein: The generating of a defect detection report based on the defect analysis model and in combination with the preliminary defect detection information and the defect re-inspection information includes: Determining the inspection multimodal data having defect confidence information greater than a first preset confidence level in the defect detection information as first data to be analyzed; When the defect re-inspection type information in the defect re-inspection information is consistent with the re-inspection type information, determining that the data to be re-inspected is the second data to be analyzed; The first data to be analyzed and preliminary defect detection information corresponding to the first data to be analyzed, as well as the second data to be analyzed and preliminary defect detection information corresponding to the second data to be analyzed are input into the defect analysis model to generate the defect detection report.

4. The defect detection method according to claim 3, characterized in that: The defect analysis model includes a feature recognition model, an analysis model and a report generation model; The step of inputting the first data to be analyzed and the preliminary defect detection information corresponding to the first data to be analyzed, and the second data to be analyzed and the preliminary defect detection information corresponding to the second data to be analyzed into the defect analysis model to generate the defect detection report includes: Inputting the first data to be analyzed and the preliminary defect detection information corresponding to the first data to be analyzed, as well as the second data to be analyzed and the preliminary defect detection information corresponding to the second data to be analyzed into the feature recognition model for feature recognition processing to obtain defect feature recognition information, wherein the defect feature recognition information includes defect feature information and defect degree information; Inputting the defect feature identification information into the analysis model for analysis and processing to obtain defect analysis information; The defect analysis information is input into the report generation model, and the defect detection report is generated by combining a preset report template and text generation technology.

5. The defect detection method according to claim 1, wherein: The defect detection model includes a feature extraction model, a feature fusion model and a detection model; Inputting the inspection multimodal data into the defect detection model to obtain preliminary defect detection information includes: Inputting the inspection multimodal data into the feature extraction model for hierarchical feature extraction processing to obtain inspection multimodal features of different levels; Inputting the inspection multimodal features of different levels into the feature fusion model for feature fusion processing to obtain inspection multimodal fusion features; The inspection multimodal fusion features are input into the detection model for defect detection processing to obtain preliminary defect detection information.

6. The defect detection method according to claim 5, characterized in that: The defect detection model is trained in the following way: Acquire sample inspection multimodal data collected by the drone inspection sample area, wherein the sample inspection multimodal data includes a sample appearance detection image and a sample thermal state detection image; Performing data enhancement processing on the sample appearance detection image to obtain an expanded appearance detection image; performing thermal diffusion enhancement processing on the sample thermal state detection image to obtain an expanded thermal state detection image; Determining the expanded appearance inspection image and the expanded thermal state inspection image as sample expanded inspection multimodal data, and determining preset defect detection information corresponding to the sample expanded inspection multimodal data; Inputting the sample expanded inspection multimodal data into the defect detection model to be trained for defect detection processing to obtain target defect detection information; The defect detection model to be trained is trained according to the preset defect detection information and the target defect detection information to obtain a defect detection model.

7. The defect detection method according to claim 1, characterized in that: The inspection multimodal data includes appearance detection images and thermal state detection images; The acquisition of multimodal inspection data collected by the drone in the target inspection area includes: Acquire original inspection multimodal data collected by the drone during inspection of the target area, wherein the original inspection multimodal data includes an original appearance detection image and an original thermal state detection image; Performing sliding block processing on the original appearance detection image according to a preset window size and a preset overlap rate to obtain an appearance detection block image; Performing illumination equalization processing on the appearance detection block image to obtain the appearance detection image; Performing color enhancement processing on the original thermal state detection image to obtain the thermal state detection image; Image alignment processing is performed on the appearance detection image and the thermal state detection image to obtain the inspection multimodal data.

8. A defect detection device, characterized in that: The device comprises: The data acquisition module is used to obtain the multimodal inspection data collected by the drone in the inspection target area; A defect detection module is used to input the inspection multimodal data into a defect detection model to obtain preliminary defect detection information; the preliminary defect detection information includes preliminary defect type information, defect location information and defect confidence information; a defect re-inspection module, configured to input the data to be re-inspected and the re-inspection type information into the defect re-inspection model to obtain defect re-inspection information; wherein the data to be re-inspected is the inspection multimodal data corresponding to the defect confidence information being less than or equal to the first preset confidence and greater than or equal to the second preset confidence, and the re-inspection type information is the preliminary defect type information corresponding to the data to be re-inspected; A report generation module is used to generate a defect detection report based on the defect analysis model and in combination with the preliminary defect detection information and the defect re-inspection information.

9. An electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction and at least one program, and the at least one instruction and the at least one program are loaded and executed by the processor to implement the defect detection method according to any one of claims 1 to 7.

10. A computer storage medium, wherein at least one instruction and at least one program are stored in the computer storage medium, wherein the at least one instruction and the at least one program are loaded and executed by a processor to implement the defect detection method according to any one of claims 1 to 7.

Images