Unmanned aerial vehicle photovoltaic inspection defect identification method based on multi-mode large model and real-time detection system

Abstract

The invention discloses an unmanned aerial vehicle photovoltaic inspection defect identification method based on a multi-mode large model and a real-time detection system, and the method comprises the following steps: data collection: an unmanned aerial vehicle collects a visible light image, an infrared thermogram and three-dimensional point cloud data of each photovoltaic panel along a flight path, and collects environmental condition parameters at the same time; data preprocessing: carrying out image enhancement on the acquired image data of each photovoltaic panel by using a Stable Diffusion model; segmentation and positioning: fine segmentation and positioning of a defect area are realized on the preprocessed data of each photovoltaic panel by using an SAM model; feature extraction: extracting comprehensive features of the segmented and positioned image in combination with text description through a trained CLIP model by using a CLIP model; and classification and identification: matching the comprehensive features by using a trained CLIP model, and outputting and judging the defect type and severity according to a matching result, so that the identification precision is high, and the misjudgment rate is low.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) photovoltaic inspection, and particularly to a method for identifying defects in UAV photovoltaic inspection based on a multimodal large model and a real-time detection system. Background Art

[0002] With the development of smart grids, regular inspection of power facilities has become increasingly important, especially for large-scale distributed energy facilities such as photovoltaic power plants. Traditional manual inspection methods are not only inefficient but also pose safety hazards under adverse weather and complex terrain conditions. In recent years, the application of UAVs equipped with high-definition cameras and sensors for power inspection has become increasingly widespread, greatly improving the inspection efficiency and safety. However, UAV-based image recognition technology still faces a series of challenges, especially in the identification of tiny defects in complex environments. To address the above challenges, the present invention proposes a method for identifying defects in UAV photovoltaic inspection based on a multimodal large model, which demonstrates significant advantages in terms of creativity, novelty, and practicality. The deficiencies of traditional technologies in photovoltaic inspection defect identification are as follows: 1. Insufficient recognition accuracy: Traditional image recognition faces the problem of low accuracy in complex scenarios of power inspection. Especially under the interference of natural environments such as tree branch occlusion and light changes, it is easy to cause misjudgment or missed detection of tiny defects such as small cracks, affecting the accuracy of safety assessment. The recognition rate of some algorithms is lower than 70%; 2. High dependence on manpower: Although the automation of UAV inspection improves efficiency, subsequent manual review and confirmation of defects are required. This process is time-consuming and costly, and professional personnel need to check frame by frame, slowing down the entire inspection process and increasing the operation burden; 3. Rigid inspection reports: Inspection reports with fixed templates cannot well express the real situation during the actual inspection process; 4. High cost for iterative optimization: Usually, it costs a huge amount of money (in the tens of millions) to implement an algorithm for a new scenario. Summary of the Invention

[0003] The present invention provides a method for identifying defects in UAV photovoltaic inspection based on a multimodal large model and a real-time detection system, aiming to solve the problems of insufficient recognition accuracy, high dependence on manpower, rigid inspection reports, and high cost for iterative optimization in existing inspections.

[0004] To solve the above technical problems, the technical solutions adopted by the present invention are as follows: A method for identifying defects in UAV photovoltaic inspection based on a multimodal large model includes the following steps: Step1. Data collection: The UAV collects visible light images, infrared thermal images, and three-dimensional point cloud data of each photovoltaic panel along the flight route, and simultaneously collects environmental condition parameters; Step 2. Data preprocessing: Use the Stable Diffusion model to perform image enhancement on the image data of each photovoltaic panel collected. Step 3. Segmentation and localization: Use the SAM model to achieve fine-grained segmentation and localization of the defect areas in the preprocessed data of each photovoltaic panel. Step 4. Feature extraction: Use the CLIP model to train the CLIP model through existing defects and their corresponding CLIP model feature vectors, and extract comprehensive features from the segmented and localized images combined with text descriptions through the trained CLIP model. Step 5. Classification and recognition: Use the trained CLIP model to match the comprehensive features, output the judgment of the defect type and severity according to the matching results, and generate a detailed defect report, including the type, location, size, and recommended repair measures.

[0005] Preferably, Step 1 includes the following steps: Step 101. Flight path planning: Generate a flight route according to the photovoltaic panel layout to ensure coverage of each photovoltaic panel. Step 102. Synchronous acquisition: Multiple sensors work synchronously to obtain visible light images, infrared thermal images, and three-dimensional point cloud data. Step 103. Environmental condition monitoring: Real-time environmental parameters, including monitoring wind speed and light intensity, and adjust the flight height and speed to ensure data quality. Step 104. Data preprocessing: During the flight, preliminarily filter out useless data, including a large number of duplicate or low-quality images.

[0006] Preferably, Step 2 includes the following steps: Step 201. Image preprocessing: Include denoising and contrast enhancement of the collected image data. Step 202. Feature enhancement and recognition: Remove image noise through the Stable Diffusion model and simultaneously enhance the feature expression of the image.

[0007] More preferably, Step 201 and Step 202 also include the following steps: Step 2011. Image denoising: Perform denoising processing on the collected images to remove noise introduced by external factors. The external factors include drone flight vibration and atmospheric disturbance to ensure image quality. Step 2012. Contrast enhancement: Enhance the image contrast to make the texture and details on the surface of the photovoltaic panel more distinct for subsequent feature extraction. Step2021. Application of the Stable Diffusion model: Input the denoised and contrast-enhanced image into the Stable Diffusion model. Through the noise injection and diffusion process, the model gradually removes the image noise and enhances the feature expression of the image, especially those subtle defect features. Step2022. Generation of the feature map: The Stable Diffusion model outputs an enhanced feature map, which not only highlights the potential defect areas on the photovoltaic panel but also preserves the global structural information of the image.

[0008] Preferably, the Step3 includes the following steps: Step301. Generation of the attention weight map: Generate the attention weight map through attention weight calculation and further optimize the weight map. Step302. Refined region segmentation: Use the optimized attention weight map to perform region segmentation on the original image and refine the boundaries of the segmented regions.

[0009] More preferably, the Step301 and Step302 also include the following steps: Step3011. Attention weight calculation: Input the enhanced feature map into the SAM model and generate the attention weight map by calculating the similarity between features. Step3012. Weight map optimization: Further, adjust the attention weight through an iterative optimization algorithm to ensure that the weight map can accurately reflect the distribution of the defect regions and suppress the influence of the background regions. Step3021. Region segmentation: Use the optimized attention weight map to perform region segmentation on the original image and separate the suspected defect regions on the photovoltaic panel from the background. Step3022. Boundary refinement: Refine the boundaries of the segmented defect regions to ensure that the boundaries are clear and accurate, providing a reliable data basis for subsequent defect type recognition.

[0010] Preferably, the Step4 includes the following steps: Step401. Image and text input: Input the defect region image segmented by the SAM model and the relevant context description text into the trained CLIP model. Step402. Feature fusion: The CLIP model automatically performs the fusion of image and text features internally to construct a unified representation space for subsequent matching and classification.

[0011] Preferably, the Step5 includes the following steps: Step 501, Matching and Classification: In the fused feature space, the CLIP model can identify the defect type, match it with the known defect patterns in the database, and determine the severity of the defect; Step 502, Detailed Description Generation: For each identified defect type, the model generates a detailed text description, including the nature of the defect, possible causes, and recommended maintenance measures, to provide guidance for subsequent maintenance work.

[0012] The UAV photovoltaic inspection real-time detection system based on the multimodal large model is used to apply the above-mentioned UAV photovoltaic inspection defect recognition method based on the multimodal large model, and includes the following modules installed on the UAV: Data Acquisition and Transmission Module: It includes a high-definition camera, an infrared thermal imager, a LiDAR sensor, and a weather sensor installed on the UAV, which are used to collect visible light images, infrared thermal images, and three-dimensional point cloud data of the photovoltaic panels in real time, and at the same time collect environmental condition parameters, and transmit the data to the backend server through the wireless communication module; Wireless Communication Module: It includes a 5G module installed on the UAV, which is used for data transfer between modules; Storage Module: It includes a high-performance CPU, high-speed storage, and large-capacity storage installed on the UAV, which are used to receive the data collected by the preprocessed data acquisition and transmission module, and the data is stored in the cache for subsequent processing; Model Inference Engine: It includes a graphics card and memory installed on the UAV. The graphics card integrates the Stable Diffusion model, the SAM model, and the CLIP model, and performs real-time analysis and recognition on the preprocessed acquisition data in the storage module. The GPU acceleration technology is adopted to ensure high computing performance; Decision-making and Feedback Module: It includes a real-time operating system and data management software integrated on the high-performance CPU, which are used to automatically identify the defect type and its location on the photovoltaic panel according to the output result of the model inference engine, generate maintenance suggestions, and feedback this information to the on-site staff in real time to guide them to perform timely repairs or replacements; User Interface and Data Visualization Module: It includes a Web server, a data visualization tool, and a front-end development framework. The data acquisition and transmission module, the storage module, the model inference engine, and the decision-making and feedback module are all connected to the Web server through the wireless communication module, which is used to provide a friendly user interface, allowing the staff to view the detection results in real time, including defect images, detailed reports, and maintenance suggestions. Through data visualization technologies, including heat maps and trend charts, it helps the staff quickly understand the overall health status of the photovoltaic panels.

[0013] Preferably, a data preprocessing module is further connected in front of the storage module, which is used to filter a large number of duplicate or low-quality images by using a high-performance CPU.

[0014] Advantages of the present invention: The present invention greatly improves the recognition accuracy in photovoltaic inspection by using three models, which is not easily obtained by combination in the art, and achieves the following remarkable effects by using three models: 1. Improve recognition accuracy: Intelligently inspect the power distribution network by using drones to achieve efficient defect recognition. The core objectives include: constructing an intelligent recognition engine that integrates visible light, infrared, and acoustic wave data, and improving the recognition accuracy to more than 95%; 2. Reduce the R & D cost of new models: Greatly reduce the dependence on manpower; build a model with strong self-adaptability and fast iterative upgrade, and reduce the cost of new algorithm iteration by more than 90%; 3. Reduce labor costs: Reduce the false alarm rate of the original algorithm, reduce the original false alarm rate by 99%, and reduce the cost of manual recheck by 90%. Use a large model to generate inspection reports in the way of generating text from pictures and generating pictures from pictures according to the shooting content. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic flow chart of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0016] As follows, the embodiments will be further described with reference to the drawings.

[0017] As Figure 1 shown, as a preferred Embodiment 1, a method for identifying defects in photovoltaic inspection by using a drone based on a multi-modal large model includes the following steps: Step1. Data collection: The drone collects visible light images, infrared thermal images, and three-dimensional point cloud data of each photovoltaic panel along the flight route, and simultaneously collects environmental condition parameters; Step2. Data preprocessing: Use the Stable Diffusion model to enhance the image data of each photovoltaic panel collected; Step3. Segmentation and positioning: Use the SAM model to achieve fine segmentation and positioning of the defect area for the preprocessed data of each photovoltaic panel; Step4. Feature extraction: Use the CLIP model to train the CLIP model through existing defects and their corresponding CLIP model feature vectors, and extract comprehensive features from the segmented and positioned images combined with text descriptions through the trained CLIP model; Step 5. Classification and Recognition: Use the trained CLIP model to match the comprehensive features, and output the judgment of the defect type and severity according to the matching results, and generate a detailed defect report, including the type, location, size, and recommended repair measures.

[0018] Preferably, the Step 1 includes the following steps: Step 101. Flight Path Planning: Generate a flight route according to the photovoltaic panel layout to ensure coverage of each photovoltaic panel; Step 102. Synchronous Acquisition: Multiple sensors work synchronously to obtain visible light images, infrared thermal images, and three-dimensional point cloud data; Step 103. Environmental Condition Monitoring: Real-time environmental parameters, including monitoring wind speed and light intensity, and adjust the flight altitude and speed to ensure data quality; Step 104. Data Preprocessing: During the flight, preliminarily filter out useless data, including a large number of repeated or low-quality images.

[0019] In the modern photovoltaic panel detection process, using an unmanned aerial vehicle as the front-end data acquisition platform can quickly cover a large area of photovoltaic panel arrays and collect high-definition image data. However, the identification of tiny defects on the surface of photovoltaic panels remains a technical challenge, especially under changing lighting conditions and complex background interference. To solve this problem, the present invention proposes an image enhancement and defect recognition scheme based on the Stable Diffusion model.

[0020] Preferably, the Step 2 includes the following steps: Step 201. Image Preprocessing: Include denoising and contrast enhancement of the collected image data; Step 202. Feature Enhancement and Recognition: Remove image noise through the Stable Diffusion model and at the same time enhance the feature expression of the image.

[0021] More preferably, the Step 201 and Step 202 further include the following steps: Step 2011. Image Denoising: Perform denoising processing on the collected images to remove the noise introduced by external factors. The external factors include the vibration of the unmanned aerial vehicle during flight and atmospheric disturbance to ensure image quality; Common denoising methods include, but are not limited to, bilateral filtering, median filtering, or non-local means denoising techniques.

[0022] Step 2012. Contrast Enhancement: Enhance the image contrast to make the texture and details on the surface of the photovoltaic panel more distinct for subsequent feature extraction; Techniques such as histogram equalization or adaptive histogram equalization (CLAHE) are used.

[0023] Step2021. Application of Stable Diffusion Model: Input the denoised and contrast-enhanced image into the Stable Diffusion model. Through the noise injection and diffusion process, the model gradually removes the image noise while enhancing the feature expression of the image, especially those subtle defect features. This process helps to maintain the recognition stability of the model in complex backgrounds.

[0024] Step2022. Feature Map Generation: The Stable Diffusion model outputs an enhanced feature map, which not only highlights the potential defect areas on the photovoltaic panel but also preserves the global structural information of the image, providing a high-quality basis for subsequent fine segmentation.

[0025] After obtaining the enhanced feature map output by the Stable Diffusion model, the next task is to perform fine segmentation on the defect areas on the photovoltaic panel. The present invention uses the Segmentation Attention Mechanism (SAM) to focus on key areas through the attention mechanism and improve the segmentation accuracy.

[0026] Preferably, Step 3 includes the following steps: Step301. Generate Attention Weight Map: Generate an attention weight map through attention weight calculation and further optimize the weight map. Step302. Fine-grained Region Segmentation: Use the optimized attention weight map to perform region segmentation on the original image and refine the boundaries of the segmented regions.

[0027] More preferably, Step 301 and Step 302 further include the following steps: Step3011. Attention Weight Calculation: Input the enhanced feature map into the SAM model and generate an attention weight map by calculating the similarity between features. This process is similar to assigning a weight value to each pixel point to reflect its importance in defect detection.

[0028] Step3012. Weight Map Optimization: Further, adjust the attention weights through an iterative optimization algorithm to ensure that the weight map can accurately reflect the distribution of defect areas while suppressing the influence of background areas. Step3021. Region Segmentation: Use the optimized attention weight map to perform region segmentation on the original image and separate the suspected defect areas on the photovoltaic panel from the background. Step3022. Boundary Refinement: Refine the boundaries of the segmented defect areas to ensure that the boundaries are clear and accurate, providing a reliable data basis for subsequent defect type recognition.

[0029] To further improve the accuracy and robustness of defect recognition, the present invention introduces a multimodal fusion strategy of the CLIP model. The CLIP model can process image and text data simultaneously. Through cross-modal matching, it can not only identify defect types but also understand the context information of the environment where they are located, thereby enhancing the recognition ability for new or disguised defects.

[0030] Preferably, Step 4 is cross-modal input, including the following steps: Step 401, Image and text input: Input the defect area image segmented by the SAM model and the relevant context description text (e.g., "shadow area", "temperature anomaly", etc.) into the trained CLIP model together. Step 402, Feature fusion: The CLIP model automatically performs the fusion of image and text features internally to construct a unified representation space for subsequent matching and classification.

[0031] Preferably, Step 5 is defect type recognition and description, including the following steps: Step 501, Matching and classification: In the fused feature space, the CLIP model can identify the defect type and match it with the known defect patterns in the database to judge the severity of the defect. Step 502, Detailed description generation: For each identified defect type, the model generates a detailed text description, including the nature of the defect, possible causes, and recommended maintenance measures, to provide guidance for subsequent maintenance work.

[0032] As a preferred embodiment 2, a real-time detection system for UAV photovoltaic inspection based on a multimodal large model is used to apply the above-mentioned UAV photovoltaic inspection defect recognition method based on a multimodal large model, including the following modules installed on the UAV: Data acquisition and transmission module: It includes a high-definition camera, an infrared thermal imager, a LiDAR sensor, and a weather sensor installed on the UAV, which are used to collect visible light images, infrared thermal images, and three-dimensional point cloud data of the photovoltaic panels in real time, and at the same time collect environmental condition parameters, and transmit the data to the backend server through the wireless communication module. Wireless communication module: It includes a 5G module installed on the UAV, which is used for data transfer between modules. Storage module: It includes a high-performance CPU, high-speed storage, and large-capacity storage installed on the UAV, which are used to receive the data collected by the preprocessed data acquisition and transmission module, and store the data in the cache for subsequent processing. Model Inference Engine: It includes a graphics card and memory installed on the drone. The graphics card integrates the Stable Diffusion model, SAM model, and CLIP model, and performs real-time analysis and recognition on the preprocessed acquisition data in the storage module. It uses GPU acceleration technology to ensure efficient computing performance; Decision and Feedback Module: It includes a real-time operating system and data management software integrated on a high-performance CPU. It is used to automatically identify the defect types and their locations on the photovoltaic panel according to the output results of the model inference engine, generate maintenance suggestions, and feedback this information to the on-site staff in real time to guide them to perform timely repairs or replacements; User Interface and Data Visualization Module: It includes a Web server, data visualization tools, and a front-end development framework. The data acquisition and transmission module, storage module, model inference engine, and decision and feedback module are all connected to the Web server through a wireless communication module. It is used to provide a friendly user interface, allowing staff to view the detection results in real time, including defect images, detailed reports, and maintenance suggestions. Through data visualization technologies, including heat maps and trend charts, it helps staff quickly understand the overall health status of the photovoltaic panel.

[0033] Preferably, a data preprocessing module is also connected in front of the storage module, which is used to filter a large number of duplicate or low-quality images using a high-performance CPU.

[0034] As a preferred Embodiment 3, this embodiment is further optimized on the basis of Embodiment 2 and provides the following specific configurations: (1) Data Acquisition and Transmission Module Drone-mounted Equipment: High-definition camera (at least 12MP, supporting 4K video recording); Infrared thermal imager (high resolution, wide temperature range); LiDAR sensor (for terrain and obstacle detection); Wireless Communication Technology: 5G module (supporting high-speed data transmission); Wi-Fi 6E (for backup short-distance high-speed transmission); (2) Data Preprocessing and Storage Module High-performance CPU: Intel Xeon or AMD EPYC series processors, at least 16 cores and 32 threads; High-speed storage: NVMe SSD, at least 1TB for cache and temporary file storage; Large-capacity storage: SAS HDD, at least 10TB for long-term data storage; (3) Model Inference Engine GPU Acceleration: NVIDIA RTX 3090 or higher-level GPU for deep learning model inference; At least 24GB of video memory, supporting TensorFlow and PyTorch frameworks; Memory: DDR4 ECC RAM, at least 64GB, ensuring data processing speed and stability; (4) Decision and Feedback Module Real-time operating system: Linux distribution such as Ubuntu Server, supporting real-time processing tasks; Data management software: PostgreSQL or MongoDB database for storing and querying detection results; (5) User Interface and Data Visualization Module Web server: Apache or Nginx for hosting the user interface and data interfaces; Data visualization tool: Grafana or Tableau for creating heat maps, trend charts, etc.; Front-end development framework: React.js or Vue.js for building the user interface; (6) Other Requirements Network security: Firewall and encryption technology to ensure secure data transmission; Power management: UPS uninterruptible power supply to prevent sudden power outages from affecting data integrity; Cooling system: Efficient heat dissipation solution to keep the server running stably.

[0035] As a preferred Embodiment 4, a comparative test is set up to compare the data using the previous identification method with the data using the identification method in Embodiment 1, and the following data is obtained, as shown in Table 1: Table 1

[0036] The following results are obtained: 1. Improve identification accuracy: Through intelligent inspection of the power distribution network by drones, efficient defect identification is achieved. The core objectives include: constructing an intelligent identification engine that integrates visible light, infrared, and acoustic data, and improving the identification accuracy to over 95%; 2. Reduce the R & D cost of new models: Greatly reduce human dependence; build a model with strong adaptability and rapid iterative upgrade, reducing the cost of new algorithm iteration by over 90%; 3. Reduce labor costs: Reduce the false alarm rate of the original algorithm by 99%, and reduce the cost of manual rechecking by 90%. Use a large model to generate inspection reports in the form of generating text from images and generating images from images according to the captured content.

Claims

1. A method for identifying photovoltaic inspection defects using drones based on a multi-modal large model, characterized in that: The following steps are involved: Step 1, data collection: The drone collects visible light images, infrared thermal images and three-dimensional point cloud data of each photovoltaic panel along the flight route, and simultaneously collects environmental condition parameters; Step 2, data preprocessing: Use the Stable Diffusion model to enhance the image data of each photovoltaic panel collected; Step 3, segmentation and positioning: Use the SAM model to perform refined segmentation and positioning of defective areas on the pre-processed data of each photovoltaic panel; Step 4, feature extraction: Using the CLIP model, the CLIP model is trained through existing defects and one-to-one corresponding CLIP model feature vectors, and the trained CLIP model is used to extract comprehensive features from the segmented and located images combined with text descriptions; Step 5, Classification and Identification: Use the trained CLIP model to match the comprehensive features, determine the defect type and severity based on the matching results, and generate a detailed defect report including type, location, size and recommended repair measures.

2. The method for identifying photovoltaic inspection defects by unmanned aerial vehicle based on a multimodal large model according to claim 1 is characterized in that: Step 1 includes the following steps: Step 101, flight path planning: Generate a flight route based on the layout of photovoltaic panels to ensure that every photovoltaic panel is covered; Step 102, synchronous acquisition: multiple sensors work synchronously to obtain visible light images, infrared thermal images and three-dimensional point cloud data; Step 103, Environmental Condition Monitoring: Real-time environmental parameters, including wind speed, light intensity, and adjusting flight altitude and speed to ensure data quality; Step 104, data preprocessing: During the flight, preliminary filtering of useless data, including a large number of repeated or low-quality images.

3. The method for identifying photovoltaic inspection defects by unmanned aerial vehicle based on a multimodal large model according to claim 1 is characterized in that: The Step 2 includes the following steps: Step 201, image preprocessing: including denoising and contrast enhancement of collected image data; Step 202, feature enhancement and recognition: Remove image noise through the Stable Diffusion model and enhance the feature expression of the image.

4. The method for identifying photovoltaic inspection defects by unmanned aerial vehicle based on a multimodal large model according to claim 3 is characterized in that: The Step 201 and Step 202 further include the following steps: Step 2011, Image denoising: De-noising the collected images to remove the noise introduced by external factors, such as the flight vibration of the drone and atmospheric disturbance, to ensure image quality; Step 2012, contrast enhancement: enhance the image contrast to make the texture and details of the photovoltaic panel surface more distinct, which is convenient for subsequent feature extraction; Step 2021. Application of Stable Diffusion model: The denoised and contrast-enhanced image is input into the Stable Diffusion model. The model gradually removes image noise through noise injection and diffusion processes, while enhancing the feature expression of the image, especially those subtle defect features. Step 2022. Feature map generation: The Stable Diffusion model outputs an enhanced feature map, which not only highlights the potential defect areas on the photovoltaic panel, but also maintains the global structural information of the image.

5. The method for identifying photovoltaic inspection defects by unmanned aerial vehicle based on a multimodal large model according to claim 1 is characterized in that: Step 3 includes the following steps: Step 301, generate attention weight map: generate attention weight map through attention weight calculation, and further optimize the weight map; Step 302, refined region segmentation: Use the optimized attention weight map to segment the original image and refine the boundaries of the segmented regions.

6. The method for identifying photovoltaic inspection defects by unmanned aerial vehicle based on a multimodal large model according to claim 5 is characterized in that: The Step 301 and Step 302 further include the following steps: Step 3011, attention weight calculation: input the enhanced feature map into the SAM model, and generate the attention weight map by calculating the similarity between the features; Step 3012, weight map optimization: Further, the attention weight is adjusted through an iterative optimization algorithm to ensure that the weight map can accurately reflect the distribution of defect areas while suppressing the influence of background areas; Step 3021, region segmentation: Use the optimized attention weight map to perform region segmentation on the original image and separate the suspected defective area on the photovoltaic panel from the background; Step 3022, boundary refinement: refine the boundary of the segmented defect area to ensure that the boundary is clear and accurate, providing a reliable data basis for subsequent defect type identification.

7. The method for identifying photovoltaic inspection defects by unmanned aerial vehicle based on a multi-modal large model according to claim 1 is characterized in that: The Step 4 includes the following steps: Step 401, image and text input: the defect area image segmented by the SAM model and the related context description text are input into the trained CLIP model; Step 402, feature fusion: The CLIP model automatically fuses image and text features to build a unified representation space for subsequent matching and classification.

8. The method for identifying photovoltaic inspection defects by unmanned aerial vehicle based on a multi-modal large model according to claim 1 is characterized in that: Step 5 includes the following steps: Step 501, matching and classification: In the fused feature space, the CLIP model can identify the defect type and match it with the known defect patterns in the database to determine the severity of the defect; Step 502, detailed description generation: For each identified defect type, the model will generate a detailed text description, including the nature of the defect, possible causes and recommended maintenance measures, to provide guidance for subsequent maintenance work.

9. The real-time detection system for photovoltaic inspection by unmanned aerial vehicles based on multi-modal large models is characterized by: The method for identifying photovoltaic inspection defects by a drone based on a multimodal large model according to any one of claims 1 to 8 comprises the following modules arranged on the drone: Data acquisition and transmission module: including high-definition cameras, infrared thermal imagers, LiDAR sensors and weather sensors on drones, which are used to collect visible light images, infrared thermal images and three-dimensional point cloud data of photovoltaic panels in real time, and collect environmental condition parameters at the same time, and transmit the data to the back-end server through the wireless communication module; Wireless communication module: including the 5G module on the drone, used for data transmission between modules; Storage module: including high-performance CPU, high-speed storage and large-capacity storage on the drone, used to receive the pre-processed data collected by the data acquisition and transmission module, and the data is stored in the cache for subsequent processing; Model inference engine: including the graphics card and memory on the drone. The graphics card integrates the Stable Diffusion model, SAM model and CLIP model, performs real-time analysis and recognition of the collected data pre-processed in the storage module, and uses GPU acceleration technology to ensure efficient computing performance; Decision-making and feedback module: including real-time operating system and data management software integrated on high-performance CPU, which is used to automatically identify the defect type and location on the photovoltaic panel according to the output results of the model reasoning engine, generate maintenance suggestions, and feed back this information to the on-site staff in real time to guide them to carry out timely repairs or replacements; User interface and data visualization module: including Web server, data visualization tools and front-end development framework. Data acquisition and transmission module, storage module, model inference engine and decision and feedback module are all connected to the Web server through wireless communication module to provide a friendly user interface, allowing staff to view the inspection results in real time, including defect images, detailed reports and maintenance suggestions. Through data visualization technology, including thermal maps and trend maps, it helps staff quickly understand the overall health status of photovoltaic panels.

10. The UAV photovoltaic inspection real-time detection system based on multi-modal large model according to claim 9 is characterized in that: The storage module is also connected to a data preprocessing module in front of the storage module, which is used to filter a large number of repeated or low-quality images using a high-performance CPU.

Images