A photovoltaic module surface obscuration detection system based on a drone

Abstract

The application discloses a photovoltaic module surface shelter detection system based on a UAV, which comprises a data acquisition module for acquiring original image data and radar point cloud data of a photovoltaic module surface; a data processing module for obtaining a radar feature map according to the image data and the radar point cloud data; a multi-scale feature enhancement module for obtaining texture enhanced features and surface enhanced features according to the image data; and a multi-modal fusion detection module for constructing a multi-modal fusion detection model, inputting the radar feature map, the texture enhanced features and the surface enhanced features into the multi-modal fusion detection model, obtaining a fusion feature map, and realizing surface shelter detection based on the fusion feature map. Through multi-modal data fusion and feature enhancement, the application significantly improves the detection accuracy and efficiency of the photovoltaic module surface shelter, and provides reliable technical support for intelligent operation and maintenance of a photovoltaic power station.

Description

Technical Field

[0001] This invention belongs to the field of intelligent detection technology of millimeter-wave radar and image fusion, and particularly relates to a photovoltaic module surface obstruction detection system based on unmanned aerial vehicles. Background Technology

[0002] Currently, photovoltaic (PV) power generation is a type of renewable energy generation, and offshore PV is increasingly being applied. Due to the complex operating environment of offshore PV modules, many interference factors need to be considered. Among these, shading is a key factor affecting solar panel power generation. Common shading factors include bird droppings, seabird shadows, and algae. When PV modules are shaded, sunlight cannot fully reach the panels, leading to imbalances in current and voltage within the panels. This increase in localized current and voltage causes localized temperature rises, a phenomenon known as the "hot spot effect." When the hot spot effect reaches a certain level, it will inevitably damage the panel grid and solder joints. Once these are damaged, the destructive effect will gradually amplify until the module is unusable, severely impacting the power generation efficiency of the PV power plant.

[0003] Intelligent drones are flexible, easy to operate, and highly efficient. Compared to conventional manual inspections, intelligent drone inspections offer the following advantages: they can quickly cover large areas of photovoltaic power plants, providing high inspection efficiency, significantly reducing the number of personnel and inspection time required, saving labor costs, and resulting in higher economic benefits. Intelligent drones are highly maneuverable, flying freely without being restricted by obstacles. Through image recognition technology, drones can accurately locate obstructions, providing accurate information for subsequent cleaning and maintenance. Drone inspections not only improve the efficiency and safety of photovoltaic power plants but also monitor and assess the environment, ensuring the environmental protection and energy-saving performance of photovoltaic power plants. Drone inspections can reduce damage and pollution to marine ecosystems, aligning with the concept of sustainable development.

[0004] By integrating and fusing information from different sensors and data sources, multimodal fusion technology can increase the diversity and coverage of data. Fusion between different data sources and sensors can improve the accuracy and robustness of target detection and identification. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a drone-based photovoltaic module surface obstruction detection system, thereby resolving the issues present in the prior art.

[0006] To achieve the above objectives, the present invention provides a photovoltaic module surface obstruction detection system based on unmanned aerial vehicles (UAVs), comprising:

[0007] Data acquisition module, data processing module, multi-scale feature enhancement module, multi-modal fusion detection module;

[0008] The data acquisition module is used to acquire raw image data and radar point cloud data of the photovoltaic module surface;

[0009] The data processing module is used to obtain a radar feature map based on the image data and the radar point cloud data;

[0010] The multi-scale feature enhancement module is used to obtain texture enhancement features and surface enhancement features based on the image data;

[0011] The multimodal fusion detection module is used to construct a multimodal fusion detection model. It inputs radar feature maps, texture enhancement features, and surface enhancement features into the multimodal fusion detection model to obtain a fused feature map, and realizes surface occlusion detection based on the fused feature map.

[0012] Optionally, the data processing module includes a radar point cloud data processing unit, an image processing unit, and a mapping unit;

[0013] The radar point cloud data processing unit is used to preprocess the radar point cloud data.

[0014] The image processing unit is used to perform HSV segmentation and morphological closing operation on the original image data to obtain the photovoltaic module region;

[0015] The mapping unit is used to align the displacement-compensated radar point cloud data with the image data in time and space, and after filling with weighted RCS semantic information, map the radar point cloud data to the photovoltaic module area to obtain a radar pseudo-image and extract radar feature map.

[0016] Optionally, the radar point cloud data processing unit calculates the floating displacement vector of the photovoltaic array based on the tidal table data to perform displacement compensation on the radar point cloud coordinates.

[0017] Optionally, the multi-scale feature enhancement module includes a separation unit, a texture enhancement unit, and a surface enhancement unit;

[0018] The separation unit is used to perform polarization feature separation on the original image data to obtain S-polarization feature map and P-polarization feature map;

[0019] The texture enhancement unit is used to perform grouped convolutional algal texture extraction on the P-polarized light feature map to obtain texture enhancement features;

[0020] The surface enhancement unit is used to perform reflection intensity weight enhancement and multi-scale weight enhancement on the S-polarized light feature map. The two enhancement results are weighted and fused to obtain the surface enhancement feature.

[0021] Optionally, the surface enhancement unit includes a max pooling branch and a SimAM attention branch; the max pooling branch includes a max pooling layer and a convolutional layer, and the SimAM attention branch includes a SimAM attention mechanism and a convolutional layer; the outputs of the two branches are concatenated to obtain the surface enhancement features.

[0022] Optionally, the multimodal fusion detection module includes a multi-level feature extraction and fusion unit, a processing unit, a multimodal feature fusion unit, and a detection unit;

[0023] The multi-level feature fusion unit includes a VGG16 backbone network and a feature enhancement submodule. After extracting multi-level feature maps through the VGG16 backbone network, feature enhancement is performed through the feature enhancement submodule to obtain multi-level enhanced features. The multi-level enhanced features include VGG Block3 features and VGG Block5 features. The feature enhancement submodule has the same structure as the surface enhancement unit.

[0024] The processing unit is used to unify the scale of the multi-level enhancement features, radar feature maps, texture enhancement features, and surface enhancement features through sampling.

[0025] The multimodal feature fusion unit is used to fuse features at the same scale to obtain a fused feature map;

[0026] The detection unit performs surface occlusion detection based on the fused feature map.

[0027] Optionally, the multimodal feature fusion unit includes: shallow fusion unit, deep fusion unit, and photovoltaic optimized FPN unit;

[0028] The shallow fusion unit is used to fuse VGG Block3 features, surface enhancement features, and radar feature maps;

[0029] The deep fusion unit is used to fuse VGG Block5 features, texture enhancement features, radar feature maps and infrared features, wherein the infrared features are obtained based on infrared hotspot images;

[0030] The photovoltaic optimized FPN unit is used to generate a two-level feature pyramid based on the outputs of the shallow fusion unit and the deep fusion unit.

[0031] Optionally, an evaluation module may also be included, the evaluation module comprising:

[0032] The parameter setting subunit is used to set the parameters of the multimodal fusion target detection model;

[0033] The model training subunit is used to input data into the fusion model for training;

[0034] The classification and regression subunit is used to input the fused feature map into the evaluation prediction model, and output the evaluation index results after classification and regression.

[0035] Compared with the prior art, the present invention has the following advantages and technical effects:

[0036] This innovative approach combines a parameter-free attention mechanism (SimAM) with a max-pooling layer. By weighting and enhancing feature maps in a parameter-free manner, it avoids additional parameter learning, reduces computational overhead, and adaptively adjusts the region of interest, thereby improving the model's efficiency and flexibility. Through dual feature enhancement using max-pooling and SimAM, key information is extracted from both local and global information, effectively improving the model's ability to capture important features and enhancing its adaptability to complex tasks. The feature maps from the two branches are then concatenated and fused via channels, preserving not only the effective capture of local features by max-pooling but also incorporating the weighted global and local information from SimAM, thus improving the expressive power of the feature maps. This design has significant application value in UAV-based photovoltaic module surface occlusion detection, capable of handling complex environmental changes and lighting conditions, and improving the performance of UAV vision systems in real-world scenarios.

[0037] This invention significantly improves the detection capability of obstructions on the surface of photovoltaic modules through innovative multimodal data fusion and feature enhancement methods. The system utilizes polarization characteristic separation and tidal compensation technology to effectively solve the problem of insufficient differentiation of surface reflection and micro-texture in traditional detection methods, greatly improving the accuracy of identifying typical obstructions such as bird droppings and algae. A hierarchical attention fusion strategy and a dedicated feature pyramid design optimize computational efficiency while ensuring detection accuracy. A unique displacement compensation mechanism ensures accurate alignment of multi-source data, while specially designed evaluation indicators provide a scientific basis for the operation and maintenance of photovoltaic power plants. Attached Figure Description

[0038] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0039] Figure 1 This is a system schematic diagram according to an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of the SimAM attention structure according to an embodiment of the present invention;

[0041] Figure 3 This is an example diagram of the MP-SimAM module according to an embodiment of the present invention. Detailed Implementation

[0042] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0043] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0044] Example 1

[0045] like Figure 1-3 As shown, this embodiment provides a photovoltaic module surface obstruction detection system based on a drone, including:

[0046] Data acquisition module, data processing module, multi-scale feature enhancement module, multi-modal fusion detection module;

[0047] The data acquisition module is used to collect raw image data and radar point cloud data of the photovoltaic module surface;

[0048] The data processing module is used to obtain radar feature maps based on image data and radar point cloud data;

[0049] As one specific implementation method, the data processing module includes a radar point cloud data processing unit, an image processing unit, and a mapping unit;

[0050] The radar point cloud data processing unit is used to preprocess the radar point cloud data; specifically, it calculates the floating displacement vector of the photovoltaic array based on the tidal table data to compensate for the displacement of the radar point cloud coordinates.

[0051] The image processing unit is used to perform HSV segmentation and morphological closing operations on the raw image data to obtain the photovoltaic module region;

[0052] The mapping unit is used to align the displacement-compensated radar point cloud data with the image data in time and space, and after weighted RCS semantic information filling, map the radar point cloud data to the photovoltaic module area to obtain radar pseudo-images and extract radar feature maps.

[0053] Specifically, the radar point cloud mapping and feature extraction module first jointly annotates the radar point cloud with image data and performs spatiotemporal alignment calibration. This maps the radar point cloud onto the image, obtaining a radar pseudo-image. Preliminary segmentation is then performed using the HSV color space to segment the image region corresponding to the photovoltaic module from the original image. Morphological closing operations are used to eliminate noise and gaps. Filtering is then applied to extract discriminative and representative feature information from the acquired point cloud data. This aims to make the point cloud data easier to fuse and process for object detection and recognition.

[0054] First, image correction is performed based on camera extrinsic and intrinsic parameters. Then, radar data and image data are aligned temporally and spatially. Radar data includes semantic features such as azimuth, range, and radar cross-section (RCS), transforming the current radar data from a two-dimensional ground plane into an imaging plane with vertical lines. An uncertain weighted RCS channel is added based on the baseline network. This uncertain weighted RCS channel is formed by multiplying the density value of an uncertain azimuth channel by its corresponding RCS channel value. A total of four channels are input to the radar branch.

[0055] According to the principle of radar point cloud mapping, the characteristics of radar echoes are stored as pixel values ​​in the enhanced image. The radar channel value projected at the image pixel position with no radar reflection intensity is set to 0; other positions are set to the corresponding pixel value. That is, the radar channel is mapped to the corresponding position and displayed with a uniform color.

[0056] Specifically, due to the sparsity problem of radar data, the density of radar data is increased by fusing the past 13 radar cycles (about 1 second) into the radar data format. This projection method compensates for self-motion and finally forms a radar feature image as input.

[0057] To extract more accurate radar point cloud feature information and reduce the impact of interference noise, filtering is required. Since radar echo signals contain many detection results irrelevant to the target, an annotation filter (AF) is used to filter the radar feature image, ultimately yielding more accurate radar feature information.

[0058] The multi-scale feature enhancement module is used to obtain texture enhancement features and surface enhancement features based on image data;

[0059] As one specific implementation, the multi-scale feature enhancement module includes a separation unit, a texture enhancement unit, and a surface enhancement unit;

[0060] The separation unit is used to separate polarization features from the original image data to obtain S-polarization feature maps and P-polarization feature maps.

[0061] The texture enhancement unit is used to perform grouped convolution on the P-polarized light feature map to extract algal texture and obtain texture enhancement features;

[0062] The surface enhancement unit is used to perform reflection intensity weight enhancement and multi-scale weight enhancement on the S-polarized light feature map. The two enhancement results are weighted and fused to obtain the surface enhancement feature.

[0063] As a specific implementation, the surface enhancement unit includes a max pooling branch and a SimAM attention branch; the max pooling branch includes a max pooling layer and a convolutional layer, and the SimAM attention branch includes a SimAM attention mechanism and a convolutional layer; the outputs of the two branches are concatenated to obtain the surface enhancement features.

[0064] For example, the SimAM attention mechanism is a parameterless 3D attention module, such as... Figure 2 As shown. Unlike existing channel attention and spatial attention methods, it can simultaneously focus on the importance of channel and spatial features, inferring the three-dimensional weights of the feature map without increasing the network parameters. See the appendix for the specific structure. Figure 3 .

[0065] First, the input features X∈R of image R C×H×W The first branch uses a max-pooling layer to downsample the feature map's length and width to twice their original size, followed by a 1×1 convolution to change the number of channels, resulting in the first branch. For the feature map, to align the input and output channels of the module with the number of channels in the original network, we take r = 1. Then, for the second branch, the input feature X ∈ R C×H×W First, the feature map is enhanced using the SimAM attention mechanism to obtain the enhanced feature map. Then, a convolutional layer with a kernel size of 3×3 and a stride of 2 is used to downsample the length and width by a factor of 2, resulting in the feature map of the second branch. Finally, the feature map channels of the two branches are concatenated to obtain the final output result:

[0066] The multimodal fusion detection module is used to construct a multimodal fusion detection model. It inputs radar feature maps, texture enhancement features, and surface enhancement features into the multimodal fusion detection model to obtain a fused feature map, and then realizes the detection of surface occlusions based on the fused feature map.

[0067] As a specific implementation method, the multimodal fusion detection module includes a multi-level feature extraction and fusion unit, a processing unit, a multimodal feature fusion unit, and a detection unit;

[0068] The multi-level feature fusion unit includes a VGG16 backbone network and a feature enhancement submodule. After extracting multi-level feature maps through the VGG16 backbone network, the feature enhancement submodule performs feature enhancement to obtain multi-level enhanced features. Among them, the multi-level enhanced features include VGG Block3 features and VGG Block5 features. The feature enhancement submodule has the same structure as the surface enhancement unit.

[0069] The processing unit is used to unify the scale of multi-level enhancement features, radar feature maps, texture enhancement features, and surface enhancement features through sampling;

[0070] The multimodal feature fusion unit is used to fuse features at the same scale to obtain a fused feature map;

[0071] The detection unit performs surface occlusion detection based on the fused feature map.

[0072] As a specific implementation method, the multimodal feature fusion unit includes: a shallow fusion unit, a deep fusion unit, and a photovoltaic optimized FPN unit;

[0073] Shallow fusion units are used to fuse VGG Block3 features, surface enhancement features, and radar feature maps;

[0074] The deep fusion unit is used to fuse VGG Block5 features, texture enhancement features, radar feature maps and infrared features, where the infrared features are obtained based on infrared hotspot images;

[0075] The photovoltaic optimized FPN unit is used to generate a two-level feature pyramid based on the outputs of the shallow fusion unit and the deep fusion unit.

[0076] Specifically, it also includes an evaluation module, which includes:

[0077] The parameter setting subunit is used to set the parameters of the multimodal fusion target detection model;

[0078] The model training subunit is used to input data into the fusion model for training;

[0079] The classification and regression subunit is used to input the fused feature map into the evaluation prediction model, and output the evaluation index results after classification and regression.

[0080] Specifically, parameters are set for the multimodal fusion target detection model, and the fused feature map is input into the evaluation prediction model. After classification and regression, the category of each detected occlusion is output, mainly including bird shadows, bird droppings, moss and algae, etc., as well as the results of the selected evaluation indicators, including accuracy, mean precision, mAP, recall, mean squared error, and mean absolute error data.

[0081] This embodiment primarily detects occlusions such as bird shadows, bird droppings, and mosses and algae. The steps include: linearly mapping radar point clouds onto the image for joint annotation; utilizing a baseline fusion network based on radar channels and weighted RCS channels to fully leverage radar information; performing multi-level feature extraction on both the radar and original images; fusing features from different modalities using a fusion model based on VGG16 and a feature pyramid backbone network; and employing a general, simple, effective, and parameter-free hybrid attention mechanism, the MP-SimAM module. Emphasis is placed on meaningful features along the channel and spatial dimensions to guide the feature extraction network to more accurately focus on the target object, achieving higher accuracy in target detection and improving model robustness.

[0082] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A photovoltaic module surface obstruction detection system based on unmanned aerial vehicles (UAVs), characterized in that, include: The system comprises a data acquisition module, a data processing module, a multi-scale feature enhancement module, and a multi-modal fusion detection module. The data acquisition module is used to acquire image data and radar point cloud data of the photovoltaic module surface; The data processing module is used to obtain a radar feature map based on the image data and the radar point cloud data; The multi-scale feature enhancement module is used to obtain texture enhancement features and surface enhancement features based on the image data; The multimodal fusion detection module is used to construct a multimodal fusion detection model. It inputs radar feature maps, texture enhancement features, and surface enhancement features into the multimodal fusion detection model to obtain a fused feature map, and realizes surface occlusion detection based on the fused feature map. The multi-scale feature enhancement module includes a separation unit, a texture enhancement unit, and a surface enhancement unit; The separation unit is used to perform polarization feature separation on the image data to obtain S-polarization feature map and P-polarization feature map; The texture enhancement unit is used to perform grouped convolutional algal texture extraction on the P-polarized light feature map to obtain texture enhancement features; The surface enhancement unit is used to perform reflection intensity weight enhancement and multi-scale weight enhancement on the S-polarized light feature map. The two enhancement results are weighted and fused to obtain the surface enhancement feature. The surface enhancement unit includes a max pooling branch and a SimAM attention branch; The max pooling branch includes a max pooling layer and a convolutional layer, and the SimAM attention branch includes a SimAM attention mechanism and a convolutional layer; the outputs of the two branches are concatenated to obtain surface enhancement features; The multimodal fusion detection module includes a multi-level feature extraction and fusion unit, a processing unit, a multimodal feature fusion unit, and a detection unit; The multi-level feature extraction and fusion unit includes a VGG16 backbone network and a feature enhancement submodule. After extracting multi-level feature maps through the VGG16 backbone network, feature enhancement is performed through the feature enhancement submodule to obtain multi-level enhanced features. Among them, the multi-level enhanced features include VGG Block3 features and VGG Block5 features. The feature enhancement submodule has the same structure as the surface enhancement unit; The processing unit is used to unify the scale of the multi-level enhancement features, radar feature maps, texture enhancement features, and surface enhancement features through sampling. The multimodal feature fusion unit is used to fuse features after unification to obtain a fused feature map; The detection unit performs surface occlusion detection based on the fused feature map.

2. The photovoltaic module surface obstruction detection system based on UAV according to claim 1, characterized in that, The data processing module includes a radar point cloud data processing unit, an image processing unit, and a mapping unit. The radar point cloud data processing unit is used to preprocess the radar point cloud data. The image processing unit is used to perform HSV segmentation and morphological closing operation on the image data to obtain the photovoltaic module region; The mapping unit is used to align the displacement-compensated radar point cloud data with the image data in time and space, and after filling with weighted RCS semantic information, map the radar point cloud data to the photovoltaic module area to obtain a radar pseudo-image and extract radar feature map.

3. The photovoltaic module surface obstruction detection system based on unmanned aerial vehicles according to claim 2, characterized in that, The radar point cloud data processing unit calculates the floating displacement vector of the photovoltaic array based on the tidal table data to compensate for the displacement of the radar point cloud coordinates.

4. The photovoltaic module surface obstruction detection system based on UAV according to claim 1, characterized in that, The multimodal feature fusion unit includes: shallow fusion unit, deep fusion unit, and photovoltaic optimized FPN unit; The shallow fusion unit is used to fuse VGG Block3 features, surface enhancement features, and radar feature maps; The deep fusion unit is used to fuse VGG Block5 features, texture enhancement features, radar feature maps and infrared features, wherein the infrared features are obtained based on infrared hotspot images; The photovoltaic optimized FPN unit is used to generate a two-level feature pyramid based on the outputs of the shallow fusion unit and the deep fusion unit.

5. The photovoltaic module surface obstruction detection system based on UAV according to claim 1, characterized in that, It also includes an evaluation module, which comprises: The parameter setting subunit is used to set the parameters of the multimodal fusion detection model; The model training subunit is used to input data into the fusion model for training; The classification and regression subunit is used to input the fused feature map into the evaluation prediction model, and output the evaluation index results after classification and regression.

Images