Photovoltaic module surface shielding object detection system based on unmanned aerial vehicle

Abstract

The invention discloses a photovoltaic module surface shielding object detection system based on an unmanned aerial vehicle. The system comprises a data acquisition module used for acquiring original image data and radar point cloud data of the surface of a photovoltaic module; the data processing module is used for obtaining a radar feature map according to the image data and the radar point cloud data; the multi-scale feature enhancement module is used for obtaining texture enhancement features and surface enhancement features according to the image data; and the multi-modal fusion detection module is used for constructing a multi-modal fusion detection model, inputting the radar feature map, the texture enhancement features and the surface enhancement features into the multi-modal fusion detection model to obtain a fusion feature map, and realizing surface shielding object detection based on the fusion feature map. According to the invention, through multi-modal data fusion and feature enhancement, the detection precision and efficiency of the shielding object on the surface of the photovoltaic module are significantly improved, and reliable technical support is provided for intelligent operation and maintenance of a photovoltaic power station.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent detection of millimeter wave radar and image fusion, and particularly relates to a photovoltaic module surface shelter detection system based on a UAV. BACKGROUND

[0002] At present, photovoltaic power generation is one of the renewable energy power generation methods, and offshore photovoltaics are increasingly applied. Due to the complex operating environment of offshore photovoltaic modules, many interference factors need to be considered, among which the shelter is a key factor affecting the power generation of solar panels. Common shelters such as bird droppings and shadows of seabirds, algae, etc. When the photovoltaic module is sheltered, sunlight cannot fully irradiate the panel, resulting in uneven current and voltage in the panel, an increase in local current and voltage, and a local temperature rise. This phenomenon is called "hot spot effect". When the hot spot effect reaches a certain degree, it will inevitably cause damage to the panel grid, solder joints, etc. Once the point is damaged, the damage effect will be gradually amplified until the module is scrapped, which will seriously affect the power generation efficiency of the photovoltaic power station.

[0003] The intelligent UAV is flexible, simple, and efficient to operate, and has the following advantages compared with conventional manual inspection: it can quickly cover a large area of photovoltaic power station, has high inspection efficiency, greatly reduces the number of personnel and inspection time required for photovoltaic power station inspection, saves labor operation and maintenance cost, and has higher economic benefit. The intelligent UAV has high mobility and can fly freely in the air without being limited by obstacles. Through image recognition technology, the UAV can accurately locate the position of the shelter to provide accurate information for subsequent cleaning and maintenance. UAV detection can not only improve the efficiency and safety of photovoltaic power stations, but also monitor and evaluate the environment to ensure the environmental protection and energy saving performance of photovoltaic power stations. UAV detection can reduce damage and pollution to the marine ecosystem, in line with the concept of sustainable development.

[0004] The use of multi-modal fusion technology can increase the diversity and coverage of data by integrating and fusing information from different sensors and different data sources. Fusion between different data sources and sensors can improve the accuracy and robustness of target detection and recognition. SUMMARY

[0005] To solve the above technical problems, the application provides a photovoltaic module surface shelter detection system based on a UAV to solve the problems existing in the prior art.

[0006] To achieve the above purpose, the application provides a photovoltaic module surface shelter detection system based on a UAV, which comprises:

[0007] a data acquisition module, a data processing module, a multi-scale feature enhancement module, and a multi-modal fusion detection module.

[0008] The data acquisition module is configured to acquire original image data and radar point cloud data of a surface of a photovoltaic module;

[0009] The data processing module is configured to obtain a radar feature map according to the image data and the radar point cloud data;

[0010] The multi-scale feature enhancement module is configured to obtain texture enhanced features and surface enhanced features according to the image data;

[0011] The multi-modal fusion detection module is configured to construct a multi-modal fusion detection model, input the radar feature map, the texture enhanced features and the surface enhanced features into the multi-modal fusion detection model, obtain a fusion feature map, and realize surface occlusion detection based on the fusion feature map.

[0012] Optionally, the data processing module comprises a radar point cloud data processing unit, an image processing unit and a mapping unit.

[0013] The radar point cloud data processing unit is configured to pre-process the radar point cloud data.

[0014] The image processing unit is configured to perform HSV segmentation and morphological closing operation on the original image data to obtain a photovoltaic module region.

[0015] The mapping unit is configured to align the radar point cloud data after displacement compensation and the image data in time and space, fill weighted RCS semantic information, map the radar point cloud data to the photovoltaic module region to obtain a radar pseudo-image and extract a radar feature map.

[0016] Optionally, the radar point cloud data processing unit calculates a photovoltaic array floating displacement vector according to tidal table data to perform displacement compensation on radar point cloud coordinates.

[0017] Optionally, the multi-scale feature enhancement module comprises a separation unit, a texture enhancement unit and a surface enhancement unit.

[0018] The separation unit is configured to perform polarization feature separation on the original image data to obtain an S-polarized light feature map and a P-polarized light feature map.

[0019] The texture enhancement unit is configured to perform grouped convolution algal texture extraction on the P-polarized light feature map to obtain texture enhanced features.

[0020] The surface enhancement unit is configured to perform reflection intensity weight enhancement and multi-scale weighted enhancement on the S-polarized light feature map, and perform weighted fusion on the two enhancement results to obtain surface enhanced features.

[0021] Optionally, the surface enhancement unit comprises a max pooling branch and a SimAM attention branch; the max pooling branch comprises a max pooling layer and a convolution layer, and the SimAM attention branch comprises a SimAM attention mechanism and a convolution layer; the outputs of the two branches are spliced to obtain surface enhancement features.

[0022] Optionally, the multi-modal fusion detection module comprises a multi-level feature extraction and fusion unit, a processing unit, a multi-modal feature fusion unit, and a detection unit.

[0023] The multi-level feature fusion unit comprises a VGG16 backbone network and a feature enhancement submodule. After extracting multi-level feature maps through the VGG16 backbone network, the feature enhancement submodule is used for feature enhancement to obtain multi-level enhanced features. The multi-level enhanced features comprise VGG Block3 features and VGG Block5 features. The feature enhancement submodule has the same structure as the surface enhancement unit.

[0024] The processing unit is configured to unify the scales of the multi-level enhanced features, the radar feature maps, the texture enhanced features, and the surface enhanced features through sampling.

[0025] The multi-modal feature fusion unit is configured to fuse the features of the same scale to obtain a fused feature map.

[0026] The detection unit is configured to perform surface occlusion detection based on the fused feature map.

[0027] Optionally, the multi-modal feature fusion unit comprises a shallow fusion unit, a deep fusion unit, and a photovoltaic optimization FPN unit.

[0028] The shallow fusion unit is configured to fuse the VGG Block3 features, the surface enhanced features, and the radar feature maps.

[0029] The deep fusion unit is configured to fuse the VGG Block5 features, the texture enhanced features, the radar feature maps, and infrared features, wherein the infrared features are obtained based on an infrared thermal spot image.

[0030] The photovoltaic optimization FPN unit is configured to generate a two-level feature pyramid according to the outputs of the shallow fusion unit and the deep fusion unit.

[0031] Optionally, the system further comprises an evaluation module, which comprises:

[0032] A parameter setting subunit configured to perform parameter setting on the multi-modal fusion target detection model.

[0033] A model training subunit configured to input data into the fusion model for training.

[0034] The classification regression subunit is used for inputting the fused feature map into an evaluation prediction model, and outputting evaluation index results through classification and regression.

[0035] Compared with the prior art, the application has the following advantages and technical effects:

[0036] The application innovatively combines the parameter-free attention mechanism (SimAM) with the maximum pooling layer, weights and enhances the feature map in a parameter-free manner, avoids additional parameter learning, reduces the computational overhead, and can adaptively adjust the attention area, thereby improving the efficiency and flexibility of the model. Through the dual feature enhancement of maximum pooling and SimAM, key information is extracted from local and global information respectively, effectively improving the model's ability to capture important features and enhancing its adaptability to complex tasks. The feature maps of the two branches are fused by channel splicing, which not only retains the effective capture of local features by the maximum pooling, but also fuses the weighting of global and local information by SimAM, improving the expression ability of the feature map. This design has great application value in photovoltaic component surface occlusion detection based on unmanned aerial vehicles, and can cope with complex environmental changes and lighting conditions, improving the performance of the unmanned aerial vehicle vision system in actual scenarios.

[0037] The application significantly improves the detection capability of photovoltaic component surface occlusions through the innovative multi-modal data fusion and feature enhancement method. The system uses polarization light characteristics separation and tidal compensation technology to effectively solve the problem of insufficient distinction between surface reflection and micro-texture in traditional detection methods, greatly improving the recognition accuracy of typical occlusions such as bird droppings and algae. The hierarchical attention fusion strategy and special feature pyramid design optimize the computational efficiency while ensuring detection accuracy. The unique displacement compensation mechanism ensures the precise alignment of multi-source data, and the specially designed evaluation index provides a scientific basis for photovoltaic power station operation and maintenance. BRIEF DESCRIPTION OF DRAWINGS

[0038] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application, and their description, are used to explain the application and are not intended to limit the application unduly.

[0039] Fig. 1 The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application, and their description, are used to explain the application and are not intended to limit the application unduly.

[0040] Fig. 2 The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application, and their description, are used to explain the application and are not intended to limit the application unduly.

[0041] Fig. 3 The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application, and their description, are used to explain the application and are not intended to limit the application unduly. DETAILED DESCRIPTION

[0042] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other in the case of no conflict. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0043] It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in a different order.

[0044] Embodiment one

[0045] As Figs. 1-3 shown, the present embodiment provides a photovoltaic module surface shelter detection system based on unmanned aerial vehicle, comprising:

[0046] a data acquisition module, a data processing module, a multi-scale feature enhancement module, and a multi-modal fusion detection module;

[0047] The data acquisition module is used to acquire original image data and radar point cloud data of the surface of the photovoltaic module.

[0048] The data processing module is used to obtain a radar feature map according to the image data and the radar point cloud data.

[0049] As a specific implementation, the data processing module includes a radar point cloud data processing unit, an image processing unit, and a mapping unit.

[0050] The radar point cloud data processing unit is used to pre-process the radar point cloud data. Specifically, the tidal table data is used to calculate the floating displacement vector of the photovoltaic array to compensate the displacement of the radar point cloud coordinates.

[0051] The image processing unit is used to perform HSV segmentation and morphological closing operation on the original image data to obtain a photovoltaic module region.

[0052] The mapping unit is used to align the radar point cloud data after displacement compensation with the image data in time and space, fill the weighted RCS semantic information, and then map the radar point cloud data to the photovoltaic module region to obtain a radar pseudo-image and extract a radar feature map.

[0053] Specifically, the radar point cloud mapping and feature information extraction module is first jointly labeled with the image data, time and space alignment is performed, the radar point cloud is mapped to the image to obtain a radar pseudo-image, HSV color space is used for preliminary segmentation to segment the image region corresponding to the photovoltaic module from the original image. Morphological closing operation is used to eliminate noise and gaps. Feature information with distinguishability and representativeness is extracted from the obtained point cloud data through filtering processing, the purpose is to make the point cloud data more easily fused and processed for object detection and recognition.

[0054] Firstly, the image is corrected according to the camera external and internal parameters, and then the radar data is time and space aligned with the image data. The radar data includes azimuth, distance and radar cross section (RCS) and other semantic feature information, and the present radar data is converted from a two-dimensional ground plane to an imaging plane with a vertical line. According to the baseline network, an uncertain weighted RCS channel is added. The uncertain weighted RCS channel is obtained by adding an uncertain azimuth direction channel, calculating the density value of the channel, and multiplying the RCS channel value to form an uncertain weighted RCS channel. A total of 4 channels are input to the radar branch.

[0055] According to the radar point cloud mapping principle, the characteristics of the radar echo are stored as pixel values in the enhanced image. The radar channel value projected at the image pixel position without radar reflection intensity is set to 0; other positions are set to the corresponding pixel point value, that is, the radar channel is mapped to the corresponding position, and displayed in a unified color.

[0056] Specifically, due to the sparsity problem of radar data, the density of radar data is increased by fusing the past 13 radar periods (about 1s) into the radar data format. The self-motion is compensated by using this projection method, and finally the radar feature image is formed as the input.

[0057] In order to extract more accurate radar point cloud feature information and reduce the influence of interference noise information, filtering processing is needed. Since there are many detection results unrelated to the detected target in the radar echo signal, the radar feature image needs to be filtered by an annotation filter (AF) to achieve the final relatively more accurate radar feature information.

[0058] The multi-scale feature enhancement module is used to obtain texture enhanced features and surface enhanced features from the image data.

[0059] As a specific implementation, the multi-scale feature enhancement module includes a separation unit, a texture enhancement unit, and a surface enhancement unit.

[0060] The separation unit is used to separate the polarized light features of the original image data to obtain S-polarized light feature maps and P-polarized light feature maps.

[0061] The texture enhancement unit is used to perform grouped convolution algae texture extraction on the P-polarized light feature maps to obtain texture enhanced features.

[0062] The surface enhancement unit is used to perform reflection intensity weight enhancement and multi-scale weight enhancement on the S-polarized light feature maps, and the two enhancement results are weighted and fused to obtain surface enhanced features.

[0063] As a specific implementation, the surface enhancement unit includes a max pooling branch and a SimAM attention branch; the max pooling branch includes a max pooling layer and a convolution layer, and the SimAM attention branch includes a SimAM attention mechanism and a convolution layer; the outputs of the two branches are spliced to obtain surface enhancement features.

[0064] For example, the SimAM attention mechanism is a parameter-free 3D attention module, as shown in Fig. 2 Unlike existing channel attention and spatial attention, it can simultaneously pay attention to the importance of channel and spatial features, and infer the three-dimensional weights of the feature map without increasing the network parameters. For the specific structure, please refer to the attached Fig. 3 .

[0065] First, the input feature X of the image R is C×H×W The max pooling layer of the first branch down-samples the height and width of the feature map to 2 times the original, and then a 1x1 convolution is performed to change the number of channels, obtaining the first branch feature map, in order to align the input and output channels of the module with the original network channel number, r=1 is taken, then for the second branch, the input feature X is C×H×W First, the SimAM attention mechanism is used for feature enhancement to obtain the enhanced feature map Then, a convolution layer with a 3x3 kernel size and a 2-step stride is used to down-sample the height and width by 2 times, obtaining the feature map of the second branch Finally, the channel of the feature map of the two branches is spliced to obtain the final output result:

[0066] The multi-modal fusion detection module is used to construct a multi-modal fusion detection model, and the radar feature map, texture enhanced feature and surface enhanced feature are input into the multi-modal fusion detection model to obtain a fusion feature map, and the surface occlusion detection is realized based on the fusion feature map.

[0067] As a specific implementation, the multi-modal fusion detection module includes a multi-level feature extraction and fusion unit, a processing unit, a multi-modal feature fusion unit, and a detection unit.

[0068] The multi-level feature fusion unit includes a VGG16 backbone network and a feature enhancement sub-module. After extracting multi-level feature maps through the VGG16 backbone network, the feature enhancement sub-module is used for feature enhancement to obtain multi-level enhanced features; wherein the multi-level enhanced features include VGG Block3 features and VGG Block5 features; the feature enhancement sub-module has the same structure as the surface enhancement unit.

[0069] The processing unit is used to unify the scale of the multi-level enhanced features, the radar feature map, the texture enhanced feature and the surface enhanced feature through sampling.

[0070] The multi-modal feature fusion unit is used for fusing the features after the same scale to obtain a fused feature map;

[0071] The detection unit detects the surface occlusion based on the fused feature map.

[0072] As a specific implementation, the multi-modal feature fusion unit comprises a shallow fusion unit, a deep fusion unit, and a photovoltaic optimization FPN unit.

[0073] The shallow fusion unit is used for fusing VGG Block3 features, surface enhancement features, and radar feature maps.

[0074] The deep fusion unit is used for fusing VGG Block5 features, texture enhancement features, radar feature maps, and infrared features, wherein the infrared features are obtained based on an infrared thermal spot image.

[0075] The photovoltaic optimization FPN unit is used for generating a two-level feature pyramid according to the outputs of the shallow fusion unit and the deep fusion unit.

[0076] Specifically, the evaluation module comprises:

[0077] The parameter setting subunit is used for setting parameters of the multi-modal fusion target detection model.

[0078] The model training subunit is used for inputting data into the fusion model for training.

[0079] The classification regression subunit is used for inputting the output fused feature map into an evaluation prediction model to output evaluation index results through classification regression.

[0080] Specifically, the multi-modal fusion target detection model is set with parameters, the output fused feature map is input into the evaluation prediction model, and each detected occlusion category is output through classification regression, mainly including flying bird shadow, bird droppings, moss algae, etc., and the selected evaluation index results are output, specifically including accuracy, average precision, mAP, recall rate, mean square error, and average absolute error data.

[0081] The main detection of the embodiment is the shelter of flying bird shadow, bird droppings, moss and algae, etc. Including steps: linearly mapping radar point cloud to image for joint labeling, based on baseline fusion network radar channel and weighted RCS channel, radar information can be more fully utilized; multi-level feature extraction is performed on the radar image and the original image; a fusion model based on VGG16 and feature pyramid backbone network is used to splice and fuse the features of different modalities; a general, simple and effective parameter-free hybrid attention mechanism, MP-SimAM module, is used. The meaningful features along the channel and spatial dimensions are emphasized to guide the feature extraction network to focus more accurately on the target object, so that higher accuracy detection results can be achieved for the target, and the robustness of the model is improved.

[0082] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A photovoltaic module surface obstruction detection system based on drone, characterized in that: include: Data acquisition module, data processing module, multi-scale feature enhancement module, and multimodal fusion detection module; The data acquisition module is used to collect original image data and radar point cloud data of the photovoltaic module surface; The data processing module is used to obtain a radar feature map based on the image data and the radar point cloud data; The multi-scale feature enhancement module is used to obtain texture enhancement features and surface enhancement features according to the image data; The multimodal fusion detection module is used to construct a multimodal fusion detection model, input the radar feature map, texture enhancement feature and surface enhancement feature into the multimodal fusion detection model, obtain a fusion feature map, and realize surface occlusion detection based on the fusion feature map.

2. The photovoltaic module surface obstruction detection system based on drone according to claim 1 is characterized in that: The data processing module includes a radar point cloud data processing unit, an image processing unit, and a mapping unit; The radar point cloud data processing unit is used to pre-process the radar point cloud data; The image processing unit is used to perform HSV segmentation and morphological closing operations on the original image data to obtain the photovoltaic module area; The mapping unit is used to align the displacement-compensated radar point cloud data with the image data in time and space, and map the radar point cloud data to the photovoltaic module area after filling in weighted RCS semantic information to obtain a radar pseudo image and extract a radar feature map.

3. The photovoltaic module surface obstruction detection system based on drone according to claim 2, characterized in that: The radar point cloud data processing unit calculates the photovoltaic array floating displacement vector according to the tide table data and performs displacement compensation on the radar point cloud coordinates.

4. The photovoltaic module surface obstruction detection system based on drone according to claim 1, characterized in that: The multi-scale feature enhancement module includes a separation unit, a texture enhancement unit, and a surface enhancement unit; The separation unit is used to perform polarization feature separation on the original image data to obtain an S-polarization feature map and a P-polarization feature map; The texture enhancement unit is used to perform group convolution on the P-polarized light feature map to extract algae textures and obtain texture enhancement features; The surface enhancement unit is used to perform reflection intensity weight enhancement and multi-scale weighted enhancement on the S-polarized light feature map, and the two enhancement results are weightedly fused to obtain surface enhancement features.

5. The photovoltaic module surface obstruction detection system based on drone according to claim 4 is characterized in that: The surface enhancement unit includes a maximum pooling branch and a SimAM attention branch; The maximum pooling branch includes a maximum pooling layer and a convolution layer, and the SimAM attention branch includes a SimAM attention mechanism and a convolution layer; the outputs of the two branches are spliced ​​to obtain surface enhancement features.

6. The photovoltaic module surface obstruction detection system based on drone according to claim 5, characterized in that: The multimodal fusion detection module includes a multi-level feature extraction and fusion unit, a processing unit, a multimodal feature fusion unit, and a detection unit; The multi-level feature fusion unit includes a VGG16 backbone network and a feature enhancement submodule. After extracting a multi-level feature map through the VGG16 backbone network, the feature enhancement submodule performs feature enhancement to obtain multi-level enhanced features; wherein the multi-level enhanced features include VGG Block3 features and VGG Block5 features; the feature enhancement submodule has the same structure as the surface enhancement unit; The processing unit is used to unify the scale of the multi-level enhancement features, radar feature map, texture enhancement features and surface enhancement features through sampling; The multimodal feature fusion unit is used to fuse features at the same scale to obtain a fused feature map; The detection unit performs surface occlusion detection based on the fused feature map.

7. The photovoltaic module surface obstruction detection system based on drone according to claim 6, characterized in that: The multimodal feature fusion unit includes: shallow fusion unit, deep fusion unit, and photovoltaic optimized FPN unit; The shallow fusion unit is used to fuse VGG Block3 features, surface enhancement features and radar feature maps; The deep fusion unit is used to fuse VGG Block5 features, texture enhancement features, radar feature maps and infrared features, wherein the infrared features are obtained based on infrared hot spot images; The photovoltaic optimized FPN unit is used to generate a two-level feature pyramid according to the outputs of the shallow fusion unit and the deep fusion unit.

8. The photovoltaic module surface obstruction detection system based on drone according to claim 1, characterized in that: Also included is an evaluation module, the evaluation module comprising: A parameter setting subunit, used to set parameters of the multimodal fusion target detection model; The model training subunit is used to input data into the fusion model for training; The classification and regression subunit is used to input the output fused feature map into the evaluation prediction model and output the evaluation index results through classification and regression.

Images