Warehouse roof solar panel defect detection method based on artificial intelligence

Abstract

The application discloses a warehouse roof solar panel defect detection method based on artificial intelligence, and belongs to the technical field of artificial intelligence and image processing. The defect detection method comprises the following steps: acquiring a collected solar panel image, acquiring a trained image reconstruction network; inputting the solar panel image into the image reconstruction network, and generating and outputting an abstract feature map after operation of a feature extraction module; inputting the abstract feature map into a reconstruction module, and then reconstructing and outputting a remade image; acquiring a trained defect detection network, and detecting possible defects on the solar panel by using the defect detection network, and the like. The application reconstructs the collected solar panel image by using the image reconstruction network, increases the resolution of the image, enriches the image features of the defects with unobvious visual performance in the original image, reduces the probability of missed detection and wrong detection, and does not need to upgrade the existing hardware devices, so that the use and operation costs are low.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of artificial intelligence and image processing, and in particular relates to a warehouse roof solar panel defect detection method based on artificial intelligence. BACKGROUND

[0002] In order to achieve the goal of carbon peak and carbon neutral, various industries in China are adjusting the energy structure. Due to the large land area of the warehouse, in recent years, the traditional roof is replaced by solar panels, and the electric energy generated by the solar panels can also save part of the energy expenditure for the enterprise. After the solar panels run for a period of time, there will be a certain probability of hot spot, crack or snail pattern defects, in order to ensure the normal operation of the solar panel power generation system, and at the same time avoid the influence of these defects on the warehouse itself, it is necessary to regularly inspect and maintain the solar panels. Due to the special position of the solar panels installed on the roof of the warehouse, a camera is usually carried on an unmanned aerial vehicle to take images of the solar panels, and artificial intelligence technology is used to automatically identify and locate defects. However, from the actual application effect, some smaller defects are easily ignored or misjudged as normal, and there is a large probability of missed detection and misdiagnosis. SUMMARY

[0003] In view of the above shortcomings in the prior art, the present application provides a warehouse roof solar panel defect detection method based on artificial intelligence, which reconstructs the collected solar panel images to improve the resolution of the images and thereby reduce the probability of missed detection and misdiagnosis in subsequent defect detection.

[0004] In order to achieve the above purpose, the present application adopts the following solutions:

[0005] A warehouse roof solar panel defect detection method based on artificial intelligence, comprising the following steps:

[0006] S100, acquiring the solar panel image collected on the roof of the warehouse, and acquiring the trained image reconstruction network; wherein the solar panel image herein refers to the image of the top panel part of the solar panel, and the image reconstruction network is provided with a feature extraction module and a reconstruction module;

[0007] S200, inputting the solar panel image into the image reconstruction network, the feature extraction module generates and outputs an abstract feature map after operation, and the abstract feature map contains the feature information of the solar panel image;

[0008] S300, the reconstruction module takes the abstract feature map as input, then reconstructs and outputs a reconstructed image, and the resolution of the reconstructed image is greater than that of the collected solar panel image;

[0009] S400: Obtain the trained defect detection network, input the reconstructed image into the defect detection network, and use the defect detection network to detect possible defects on the solar panel;

[0010] The feature extraction module includes a primary feature extraction mechanism, an amplification mechanism, a secondary feature extraction mechanism, and a reduction mechanism. The primary feature extraction mechanism performs feature extraction on the feature map input to the feature extraction module. The amplification mechanism increases the width and height of the feature map output by the primary feature extraction mechanism. The secondary feature extraction mechanism performs feature extraction on the feature map output by the amplification mechanism. The reduction mechanism reduces the width and height of the feature map output by the secondary feature extraction mechanism. The size of the feature map output by the reduction mechanism is equal to the size of the feature map output by the primary feature extraction mechanism.

[0011] Furthermore, the primary feature extraction mechanism includes multiple parallel primary convolutional blocks, each comprising a primary convolutional layer and a primary activation layer arranged sequentially, with each primary convolutional layer in the primary feature extraction mechanism having a different kernel size.

[0012] Furthermore, the amplification mechanism includes a first stitching layer, a first convolutional layer, a first activation layer, a first subpixel convolutional layer, and a second activation layer arranged sequentially. The first stitching layer is used to stitch together the feature maps output by each of the first-level convolutional blocks.

[0013] Furthermore, the secondary feature extraction mechanism includes multiple parallel secondary convolutional blocks, each comprising a secondary convolutional layer and a secondary activation layer arranged sequentially, with each secondary convolutional layer in the secondary feature extraction mechanism having a different kernel size.

[0014] Furthermore, the shrinking mechanism includes multiple strided convolutional layers, each strided convolutional layer corresponding to one of the secondary convolutional blocks.

[0015] Furthermore, the feature extraction module also includes an end-integration block, which is used to fuse the feature maps output by each of the straddle convolutional layers.

[0016] Furthermore, the number of the second-level convolutional blocks is equal to the number of the first-level convolutional blocks, and the second-level convolutional blocks are configured in a one-to-one correspondence with the first-level convolutional blocks; the feature extraction module also includes multiple size transformation attention mechanisms, which are configured in a one-to-one correspondence with the strided convolutional layers; the size transformation attention mechanism takes the feature map output by the magnification mechanism and the feature map output by the corresponding first-level convolutional block as input, and then generates a local attention map; the local attention map is used to modulate the feature map output by the corresponding strided convolutional layer.

[0017] Furthermore, the calculation process within the size transformation attention mechanism is represented by the following mathematical model:

[0018] Wherein, DT represents the feature map output from the first-level convolutional block and input into the size transformation attention mechanism, BT represents the feature map output from the magnification mechanism and input into the size transformation attention mechanism; gvp represents global variance pooling of the feature map in the channel direction, gap represents global max pooling of the feature map in the channel direction, gep represents global average pooling of the feature map in the channel direction, fp represents average pooling with a pooling window size of 2*2 and a stride of 2, × represents element-wise multiplication, [] represents concatenation of the feature maps, fcv represents convolution with a kernel size of 1*1 and a stride of 1, and sig represents sigmoid. The activation function is defined as follows: L1 represents the feature map generated after performing global variance pooling on the DT feature map in the channel direction; L2 represents the result obtained by subtracting the feature map generated by performing global max pooling on DT and the feature map generated by performing global average pooling on DT; L3 represents the feature map generated after performing pooling on the feature map BT; L4 represents the result obtained by adding the feature map generated by performing global variance pooling on L3 and L1; the two feature maps obtained by performing global max pooling and global average pooling on L3 are subtracted respectively, and then the result of the subtraction is multiplied element-wise with L2 to generate feature map L5; LAT represents the local attention map generated by the size transformation attention mechanism.

[0019] Furthermore, the image reconstruction network also includes a comprehensive attention module, which takes all the local attention maps as input, and then generates and outputs a comprehensive attention map, which is used to modulate the reconstructed image.

[0020] Furthermore, the integrated attention module includes a second splicing layer, an integrated convolutional layer, a first integrated activation layer, a second subpixel convolutional layer, and a second integrated activation layer arranged sequentially.

[0021] The beneficial effects of this invention are:

[0022] This invention utilizes an image reconstruction network to reconstruct the acquired solar panel images, increasing the image resolution, enriching the image features of some visually inconspicuous defects in the original image, reducing the probability of missed detections and false detections, without requiring upgrades to existing hardware equipment, and with low usage and operating costs.

[0023] The feature extraction module is internally set with a primary feature extraction mechanism, a magnification mechanism, a secondary feature extraction mechanism, and a reduction mechanism. After the image reconstruction network is trained, the feature extraction module can learn the mapping relationship between the width and height of the feature map before and after the change in the field of view at different sizes. Compared with the existing technology (which basically only performs a magnification operation on the feature map once), multiple scaling operations enable the network to learn a more stable and comprehensive high and low resolution image feature mapping relationship, thereby improving the quality of the reconstructed image.

[0024] The size transformation attention mechanism takes the feature map output by the magnification mechanism and the feature map output by the corresponding first-level convolutional block as input. The generated local attention map can selectively strengthen the parts that affect the image quality after the resolution change from many features, reduce the distortion caused by the interaction of different features when reconstructing the image, improve the reconstruction effect of the network in the details, and enhance the robustness of the network.

[0025] This invention not only utilizes the modulation information learned by the size transformation attention mechanism in the feature extraction stage, but also sets up a comprehensive attention module to fuse the modulation information at various levels for use in the image reconstruction stage. Tests show that the comprehensive attention module can fine-tune the image output by the reconstruction module, which can further improve the image reconstruction quality.

[0026] Inside the size transformation attention mechanism, the difference between the two matrices generated by global max pooling and global average pooling of the L1 feature map is first calculated, and the difference between the two matrices generated by global max pooling and global average pooling of the L3 feature map is also calculated. The results of the two differences are then fused by element-wise multiplication. This can eliminate some feature information in the input size transformation attention mechanism that does not change with resolution, allowing the attention mechanism to eliminate interference and better focus on learning feature information that affects the size transformation process. In conjunction with the global variance pooling operation, it can achieve efficient learning and effective modulation of the feature map size transformation process. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the internal computational operation process of the image reconstruction network in Example 1;

[0028] Figure 2 This is a schematic diagram of the internal operation process of the feature extraction module in Example 1;

[0029] Figure 3 This is a schematic diagram of the internal calculation process of the size transformation attention mechanism in Example 1;

[0030] Figure 4 This is a schematic diagram of the internal operation process of the reconstruction module in Example 1;

[0031] Figure 5This is a schematic diagram of the internal computational operation process of the integrated attention module in Example 1;

[0032] Figure 6 This is a schematic diagram illustrating the internal computational processes of the image reconstruction network in the comparison example;

[0033] Attached Figure

[0034] Solar panel image, 2-Feature extraction module, 21-First-level convolutional block, 22-Magnification mechanism, 23-Second-level convolutional block, 24-Stepping convolutional layer, 25-End integration block, 26-Size transformation attention mechanism, 3-Reconstruction module, 4-Integration attention module, 5-High-resolution image, 6-Pre-convolutional layer. Detailed Implementation

[0035] The present invention will be further described below with reference to the accompanying drawings: Example

[0036] Figure 1 The diagram visually illustrates the data processing of the image reconstruction network in this embodiment. A front-end convolutional layer 6 with a kernel size of 3*3 and a stride of 1 is set at the front end. Assuming that the size of the solar panel image 1 input to the image reconstruction network is K*P*3 (width*height*number of channels), the size of the feature map output by the front convolutional layer 6 is K*P*64.

[0037] Six feature extraction modules 2 are sequentially arranged in the middle of the image reconstruction network. The feature map output by the last feature extraction module 2 is used as the abstract feature map input to the reconstruction module 3. Figure 2 As shown, within each feature extraction module 2, the primary feature extraction mechanism includes two parallel primary convolutional blocks 21. In one primary convolutional block 21, the primary convolutional layer kernel size is 3*3 with a stride of 1, and the primary activation layer uses the ReLU function. In the other primary convolutional block 21, the primary convolutional layer kernel size is 5*5 with a stride of 1, and the primary activation layer uses the ReLU function. The output feature map size of both primary convolutional blocks 21 is K*P*64.

[0038] The magnification mechanism 22 includes a first stitching layer, a first convolutional layer (with a kernel size of 3*3 and a stride of 1), a first activation layer (ReLU function), a first subpixel convolutional layer, and a second activation layer (ReLU function) arranged sequentially. The first stitching layer is used to stitch together the feature maps output by the two first-level convolutional blocks 21. The output feature map size of the first convolutional layer is K*P*64*4, and the output feature map size after the operation of the first subpixel convolutional layer is 2K*2P*64 (the width and height of the feature map are doubled).

[0039] The secondary feature extraction mechanism comprises two parallel secondary convolutional blocks 23. In one secondary convolutional block 23, the kernel size of the secondary convolutional layer is 3*3 with a stride of 1, and the secondary activation layer uses the ReLU function. In the other secondary convolutional block 23, the kernel size of the secondary convolutional layer is 5*5 with a stride of 1, and the secondary activation layer uses the ReLU function. The output feature map size of both secondary convolutional blocks 23 is 2K*2P*64.

[0040] The reduction mechanism includes two strided convolutional layers 24, which are connected downstream of two secondary convolutional blocks 23. Both strided convolutional layers 24 have a kernel size of 3*3 and a stride of 2, outputting feature maps of size K*P*64. Each feature extraction module 2 includes an end-merging block 25, which fuses the feature maps output by each strided convolutional layer 24. The feature map output by the end-merging block 25 is used as the output of the feature extraction module 2. In this embodiment, the end-merging block 25 includes a merging and splicing operation layer, a merging convolutional layer (stride 1, kernel size 1*1), and a merging activation layer (ReLU function) arranged sequentially. The merging and splicing operation layer concatenates the feature maps output by each strided convolutional layer 24, and the merging activation layer outputs a feature map of size K*P*64.

[0041] Each feature extraction module 2 contains two size-shifting attention mechanisms 26, and the internal operations of the two size-shifting attention mechanisms 26 are identical. One size-shifting attention mechanism 26 takes the feature map output by the magnification mechanism 22 and the (activated) feature map output by a first-level convolutional layer with a kernel size of 3*3 as input, generates a local attention map, and performs an element-wise multiplication operation with the feature map output by one of the strided convolutional layers 24 (located downstream of a second-level convolutional layer with a kernel size of 3*3) to modulate the feature map. The other size-shifting attention mechanism 26 takes the feature map output by the magnification mechanism 22 and the (activated) feature map output by a first-level convolutional layer with a kernel size of 5*5 as input, generates a local attention map, and performs an element-wise multiplication operation with the feature map output by another strided convolutional layer 24 (located downstream of a second-level convolutional layer with a kernel size of 5*5) to modulate the feature map.

[0042] Figure 3The computational operations within the size-shifting attention mechanism 26 are visualized. DT represents the feature map output from the first-level convolutional block 21 and input into the size-shifting attention mechanism 26, with a size of K*P*64. BT represents the feature map output from the amplification mechanism 22 and input into the size-shifting attention mechanism 26, with a size of 2K*2P*64. After performing global variance pooling, global average pooling, or global max pooling on the feature map along the channel direction, the number of channels in the feature map becomes 1, while the width and height remain unchanged. fp represents an average pooling operation with a pooling window size of 2*2 and a stride of 2, where the pooling window slides along the feature map space direction. Therefore, the size of L1, L2, L4, L5, and the local attention map LAT is all K*P*1, and the size of L3 is K*P*64.

[0043] like Figure 4 As shown, reconstruction module 3 includes a first reconstruction convolutional layer (with a kernel size of 3*3 and a stride of 1), a first reconstruction activation function (ReLU function), a reconstruction sub-pixel convolutional layer, a second reconstruction activation function (ReLU function), a second reconstruction convolutional layer (with a kernel size of 3*3 and a stride of 1), and a third reconstruction activation function (ReLU function) arranged sequentially. The feature map size output by the first reconstruction convolutional layer is K*P*64X. 2 (X represents the factor by which the image reconstruction network enlarges the image resolution, and X is a positive integer greater than 1). The output feature map size of the reconstruction subpixel convolutional layer is XK*XP*64, and the third reconstruction activation function generates a reconstructed image with a size of XK*XP*3.

[0044] like Figure 5 As shown, the integrated attention module 4 includes a second concatenation layer, a comprehensive convolutional layer (with a kernel size of 3*3 and a stride of 1), a first comprehensive activation layer (ReLU function), a second sub-pixel convolutional layer, and a second comprehensive activation layer (sigmoid function), arranged sequentially. For the integrated attention module 4, the 12 local attention maps generated by the 12 size transformation attention mechanisms 26 serve as the input to the integrated attention module 4. The kernel size of the comprehensive convolutional layer is 3*3 with a stride of 1, and the output feature map size of the comprehensive convolutional layer is K*P*X. 2 The second subpixel convolutional layer outputs a feature map of size XK*XP*1. After activation by the second comprehensive activation layer, a comprehensive attention map of size XK*XP*1 is obtained. The comprehensive attention map is element-wise multiplied with the reconstructed image to fine-tune different spatial locations of the reconstructed image, generating a high-resolution image 5, which serves as the input to the defect detection network.

[0045] The image reconstruction network of this embodiment and some existing image reconstruction models are trained and tested using a self-built training set and a self-built test set.

[0046] The image reconstructed by the SRMDNF model shows significantly lower peak signal-to-noise ratio (PSNR) and structural similarity than the image reconstructed by the image reconstruction network in Example 1, fully demonstrating that the image reconstruction network proposed in this invention has excellent reconstruction performance for low-resolution images. Comparative Example:

[0047] This comparative example is based on Example 1, except that the comprehensive attention module 4 in the image reconstruction network is removed, while the rest of the network remains unchanged (e.g., ...). Figure 6 (as shown), and the modified network was trained and tested in the same way. Example

[0048] Without the integrated attention module 4, the quality of the reconstructed images by the image reconstruction network is significantly reduced. Comparative experiments demonstrate that the integrated attention module 4 plays an important role in improving network performance.

[0049] The image reconstruction network trained in Example 1 is used to reconstruct the self-built validation set images, and the output is named Reconstructed Validation Set 1. The SRMDNF trained in Example 1 is used to reconstruct the self-built validation set images, and the output is named Reconstructed Validation Set 2. The image reconstruction network trained in the comparative example (with the integrated attention module 4 removed) is used to reconstruct the validation set images, and the output is named Reconstructed Validation Set 3. Then, the trained YOLO V4 network is used to perform object detection tests on the original self-built validation set (without image reconstruction), Reconstructed Validation Set 1, Reconstructed Validation Set 2, and Reconstructed Validation Set 3, respectively.

[0050] The YOLO V4 network achieved the best detection performance on the reconstruction validation set 1, further demonstrating the effectiveness and advancement of the warehouse roof solar panel defect detection method provided by this invention. Furthermore, the YOLO V4 network's detection performance on the reconstruction validation set 3 was significantly lower than its performance on the reconstruction validation set 1, which also illustrates from another perspective that setting the integrated attention module 4 can effectively improve the reliability of defect detection results.

[0051] The embodiments described above are merely illustrative of specific implementations of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. An artificial intelligence-based warehouse roof solar panel defect detection method, characterized by: The method comprises the following steps: S100, acquiring the collected solar panel image on the warehouse roof, and acquiring the trained image reconstruction network; the image reconstruction network is provided with a feature extraction module and a reconstruction module; S200, inputting the solar panel image into the image reconstruction network, the feature extraction module generates and outputs an abstract feature map after operation, and the abstract feature map contains the feature information of the solar panel image; S300, the reconstruction module takes the abstract feature map as input, then reconstructs and outputs a remade image, and the resolution of the remade image is greater than that of the collected solar panel image; S400, acquiring the trained defect detection network, inputting the remade image into the defect detection network, and detecting the defects on the solar panel by using the defect detection network; The feature extraction module is sequentially provided with a first feature extraction mechanism, an enlargement mechanism, a second feature extraction mechanism and a reduction mechanism, the first feature extraction mechanism is used for feature extraction operation on the feature map input into the feature extraction module, the enlargement mechanism is used for increasing the width and height size of the feature map output by the first feature extraction mechanism, the second feature extraction mechanism is used for feature extraction operation on the feature map output by the enlargement mechanism, and the reduction mechanism is used for reducing the width and height size of the feature map output by the second feature extraction mechanism, the size of the feature map output by the reduction mechanism is equal to that of the feature map output by the first feature extraction mechanism; The first feature extraction mechanism is provided with a plurality of first convolution blocks in parallel, the second feature extraction mechanism is provided with a plurality of second convolution blocks in parallel, the number of the second convolution blocks is equal to that of the first convolution blocks, and the second convolution blocks are arranged one by one corresponding to the first convolution blocks; The reduction mechanism comprises a plurality of stride convolution layers, and the stride convolution layers are arranged one by one corresponding to the second convolution blocks; The feature extraction module is further provided with a plurality of size transformation attention mechanisms, the size transformation attention mechanisms are arranged one by one corresponding to the stride convolution layers, the size transformation attention mechanisms take the feature map output by the enlargement mechanism and the feature map output by the corresponding first convolution block as input, and then generate a local attention map; the local attention map is used for modulating the feature map output by the corresponding stride convolution layer.

2. The warehouse rooftop solar panel defect detection method based on artificial intelligence according to claim 1, characterized in that: The first convolution block comprises a first convolution layer and a first activation layer arranged in sequence, and the convolution kernel sizes of the first convolution layers in the first feature extraction mechanism are different.

3. The warehouse rooftop solar panel defect detection method based on artificial intelligence according to claim 2, characterized in that: The enlargement mechanism comprises a first splicing layer, a first convolution layer, a first activation layer, a first sub-pixel convolution layer and a second activation layer arranged in sequence, and the first splicing layer is used for splicing the feature maps output by the first convolution blocks.

4. The warehouse rooftop solar panel defect detection method based on artificial intelligence according to claim 3, characterized in that: The second convolution block comprises a second convolution layer and a second activation layer arranged in sequence, and the convolution kernel sizes of the second convolution layers in the second feature extraction mechanism are different.

5. The warehouse rooftop solar panel defect detection method based on artificial intelligence according to claim 4, characterized in that: The feature extraction module is further provided with a terminal integration block, and the terminal integration block is used for fusing the feature maps output by the stride convolution layers.

6. The warehouse rooftop solar panel defect detection method based on artificial intelligence according to claim 5, characterized in that: The calculation process inside the size transformation attention mechanism is represented as a mathematical model as follows: ； ； ； ； ； ； wherein, DT represents the feature map output from the first convolutional block and input into the size transformation attention mechanism, BT represents the feature map output from the upsampling mechanism and input into the size transformation attention mechanism; gvp represents the global variance pooling operation on the channel direction of the feature map, gap represents the global max pooling operation on the channel direction of the feature map, gep represents the global average pooling operation on the channel direction of the feature map, and fp represents the pooling window size of 2 2, the average pooling operation with a step size of 2, x represents the element corresponding multiplication operation, [] represents the splicing operation on the feature map, and fcv represents the convolution kernel size of 1 1, the convolution operation with a step size of 1, sig represents the sigmoid activation function; L1 represents the feature map generated after the global variance pooling operation on the channel direction of the DT feature map; L2 represents the result obtained after the difference between the feature map generated by the global max pooling operation on the DT and the feature map generated by the global average pooling operation on the DT; L3 represents the feature map generated after the pooling operation on the feature map BT; L4 represents the result obtained after the addition of the feature map generated by the global variance pooling operation on L3 and L1; the difference between the two feature maps obtained after the global max pooling operation and the global average pooling operation on L3, respectively, and the element corresponding multiplication between the result obtained after the difference and L2 generate the feature map L5; and LAT represents the local attention map generated and output by the size transformation attention mechanism. 7.The warehouse rooftop solar panel defect detection method based on artificial intelligence according to claim 1 or 6, characterized in that: The image reconstruction network is further provided with a comprehensive attention module, which takes all the local attention maps as input, then generates and outputs a comprehensive attention map, which is used to modulate the reconstructed image.

8. The warehouse rooftop solar panel defect detection method based on artificial intelligence according to claim 7, characterized in that: The comprehensive attention module comprises a second splicing layer, a comprehensive convolution layer, a first comprehensive activation layer, a second sub-pixel convolution layer and a second comprehensive activation layer arranged in sequence.

Images