A photovoltaic and fault detection method based on multi-modal remote sensing images

Abstract

The application relates to the field of traffic prediction interpretability research, and discloses a photovoltaic and fault detection method based on multi-modal remote sensing images, which comprises the following steps: S1, acquiring a multi-modal remote sensing image training sample set, each group of multi-modal remote sensing image training sample comprising multi-modal remote sensing images and photovoltaic label images corresponding to the multi-modal remote sensing images; S2, constructing an initial photovoltaic fault detection model, training the initial photovoltaic fault detection model through the acquired multi-modal remote sensing image training sample set to obtain a trained photovoltaic fault detection model; and S3, performing photovoltaic fault detection on the multi-modal remote sensing images to be detected based on the trained photovoltaic fault detection model, and performing vectorization to obtain type data and position data of photovoltaic faults. The application improves the detection precision of deep learning in the photovoltaic fault detection scene, realizes photovoltaic fault detection through images, reduces manual patrol, and reduces the uncertainty caused by different image noises.

Description

Technical Field

[0001] This application relates to the field of image-based photovoltaic fault detection, and in particular to a photovoltaic fault detection method based on multimodal remote sensing images. Background Technology

[0002] With increasingly prominent environmental issues, countries are placing greater emphasis on the solar photovoltaic (PV) industry, leading to more complex challenges. For example, PV module failures, if not addressed promptly, can not only reduce power generation efficiency but also shorten the module's lifespan. When a PV module malfunctions, localized resistance increases, consuming energy as a load, creating temperature differences, and forming hot spots. Damage to individual cells, localized dirt on the module surface, and shading from surrounding vegetation can all contribute to hot spots. Failure to address these issues promptly will affect the module's performance and lifespan, and may even cause combustion, posing safety risks. Therefore, the detection and location of hot spot defects are crucial for the normal operation of PV modules and the maintenance of PV power plants.

[0003] However, due to the nature of photovoltaic (PV) power generation equipment, PV systems are often required to be installed in sparsely populated areas without vegetation, which brings many difficulties to PV fault monitoring and maintenance. Currently, fault diagnosis of PV power plants mainly relies on the analysis of string data from the power plant's backend system, which suffers from coarse resolution and poor real-time performance, making it difficult to meet the needs of real-time intelligent diagnosis. With the rapid development of deep learning and drone aerial photography technology, using deep learning to detect hot spot defects in PV modules from aerial infrared images has gradually become mainstream.

[0004] With the rapid development of deep learning and drone aerial photography technology, using deep learning to detect hot spot defects in photovoltaic modules from aerial infrared images has gradually become mainstream. Therefore, photovoltaic fault detection methods based on drone and multimodal data fusion are being deployed on edge devices to efficiently and stably monitor photovoltaic equipment. However, the detection accuracy of deep learning in photovoltaic fault detection scenarios still needs improvement. Summary of the Invention

[0005] To address the aforementioned technical issues, this application proposes a photovoltaic fault detection method based on multimodal remote sensing imagery. This method aims to improve the detection accuracy of deep learning in photovoltaic fault detection scenarios, enabling the detection of photovoltaic faults using images, reducing manual inspections, and mitigating the uncertainty caused by noise in different images.

[0006] This application provides a photovoltaic and fault detection method based on multimodal remote sensing imagery, which adopts the following technical solution:

[0007] A photovoltaic and fault detection method based on multimodal remote sensing imagery includes the following steps:

[0008] S1, Obtain a multimodal remote sensing image training sample set, which includes multiple sets of multimodal remote sensing image training samples. Each set of multimodal remote sensing image training samples includes a multimodal remote sensing image and a photovoltaic tag image corresponding to the multimodal remote sensing image.

[0009] The multimodal remote sensing images include optical remote sensing images and thermal infrared remote sensing images;

[0010] The photovoltaic tag image includes multi-level photovoltaic tag files corresponding to multimodal remote sensing images;

[0011] S2, Construct an initial photovoltaic fault detection model. Train the initial photovoltaic fault detection model using the acquired multimodal remote sensing image training sample set until the loss function converges to a preset value to obtain the trained photovoltaic fault detection model.

[0012] S3. Based on the trained photovoltaic fault detection model, photovoltaic fault detection is performed on the multimodal remote sensing images to be detected, and the images are vectorized to obtain photovoltaic fault type data and location data.

[0013] Preferably, the structure of the photovoltaic fault detection model is as follows:

[0014] The photovoltaic fault detection model includes a visible light branch network, a thermal infrared branch network, a feature fusion network, a gradient learning module, and a result output module.

[0015] The visible light branching network includes convolutional layer 1, high-efficiency aggregation layer 1, high-efficiency aggregation layer 2, high-efficiency aggregation layer 3, and high-efficiency aggregation layer 4.

[0016] Convolutional layer 1 includes one convolutional module, efficient aggregation layer 1 includes one convolutional module and one efficient aggregation module, and efficient aggregation layer 2, efficient aggregation layer 3, and efficient aggregation layer 4 each include one downsampling convolutional module and one efficient aggregation module.

[0017] The thermal infrared branch network includes convolutional layer 2, high-efficiency aggregation layer 5, high-efficiency aggregation layer 6, high-efficiency aggregation layer 7, and high-efficiency aggregation layer 8;

[0018] Convolutional layer 2 includes one convolutional module, efficient aggregation layer 5 includes one convolutional module and one efficient aggregation module, and efficient aggregation layers 6, 7, and 8 each include one downsampling convolutional module and one efficient aggregation module.

[0019] The detection process of the photovoltaic fault detection model is as follows:

[0020] Optical remote sensing images are input into a visible light branch network to obtain optical image coding features;

[0021] The optical remote sensing image is input into convolutional layer one. Convolutional layer one performs a 3×3 convolution on the optical remote sensing image, converting the 640×640 image into a 320×320 first optical image feature. Then, the 320×320 first optical image feature is input into efficient aggregation layer one, outputting a 160×160 second optical image feature. Next, the 160×160 second optical image feature is input into efficient aggregation layer two, outputting an 80×80 third optical image feature. Then, the 80×80 third optical image feature is input into efficient aggregation layer three, outputting a 40×40 fourth optical image feature. Finally, the 40×40 fourth optical image feature is input into efficient aggregation layer four, outputting a 20×20 optical image coding feature.

[0022] The thermal infrared remote sensing image is input into convolutional layer two, which performs a 3×3 convolution on the image, converting the 640×640 image into a 320×320 first thermal infrared image feature. This 320×320 first thermal infrared image feature is then input into efficient aggregation layer five, outputting a 160×160 second thermal infrared image feature. Next, the 160×160 second thermal infrared image feature is input into efficient aggregation layer six, outputting an 80×80 third thermal infrared image feature. This 80×80 third thermal infrared image feature is then input into efficient aggregation layer seven, outputting a 40×40 fourth thermal infrared image feature. Finally, the 40×40 third thermal infrared image feature is input into efficient aggregation layer eight, outputting a 20×20 thermal infrared image encoded feature.

[0023] The feature fusion network performs feature fusion on the first optical image feature and the first thermal infrared image feature, the second optical image feature and the second thermal infrared image feature, the third optical image feature and the third thermal infrared image feature, the fourth optical image feature and the fourth thermal infrared image feature, and the optical image coding feature and the thermal infrared image coding feature, respectively, to obtain the first fused feature, the second fused feature, the third fused feature, the fourth fused feature and the fifth fused feature.

[0024] The feature fusion network fuses the first, second, third, fourth, and fifth fusion features to obtain the sixth fusion feature;

[0025] The feature fusion network fuses the second, third, fourth, and fifth fusion features to obtain the seventh fusion feature;

[0026] The feature fusion network fuses the third, fourth, and fifth fusion features to obtain the eighth fusion feature;

[0027] The feature fusion network fuses the fourth and fifth fusion features to obtain the ninth fusion feature;

[0028] The feature fusion network fuses the fifth fusion feature to obtain the tenth fusion feature;

[0029] The gradient learning network extracts and integrates features from the outputs of the feature fusion network, the visible light branch network, and the thermal infrared branch network, and finally outputs the prediction result.

[0030] The results output module converts the prediction results into detection boxes and outputs the detection results in visible light images and thermal infrared images.

[0031] Preferably, the structure of the feature fusion network is as follows:

[0032] The feature fusion network includes a layer-by-layer feature branching module and a layer-by-layer feature fusion module. The feature branching module is a brancher with five feature layers. The first layer of the feature branching module includes a convolution module and a feature extraction module. The second layer of the feature branching module consists of a convolution module, a feature extraction module, and an efficient aggregation module. The third, fourth, and fifth layers of the feature branching module each include a downsampling module, a feature extraction module, and an efficient aggregation module.

[0033] The processing procedure of the feature fusion network is as follows:

[0034] The first layer of the feature branching module fuses the first optical image feature and the first thermal infrared image feature into a first fused feature;

[0035] The second layer of the feature branching module fuses the second optical image features and the second thermal infrared image features into a second fused feature;

[0036] The third layer of the feature branch module fuses the third optical image feature and the third thermal infrared image feature into a third fused feature;

[0037] The fourth layer of the feature branch module fuses the fourth optical image feature and the fourth thermal infrared image feature into a fourth fused feature;

[0038] The fifth layer of the feature branching module fuses optical image coding features and thermal infrared image coding features into a fifth fused feature;

[0039] The feature fusion module fuses the first, second, third, fourth, and fifth fusion features to obtain the sixth fusion feature;

[0040] The feature fusion module fuses the second, third, fourth, and fifth fusion features to obtain the seventh fusion feature;

[0041] The feature fusion module fuses the third, fourth, and fifth fusion features to obtain the eighth fusion feature;

[0042] The feature fusion module fuses the fourth and fifth fusion features to obtain the ninth fusion feature;

[0043] The feature fusion module fuses the fifth fusion feature to obtain the tenth fusion feature.

[0044] Preferably, the structure of the gradient learning module is as follows:

[0045] The gradient learning module includes one main branch module and one auxiliary branch module, which are used to perform forward propagation learning and backward propagation learning on the features;

[0046] The auxiliary branch module includes one pyramid aggregation module, two upsampling modules, two high-efficiency aggregation modules, and two connection layer modules;

[0047] The main branch module includes one pyramid aggregation module, two upsampling modules, four connection layer modules, four high-efficiency aggregation modules, and two downsampling modules;

[0048] The processing procedure of the gradient learning module is as follows:

[0049] The main branch module's processing procedure is as follows: the pyramid aggregation module uses max pooling with different windows to re-integrate the tenth fusion feature to form the eleventh fusion feature; the upsampling module upsamples the eleventh fusion feature and then outputs the twelfth fusion feature (40×40); the connection layer module connects the twelfth fusion feature with the ninth fusion feature; the efficient aggregation module extracts features from the connected features and inputs the features to the upsampling module; the upsampling module outputs the thirteenth fusion feature (80×80); the connection layer module connects the thirteenth fusion feature with the seventh fusion feature; and the efficient aggregation module extracts features from the connected features to obtain the fourteenth fusion feature.

[0050] The fourteenth fusion feature is downsampled to form a 40×40 fifteenth fusion feature. The fifteenth fusion feature is then connected with the twelfth fusion feature to form the sixteenth fusion feature. The efficient aggregation module extracts features from the sixteenth fusion feature. Then, the downsampling module downsamples the extracted features to output a 20×20 seventeenth fusion feature, which is then connected with the eleventh fusion feature. This connection is then input into the efficient aggregation module for feature extraction, outputting the eighteenth fusion feature.

[0051] The processing procedure of the auxiliary branch module is as follows:

[0052] The pyramid aggregation module uses max pooling with different windows to integrate optical image coding features and thermal infrared image coding features into the nineteenth fusion feature; the upsampling module upsamples the nineteenth fusion feature and outputs a 40×40 twentieth fusion feature; the twentieth fusion feature is connected and integrated with the fourth optical image feature and the fourth thermal infrared image feature, and then the efficient aggregation module performs feature extraction, followed by upsampling and outputting an 80×80 twentieth fusion feature; the connection layer module connects and reconstructs the twentieth fusion feature with the third optical image feature and the third thermal infrared image feature, and uses the efficient aggregation module to perform feature extraction and output the twenty-second fusion feature.

[0053] Preferably, the structure of the result output module is as follows:

[0054] The results output module includes 6 feature detection heads. Each feature detection head includes an improved loss function module, a visible light results output module, a thermal infrared results output module, and a feature detection module.

[0055] The output process of the result output module is as follows:

[0056] The detection head obtains the nineteenth, twentieth, and twenty-first fusion features from the auxiliary branch module for target detection; the feature detection module forms adaptive detection anchor boxes for feature detection, compares the detection results with the training data, obtains the loss function, and then updates the weights of the backbone network with the loss function;

[0057] The detection head obtains the fourteenth, sixteenth, and eighteenth fusion features from the main branch module for target detection, adjusts the weights of the detection results and the auxiliary branch results, completes the prediction, and generates the prediction result.

[0058] The visible light result output module and the thermal infrared result output module convert the prediction results into detection boxes and output the RGB visible light image detection results and the thermal infrared IR image detection results, respectively.

[0059] The loss function is calculated as follows:

[0060] The formula for calculating the overall loss function is:

[0061] ;

[0062] The loss function includes localization loss ( ), classification loss ( ), confidence loss ( ) and cross-level consistency loss ( );in The weights of each loss function component in the total loss function are represented by the following values: In the weight configuration, the localization loss has a weight of 0.75, the classification loss and confidence loss both have a weight of 0.25, and the cross-level loss is used to enhance feature consistency with a weight of 0.1.

[0063] The method for calculating the localization loss EIoU is as follows:

[0064] ;

[0065] EIOU was split into L IOU Loss, distance loss L dis And side length loss L asp EioU represents the condition where the side length is used as a penalty term, b gt and b p d(b) represents the coordinates of the center points of the predicted bounding box and the target bounding box, respectively. p ,b gt ) represents the Euclidean distance between the two, w c and h c d(w) is the width and height of the smallest bounding rectangle of the predicted bounding box and the target bounding box. p ,w gt The value represents the difference between the width of the predicted bounding box and the width of the target bounding box.

[0066] In summary, this application includes the following beneficial technical effects:

[0067] 1. This application improves the detection accuracy of deep learning in photovoltaic fault detection scenarios. It can detect photovoltaic faults by utilizing multimodal remote sensing images, reducing manual inspections.

[0068] 2. This application improves the accuracy of photovoltaic fault detection and positioning by adjusting the positioning loss, and reduces the detection area error caused by similar detection frames.

[0069] 3. This application uses a multimodal data extraction and fusion technique to filter out noise from the images of each branch network, thereby reducing the error caused by image noise during feature extraction.

[0070] 4. This application improves the problem of bright spots caused by solar reflection affecting the detection results in UAV patrol and detection by combining visible light and thermal infrared data. Attached Figure Description

[0071] Figure 1 This is a system diagram of photovoltaic fault detection in an embodiment of this application.

[0072] Figure 2 This is a flowchart of photovoltaic fault detection in the embodiments of this application.

[0073] Figure 3 This is a schematic diagram of the visible light branch network and the thermal infrared branch network in the embodiments of this application.

[0074] Figure 4 This is a schematic diagram of the feature fusion network in an embodiment of this application.

[0075] Figure 5 This is a schematic diagram of the dataset for photovoltaic fault detection under dual-modal data in the embodiments of this application.

[0076] Figure 6 This is a schematic diagram of the photovoltaic fault detection results in an embodiment of this application. Detailed Implementation

[0077] The present application will be further described in detail below with reference to the accompanying drawings.

[0078] This application discloses a photovoltaic and fault detection method based on multimodal remote sensing images.

[0079] Reference Figure 1 and Figure 2 The photovoltaic and fault detection method based on multimodal remote sensing imagery includes the following steps:

[0080] S1, Obtain a multimodal remote sensing image training sample set, which includes multiple sets of multimodal remote sensing image training samples. Each set of multimodal remote sensing image training samples includes a multimodal remote sensing image and a photovoltaic tag image corresponding to the multimodal remote sensing image.

[0081] The multimodal remote sensing images include optical remote sensing images and thermal infrared remote sensing images;

[0082] The photovoltaic tag image includes multi-level photovoltaic tag files corresponding to multimodal remote sensing images;

[0083] S2, Construct an initial photovoltaic fault detection model. Train the initial photovoltaic fault detection model using the acquired multimodal remote sensing image training sample set until the loss function converges to a preset value to obtain the trained photovoltaic fault detection model.

[0084] S3. Based on the trained photovoltaic fault detection model, photovoltaic fault detection is performed on the multimodal remote sensing images to be detected, and the images are vectorized to obtain photovoltaic fault type data and location data.

[0085] Specifically, the structure of the photovoltaic fault detection model is as follows:

[0086] The photovoltaic fault detection model includes a visible light branch network, a thermal infrared branch network, a feature fusion network, a gradient learning module, and a result output module.

[0087] The visible light branching network includes convolutional layer 1, high-efficiency aggregation layer 1, high-efficiency aggregation layer 2, high-efficiency aggregation layer 3, and high-efficiency aggregation layer 4.

[0088] Convolutional layer 1 includes one convolutional module, efficient aggregation layer 1 includes one convolutional module and one efficient aggregation module, and efficient aggregation layer 2, efficient aggregation layer 3, and efficient aggregation layer 4 each include one downsampling convolutional module and one efficient aggregation module.

[0089] The processing procedure for visible light branching networks is as follows:

[0090] Optical remote sensing images are input into a visible light branch network to obtain optical image coding features;

[0091] The optical remote sensing image is input into convolutional layer one. Convolutional layer one performs a 3×3 convolution on the optical remote sensing image, converting the 640×640 image into a 320×320 first optical image feature. Then, the 320×320 first optical image feature is input into efficient aggregation layer one, outputting a 160×160 second optical image feature. Next, the 160×160 second optical image feature is input into efficient aggregation layer two, outputting an 80×80 third optical image feature. Then, the 80×80 third optical image feature is input into efficient aggregation layer three, outputting a 40×40 fourth optical image feature. Finally, the 40×40 fourth optical image feature is input into efficient aggregation layer four, outputting a 20×20 optical image encoded feature.

[0092] The thermal infrared branch network includes convolutional layer 2, high-efficiency aggregation layer 5, high-efficiency aggregation layer 6, high-efficiency aggregation layer 7, and high-efficiency aggregation layer 8;

[0093] Convolutional layer 2 includes one convolutional module, efficient aggregation layer 5 includes one convolutional module and one efficient aggregation module, and efficient aggregation layers 6, 7, and 8 each include one downsampling convolutional module and one efficient aggregation module.

[0094] The processing procedure of the thermal infrared branch network is as follows:

[0095] The thermal infrared remote sensing image is input into convolutional layer two, which performs a 3×3 convolution on the image, converting the 640×640 image into a 320×320 first thermal infrared image feature. This 320×320 first thermal infrared image feature is then input into efficient aggregation layer five, outputting a 160×160 second thermal infrared image feature. Next, the 160×160 second thermal infrared image feature is input into efficient aggregation layer six, outputting an 80×80 third thermal infrared image feature. This 80×80 third thermal infrared image feature is then input into efficient aggregation layer seven, outputting a 40×40 fourth thermal infrared image feature. Finally, the 40×40 third thermal infrared image feature is input into efficient aggregation layer eight, outputting a 20×20 thermal infrared image encoded feature.

[0096] The feature fusion network performs feature fusion on the first optical image feature and the first thermal infrared image feature, the second optical image feature and the second thermal infrared image feature, the third optical image feature and the third thermal infrared image feature, the fourth optical image feature and the fourth thermal infrared image feature, and the optical image coding feature and the thermal infrared image coding feature, respectively, to obtain the first fused feature, the second fused feature, the third fused feature, the fourth fused feature and the fifth fused feature.

[0097] The feature fusion network fuses the first, second, third, fourth, and fifth fusion features to obtain the sixth fusion feature;

[0098] The feature fusion network fuses the second, third, fourth, and fifth fusion features to obtain the seventh fusion feature;

[0099] The feature fusion network fuses the third, fourth, and fifth fusion features to obtain the eighth fusion feature;

[0100] The feature fusion network fuses the fourth and fifth fusion features to obtain the ninth fusion feature;

[0101] The feature fusion network fuses the fifth fusion feature to obtain the tenth fusion feature.

[0102] Specifically, the structure of the feature fusion network is as follows:

[0103] The feature fusion network includes a layer-by-layer feature branching module and a layer-by-layer feature fusion module. The feature branching module is a brancher with five feature layers. The first layer of the feature branching module includes a convolution module and a feature extraction module. The second layer of the feature branching module consists of a convolution module, a feature extraction module, and an efficient aggregation module. The third, fourth, and fifth layers of the feature branching module each include a downsampling module, a feature extraction module, and an efficient aggregation module.

[0104] The specific processing steps of the feature fusion network are as follows:

[0105] The first layer of the feature branching module fuses the first optical image feature and the first thermal infrared image feature into a first fused feature;

[0106] The second layer of the feature branching module fuses the second optical image features and the second thermal infrared image features into a second fused feature;

[0107] The third layer of the feature branch module fuses the third optical image feature and the third thermal infrared image feature into a third fused feature;

[0108] The fourth layer of the feature branch module fuses the fourth optical image feature and the fourth thermal infrared image feature into a fourth fused feature;

[0109] The fifth layer of the feature branching module fuses optical image coding features and thermal infrared image coding features into a fifth fused feature;

[0110] The feature fusion module fuses the first, second, third, fourth, and fifth fusion features to obtain the sixth fusion feature;

[0111] The feature fusion module fuses the second, third, fourth, and fifth fusion features to obtain the seventh fusion feature;

[0112] The feature fusion module fuses the third, fourth, and fifth fusion features to obtain the eighth fusion feature;

[0113] The feature fusion module fuses the fourth and fifth fusion features to obtain the ninth fusion feature;

[0114] The feature fusion module fuses the fifth fusion feature to obtain the tenth fusion feature.

[0115] The gradient learning network extracts and integrates features from the outputs of the feature fusion network, the visible light branch network, and the thermal infrared branch network, and finally outputs the prediction result.

[0116] The structure of the gradient learning module is as follows:

[0117] The gradient learning module includes one main branch module and one auxiliary branch module, which are used to perform forward propagation learning and backward propagation learning on the features;

[0118] The auxiliary branch module includes one pyramid aggregation module, two upsampling modules, two high-efficiency aggregation modules, and two connection layer modules;

[0119] The main branch module includes one pyramid aggregation module, two upsampling modules, four connection layer modules, four high-efficiency aggregation modules, and two downsampling modules;

[0120] The specific processing procedure of the gradient learning module is as follows:

[0121] The main branch module's processing procedure is as follows: the pyramid aggregation module uses max pooling with different windows to re-integrate the tenth fusion feature to form the eleventh fusion feature; the upsampling module upsamples the eleventh fusion feature and then outputs the twelfth fusion feature (40×40); the connection layer module connects the twelfth fusion feature with the ninth fusion feature; the efficient aggregation module extracts features from the connected features and inputs the features to the upsampling module; the upsampling module outputs the thirteenth fusion feature (80×80); the connection layer module connects the thirteenth fusion feature with the seventh fusion feature; and the efficient aggregation module extracts features from the connected features to obtain the fourteenth fusion feature.

[0122] The fourteenth fusion feature is downsampled to form a 40×40 fifteenth fusion feature. The fifteenth fusion feature is then connected with the twelfth fusion feature to form the sixteenth fusion feature. The efficient aggregation module extracts features from the sixteenth fusion feature. Then, the downsampling module downsamples the extracted features to output a 20×20 seventeenth fusion feature, which is then connected with the eleventh fusion feature. This connection is then input into the efficient aggregation module for feature extraction, outputting the eighteenth fusion feature.

[0123] The processing procedure of the auxiliary branch module is as follows:

[0124] The pyramid aggregation module uses max pooling with different windows to integrate optical image coding features and thermal infrared image coding features into the nineteenth fusion feature; the upsampling module upsamples the nineteenth fusion feature and outputs a 40×40 twentieth fusion feature; the twentieth fusion feature is connected and integrated with the fourth optical image feature and the fourth thermal infrared image feature, and then the efficient aggregation module performs feature extraction, followed by upsampling and outputting an 80×80 twentieth fusion feature; the connection layer module connects and reconstructs the twentieth fusion feature with the third optical image feature and the third thermal infrared image feature, and uses the efficient aggregation module to perform feature extraction and output the twenty-second fusion feature.

[0125] The results output module converts the prediction results into detection boxes and outputs the detection results in visible light images and thermal infrared images.

[0126] The structure of the result output module is as follows:

[0127] The results output module includes 6 feature detection heads. Each feature detection head includes an improved loss function module, a visible light results output module, a thermal infrared results output module, and a feature detection module.

[0128] The specific output process of the result output module is as follows:

[0129] The detection head obtains the nineteenth, twentieth, and twenty-first fusion features from the auxiliary branch module for target detection; the feature detection module forms adaptive detection anchor boxes for feature detection, compares the detection results with the training data, obtains the loss function, and then updates the weights of the backbone network with the loss function;

[0130] The detection head obtains the fourteenth, sixteenth, and eighteenth fusion features from the main branch module for target detection, adjusts the weights of the detection results and the auxiliary branch results, completes the prediction, and generates the prediction result.

[0131] The visible light result output module and the thermal infrared result output module convert the prediction results into detection boxes and output the RGB visible light image detection results and the thermal infrared IR image detection results, respectively.

[0132] The loss function is calculated as follows:

[0133] The formula for calculating the overall loss function is:

[0134] ;

[0135] The loss function includes localization loss ( ), classification loss ( ), confidence loss ( ) and cross-level consistency loss ( );in The weights of each loss function component in the total loss function are represented by the following values: In the weight configuration, the localization loss has a weight of 0.75, the classification loss and confidence loss both have a weight of 0.25, and the cross-level loss is used to enhance feature consistency with a weight of 0.1.

[0136] The method for calculating the localization loss EIoU is as follows:

[0137] ;

[0138] EIOU was split into L IOU Loss, distance loss L dis And side length loss L asp EIoU directly uses the side length as a penalty term, which can, to some extent, solve the problem of the side length being erroneously amplified in traditional loss functions. gt and b p d(b) represents the coordinates of the center points of the predicted bounding box and the target bounding box, respectively. p ,b gt ) represents the Euclidean distance between the two, w c and h c d(w) is the width and height of the smallest bounding rectangle of the predicted bounding box and the target bounding box. p ,w gt The value represents the difference between the width of the predicted bounding box and the width of the target bounding box.

[0139] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A photovoltaic and fault detection method based on multimodal remote sensing imagery, characterized in that, Includes the following steps: S1, Obtain a multimodal remote sensing image training sample set, which includes multiple sets of multimodal remote sensing image training samples. Each set of multimodal remote sensing image training samples includes a multimodal remote sensing image and a photovoltaic tag image corresponding to the multimodal remote sensing image. The multimodal remote sensing images include optical remote sensing images and thermal infrared remote sensing images; The photovoltaic tag image includes multi-level photovoltaic tag files corresponding to multimodal remote sensing images; S2, Construct an initial photovoltaic fault detection model. Train the initial photovoltaic fault detection model using the acquired multimodal remote sensing image training sample set until the loss function converges to a preset value to obtain the trained photovoltaic fault detection model. S3. Based on the trained photovoltaic fault detection model, photovoltaic fault detection is performed on the multimodal remote sensing images to be detected, and the images are vectorized to obtain photovoltaic fault type data and location data.

2. The photovoltaic and fault detection method based on multimodal remote sensing imagery according to claim 1, characterized in that, The structure of the photovoltaic fault detection model is as follows: The photovoltaic fault detection model includes a visible light branch network, a thermal infrared branch network, a feature fusion network, a gradient learning module, and a result output module. The visible light branching network includes convolutional layer 1, high-efficiency aggregation layer 1, high-efficiency aggregation layer 2, high-efficiency aggregation layer 3, and high-efficiency aggregation layer 4. Convolutional layer 1 includes one convolutional module, efficient aggregation layer 1 includes one convolutional module and one efficient aggregation module, and efficient aggregation layer 2, efficient aggregation layer 3, and efficient aggregation layer 4 each include one downsampling convolutional module and one efficient aggregation module. The thermal infrared branch network includes convolutional layer 2, high-efficiency aggregation layer 5, high-efficiency aggregation layer 6, high-efficiency aggregation layer 7, and high-efficiency aggregation layer 8; Convolutional layer 2 includes one convolutional module, efficient aggregation layer 5 includes one convolutional module and one efficient aggregation module, and efficient aggregation layers 6, 7, and 8 each include one downsampling convolutional module and one efficient aggregation module. The detection process of the photovoltaic fault detection model is as follows: Optical remote sensing images are input into a visible light branch network to obtain optical image coding features; The optical remote sensing image is input into convolutional layer one. Convolutional layer one performs a 3×3 convolution on the optical remote sensing image, converting the 640×640 image into a 320×320 first optical image feature. Then, the 320×320 first optical image feature is input into efficient aggregation layer one, outputting a 160×160 second optical image feature. Next, the 160×160 second optical image feature is input into efficient aggregation layer two, outputting an 80×80 third optical image feature. Then, the 80×80 third optical image feature is input into efficient aggregation layer three, outputting a 40×40 fourth optical image feature. Finally, the 40×40 fourth optical image feature is input into efficient aggregation layer four, outputting a 20×20 optical image coding feature. The thermal infrared remote sensing image is input into convolutional layer two, which performs a 3×3 convolution on the image, converting the 640×640 image into a 320×320 first thermal infrared image feature. This 320×320 first thermal infrared image feature is then input into efficient aggregation layer five, outputting a 160×160 second thermal infrared image feature. Next, the 160×160 second thermal infrared image feature is input into efficient aggregation layer six, outputting an 80×80 third thermal infrared image feature. This 80×80 third thermal infrared image feature is then input into efficient aggregation layer seven, outputting a 40×40 fourth thermal infrared image feature. Finally, the 40×40 third thermal infrared image feature is input into efficient aggregation layer eight, outputting a 20×20 thermal infrared image encoded feature. The feature fusion network performs feature fusion on the first optical image feature and the first thermal infrared image feature, the second optical image feature and the second thermal infrared image feature, the third optical image feature and the third thermal infrared image feature, the fourth optical image feature and the fourth thermal infrared image feature, and the optical image coding feature and the thermal infrared image coding feature, respectively, to obtain the first fused feature, the second fused feature, the third fused feature, the fourth fused feature and the fifth fused feature. The feature fusion network fuses the first, second, third, fourth, and fifth fusion features to obtain the sixth fusion feature; The feature fusion network fuses the second, third, fourth, and fifth fusion features to obtain the seventh fusion feature; The feature fusion network fuses the third, fourth, and fifth fusion features to obtain the eighth fusion feature; The feature fusion network fuses the fourth and fifth fusion features to obtain the ninth fusion feature; The feature fusion network fuses the fifth fusion feature to obtain the tenth fusion feature; The gradient learning network extracts and integrates features from the outputs of the feature fusion network, the visible light branch network, and the thermal infrared branch network, and finally outputs the prediction result. The results output module converts the prediction results into detection boxes and outputs the detection results in visible light images and thermal infrared images.

3. The photovoltaic and fault detection method based on multimodal remote sensing imagery according to claim 2, characterized in that, The structure of the feature fusion network is as follows: The feature fusion network includes a layer-by-layer feature branching module and a layer-by-layer feature fusion module. The feature branching module is a brancher with five feature layers. The first layer of the feature branching module includes a convolution module and a feature extraction module. The second layer of the feature branching module consists of a convolution module, a feature extraction module, and an efficient aggregation module. The third, fourth, and fifth layers of the feature branching module each include a downsampling module, a feature extraction module, and an efficient aggregation module. The processing procedure of the feature fusion network is as follows: The first layer of the feature branching module fuses the first optical image feature and the first thermal infrared image feature into a first fused feature; The second layer of the feature branching module fuses the second optical image features and the second thermal infrared image features into a second fused feature; The third layer of the feature branching module fuses the third optical image feature and the third thermal infrared image feature into a third fused feature; The fourth layer of the feature branch module fuses the fourth optical image feature and the fourth thermal infrared image feature into a fourth fused feature; The fifth layer of the feature branching module fuses optical image coding features and thermal infrared image coding features into a fifth fused feature; The feature fusion module fuses the first, second, third, fourth, and fifth fusion features to obtain the sixth fusion feature; The feature fusion module fuses the second, third, fourth, and fifth fusion features to obtain the seventh fusion feature; The feature fusion module fuses the third, fourth, and fifth fusion features to obtain the eighth fusion feature; The feature fusion module fuses the fourth and fifth fusion features to obtain the ninth fusion feature; The feature fusion module fuses the fifth fusion feature to obtain the tenth fusion feature.

4. The photovoltaic and fault detection method based on multimodal remote sensing imagery according to claim 3, characterized in that, The gradient learning module has the following structure: The gradient learning module includes one main branch module and one auxiliary branch module, which are used to perform forward propagation learning and backward propagation learning on the features; The auxiliary branch module includes one pyramid aggregation module, two upsampling modules, two high-efficiency aggregation modules, and two connection layer modules; The main branch module includes one pyramid aggregation module, two upsampling modules, four connection layer modules, four high-efficiency aggregation modules, and two downsampling modules; The processing procedure of the gradient learning module is as follows: The main branch module's processing procedure is as follows: the pyramid aggregation module uses max pooling with different windows to re-integrate the tenth fusion feature to form the eleventh fusion feature; the upsampling module upsamples the eleventh fusion feature and then outputs the twelfth fusion feature (40×40); the connection layer module connects the twelfth fusion feature with the ninth fusion feature; the efficient aggregation module extracts features from the connected features and inputs the features to the upsampling module; the upsampling module outputs the thirteenth fusion feature (80×80); the connection layer module connects the thirteenth fusion feature with the seventh fusion feature; and the efficient aggregation module extracts features from the connected features to obtain the fourteenth fusion feature. The fourteenth fusion feature is downsampled to form a 40×40 fifteenth fusion feature. The fifteenth fusion feature is then connected with the twelfth fusion feature to form the sixteenth fusion feature. The efficient aggregation module extracts features from the sixteenth fusion feature. Then, the downsampling module downsamples the extracted features to output a 20×20 seventeenth fusion feature, which is then connected with the eleventh fusion feature. This connection is then input into the efficient aggregation module for feature extraction, outputting the eighteenth fusion feature. The processing procedure of the auxiliary branch module is as follows: The pyramid aggregation module uses max pooling of different windows to integrate optical image coding features and thermal infrared image coding features into the nineteenth fusion feature; The upsampling module upsamples the nineteenth fusion feature and outputs a 40×40 fusion feature. The twentieth fusion feature is then connected and integrated with the fourth optical image feature and the fourth thermal infrared image feature. The efficient aggregation module then extracts the features. Next, the upsampling module upsamples the features and outputs an 80×80 fusion feature. The connection layer module connects and reconstructs the fusion feature with the third optical image feature and the third thermal infrared image feature, and uses the efficient aggregation module to extract the features, outputting the twenty-second fusion feature.

5. The photovoltaic and fault detection method based on multimodal remote sensing imagery according to claim 4, characterized in that, The structure of the result output module is as follows: The results output module includes 6 feature detection heads. Each feature detection head includes an improved loss function module, a visible light results output module, a thermal infrared results output module, and a feature detection module. The output process of the result output module is as follows: The detection head obtains the nineteenth, twentieth, and twenty-first fusion features from the auxiliary branch module for target detection; the feature detection module forms adaptive detection anchor boxes for feature detection, compares the detection results with the training data, obtains the loss function, and then updates the weights of the backbone network with the loss function; The detection head obtains the fourteenth, sixteenth, and eighteenth fusion features from the main branch module for target detection, adjusts the weights of the detection results and the auxiliary branch results, completes the prediction, and generates the prediction result. The visible light result output module and the thermal infrared result output module convert the prediction results into detection boxes and output the RGB visible light image detection results and the thermal infrared IR image detection results, respectively. The loss function is calculated as follows: The formula for calculating the overall loss function is: ； The loss function includes localization loss ( ), classification loss ( ), confidence loss ( ) and cross-level consistency loss ( );in The weights of each loss function component in the total loss function are represented by the following values: In the weight configuration, the localization loss has a weight of 0.75, the classification loss and confidence loss both have a weight of 0.25, and the cross-level loss is used to enhance feature consistency with a weight of 0.

1. The method for calculating the localization loss EIoU is as follows: ； EIOU was split into L IOU Loss, distance loss L dis And side length loss L asp EioU represents the condition where the side length is used as a penalty term, b gt and b p d(b) represents the coordinates of the center points of the predicted bounding box and the target bounding box, respectively. p ,b gt ) represents the Euclidean distance between the two, w c and h c d(w) is the width and height of the smallest bounding rectangle of the predicted bounding box and the target bounding box. p ,w gt The value represents the difference between the width of the predicted bounding box and the width of the target bounding box.

Images