Photovoltaic panel fault detection method, device and equipment based on multi-branch feature interaction and medium

Abstract

The invention discloses a photovoltaic panel fault detection method and device based on multi-branch feature interaction, equipment and a medium. The method comprises the following steps: acquiring a visible light-infrared image pair of a photovoltaic panel to be detected; through a preset target multi-branch fault detection model, image features are extracted from the visible light-infrared image pair, iterative feature interaction is carried out, and a plurality of input feature maps are generated; performing fault detection by adopting all the input feature maps through the target multi-branch fault detection model, and judging whether the photovoltaic panel to be detected has a fault or not; if yes, the fault position is marked through a target multi-branch fault detection model, and the fault category is output; and if not, continuously monitoring whether a new visible light-infrared image pair is received or not. Therefore, through the multi-branch structure of the fault detection model, feature interaction is carried out from different image dimensions to adapt to the condition of diversified arrangement scenes of the photovoltaic panel, and the fault detection precision of the photovoltaic panel is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a photovoltaic panel fault detection method, device, equipment and medium based on multi-branch feature interaction. Background Art

[0002] With the increasing global demand for clean energy, photovoltaic power generation technology has developed rapidly. As the core component of the photovoltaic power generation system, the performance of photovoltaic panels directly affects the power generation efficiency and economic benefits of the entire system. However, since photovoltaic panels are exposed to the outdoor environment for a long time, their performance may be affected by weather such as temperature, rain, wind, human work activities or biological damage, as well as mechanical damage during transportation and installation. These factors may cause various failures of photovoltaic panels, thereby reducing their power generation efficiency and may even cause safety hazards.

[0003] Traditional methods rely on manual inspections, which are not only time-consuming and labor-intensive, but also prone to false detection and missed detection. With the continuous development of computer vision technology in recent years, it has achieved good application results in many fields. There are also solutions for fault detection of photovoltaic panels through computer vision, such as fault detection of photovoltaic panels through visible light images.

[0004] When performing fault detection, the existing technology usually only uses single-image single-branch detection, which cannot adapt to the diverse scenarios of photovoltaic panel layout, resulting in reduced detection accuracy. Summary of the invention

[0005] The present invention provides a photovoltaic panel fault detection method, device, equipment and medium based on multi-branch feature interaction, which solves the technical problem that the prior art usually only uses single-image single-branch detection when performing fault detection, and cannot adapt to the diverse photovoltaic panel layout scenarios, resulting in reduced detection accuracy.

[0006] A first aspect of the present invention provides a photovoltaic panel fault detection method based on multi-branch feature interaction, comprising:

[0007] Acquire a visible light-infrared image pair of the photovoltaic panel to be inspected;

[0008] Extracting image features from the visible light-infrared image pair and performing iterative feature interaction through a preset target multi-branch fault detection model to generate a plurality of input feature maps;

[0009] Using the target multi-branch fault detection model to perform fault detection using all the input feature graphs, determining whether the photovoltaic panel to be detected has a fault;

[0010] If yes, then mark the fault location and output the fault category through the target multi-branch fault detection model;

[0011] If not, continue to monitor whether a new visible light-infrared image pair is received.

[0012] Optionally, the method further comprises:

[0013] Obtain multiple visible-infrared training image pairs of photovoltaic panels to create a detection dataset;

[0014] Embed multiple stage output positions of the first visible light branch, the second visible light branch and the infrared branch with multiple feature interaction modules;

[0015] Connecting the neck network and the detection head network in sequence at the output position of the stage of generating the input feature map to create an initial multi-branch fault detection model;

[0016] The initial multi-branch fault detection model is trained using the detection data set to obtain a target multi-branch fault detection model.

[0017] Optionally, the adopting the detection data set to train the initial multi-branch fault detection model to obtain a target multi-branch fault detection model includes:

[0018] Dividing the detection data set into a training set, a validation set and a test set according to a preset ratio;

[0019] Inputting the training set into the initial multi-branch fault detection model to generate a fault prediction result;

[0020] According to the fault prediction result and the actual fault result corresponding to the training set, respectively calculate the rectangular box loss, the confidence loss and the classification loss and perform weighted superposition to obtain a target loss value;

[0021] Adjusting parameters of the initial multi-branch fault detection model according to the target loss value until the target loss value converges to obtain a first multi-branch fault detection model;

[0022] Using the validation set to adjust the hyperparameters of the first multi-branch fault detection model, to obtain a plurality of second multi-branch fault detection models with different hyperparameters;

[0023] The test set is used to input each of the second multi-branch fault detection models respectively, and the second multi-branch fault detection model with the highest average accuracy is selected as the target multi-branch fault detection model.

[0024] Optionally, the visible light-infrared image pair includes a visible light image and an infrared image, and the target multi-branch fault detection model includes a first stage module and multiple second stage modules respectively connected to feature interaction modules; extracting image features from the visible light-infrared image pair and performing iterative feature interaction through a preset target multi-branch fault detection model to generate multiple input feature maps includes:

[0025] Extracting image features from the visible light image and the infrared image respectively through the first stage module and performing feature interaction in the feature interaction module to generate a plurality of interactive features and input them into the second stage module;

[0026] Iterative feature interaction is performed on the plurality of interactive features through the plurality of the second-stage modules to generate input feature maps respectively.

[0027] Optionally, the first-stage module includes a first CNN branch, a first Swin Transformer branch, and a first infrared branch connected in parallel to a first feature interaction module; extracting image features from the visible light image and the infrared image respectively through the first-stage module and performing feature interaction in the feature interaction module to generate a plurality of interaction features and input them into the second-stage module, including:

[0028] Extracting a first visible light feature from the visible light image through the first CNN branch and inputting the first visible light feature into the first feature interaction module;

[0029] Extracting a second visible light feature from the visible light image through the first Swin Transformer branch and inputting the second visible light feature into the first feature interaction module;

[0030] Extracting a first infrared feature from the infrared image through the first infrared branch and inputting the first infrared feature into the first feature interaction module;

[0031] The first feature interaction module uses the first visible light feature, the second visible light feature and the first infrared feature to perform feature interaction to generate a single fusion feature;

[0032] Performing a channel separation operation on the fused features through the first feature interaction module to obtain a first shared feature, a second shared feature, and a third shared feature;

[0033] The first visible light feature and the first shared feature, the second visible light feature and the second shared feature, and the first infrared feature and the third shared feature are respectively superimposed through the first feature interaction module to obtain multiple interactive features and input them into the second stage module.

[0034] Optionally, the first feature interaction module includes a fusion part and a dense splicing part connected in sequence; the first feature interaction module uses the first visible light feature, the second visible light feature and the first infrared feature to perform feature interaction to generate a single fusion feature, including:

[0035] After sequentially performing convolution, batch normalization, and function activation on the first visible light feature, the second visible light feature, and the first infrared feature through the fusion part, a single first intermediate feature is obtained by splicing them;

[0036] Performing batch normalization, function activation, and convolution on the first intermediate feature through the dense concatenation part to obtain a second intermediate feature;

[0037] Splicing the second intermediate feature with all the existing first intermediate features through the dense splicing part to obtain a third intermediate feature as a new first intermediate feature;

[0038] Jump to execute the step of performing batch normalization, function activation and convolution on the first intermediate feature through the dense splicing part to obtain the second intermediate feature until the preset number of iterations is reached, and determine the third intermediate feature at the current moment as the fusion feature.

[0039] Optionally, the second stage module includes multiple groups of second CNN branches, second Swin Transformer branches and second infrared branches connected in parallel to the second feature interaction module, and the branch architecture of each group of the second Swin Transformer branches is different; the iterative feature interaction of the multiple interactive features through the multiple second stage modules to generate input feature maps respectively includes:

[0040] Extract the third visible light feature, the fourth visible light feature and the second infrared feature from each of the interaction features through the second CNN branch, the second Swin Transformer branch and the second infrared branch, and input them into the second feature interaction module;

[0041] The second feature interaction module uses the third visible light feature, the fourth visible light feature and the second infrared feature to perform feature interaction to generate an input feature map;

[0042] Performing a channel separation operation on the input feature map through the second feature interaction module to obtain a fourth shared feature, a fifth shared feature, and a sixth shared feature;

[0043] Superimposing the third visible light feature and the fourth shared feature, the fourth visible light feature and the fifth shared feature, and the second infrared feature and the sixth shared feature respectively through the second feature interaction module to obtain a plurality of new interaction features;

[0044] The jump execution extracts the third visible light feature, the fourth visible light feature and the second infrared feature from the interaction feature respectively through the second CNN branch, the second Swin Transformer branch and the second infrared branch, and inputs them into the second feature interaction module until a preset number of input feature maps are generated.

[0045] Optionally, the target multi-branch fault detection model further includes a neck network and a detection head network connected in sequence; the target multi-branch fault detection model uses all the input feature graphs to perform fault detection to determine whether the photovoltaic panel to be detected has a fault, including:

[0046] The neck network receives the plurality of input feature maps and sequentially performs up-sampling feature fusion and down-sampling feature fusion to obtain a plurality of enhanced feature maps;

[0047] Re-parameterizing each of the enhanced feature maps through the neck network and inputting the re-parameterized feature maps into the detection head network;

[0048] Determine whether there is a target that meets the preset fault category from each of the enhanced feature maps through the detection head network;

[0049] If yes, it is determined that the photovoltaic panel to be detected has a fault;

[0050] If not, it is determined that the photovoltaic panel to be detected has no fault.

[0051] A second aspect of the present invention provides a photovoltaic panel fault detection device based on multi-branch feature interaction, comprising:

[0052] An image pair acquisition module, used to acquire a visible light-infrared image pair of the photovoltaic panel to be inspected;

[0053] A feature map generation module is used to call a preset target multi-branch fault detection model to extract image features from the visible light-infrared image pair and perform iterative feature interaction to generate multiple input feature maps;

[0054] A fault judgment module, used to call the target multi-branch fault detection model to use all the input feature graphs to perform fault detection, and judge whether the photovoltaic panel to be detected has a fault;

[0055] A first determination module, configured to, if yes, mark the fault location and output the fault category through the target multi-branch fault detection model;

[0056] The second determination module is used for, if not, continuously monitoring whether a new visible light-infrared image pair is received.

[0057] The third aspect of the present invention provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the photovoltaic panel fault detection method based on multi-branch feature interaction as described in any one of the first aspect of the present invention.

[0058] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the photovoltaic panel fault detection method based on multi-branch feature interaction as described in any one of the first aspects of the present invention.

[0059] It can be seen from the above technical solutions that the present invention has the following advantages:

[0060] The present invention obtains a visible light-infrared image pair of a photovoltaic panel to be detected; extracts image features from the visible light-infrared image pair through a preset target multi-branch fault detection model and performs iterative feature interaction to generate multiple input feature maps; uses the target multi-branch fault detection model to use all input feature maps for fault detection to determine whether the photovoltaic panel to be detected has a fault; if so, marks the fault location and outputs the fault category through the target multi-branch fault detection model; if not, continuously monitors whether a new visible light-infrared image pair is received. Thus, through the multi-branch structure of the fault detection model, feature interaction is performed from different image dimensions to adapt to the diversified photovoltaic panel layout scenarios, effectively improving the accuracy of photovoltaic panel fault detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0062] Figure 1 A flowchart of a photovoltaic panel fault detection method based on multi-branch feature interaction provided by an embodiment of the present invention;

[0063] Figure 2 A model structure diagram of a target multi-branch fault detection model provided by an embodiment of the present invention;

[0064] Figure 3 A schematic diagram of the original structure of the second visible light branch provided by an embodiment of the present invention;

[0065] Figure 4 A module architecture diagram of the MV module provided in an embodiment of the present invention;

[0066] Figure 5 A module architecture diagram of a TADM module provided in an embodiment of the present invention;

[0067] Figure 6 A module architecture diagram of a feature interaction module provided in an embodiment of the present invention;

[0068] Figure 7 A schematic diagram of the structure of a neck network and a detection head network provided by an embodiment of the present invention;

[0069] Figure 8 A schematic diagram of the structure of a feature extraction network provided by an embodiment of the present invention;

[0070] Fig. 9 A structural block diagram of a photovoltaic panel fault detection device based on multi-branch feature interaction provided by an embodiment of the present invention;

[0071] Fig.10 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0072] The embodiments of the present invention provide a photovoltaic panel fault detection method, device, equipment and medium based on multi-branch feature interaction, which is used to solve the technical problem that the prior art usually only uses single-image single-branch detection when performing fault detection, and cannot adapt to the diverse situations of photovoltaic panel layout scenarios, resulting in reduced detection accuracy.

[0073] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] See also Figure 1 , Figure 1 A flowchart of the steps of a photovoltaic panel fault detection method based on multi-branch feature interaction provided by an embodiment of the present invention.

[0075] The present invention provides a photovoltaic panel fault detection method based on multi-branch feature interaction, comprising:

[0076] Step 101, obtaining a visible light-infrared image pair of a photovoltaic panel to be inspected;

[0077] The visible light-infrared image pair refers to an image pair including a visible light image and an infrared image of the same photovoltaic panel to be inspected, wherein the visible light image and the infrared image can be obtained by shooting from the same angle by a visible light camera and an infrared camera respectively, and the image resolution and size can be the same.

[0078] Photovoltaic panels absorb solar radiation and use the photoelectric effect to convert solar energy into electrical energy for users. In order to detect photovoltaic panel faults in a timely and accurate manner, visible light cameras and infrared cameras can be used to obtain visible light images and infrared images of the photovoltaic panels to be tested, and visible light-infrared image pairs can be constructed as the data basis for fault detection.

[0079] In an example of the present application, before executing step 102, the target multi-branch fault detection model can be constructed and trained through the following steps S11-S14, and its model architecture is as follows: Figure 2 shown.

[0080] S11, obtaining multiple visible light-infrared training image pairs of photovoltaic panels to create a detection data set;

[0081] When collecting visible light-infrared training image pairs, the visible light camera and the infrared camera should be used to shoot the photovoltaic panel at the same angle to form an image pair. For different image pairs, the diversity of lighting environment and angle should be ensured as much as possible. Among them, photovoltaic panels include normal photovoltaic panels and faulty photovoltaic panels.

[0082] To ensure the accuracy of the model, the number of visible light-infrared training image pairs for each type of photovoltaic panel should be no less than 1000. Since there may be fewer samples of faulty photovoltaic panels and it is difficult to capture a sufficient number of faulty photovoltaic panel image pairs, data enhancement can be used to increase the number of samples, such as flipping, rotating, stretching, and randomly cropping the images.

[0083] According to the type of fault, the photovoltaic panel image pairs are classified as having some kind of fault or being normal, and the images are annotated using labelimg. The annotation content includes the annotation anchor box and the category. Each anchor box is an image sample of a photovoltaic panel. All annotated image samples constitute a photovoltaic panel detection dataset, and the photovoltaic panel detection dataset is randomly divided into a training set, a test set, and a validation set in proportion. This embodiment is randomly divided into a training set, a validation set, and a test set in a ratio of 7:2:1.

[0084] S12, embedding multiple stage output positions of the first visible light branch, the second visible light branch and the infrared branch with multiple feature interaction modules;

[0085] like Figure 3 As shown, Figure 3 This is a schematic diagram of the original structure of the second visible light branch in the embodiment of the present application. The Swin Transformer network can be used. The network performs feature segmentation and feature mapping on the input in the Patch Partition layer and the Linear Embedding layer, then extracts features through the Swin Transformer Block, and uses the PatchMerging layer for downsampling. Compared with the general Transformer, this network can reduce complexity, avoid the problem of too long sequences, and has better scalability. At the same time, since CNN has good global invariance, CNN is selected to form the first visible light branch to enhance the global invariance of the model. The structure of the first visible light branch can be seen in Figure 2 The leftmost branch.

[0086] The specific structure of the infrared branch can be found in Figure 2 The rightmost branch of , for the TDAM module and MV module, can be constructed using different module structures.

[0087] The module architecture of the MV module is as follows Figure 4 As shown, in this embodiment, the MV modules have different step sizes. For the MV module with a step size of 1, its module architecture is as follows: Figure 4 As shown in the structure (a) in Figure 1, the second convolution of the main branch is replaced by a depth convolution based on the general residual module, which improves the computational efficiency and reduces the number of model parameters. For the MV module with a stride of 2, the depth convolution stride of the main branch is 2, and a convolution layer with a stride of 2 is also added to the residual side to keep the residual side consistent with the feature map size output by the main branch.

[0088] The module architecture of TDAM module is as follows Figure 5As shown in the figure, in this module, the input features are convolved with a convolution kernel size of 1×1 to obtain a feature map of size C×H×W, and then a global average pooling (GAP) operation is performed on each of the three dimensions to obtain three feature maps of size C×1×1, 1×H×1 and 1×1×W. Then, the two feature maps of size 1×H×1 and 1×1×W are concatenated and fully connected to obtain the information between adjacent elements in the features. Then, the feature map of size 1×1×(W+H) and the feature map of size C×1×1 are concatenated and fully connected to obtain a feature map of size 1×1×(C+W+H). After splitting, it is multiplied with the input features to dynamically allocate weights.

[0089] It should be noted that the feature interaction module can be divided into a first feature interaction module and a second feature interaction module. The first feature interaction module does not have an output of the input feature map, but only outputs shared features, while the second feature interaction module outputs both the input feature map and the interactive features. Figure 6 As shown, the specific architecture of the first feature interaction module is basically similar to that of the second feature interaction module, except that the output of Si is missing. The stage output position refers to the position where there is a shared feature or interactive feature output.

[0090] In an embodiment of the present application, the first visible light branch, the second visible light branch and the infrared branch are respectively divided by feature interaction modules, and the feature interaction modules are embedded in the stage output position of each branch to form a feature extraction network.

[0091] S13, connecting the neck network and the detection head network in sequence at the output position of the stage of generating the input feature map to create an initial multi-branch fault detection model;

[0092] In order to obtain the input feature map for subsequent fault detection, the output position of the input feature map generation stage is sequentially connected to the neck network and the detection head network to construct an initial multi-branch fault detection model.

[0093] The neck network is used to perform up-sampling feature fusion and down-sampling feature fusion on each input feature map, thereby obtaining multiple enhanced feature maps of different scales. These maps are then input into the detection head network for further fault prediction, from which the operating status of the photovoltaic panel is identified, and whether the photovoltaic panel has a fault and the specific fault type is determined. The specific architecture of the neck network and the detection head network is shown below. Figure 7As shown in Figure 2, the detection head network, i.e., the Head module, contains three detection heads, each of which consists of three convolutional layers, each of which is followed by a BN layer and an activation function. The Head module converts the three enhanced feature maps into detection results, including bounding boxes, categories, and confidence levels.

[0094] The initial multi-branch fault detection model constructed in this embodiment is improved on the basis of the network structure of YOLOv7 and Swin Transformer, realizing the complementary advantages of CNN and Swin Transformer, and the detection accuracy of the model is higher; in addition, this embodiment effectively improves the detection accuracy of the model under different lighting conditions through dual-band detection of visible light and infrared light, making the model more robust; this embodiment designs a TDAM module, which enables the model to better focus on the more important parts of the input image by dynamically allocating weights; this embodiment also designs a splicing-based feature interaction module, which fully utilizes the features through a mechanism of densely connecting all the previous layers with the subsequent layers, thereby improving the detection accuracy; since the feature interaction module performs multiple feature interactions at different stages, the detection performance of the model is better; the feature interaction module is an intermediate-level interaction, and the detection speed is faster than pixel-level fusion.

[0095] S14. Use the detection data set to train the initial multi-branch fault detection model to obtain the target multi-branch fault detection model.

[0096] Optionally, S14 may include the following sub-steps:

[0097] Divide the detection data set into training set, validation set and test set according to the preset ratio;

[0098] Input the training set into the initial multi-branch fault detection model to generate fault prediction results;

[0099] According to the fault prediction results and the actual fault results corresponding to the training set, the rectangular box loss, confidence loss and classification loss are calculated respectively and weightedly superimposed to obtain the target loss value;

[0100] Adjusting parameters of the initial multi-branch fault detection model according to the target loss value until the target loss value converges to obtain a first multi-branch fault detection model;

[0101] Using the validation set to adjust the hyperparameters of the first multi-branch fault detection model, to obtain a plurality of second multi-branch fault detection models with different hyperparameters;

[0102] The test set is used to input each second multi-branch fault detection model respectively, and the second multi-branch fault detection model with the highest average accuracy is selected as the target multi-branch fault detection model.

[0103] In this embodiment, the detection data set is randomly divided into a training set, a validation set, and a test set in a ratio of 7:2:1.

[0104] It should be noted that the loss function of the initial multi-branch fault detection model consists of three parts, including the rectangular box loss Lrect, the confidence loss Lobj and the classification loss Lcls. CIoU loss is used as the rectangular box loss, and its calculation formula is:

[0105]

[0106] Where IoU represents the intersection-over-union ratio of the predicted box A and the real box B, and its calculation formula is:

[0107]

[0108] In addition, d is the distance between the center point of the predicted box and the real box, c is the diagonal distance of the minimum circumscribed rectangle, and v is the correction factor, which is calculated as follows:

[0109]

[0110] Among them, w G 、h G and w P 、h P are the width and height of the target box and the predicted box respectively.

[0111] Target confidence loss L obj It is calculated using binary cross entropy, and the calculation formula is:

[0112] where p i,0 is the target confidence score in the i-th prediction box, p i,iou is the IoU value between the i-th predicted box and the corresponding target box.

[0113] Classification loss L cls It is also calculated using binary cross entropy, with y and p representing the true label and predicted value respectively, and N being the number of samples. The calculation formula is:

[0114]

[0115] in, Represents the i-th true label in the dataset; Indicates the corresponding predicted value.

[0116] The above three types of losses are weighted , and Add them together, so the loss function calculation formula of the multi-branch feature interaction network is:

[0117]

[0118] in , and satisfy:

[0119]

[0120] Train on the multi-branch feature interaction network until the loss value converges and obtain the model weights.

[0121] In this embodiment, the calculation of average precision can be performed by combining the precision and recall as the basis of evaluation indicators, as shown below:

[0122]

[0123]

[0124] Among them, P represents precision; R represents recall; TP represents the number of samples correctly classified as positive examples; FP represents the number of samples incorrectly classified as positive examples; and FN represents the number of samples incorrectly classified as negative examples.

[0125] As can be seen from the formula, precision is more inclined to not make wrong predictions, while recall is more inclined to rather predict more wrong ones than miss detections. The two will change when the threshold value is different. Under the same circumstances, increasing the threshold will increase the precision and decrease the recall, and vice versa. Therefore, the PR curve is used to evaluate the model. The PR curve is a two-dimensional curve with precision P and recall R as the vertical and horizontal axis coordinates, and is drawn by selecting the corresponding precision and recall when different thresholds are selected. The area enclosed by the PR curve and the horizontal and vertical axes is AP (average precision). The calculation method of AP for a certain category is as follows:

[0126]

[0127] Among them, r1, r2, ... r n It represents the recall rate R value corresponding to the first interpolation point of each interpolation segment of Precision arranged in ascending order; Pinterp(r) represents the accuracy rate corresponding to the recall rate r. The average value of AP of multiple categories is mAP:

[0128]

[0129] Among them, C represents the number of categories in the dataset, AP i Represents the AP value of the i-th category.

[0130] Step 102, extracting image features from the visible light-infrared image pair through a preset target multi-branch fault detection model and performing iterative feature interaction to generate multiple input feature maps;

[0131] In this embodiment, the target multi-branch fault detection model includes a feature extraction network and a fault detection network. The feature extraction network is divided into multiple stage modules by a first feature interaction module and multiple second feature interaction modules, such as Figure 8 The first feature interaction module is located between the first stage module stage1 and the first second stage module stage2, and the second feature interaction module is located between the second stage modules stage2, stage3 and stage4 and at the stage output position of stage4.

[0132] The feature extraction network uses the YOLOv7 backbone as the visible light CNN branch and the Swin Transformer backbone as the visible light Swin Transformer branch, and forms a multi-branch network with the infrared branch based on the three-dimensional attention module (TDAM) designed by the present invention. The feature F outputted by the i-th stage of the three branches Ci 、F Ti and F Ii Input into the feature interaction module to make frequent interactions between features of different branches. Where i=1,2,3,4.

[0133] In general, by obtaining visible light-infrared image pairs of normal photovoltaic panels and faulty photovoltaic panels, and constructing photovoltaic panel detection data sets according to fault categories; constructing an initial multi-branch fault detection model based on the network architecture of YOLOv7 and Swin Transformer, improving the backbone network to a multi-branch feature interaction network, and using a splicing-based feature interaction module between three branches for feature interaction; in three different stages of feature extraction from shallow to deep, the three pairs of features of different branches are fused in turn through the above-mentioned feature interaction module, and the fused features S2, S3, and S4 are input into the Neck network to obtain three enhanced features in the Neck network; the model is trained based on the photovoltaic panel detection data set; the trained model is used for photovoltaic panel fault detection, and the model is deployed to a visual photovoltaic panel monitoring system to detect the captured photovoltaic panel images. The present invention constructs a multi-branch detection network for photovoltaic panels by combining and improving the YOLOv7 and Swin Transformer algorithms, solves the problem that the photovoltaic panel fault detection accuracy in photovoltaic power stations is not high and is greatly affected by environmental conditions such as light, and improves the efficiency and management level of photovoltaic power stations.

[0134] In an example of the present application, the visible light-infrared image pair includes a visible light image and an infrared image, and the target multi-branch fault detection model includes a first stage module and a plurality of second stage modules respectively connected to the feature interaction modules; step 102 may include the following S21-S22:

[0135] S21, extracting image features from the visible light image and the infrared image respectively through the first stage module and performing feature interaction in the feature interaction module, generating multiple interactive features and inputting them into the second stage module;

[0136] Optionally, the first stage module includes a first CNN branch, a first Swin Transformer branch and a first infrared branch connected in parallel to the first feature interaction module; S21 may include the following sub-steps S211-S216:

[0137] S211, extracting a first visible light feature from the visible light image through a first CNN branch and inputting the first visible light feature into a first feature interaction module;

[0138] S212, extracting a second visible light feature from the visible light image through the first Swin Transformer branch and inputting the second visible light feature into the first feature interaction module;

[0139] S213, extracting a first infrared feature from the infrared image through the first infrared branch and inputting the first infrared feature into a first feature interaction module;

[0140] After the first-stage module receives the visible light-infrared image pair, it splits it into a visible light image and an infrared image, calls the first CNN branch to extract the first visible light feature from the visible light image and inputs it into the first feature interaction module, calls the first Swin Transformer branch to extract the second visible light feature from the visible light image and inputs it into the first feature interaction module, and calls the first infrared branch to extract the first infrared feature from the infrared image and inputs it into the first feature interaction module to provide a data processing basis for the first feature interaction module.

[0141] S214, performing feature interaction using the first visible light feature, the second visible light feature, and the first infrared feature through a first feature interaction module to generate a single fusion feature;

[0142] Furthermore, the first feature interaction module includes a fusion part and a dense splicing part connected in sequence; S214 may include the following sub-steps:

[0143] The fusion part performs convolution, batch normalization and function activation on the first visible light feature, the second visible light feature and the first infrared feature in sequence, and then concatenates them to obtain a single first intermediate feature;

[0144] Performing batch normalization, function activation, and convolution on the first intermediate feature through the dense concatenation part to obtain a second intermediate feature;

[0145] Partially joining the second intermediate feature and all existing first intermediate features by dense joining to obtain a third intermediate feature as a new first intermediate feature;

[0146] The jump execution performs batch normalization, function activation and convolution on the first intermediate feature through the dense splicing part to obtain the step of the second intermediate feature until the preset number of iterations is reached, and the third intermediate feature at the current moment is determined as the fusion feature.

[0147] See also Figure 6 ,in , and They represent the features of the i-th stage output of the visible light CNN branch, the i-th stage output of the visible light Swin Transformer branch, and the i-th stage output of the infrared CNN branch. The feature interaction module performs convolution, batch normalization, and SiLU activation operations on the three inputs respectively. Among them, i=1,2,3,4.

[0148] Specifically, the fusion part first performs a 1×1 convolution operation on the three input features respectively, so that the number of channels of the input feature map is compressed to one third of the original number; then batch normalization is performed to suppress gradient explosion and gradient disappearance; then the SiLU function is used for activation, and then the three features are spliced ​​to obtain Concat(x,y) represents the concatenation of input x and input y, and cbs(x) represents the effect of Conv layer, BN layer and SiLU function on input x. The process can be expressed as:

[0149]

[0150] The first intermediate feature after splicing The fused features are input into the dense splicing part. The dense splicing part makes full use of the features by densely connecting all the previous layers with the subsequent layers. Then, the fused features are divided into three feature maps by channel through the channel separation operation Split, and the original features before the input feature interaction module of the visible light CNN branch, the visible light Swin Transformer branch, and the infrared branch are superimposed. The superimposed features not only retain the original information, but also learn the correlation information between branches.

[0151] The dense splicing part is the core of the feature interaction module. It includes four modules consisting of BN layer, ReLU function and Conv layer. It densely connects the features of all previous layers with the features obtained in the following layers, making full use of the features and effectively solving the problem of gradient disappearance. In the dense splicing part, the features of each previous layer are saved, spliced ​​with the output features of the following layer, and used in each subsequent convolution. ) represents the dense concatenation part of the input , brc(x) represents the effect of the BN layer, ReLU function and Conv layer on the input x, concat(x,y) represents the concatenation operation of the input x and the input y, then the operation of the dense concatenation part can be expressed as:

[0152]

[0153] Among them, x1, x2, and x3 represent the output of x0 after passing through the 1st, 2nd, and 3rd BN-ReLU-Conv layers respectively:

[0154]

[0155]

[0156]

[0157] In summary, use , and They represent the visible light CNN branch features, visible light Swin Transformer branch features, and infrared branch features of the input feature interaction module respectively. The function cbs(x) represents the effect of Conv, BN layer, and SiLU function on the input x. The function d(x) represents the effect of the dense concatenation part on the input x. The function concat(x, y) represents the concatenation operation of input x and input y. The relationship between the fusion feature Si and the three input features can be expressed as:

[0158]

[0159] S215, performing a channel separation operation on the fused feature through the first feature interaction module to obtain a first shared feature, a second shared feature, and a third shared feature;

[0160] S216. Superimpose the first visible light feature and the first shared feature, the second visible light feature and the second shared feature, and the first infrared feature and the third shared feature through the first feature interaction module to obtain multiple interactive features and input them into the second stage module.

[0161] After obtaining the fusion feature S iAfter that, use the split operation to fusion feature S i Perform channel separation operation and fuse the feature S i Equally divided into the first shared feature S that is superimposed with the original feature of the visible light CNN branch Ci , the second shared feature S superimposed with the original feature of the visible light SwinTransformer branch Ti And the third shared feature S superimposed with the original feature of the infrared branch Ii . S Ci , S Ti and S Ii They all carry the association information between branches learned by the feature interaction module, which is combined with the three original features F Ci 、F Ti and F Ii Superposition, we get G Ci , G Ti and G Ii , the process can be expressed as:

[0162]

[0163] The feature interaction module fuses and interacts the features of the three branches, and obtains three interactive features after split operation, which are superimposed with the original features of the three branches. CNN has good translation invariance, and SwinTransformer has a global receptive field, so feature interaction combines the advantages of the two, and their feature information can be complementary, so that the interactive features have sufficient information expression capabilities; in addition, under poor lighting conditions, the visible light pictures taken are not easy to detect, but infrared images with clear contours can be taken, so feature interaction makes the model more robust to changes in lighting conditions.

[0164] S22. Perform iterative feature interaction on multiple interactive features through multiple second-stage modules to generate input feature maps respectively.

[0165] In one example of the present application, the second stage module includes multiple groups of second CNN branches, second Swin Transformer branches and second infrared branches connected in parallel to the second feature interaction module, and the branch architecture of each group of second Swin Transformer branches is different; S22 may include sub-steps:

[0166] The third visible light feature, the fourth visible light feature and the second infrared feature are respectively extracted from each interaction feature through the second CNN branch, the second Swin Transformer branch and the second infrared branch, and input into the second feature interaction module;

[0167] The third visible light feature, the fourth visible light feature and the second infrared feature are used to perform feature interaction through the second feature interaction module to generate an input feature map;

[0168] Performing a channel separation operation on the input feature map through a second feature interaction module to obtain a fourth shared feature, a fifth shared feature, and a sixth shared feature;

[0169] The third visible light feature and the fourth shared feature, the fourth visible light feature and the fifth shared feature, and the second infrared feature and the sixth shared feature are respectively superimposed by the second feature interaction module to obtain a plurality of new interaction features;

[0170] The jump execution extracts the third visible light feature, the fourth visible light feature and the second infrared feature from the interactive features respectively through the second CNN branch, the second Swin Transformer branch and the second infrared branch, and inputs them into the second feature interaction module until a preset number of input feature maps are generated.

[0171] In this embodiment, the feature interaction fusion process of the second feature interaction module is similar to that of the first feature interaction module, except that the fusion feature S generated at the output position of each stage is i They are respectively determined as input feature maps S2, S3 and S4, and are respectively input into the neck network for subsequent fault detection processing. The specific processing process will not be repeated here.

[0172] Step 103, using the target multi-branch fault detection model to perform fault detection using all input feature maps to determine whether the photovoltaic panel to be detected has a fault;

[0173] The target multi-branch fault detection model refers to a model that includes multiple branch networks and fault detection networks. Each branch network has interactive fusion of features at the output position of each stage and outputs the fused input feature map respectively, and performs fault detection in combination with each input feature map. For the specific model structure, see Figure 2 As shown in the figure, the model includes at least three feature extraction branches, a neck network and a detection head network. The feature extraction branch includes an even-numbered visible light branch and an odd-numbered infrared branch as well as a feature interaction module that associates each branch. Among them, the visible light branch includes a visible light CNN branch using YOLOv7 as the main trunk and a visible light Swin Transformer branch using Swin Transformer as the main trunk. The infrared branch is an infrared branch improved on the three-dimensional attention module (TDAM).

[0174] In this embodiment, after completing the construction and training of the target multi-branch fault detection model, the visible light-infrared image pair is input into the model, and the visible light-infrared image pair is split into visible light images and infrared images through the model for feature extraction respectively, and iterative feature interaction is performed at the output position of each stage to generate an input feature map, and the input feature map of each stage is input into the fault detection network in the model, and fault detection is performed by the neck network and the detection head network of the fault detection network combined with input feature maps of different scales, so as to determine whether there is a fault in the photovoltaic panel to be detected.

[0175] In an example of the present application, the target multi-branch fault detection model further includes a neck network and a detection head network connected in sequence; step 103 may include the following sub-steps:

[0176] The neck network receives multiple input feature maps and sequentially performs up-sampling feature fusion and down-sampling feature fusion to obtain multiple enhanced feature maps;

[0177] Each enhanced feature map is reparameterized through the neck network and then input into the detection head network;

[0178] The detection head network is used to determine whether there is a target that meets the preset fault category from each enhanced feature map;

[0179] If so, it is determined that the photovoltaic panel to be tested is faulty;

[0180] If not, it is determined that the photovoltaic panel to be tested has no fault.

[0181] like Figure 7 As shown, in the embodiment of the present application, the second, third and fourth feature interaction modules are input into the neck network through the input feature maps S2, S3 and S4 before split.

[0182] The Neck network uses the FPN and PANet structures. In FPN, for the features S2, S3, and S4 input to the Neck, S4 is first upsampled and then fused with S3 to obtain the intermediate feature P; then the fused feature P is upsampled and fused with S2 to obtain the feature H1. In the PANet structure, H1 is downsampled and fused with P to obtain the feature H2; then the feature H2 is downsampled and fused with S3 to obtain the feature H3. The three enhanced feature maps H1, H2, and H3 of different sizes are respectively processed by the RepConv layer for reparameterization and input into the detection head network Head module. The Head module contains three detection heads, each of which consists of three convolutional layers, and each convolutional layer is followed by a BN layer and an activation function. The Head module converts the three input feature maps into detection results, including bounding boxes, categories, and confidences.

[0183] Step 104: If yes, the fault location is marked and the fault category is output through the target multi-branch fault detection model;

[0184] When the target multi-branch fault detection model determines that there is a fault in the photovoltaic panel to be detected, the model will further mark the fault location in the fused feature image input by the fault detection, and classify it according to the different fault locations to determine the fault category corresponding to the fault location.

[0185] Optionally, during the detection process, if a fault is detected in the photovoltaic panel to be detected, an alarm may be issued at the same time.

[0186] Step 105: If not, continue to monitor whether a new visible light-infrared image pair is received.

[0187] If the target multi-branch fault detection model determines that the photovoltaic panel to be detected has no fault, in order to facilitate periodic fault monitoring of the photovoltaic panel, it can continuously monitor whether a new visible light-infrared image pair is received, and execute the image pair reception trigger method.

[0188] In the embodiment of the present application, a visible light-infrared image pair of the photovoltaic panel to be detected is obtained; image features are extracted from the visible light-infrared image pair through a preset target multi-branch fault detection model and iterative feature interaction is performed to generate multiple input feature maps; fault detection is performed using all input feature maps through the target multi-branch fault detection model to determine whether the photovoltaic panel to be detected has a fault; if so, the fault location is marked and the fault category is output through the target multi-branch fault detection model; if not, continuous monitoring is performed to determine whether a new visible light-infrared image pair is received. Thus, through the multi-branch structure of the fault detection model, feature interaction is performed from different image dimensions to adapt to the diverse situations of photovoltaic panel layout scenarios, effectively improving the accuracy of photovoltaic panel fault detection.

[0189] See also Fig. 9 , Fig. 9 A structural block diagram of a photovoltaic panel fault detection device based on multi-branch feature interaction according to an embodiment of the present application is shown.

[0190] The embodiment of the present invention provides a photovoltaic panel fault detection device based on multi-branch feature interaction, comprising:

[0191] An image pair acquisition module 901 is used to acquire a visible light-infrared image pair of a photovoltaic panel to be inspected;

[0192] A feature map generation module 902 is used to call a preset target multi-branch fault detection model to extract image features from a visible light-infrared image pair and perform iterative feature interaction to generate multiple input feature maps;

[0193] The fault judgment module 903 is used to call the target multi-branch fault detection model to use all input feature graphs to perform fault detection and determine whether the photovoltaic panel to be detected has a fault;

[0194] A first determination module 904 is used to mark the fault location and output the fault category through the target multi-branch fault detection model if yes;

[0195] The second determination module 905 is configured to, if not, continue to monitor whether a new visible light-infrared image pair is received.

[0196] Optionally, the device further comprises:

[0197] A data set creation module, used to obtain multiple visible light-infrared training image pairs of photovoltaic panels and create a detection data set;

[0198] A branch embedding module, used for embedding multiple stage output positions of the first visible light branch, the second visible light branch and the infrared branch with multiple feature interaction modules;

[0199] A model connection module, used to sequentially connect the neck network and the detection head network at the output position of the stage of generating the input feature map, to create an initial multi-branch fault detection model;

[0200] The model training module is used to train the initial multi-branch fault detection model using the detection data set to obtain the target multi-branch fault detection model.

[0201] Optionally, the model training module is specifically used to:

[0202] Divide the detection data set into training set, validation set and test set according to the preset ratio;

[0203] Input the training set into the initial multi-branch fault detection model to generate fault prediction results;

[0204] According to the fault prediction results and the actual fault results corresponding to the training set, the rectangular box loss, confidence loss and classification loss are calculated respectively and weightedly superimposed to obtain the target loss value;

[0205] Adjusting parameters of the initial multi-branch fault detection model according to the target loss value until the target loss value converges to obtain a first multi-branch fault detection model;

[0206] Using the validation set to adjust the hyperparameters of the first multi-branch fault detection model, to obtain a plurality of second multi-branch fault detection models with different hyperparameters;

[0207] The test set is used to input each second multi-branch fault detection model respectively, and the second multi-branch fault detection model with the highest average accuracy is selected as the target multi-branch fault detection model.

[0208] Optionally, the visible light-infrared image pair includes a visible light image and an infrared image, and the target multi-branch fault detection model includes a first stage module and a plurality of second stage modules respectively connected to the feature interaction modules; the feature map generation module 902 includes:

[0209] The first interaction submodule is used to extract image features from the visible light image and the infrared image respectively through the first stage module and perform feature interaction in the feature interaction module to generate multiple interaction features and input them into the second stage module;

[0210] The second interaction submodule is used to perform iterative feature interaction on multiple interactive features through multiple second-stage modules to generate input feature maps respectively.

[0211] Optionally, the first stage module includes a first CNN branch, a first Swin Transformer branch and a first infrared branch connected in parallel to the first feature interaction module; the first interaction submodule includes:

[0212] A first visible light extraction unit, configured to extract a first visible light feature from the visible light image through a first CNN branch and input the first visible light feature into a first feature interaction module;

[0213] A second visible light extraction unit, used for extracting a second visible light feature from the visible light image through the first Swin Transformer branch and inputting the second visible light feature into the first feature interaction module;

[0214] An infrared extraction unit, used for extracting a first infrared feature from the infrared image through a first infrared branch and inputting the first infrared feature into a first feature interaction module;

[0215] A fusion unit, configured to perform feature interaction using the first visible light feature, the second visible light feature and the first infrared feature through a first feature interaction module to generate a single fused feature;

[0216] A channel separation unit, configured to perform a channel separation operation on the fused feature through the first feature interaction module to obtain a first shared feature, a second shared feature, and a third shared feature;

[0217] The interactive superposition unit is used to superimpose the first visible light feature and the first shared feature, the second visible light feature and the second shared feature, and the first infrared feature and the third shared feature through the first feature interactive module to obtain multiple interactive features and input them into the second stage module.

[0218] Optionally, the first feature interaction module includes a fusion part and a dense splicing part connected in sequence; and a fusion unit is specifically used to:

[0219] The fusion part performs convolution, batch normalization and function activation on the first visible light feature, the second visible light feature and the first infrared feature in sequence, and then concatenates them to obtain a single first intermediate feature;

[0220] Performing batch normalization, function activation, and convolution on the first intermediate feature through the dense concatenation part to obtain a second intermediate feature;

[0221] Partially joining the second intermediate feature and all existing first intermediate features by dense joining to obtain a third intermediate feature as a new first intermediate feature;

[0222] The jump execution performs batch normalization, function activation and convolution on the first intermediate feature through the dense splicing part to obtain the step of the second intermediate feature until the preset number of iterations is reached, and the third intermediate feature at the current moment is determined as the fusion feature.

[0223] Optionally, the second stage module includes multiple groups of second CNN branches, second Swin Transformer branches and second infrared branches connected in parallel to the second feature interaction module, and the branch architecture of each group of second Swin Transformer branches is different; the second interaction submodule is specifically used for:

[0224] The third visible light feature, the fourth visible light feature and the second infrared feature are respectively extracted from each interaction feature through the second CNN branch, the second Swin Transformer branch and the second infrared branch, and input into the second feature interaction module;

[0225] The third visible light feature, the fourth visible light feature and the second infrared feature are used to perform feature interaction through the second feature interaction module to generate an input feature map;

[0226] Performing a channel separation operation on the input feature map through a second feature interaction module to obtain a fourth shared feature, a fifth shared feature, and a sixth shared feature;

[0227] The third visible light feature and the fourth shared feature, the fourth visible light feature and the fifth shared feature, and the second infrared feature and the sixth shared feature are respectively superimposed by the second feature interaction module to obtain a plurality of new interaction features;

[0228] The jump execution extracts the third visible light feature, the fourth visible light feature and the second infrared feature from the interactive features respectively through the second CNN branch, the second Swin Transformer branch and the second infrared branch, and inputs them into the second feature interaction module until a preset number of input feature maps are generated.

[0229] Optionally, the target multi-branch fault detection model further includes a neck network and a detection head network connected in sequence; a fault judgment module 903, specifically used for:

[0230] The neck network receives multiple input feature maps and sequentially performs up-sampling feature fusion and down-sampling feature fusion to obtain multiple enhanced feature maps;

[0231] Each enhanced feature map is reparameterized through the neck network and then input into the detection head network;

[0232] The detection head network is used to determine whether there is a target that meets the preset fault category from each enhanced feature map;

[0233] If so, it is determined that the photovoltaic panel to be tested is faulty;

[0234] If not, it is determined that the photovoltaic panel to be detected has no fault.

[0235] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules, sub-modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0236] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0237] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0238] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0239] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0240] This embodiment provides a terminal device, which can be a computer. Fig.10As shown, a processor 802, a memory, an input device 804, a display 805 and a network interface 806 are connected via a system bus 801. The processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium 803 and an internal memory 807. The non-volatile storage medium 803 stores an operating system, a computer program and a database. The internal memory 807 provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. When the processor 802 executes the computer program stored in the memory, the photovoltaic panel fault detection method based on multi-branch feature interaction of any of the above embodiments is implemented.

[0241] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A photovoltaic panel fault detection method based on multi-branch feature interaction, characterized in that: include: Acquire a visible light-infrared image pair of the photovoltaic panel to be inspected; Extracting image features from the visible light-infrared image pair and performing iterative feature interaction through a preset target multi-branch fault detection model to generate a plurality of input feature maps; Using the target multi-branch fault detection model to perform fault detection using all the input feature graphs, determining whether the photovoltaic panel to be detected has a fault; If yes, then mark the fault location and output the fault category through the target multi-branch fault detection model; If not, continue to monitor whether a new visible light-infrared image pair is received.

2. The method according to claim 1, characterized in that The method further comprises: Obtain multiple visible-infrared training image pairs of photovoltaic panels to create a detection dataset; Embed multiple stage output positions of the first visible light branch, the second visible light branch and the infrared branch with multiple feature interaction modules; Connecting the neck network and the detection head network in sequence at the output position of the stage of generating the input feature map to create an initial multi-branch fault detection model; The initial multi-branch fault detection model is trained using the detection data set to obtain a target multi-branch fault detection model.

3. The method according to claim 2, characterized in that The adopting the detection data set to train the initial multi-branch fault detection model to obtain a target multi-branch fault detection model includes: Dividing the detection data set into a training set, a validation set and a test set according to a preset ratio; Inputting the training set into the initial multi-branch fault detection model to generate a fault prediction result; According to the fault prediction result and the actual fault result corresponding to the training set, respectively calculate the rectangular box loss, confidence loss and classification loss and perform weighted superposition to obtain a target loss value; Adjusting parameters of the initial multi-branch fault detection model according to the target loss value until the target loss value converges to obtain a first multi-branch fault detection model; Using the validation set to adjust the hyperparameters of the first multi-branch fault detection model, to obtain a plurality of second multi-branch fault detection models with different hyperparameters; The test set is used to input each of the second multi-branch fault detection models respectively, and the second multi-branch fault detection model with the highest average accuracy is selected as the target multi-branch fault detection model.

4. The method according to claim 1, characterized in that: The visible light-infrared image pair includes a visible light image and an infrared image, and the target multi-branch fault detection model includes a first stage module and a plurality of second stage modules respectively connected to feature interaction modules; The method extracts image features from the visible light-infrared image pair through a preset target multi-branch fault detection model and performs iterative feature interaction to generate multiple input feature maps, including: Extracting image features from the visible light image and the infrared image respectively through the first stage module and performing feature interaction in the feature interaction module to generate a plurality of interactive features and input them into the second stage module; Iterative feature interaction is performed on the plurality of interactive features through the plurality of the second-stage modules to generate input feature maps respectively.

5. The method according to claim 4, characterized in that The first stage module includes a first CNN branch, a first Swin Transformer branch, and a first infrared branch connected in parallel to a first feature interaction module; extracting image features from the visible light image and the infrared image respectively through the first stage module and performing feature interaction in the feature interaction module to generate multiple interaction features and input them into the second stage module, including: Extracting a first visible light feature from the visible light image through the first CNN branch and inputting the first visible light feature into the first feature interaction module; Extracting a second visible light feature from the visible light image through the first Swin Transformer branch and inputting the second visible light feature into the first feature interaction module; Extracting a first infrared feature from the infrared image through the first infrared branch and inputting the first infrared feature into the first feature interaction module; The first feature interaction module uses the first visible light feature, the second visible light feature and the first infrared feature to perform feature interaction to generate a single fusion feature; Performing a channel separation operation on the fused features through the first feature interaction module to obtain a first shared feature, a second shared feature, and a third shared feature; The first visible light feature and the first shared feature, the second visible light feature and the second shared feature, and the first infrared feature and the third shared feature are respectively superimposed through the first feature interaction module to obtain multiple interactive features and input them into the second stage module.

6. The method according to claim 5, characterized in that The first feature interaction module includes a fusion part and a dense splicing part connected in sequence; the first feature interaction module uses the first visible light feature, the second visible light feature and the first infrared feature to perform feature interaction to generate a single fusion feature, including: After sequentially performing convolution, batch normalization, and function activation on the first visible light feature, the second visible light feature, and the first infrared feature through the fusion part, a single first intermediate feature is obtained by splicing them; Performing batch normalization, function activation, and convolution on the first intermediate feature through the dense concatenation part to obtain a second intermediate feature; Splicing the second intermediate feature with all the existing first intermediate features through the dense splicing part to obtain a third intermediate feature as a new first intermediate feature; Jump to execute the step of performing batch normalization, function activation and convolution on the first intermediate feature through the dense splicing part to obtain the second intermediate feature until the preset number of iterations is reached, and determine the third intermediate feature at the current moment as the fusion feature.

7. The method according to claim 4, characterized in that The second stage module includes multiple groups of second CNN branches, second Swin Transformer branches and second infrared branches connected in parallel to the second feature interaction module, and the branch architecture of each group of the second Swin Transformer branches is different; The iterative feature interaction of the plurality of interactive features by the plurality of the second-stage modules to generate input feature maps respectively includes: Extract the third visible light feature, the fourth visible light feature and the second infrared feature from each of the interaction features through the second CNN branch, the second Swin Transformer branch and the second infrared branch, and input them into the second feature interaction module; The second feature interaction module uses the third visible light feature, the fourth visible light feature and the second infrared feature to perform feature interaction to generate an input feature map; Performing a channel separation operation on the input feature map through the second feature interaction module to obtain a fourth shared feature, a fifth shared feature, and a sixth shared feature; Superimposing the third visible light feature and the fourth shared feature, the fourth visible light feature and the fifth shared feature, and the second infrared feature and the sixth shared feature respectively through the second feature interaction module to obtain a plurality of new interaction features; The jump execution extracts the third visible light feature, the fourth visible light feature and the second infrared feature from the interaction feature respectively through the second CNN branch, the second Swin Transformer branch and the second infrared branch, and inputs them into the second feature interaction module until a preset number of input feature maps are generated.

8. The method according to claim 1, characterized in that The target multi-branch fault detection model also includes a neck network and a detection head network connected in sequence; the target multi-branch fault detection model uses all the input feature graphs to perform fault detection to determine whether the photovoltaic panel to be detected has a fault, including: The neck network receives the plurality of input feature maps and sequentially performs up-sampling feature fusion and down-sampling feature fusion to obtain a plurality of enhanced feature maps; Re-parameterizing each of the enhanced feature maps through the neck network and inputting the re-parameterized feature maps into the detection head network; Determine whether there is a target that meets the preset fault category from each of the enhanced feature maps through the detection head network; If yes, it is determined that the photovoltaic panel to be detected has a fault; If not, it is determined that the photovoltaic panel to be detected has no fault.

9. A photovoltaic panel fault detection device based on multi-branch feature interaction, characterized in that: include: An image pair acquisition module, used to acquire a visible light-infrared image pair of the photovoltaic panel to be inspected; A feature map generation module is used to call a preset target multi-branch fault detection model to extract image features from the visible light-infrared image pair and perform iterative feature interaction to generate multiple input feature maps; A fault judgment module, used to call the target multi-branch fault detection model to use all the input feature graphs to perform fault detection, and judge whether the photovoltaic panel to be detected has a fault; A first determination module, configured to, if yes, mark the fault location and output the fault category through the target multi-branch fault detection model; The second determination module is used for, if not, continuously monitoring whether a new visible light-infrared image pair is received.

Images