Fusion method and device for infrared image and visible light image of photovoltaic module

Abstract

The invention relates to the technical field of image data processing, and discloses a fusion method and device for an infrared image and a visible light image of a photovoltaic module, and the method comprises the steps: carrying out the shooting of a whole region of a photovoltaic station based on a thermal infrared camera and a visible light camera carried by an unmanned plane, and obtaining a photovoltaic infrared image and a photovoltaic visible light image; performing registration processing on the photovoltaic infrared image and the photovoltaic visible light image according to an edge rectangular feature method to obtain a photovoltaic double-light-source image; inputting the photovoltaic dual-light-source image into a dual-discriminator generative adversarial network, outputting a target fusion low-rank image and a target fusion sparse detail image, and generating a target fusion image according to a weighted average fusion strategy; the target fusion image is subjected to fault type labeling, according to the YOLO series target detection model, the fault type and position of the target fusion image are detected after training is completed, it can be guaranteed that the obtained target fusion image has clear texture features and temperature intensity information, and powerful data support is provided for subsequent fault detection.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a method and device for fusing an infrared image and a visible light image of a photovoltaic module. Background Art

[0002] Photovoltaic modules are the core power generation components of photovoltaic power stations. Whether they are abnormal or not will directly affect the power generation and revenue of the entire photovoltaic power station, and even affect the safety of the entire photovoltaic power station. Photovoltaic modules in each photovoltaic power station will have abnormalities. Therefore, how to quickly find abnormal photovoltaic modules in a running photovoltaic power station is particularly important for the operation and maintenance of photovoltaic power stations.

[0003] In recent years, with the rapid development of deep learning technology, more and more target detection algorithms related to it have emerged. In the prior art, according to whether there is a region proposal generation stage, the detection algorithms based on deep learning can be divided into two categories: detection algorithms based on region proposals and detection algorithms not based on region proposals. The detection algorithm based on region proposals first generates region proposals, and then uses a deep learning network to extract features to obtain the category information and coordinate information corresponding to the features. This type of algorithm has high detection accuracy, but slow speed. The main algorithms include SPP-NET and Faster R-CNN. The detection algorithm not based on region proposals does not generate candidate boxes, and directly uses a deep learning network to predict the possibility of the existence of a target at each position in the image. This type of algorithm has the advantages of fast speed and strong generalization ability. The main algorithms include SSD and YOLO series. All of the above target detection algorithms detect a single image of a photovoltaic module, and do not consider combining the temperature intensity information of the infrared image with the detail texture information of the visible light image to perform fault detection on the fused image, thereby ensuring that the abnormal defects of the component are fully detected. Summary of the invention

[0004] The embodiments of the present invention provide a method and device for fusing infrared images and visible light images of photovoltaic modules, which are used to solve the problems of insufficient information fusion of infrared images and visible light images and low fault detection accuracy in the prior art.

[0005] In order to achieve the above object, the present invention provides a method for fusing infrared images and visible light images of photovoltaic modules, comprising:

[0006] Based on the pre-deployed drones equipped with thermal infrared cameras and visible light cameras, the whole area of ​​the photovoltaic station is inspected and photographed to obtain photovoltaic infrared images and photovoltaic visible light images of the photovoltaic power station strings;

[0007] Performing registration processing on the photovoltaic infrared image and the photovoltaic visible light image according to the edge rectangle feature method to obtain the photovoltaic dual light source image of the photovoltaic power station string;

[0008] The photovoltaic dual-light source image is used as input data, input into a dual-discriminator generative adversarial network, and based on the dual-discriminator generative adversarial network output, a target fused low-rank image and a target fused sparse detail image are output, and a target fused image of the photovoltaic power station string is generated according to a weighted average fusion strategy;

[0009] The target fusion image is labeled with the fault type based on a preset tool, and a YOLO series target detection model is trained according to the image data of the target fusion image. After the training is completed, the fault type and location of the target fusion image are detected.

[0010] Furthermore, when the photovoltaic infrared image and the photovoltaic visible light image are registered according to the edge rectangle feature method to obtain the photovoltaic dual light source image of the photovoltaic power station string, it includes:

[0011] Preprocessing the photovoltaic infrared image and the photovoltaic visible light image respectively, wherein the preprocessing includes removing noise and interference;

[0012] Extracting first edge information and second edge information of the photovoltaic infrared image and the photovoltaic visible light image based on an edge detection algorithm, and generating a first edge rectangle and a second edge rectangle according to the first edge information and the second edge information;

[0013] Converting the first edge rectangle into the frequency domain by Fourier transform, and extracting the corresponding first amplitude value in the frequency domain;

[0014] Converting the second edge rectangle into the frequency domain by Fourier transform, and extracting the corresponding second amplitude value in the frequency domain;

[0015] Calculating edge matching coefficients of the first edge rectangle and the second edge rectangle according to the first amplitude value and the second amplitude value;

[0016] Determining whether the photovoltaic infrared image can be mapped to the photovoltaic visible light image according to the relationship between the edge matching degree coefficient and a preset edge matching degree coefficient;

[0017] When the edge matching degree coefficient is greater than or equal to the preset edge matching degree coefficient, it is determined that the photovoltaic infrared image can be mapped to the photovoltaic visible light image to obtain the photovoltaic dual-light source image of the photovoltaic power station string;

[0018] When the edge matching degree coefficient is less than the preset edge matching degree coefficient, it is determined that the photovoltaic infrared image cannot be mapped to the photovoltaic visible light image, and the photovoltaic infrared image and the photovoltaic visible light image are re-photographed until the edge matching degree coefficient is greater than or equal to the preset edge matching degree coefficient.

[0019] Further, when calculating the edge matching degree coefficient of the first edge rectangle and the second edge rectangle according to the first amplitude value and the second amplitude value, it includes:

[0020] Constructing a first amplitude value sequence according to all first amplitude values, and constructing a second amplitude value sequence according to all second amplitude values;

[0021] Calculate a first sub-edge matching degree coefficient and a second sub-edge matching degree coefficient of the first edge rectangle and the second edge rectangle based on the first amplitude value sequence and the second amplitude value sequence;

[0022] The edge matching degree coefficients of the first edge rectangle and the second edge rectangle are calculated according to the first sub-edge matching degree coefficient and the second sub-edge matching degree coefficient.

[0023] Further, when calculating the first sub-edge matching degree coefficient of the first edge rectangle and the second edge rectangle based on the first amplitude value sequence and the second amplitude value sequence, it includes:

[0024] Calculate a first sequence mean corresponding to the first amplitude value sequence, and calculate a second sequence mean corresponding to the second amplitude value sequence;

[0025] Calculate the sequence mean difference between the first sequence mean and the second sequence mean;

[0026] Compare the first amplitude value sequence and the second amplitude value sequence one by one to determine a plurality of amplitude value differences;

[0027] Extracting a maximum amplitude value difference and a minimum amplitude value difference from all amplitude value differences, and calculating an extreme amplitude value difference of the maximum amplitude value difference and the minimum amplitude value difference;

[0028] The first sub-edge matching degree coefficient is calculated according to the sequence mean difference, extreme amplitude value difference, maximum amplitude value difference and minimum amplitude value difference.

[0029] Further, when calculating the first sub-edge matching degree coefficient according to the sequence mean difference, extreme amplitude value difference, maximum amplitude value difference and minimum amplitude value difference, it includes:

[0030] The first sub-edge matching degree coefficient is calculated according to the following formula:

[0031]

[0032] Among them, q is the first sub-edge matching degree coefficient, r1 is the sequence mean difference, r2 is the extreme amplitude difference, r3 is the maximum amplitude difference, and r4 is the minimum amplitude difference.

[0033] Further, when calculating the second sub-edge matching degree coefficient of the first edge rectangle and the second edge rectangle based on the first amplitude value sequence and the second amplitude value sequence, it includes:

[0034] Randomly matching the amplitude values ​​in the first amplitude value sequence and the second amplitude value sequence in pairs to obtain a plurality of randomly matched amplitude value sets;

[0035] The second sub-edge matching coefficient is calculated according to the following formula:

[0036]

[0037] Among them, t is the second sub-edge matching coefficient, n1 is the number of random matching sets of amplitude values, and a u is the larger amplitude value in the random matching set of the u-th amplitude value, d u is the smaller amplitude value in the random matching set of the u-th amplitude value, n2 is the number of the first amplitude values ​​in the first amplitude value sequence, g f is the fth first amplitude value in the first amplitude value sequence, n3 is the number of second amplitude values ​​in the second amplitude value sequence, k h is the hth second amplitude value in the second amplitude value sequence.

[0038] Further, when calculating the edge matching degree coefficient of the first edge rectangle and the second edge rectangle according to the first sub-edge matching degree coefficient and the second sub-edge matching degree coefficient, it includes:

[0039] configuring a first calculation coefficient for the first sub-edge matching degree coefficient, and configuring a second calculation coefficient for the first sub-edge matching degree coefficient;

[0040] The edge matching coefficient of the first edge rectangle and the second edge rectangle is calculated according to the following formula:

[0041] v = m1 × q + m2 × t;

[0042] Wherein, v is the edge matching degree coefficient of the first edge rectangle and the second edge rectangle, m3 is the first calculation coefficient, m2 is the second calculation coefficient, and m1+m2=1, m1>m2.

[0043] Further, when the photovoltaic dual-light source image is used as input data, input into a dual-discriminator generative adversarial network, and based on the dual-discriminator generative adversarial network outputting a target fused low-rank image and a target fused sparse detail image, a target fused image of the photovoltaic power station string is generated according to a weighted average fusion strategy, including:

[0044] Decomposing the photovoltaic dual-light source image by low-rank sparse decomposition to obtain a low-rank part and a sparse detail part;

[0045] Input the low-rank part as input data to a first SN-CNN discriminator, and input the sparse detail part as input data to a second SN-CNN discriminator;

[0046] Optimizing the first SN-CNN discriminator and the second SN-CNN discriminator based on a local binary pattern algorithm and an improved Wasserstein distance algorithm;

[0047] Outputting the target fused low-rank image based on the optimized first SN-CNN discriminator, wherein the target fused low-rank image can save temperature intensity information and detail information from the infrared image;

[0048] The target fused sparse detail image is output based on the optimized second SN-CNN discriminator, wherein the target fused sparse detail image can preserve texture information and detail information from the visible light image.

[0049] Further, when optimizing the first SN-CNN discriminator and the second SN-CNN discriminator based on the local binary pattern algorithm and the improved Wasserstein distance algorithm, it includes:

[0050] The expression of the local binary pattern algorithm is as follows:

[0051]

[0052] Among them, LBP(x c ,y c ) is the central pixel, p is the number of neighboring pixels of the central pixel, i p is the neighborhood pixel intensity value of the pth central pixel, i c is the pixel intensity value of the center pixel, when i p -i c ≥0, then s(i p -i c )=1, when i p -i c <0, then s(i p -i c )=0;

[0053] The expression of the improved Wasserstein distance algorithm is as follows:

[0054]

[0055] in, is the objective function of the generative adversarial network, E x～pr For the real data distribution p r The expectation is to calculate the expectation of all data x sampled from the real data distribution. D(x) is the output of the discriminator for the input x, which is used to determine whether the input x comes from real data or generated data. D is the discriminator, E z～pz To generate data distribution p z The expectation is to calculate the expectation of all noise z sampled from the generated data distribution. D(G(z)) is the output of the generator for the input noise z, that is, the generated data. G is the generator, μ is the penalty coefficient, For linear interpolation data expectations, is the change in x, P r and P z are the real data distribution and the generated data distribution respectively, The discriminator input Output.

[0056] In order to achieve the above-mentioned object, the present invention also provides a device for fusing infrared images and visible light images of photovoltaic modules, characterized in that:

[0057] Image capture module, used to inspect and capture the entire area of ​​the photovoltaic station based on the pre-deployed drone equipped with thermal infrared camera and visible light camera, and obtain photovoltaic infrared images and photovoltaic visible light images of the photovoltaic power station strings;

[0058] An image registration module is used to perform registration processing on the photovoltaic infrared image and the photovoltaic visible light image according to the edge rectangle feature method to obtain the photovoltaic dual light source image of the photovoltaic power station string;

[0059] An image processing module, used to take the photovoltaic dual-light source image as input data, input it into a dual-discriminator generative adversarial network, and output a target fused low-rank image and a target fused sparse detail image based on the dual-discriminator generative adversarial network, and generate a target fused image of the photovoltaic power station string according to a weighted average fusion strategy;

[0060] The image analysis module is used to annotate the fault type of the target fusion image based on a preset tool, and train a YOLO series target detection model according to the image data of the target fusion image, and detect the fault type and location of the target fusion image after the training is completed.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] The present invention discloses a method and device for fusing infrared images and visible light images of photovoltaic components. The method comprises the following steps: photographing the entire area of ​​a photovoltaic station based on a thermal infrared camera and a visible light camera carried by an unmanned aerial vehicle to obtain a photovoltaic infrared image and a photovoltaic visible light image; registering the photovoltaic infrared image and the photovoltaic visible light image according to an edge rectangle feature method to obtain a photovoltaic dual-light source image; inputting the photovoltaic dual-light source image into a dual-discriminator generative adversarial network to output a target fused low-rank image and a target fused sparse detail image, and generating a target fused image according to a weighted average fusion strategy; annotating the target fused image for the fault type, and detecting the fault type and position of the target fused image after training according to a YOLO series target detection model. The present invention can ensure that the obtained target fused image has clear texture features and temperature intensity information, and provide strong data support for subsequent fault detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0064] Figure 1 A schematic flow chart of a method for fusing an infrared image of a photovoltaic module with a visible light image in an embodiment of the present invention is shown;

[0065] Figure 2 A schematic diagram of the structure of a device for fusing infrared images and visible light images of a photovoltaic module in an embodiment of the present invention is shown;

[0066] Figure 3 The infrared and visible light image fusion algorithm framework of the improved dual discriminator generative adversarial network in the embodiment of the present invention is shown;

[0067] Figure 4 The generator network structure in the embodiment of the present invention is shown;

[0068] Figure 5 The discriminator network structure in the embodiment of the present invention is shown. DETAILED DESCRIPTION

[0069] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0070] In the description of the present application, it should be understood that the terms "center", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0071] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0072] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0073] The following is a description of preferred embodiments of the present invention with reference to the accompanying drawings.

[0074] like Figure 1 As shown, an embodiment of the present invention discloses a method for fusing an infrared image and a visible light image of a photovoltaic module, comprising:

[0075] S110: Based on the pre-deployed drone equipped with thermal infrared camera and visible light camera, the whole area of ​​the photovoltaic station is inspected and photographed to obtain photovoltaic infrared images and photovoltaic visible light images of the photovoltaic power station strings;

[0076] In this embodiment, a thermal infrared camera mounted on a drone is used to capture photovoltaic infrared images of photovoltaic power station strings, and a visible light camera is used to capture photovoltaic visible light images of photovoltaic power station strings.

[0077] S120: performing registration processing on the photovoltaic infrared image and the photovoltaic visible light image according to the edge rectangle feature method to obtain a photovoltaic dual-light source image of the photovoltaic power station string;

[0078] In some embodiments of the present application, when the photovoltaic infrared image and the photovoltaic visible light image are registered according to the edge rectangle feature method to obtain the photovoltaic dual light source image of the photovoltaic power station string, it includes:

[0079] Preprocessing the photovoltaic infrared image and the photovoltaic visible light image respectively, wherein the preprocessing includes removing noise and interference;

[0080] Extracting first edge information and second edge information of the photovoltaic infrared image and the photovoltaic visible light image based on an edge detection algorithm, and generating a first edge rectangle and a second edge rectangle according to the first edge information and the second edge information;

[0081] Converting the first edge rectangle into the frequency domain by Fourier transform, and extracting the corresponding first amplitude value in the frequency domain;

[0082] Converting the second edge rectangle into the frequency domain by Fourier transform, and extracting the corresponding second amplitude value in the frequency domain;

[0083] Calculating edge matching coefficients of the first edge rectangle and the second edge rectangle according to the first amplitude value and the second amplitude value;

[0084] Determining whether the photovoltaic infrared image can be mapped to the photovoltaic visible light image according to the relationship between the edge matching degree coefficient and a preset edge matching degree coefficient;

[0085] When the edge matching degree coefficient is greater than or equal to the preset edge matching degree coefficient, it is determined that the photovoltaic infrared image can be mapped to the photovoltaic visible light image to obtain the photovoltaic dual-light source image of the photovoltaic power station string;

[0086] When the edge matching degree coefficient is less than the preset edge matching degree coefficient, it is determined that the photovoltaic infrared image cannot be mapped to the photovoltaic visible light image, and the photovoltaic infrared image and the photovoltaic visible light image are re-photographed until the edge matching degree coefficient is greater than or equal to the preset edge matching degree coefficient.

[0087] In this embodiment, the edge detection algorithm includes one or more of a Canny edge detection algorithm, a Sobel edge detection algorithm, or a Laplacian edge detection algorithm.

[0088] In this embodiment, a morphological transformation method is used to connect edge points to form a closed edge rectangle.

[0089] In this embodiment, the reference points in the photovoltaic infrared image are mapped onto the photovoltaic visible light image to complete the image registration and obtain the photovoltaic dual-light source image of the photovoltaic power station string.

[0090] The beneficial effect of the above technical solution is that the present invention determines whether the photovoltaic infrared image can be mapped to the photovoltaic visible light image based on the relationship between the edge matching degree coefficient and the preset edge matching degree coefficient, thereby ensuring the determination accuracy of the photovoltaic dual-light source image and laying the foundation for subsequent image fusion.

[0091] In some embodiments of the present application, when calculating the edge matching degree coefficient of the first edge rectangle and the second edge rectangle according to the first amplitude value and the second amplitude value, it includes:

[0092] Constructing a first amplitude value sequence according to all first amplitude values, and constructing a second amplitude value sequence according to all second amplitude values;

[0093] Calculate a first sub-edge matching degree coefficient and a second sub-edge matching degree coefficient of the first edge rectangle and the second edge rectangle based on the first amplitude value sequence and the second amplitude value sequence;

[0094] The edge matching degree coefficients of the first edge rectangle and the second edge rectangle are calculated according to the first sub-edge matching degree coefficient and the second sub-edge matching degree coefficient.

[0095] In some embodiments of the present application, when calculating the first sub-edge matching degree coefficient of the first edge rectangle and the second edge rectangle based on the first amplitude value sequence and the second amplitude value sequence, it includes:

[0096] Calculate a first sequence mean corresponding to the first amplitude value sequence, and calculate a second sequence mean corresponding to the second amplitude value sequence;

[0097] Calculate the sequence mean difference between the first sequence mean and the second sequence mean;

[0098] Compare the first amplitude value sequence and the second amplitude value sequence one by one to determine a plurality of amplitude value differences;

[0099] Extracting a maximum amplitude value difference and a minimum amplitude value difference from all amplitude value differences, and calculating an extreme amplitude value difference of the maximum amplitude value difference and the minimum amplitude value difference;

[0100] The first sub-edge matching degree coefficient is calculated according to the sequence mean difference, extreme amplitude value difference, maximum amplitude value difference and minimum amplitude value difference.

[0101] In this embodiment, the method for calculating the sequence mean is not repeated.

[0102] In this embodiment, the sequence mean difference is the absolute value of the difference between the first sequence mean and the second sequence mean.

[0103] The beneficial effect of the above technical solution is: the present invention calculates the first sub-edge matching degree coefficient according to the sequence mean difference, extreme amplitude value difference, maximum amplitude value difference and minimum amplitude value difference, thereby ensuring the calculation accuracy of the first sub-edge matching degree coefficient and providing a calculation basis for the calculation of the edge matching degree coefficient.

[0104] In some embodiments of the present application, when calculating the first sub-edge matching degree coefficient according to the sequence mean difference, the extreme amplitude value difference, the maximum amplitude value difference and the minimum amplitude value difference, it includes:

[0105] The first sub-edge matching degree coefficient is calculated according to the following formula:

[0106]

[0107] Among them, q is the first sub-edge matching degree coefficient, r1 is the sequence mean difference, r2 is the extreme amplitude difference, r3 is the maximum amplitude difference, and r4 is the minimum amplitude difference.

[0108] In some embodiments of the present application, when calculating the second sub-edge matching degree coefficient of the first edge rectangle and the second edge rectangle based on the first amplitude value sequence and the second amplitude value sequence, it includes:

[0109] Randomly matching the amplitude values ​​in the first amplitude value sequence and the second amplitude value sequence in pairs to obtain a plurality of randomly matched amplitude value sets;

[0110] The second sub-edge matching coefficient is calculated according to the following formula:

[0111]

[0112] Among them, t is the second sub-edge matching coefficient, n1 is the number of random matching sets of amplitude values, and a u is the larger amplitude value in the random matching set of the u-th amplitude value, d u is the smaller amplitude value in the random matching set of the u-th amplitude value, n2 is the number of the first amplitude values ​​in the first amplitude value sequence, g f is the fth first amplitude value in the first amplitude value sequence, n3 is the number of second amplitude values ​​in the second amplitude value sequence, k h is the hth second amplitude value in the second amplitude value sequence.

[0113] In this embodiment, the larger amplitude value in the u-th amplitude value random matching set may be the first amplitude value or the second amplitude value, and the larger amplitude value in the u-th amplitude value random matching set may be the first amplitude value or the second amplitude value, and the specific comparison can be based on actual conditions.

[0114] The beneficial effect of the above technical solution is: the present invention randomly matches the amplitude values ​​in the first amplitude value sequence and the second amplitude value sequence in pairs to obtain multiple randomly matched sets of amplitude values, thereby ensuring the objectivity of the calculation and avoiding errors. By calculating the second sub-edge matching degree coefficient, another calculation basis is provided for the calculation of the edge matching degree coefficient.

[0115] In some embodiments of the present application, when calculating the edge matching degree coefficients of the first edge rectangle and the second edge rectangle according to the first sub-edge matching degree coefficient and the second sub-edge matching degree coefficient, the method includes:

[0116] configuring a first calculation coefficient for the first sub-edge matching degree coefficient, and configuring a second calculation coefficient for the first sub-edge matching degree coefficient;

[0117] The edge matching coefficient of the first edge rectangle and the second edge rectangle is calculated according to the following formula:

[0118] v = m1 × q + m2 × t;

[0119] Wherein, v is the edge matching degree coefficient of the first edge rectangle and the second edge rectangle, m3 is the first calculation coefficient, m2 is the second calculation coefficient, and m1+m2=1, m1>m2.

[0120] The beneficial effect of the above technical solution is: the present invention calculates the edge matching degree coefficient of the first edge rectangle and the second edge rectangle according to the first sub-edge matching degree coefficient and the second sub-edge matching degree coefficient, which provides a registration basis for the registration processing of photovoltaic infrared images and photovoltaic visible light images, and ensures the accuracy and efficiency of the registration processing.

[0121] S130: taking the photovoltaic dual-light source image as input data, inputting it into a dual-discriminator generative adversarial network, and based on the dual-discriminator generative adversarial network outputting a target fused low-rank image and a target fused sparse detail image, generating a target fused image of the photovoltaic power station string according to a weighted average fusion strategy;

[0122] In some embodiments of the present application, when the photovoltaic dual-light source image is used as input data, input into a dual-discriminator generative adversarial network, and based on the dual-discriminator generative adversarial network outputting a target fused low-rank image and a target fused sparse detail image, a target fused image of the photovoltaic power station string is generated according to a weighted average fusion strategy, including:

[0123] Decomposing the photovoltaic dual-light source image by low-rank sparse decomposition to obtain a low-rank part and a sparse detail part;

[0124] Input the low-rank part as input data to a first SN-CNN discriminator, and input the sparse detail part as input data to a second SN-CNN discriminator;

[0125] Optimizing the first SN-CNN discriminator and the second SN-CNN discriminator based on a local binary pattern algorithm and an improved Wasserstein distance algorithm;

[0126] Outputting the target fused low-rank image based on the optimized first SN-CNN discriminator, wherein the target fused low-rank image can save temperature intensity information and detail information from the infrared image;

[0127] The target fused sparse detail image is output based on the optimized second SN-CNN discriminator, wherein the target fused sparse detail image can preserve texture information and detail information from the visible light image.

[0128] In this embodiment, the Transformer network replaces the original GAN ​​generator network structure, and a self-attention mechanism is added to enhance the correlation between each pixel, thereby improving the quality and visual effect of the fused image and improving the accuracy of fault detection.

[0129] In this embodiment, in GAN, a more expressive Transformer network is used as a generator network, and the low-rank part / sparse detail part of infrared and visible light is connected in the channel direction as the input of the generator based on the Transformer network. The image generated by the improved generator can preserve some infrared intensity information and background information of the visible light image to a certain extent. The present invention effectively eliminates the adverse effects caused by singular sample data by introducing a spectral normalization layer, and accelerates the convergence speed. This improvement is very obvious because it not only meets the Lipschitz condition, but also maintains the stability of the parameter matrix, which is uncommon in previous GAN models. Using a spectral normalization convolutional neural network (SN-CNN) as a discriminator network, the fused image and the infrared image / visible light image are used as the input of the improved dual discriminator (SN-CNN). When both improved discriminators cannot distinguish whether the input image is a generated image or a source image, it is considered that the fused image has well preserved the infrared intensity information and the detail information of the visible light. The present invention can significantly improve the type and accuracy of photovoltaic module fault detection by fusing infrared and visible light images.

[0130] In this embodiment, compared with the CNN generator network in the traditional GAN, the Transformer can establish the global dependency of the image, thereby obtaining more global information of the image. In addition, the structure of using the Transformer as the generator can give the model stronger expressive power and generalization ability. The CBAM module can be plug-and-play as a lightweight general module, and it is placed after the feature extraction module. The self-attention mechanism is introduced in the generator to enhance the dependency between each pixel, so that the fused image can better preserve the information in the infrared image and the visible light image. In order to better adapt to the image fusion task, the model of the present invention adjusts the traditional structure, removes the decoder, and only retains the encoder. Pruning destroys the structure of the discriminator weight matrix, so that the WGAN converges slowly. For this reason, the present invention introduces a spectral normalization layer in the CNN discriminator network in the traditional GAN, and limits the weight matrix of each layer of the network to a range, that is, by dividing each element in the parameter matrix by its spectral norm, so that the Lipschitz constant is 1. The SN-CNN discriminator can not only meet the Lipschitz condition, but also maintain the stability of the parameter matrix and accelerate training.

[0131] In this embodiment, the network includes a Transformer-based generator and two discriminators based on spectral normalized convolutional neural network (SN-CNN). This structure can not only meet the Lipschitz condition, but also maintain the stability of the parameter matrix and accelerate the training process. The photovoltaic bi-light image is used as the input of the improved generator, and the image generated by the generator can preserve some infrared intensity information and background information of the visible light image to a certain extent. ID2WGAN designs two SN-CNN-based discriminators. The first discriminator is designed to distinguish between the generated image and the infrared image, so that the generated image can preserve the temperature intensity information and detail information from the infrared image; the second discriminator is designed to distinguish between the generated image and the visible light image, so that the generated image can preserve enough texture information and detail information from the visible light image.

[0132] The beneficial effects of the above technical solution are: the present invention realizes multi-type fault detection of photovoltaic modules by combining low-rank sparse decomposition with an improved dual-discriminator generative adversarial network to fuse the dual-light source images of photovoltaic modules, and no longer distinguishes between infrared faults and visible light faults, effectively improving the fault detection accuracy. By replacing the generator network and discriminator network of the traditional GAN, the proposed image fusion method can simultaneously preserve the target information in the infrared image and the background information in the visible light image, and the loss function based on the local binary pattern can better preserve the texture information in the visible light image. The fusion effect is significantly better than the image fusion result of the single discriminator generative adversarial network.

[0133] In some embodiments of the present application, when optimizing the first SN-CNN discriminator and the second SN-CNN discriminator based on the local binary pattern algorithm and the improved Wasserstein distance algorithm, it includes:

[0134] The expression of the local binary pattern algorithm is as follows:

[0135]

[0136] Among them, LBP(x c ,y c ) is the central pixel, p is the number of neighboring pixels of the central pixel, i p is the neighborhood pixel intensity value of the pth central pixel, i c is the pixel intensity value of the center pixel, when i p -i c ≥0, then s(i p -i c )=1, when i p -i c <0, then s(i p -i c )=0;

[0137] The expression of the improved Wasserstein distance algorithm is as follows:

[0138]

[0139] in, is the objective function of the generative adversarial network, E x～pr For the real data distribution p r The expectation is to calculate the expectation of all data x sampled from the real data distribution. D(x) is the output of the discriminator for the input x, which is used to determine whether the input x comes from real data or generated data. D is the discriminator, E z～pz To generate data distribution p z The expectation is to calculate the expectation of all noise z sampled from the generated data distribution. D(G(z)) is the output of the generator for the input noise z, that is, the generated data. G is the generator, μ is the penalty coefficient, For linear interpolation data expectations, is the change in x, P r and P z are the real data distribution and the generated data distribution respectively, The discriminator input Output.

[0140] S140: annotating the target fusion image with a fault type based on a preset tool, and training a YOLO series target detection model according to the image data of the target fusion image, and detecting the fault type and location of the target fusion image after the training is completed.

[0141] In this embodiment, the Labelimg labeling tool is used to label the target fusion image with the fault type.

[0142] In this embodiment, during the labeling process, the corresponding area is selected on the image according to the preset fault type (such as crack, corrosion, wear, etc.), and the corresponding label is assigned to it. After the labeling is completed, the label information is saved as a text file in YOLO format for subsequent training.

[0143] In this embodiment, the YOLO series target detection model training:

[0144] Dataset division: The labeled image dataset is divided into a training set and a validation set. Typically, the training set contains about 70% of the data, which is used for model training; the validation set contains the remaining 30% of the data, which is used to evaluate the performance of the model.

[0145] Model configuration: Select a suitable YOLO model (such as YOLOv7) as the base model as needed. Configure the model parameters, including input image size, learning rate, batch size, etc.

[0146] Model training: Use the training set to train the model. During the training process, the model will learn how to extract features from the image and predict the fault type and location based on these features. After the training is completed, the validation set is used to evaluate the performance of the model.

[0147] Model optimization: Optimize the model based on the validation results. The accuracy and stability of the model can be improved by adjusting model parameters, adding regularization terms, using more complex network structures, etc.

[0148] In this embodiment, fault type and location detection:

[0149] Load model: Load the trained YOLO model into the target detection system.

[0150] Image preprocessing: Preprocess the fused image to be detected to make it meet the input requirements of the model.

[0151] Fault detection: Use the model to perform forward propagation calculations on the image to be detected to obtain prediction results. The prediction results will include the type and location information of the fault.

[0152] Result visualization: The prediction results are visualized on an image so that users can intuitively understand the fault situation.

[0153] The beneficial effects of the above technical scheme are: the technical scheme of the present invention not only improves the type and accuracy of photovoltaic component fault detection, but also can adapt to different photovoltaic power station terrains. Whether it is a fishery-photovoltaic complementary power station or an agricultural-photovoltaic complementary power station, the method can be reasonably applied. The present invention provides strong support for the digital and intelligent development of photovoltaic stations, and greatly improves the operation and maintenance efficiency of photovoltaic power stations through automated image acquisition, fusion and fault detection processes.

[0154] In order to further explain the technical idea of ​​the present invention, the technical solution of the present invention is now described in combination with specific application scenarios.

[0155] Correspondingly, such as Figure 2 As shown, the present application also provides a device for fusing infrared images and visible light images of photovoltaic modules, characterized in that:

[0156] Image capture module, used to inspect and capture the entire area of ​​the photovoltaic station based on the pre-deployed drone equipped with thermal infrared camera and visible light camera, and obtain photovoltaic infrared images and photovoltaic visible light images of the photovoltaic power station strings;

[0157] An image registration module is used to perform registration processing on the photovoltaic infrared image and the photovoltaic visible light image according to the edge rectangle feature method to obtain the photovoltaic dual light source image of the photovoltaic power station string;

[0158] An image processing module, used to take the photovoltaic dual-light source image as input data, input it into a dual-discriminator generative adversarial network, and output a target fused low-rank image and a target fused sparse detail image based on the dual-discriminator generative adversarial network, and generate a target fused image of the photovoltaic power station string according to a weighted average fusion strategy;

[0159] The image analysis module is used to annotate the fault type of the target fusion image based on a preset tool, and train a YOLO series target detection model according to the image data of the target fusion image, and detect the fault type and location of the target fusion image after the training is completed.

[0160] In the description of the above embodiments, specific features, structures, materials or characteristics may be combined in a suitable manner in any one or more embodiments or examples.

[0161] Although the present invention has been described above with reference to the embodiments, various modifications may be made thereto and parts thereof may be replaced with equivalents without departing from the scope of the present invention. In particular, as long as there is no structural conflict, the various features in the embodiments disclosed by the present invention may be used in combination with each other in any manner, and the fact that these combinations are not fully described in this specification is only for the sake of omitting space and saving resources.

[0162] Those skilled in the art can understand that the above are only preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions recorded in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for fusing infrared images and visible light images of photovoltaic modules, characterized in that: include: Based on the pre-deployed drones equipped with thermal infrared cameras and visible light cameras, the whole area of ​​the photovoltaic station is inspected and photographed to obtain photovoltaic infrared images and photovoltaic visible light images of the photovoltaic power station strings; Performing registration processing on the photovoltaic infrared image and the photovoltaic visible light image according to the edge rectangle feature method to obtain the photovoltaic dual light source image of the photovoltaic power station string; The photovoltaic dual-light source image is used as input data, input into a dual-discriminator generative adversarial network, and based on the dual-discriminator generative adversarial network output, a target fused low-rank image and a target fused sparse detail image are output, and a target fused image of the photovoltaic power station string is generated according to a weighted average fusion strategy; The target fusion image is labeled with the fault type based on a preset tool, and a YOLO series target detection model is trained according to the image data of the target fusion image. After the training is completed, the fault type and location of the target fusion image are detected.

2. The method for fusing infrared images and visible light images of photovoltaic modules according to claim 1, characterized in that: When the photovoltaic infrared image and the photovoltaic visible light image are registered according to the edge rectangle feature method to obtain the photovoltaic dual light source image of the photovoltaic power station string, it includes: Preprocessing the photovoltaic infrared image and the photovoltaic visible light image respectively, wherein the preprocessing includes removing noise and interference; Extracting first edge information and second edge information of the photovoltaic infrared image and the photovoltaic visible light image based on an edge detection algorithm, and generating a first edge rectangle and a second edge rectangle according to the first edge information and the second edge information; Converting the first edge rectangle into the frequency domain by Fourier transform, and extracting the corresponding first amplitude value in the frequency domain; Converting the second edge rectangle into the frequency domain by Fourier transform, and extracting the corresponding second amplitude value in the frequency domain; Calculating edge matching coefficients of the first edge rectangle and the second edge rectangle according to the first amplitude value and the second amplitude value; Determining whether the photovoltaic infrared image can be mapped to the photovoltaic visible light image according to the relationship between the edge matching degree coefficient and a preset edge matching degree coefficient; When the edge matching degree coefficient is greater than or equal to the preset edge matching degree coefficient, it is determined that the photovoltaic infrared image can be mapped to the photovoltaic visible light image to obtain the photovoltaic dual-light source image of the photovoltaic power station string; When the edge matching degree coefficient is less than the preset edge matching degree coefficient, it is determined that the photovoltaic infrared image cannot be mapped to the photovoltaic visible light image, and the photovoltaic infrared image and the photovoltaic visible light image are re-photographed until the edge matching degree coefficient is greater than or equal to the preset edge matching degree coefficient.

3. The method for fusing infrared images and visible light images of photovoltaic modules according to claim 2, characterized in that: When calculating the edge matching degree coefficient of the first edge rectangle and the second edge rectangle according to the first amplitude value and the second amplitude value, it includes: Constructing a first amplitude value sequence according to all first amplitude values, and constructing a second amplitude value sequence according to all second amplitude values; Calculate a first sub-edge matching degree coefficient and a second sub-edge matching degree coefficient of the first edge rectangle and the second edge rectangle based on the first amplitude value sequence and the second amplitude value sequence; The edge matching degree coefficients of the first edge rectangle and the second edge rectangle are calculated according to the first sub-edge matching degree coefficient and the second sub-edge matching degree coefficient.

4. The method for fusing infrared images and visible light images of photovoltaic modules according to claim 3, characterized in that: When calculating the first sub-edge matching degree coefficient of the first edge rectangle and the second edge rectangle based on the first amplitude value sequence and the second amplitude value sequence, the method includes: Calculate a first sequence mean corresponding to the first amplitude value sequence, and calculate a second sequence mean corresponding to the second amplitude value sequence; Calculate the sequence mean difference between the first sequence mean and the second sequence mean; Compare the first amplitude value sequence and the second amplitude value sequence one by one to determine a plurality of amplitude value differences; Extracting a maximum amplitude value difference and a minimum amplitude value difference from all amplitude value differences, and calculating an extreme amplitude value difference of the maximum amplitude value difference and the minimum amplitude value difference; The first sub-edge matching degree coefficient is calculated according to the sequence mean difference, extreme amplitude value difference, maximum amplitude value difference and minimum amplitude value difference.

5. The method for fusing infrared images and visible light images of photovoltaic modules according to claim 4, characterized in that: When calculating the first sub-edge matching degree coefficient according to the sequence mean difference, the extreme amplitude value difference, the maximum amplitude value difference and the minimum amplitude value difference, it includes: The first sub-edge matching degree coefficient is calculated according to the following formula: Among them, q is the first sub-edge matching degree coefficient, r1 is the sequence mean difference, r2 is the extreme amplitude difference, r3 is the maximum amplitude difference, and r4 is the minimum amplitude difference.

6. The method for fusing infrared images and visible light images of photovoltaic modules according to claim 5, characterized in that: When calculating the second sub-edge matching degree coefficient of the first edge rectangle and the second edge rectangle based on the first amplitude value sequence and the second amplitude value sequence, the method includes: Randomly matching the amplitude values ​​in the first amplitude value sequence and the second amplitude value sequence in pairs to obtain a plurality of randomly matched amplitude value sets; The second sub-edge matching coefficient is calculated according to the following formula: Among them, t is the second sub-edge matching coefficient, n1 is the number of random matching sets of amplitude values, and a u is the larger amplitude value in the random matching set of the u-th amplitude value, d u is the smaller amplitude value in the random matching set of the u-th amplitude value, n2 is the number of the first amplitude values ​​in the first amplitude value sequence, g f is the fth first amplitude value in the first amplitude value sequence, n3 is the number of second amplitude values ​​in the second amplitude value sequence, k h is the hth second amplitude value in the second amplitude value sequence.

7. The method for fusing infrared images and visible light images of photovoltaic modules according to claim 6, characterized in that: When calculating the edge matching degree coefficients of the first edge rectangle and the second edge rectangle according to the first sub-edge matching degree coefficient and the second sub-edge matching degree coefficient, it includes: configuring a first calculation coefficient for the first sub-edge matching degree coefficient, and configuring a second calculation coefficient for the first sub-edge matching degree coefficient; The edge matching coefficient of the first edge rectangle and the second edge rectangle is calculated according to the following formula: v = m1 × q + m2 × t; Wherein, v is the edge matching degree coefficient of the first edge rectangle and the second edge rectangle, m3 is the first calculation coefficient, m2 is the second calculation coefficient, and m1+m2=1, m1>m2.

8. The method for fusing infrared images and visible light images of photovoltaic modules according to claim 1, characterized in that: When the photovoltaic dual-light source image is used as input data, input into a dual-discriminator generative adversarial network, and based on the dual-discriminator generative adversarial network outputting a target fused low-rank image and a target fused sparse detail image, and a target fused image of the photovoltaic power station string is generated according to a weighted average fusion strategy, the method includes: Decomposing the photovoltaic dual-light source image by low-rank sparse decomposition to obtain a low-rank part and a sparse detail part; Input the low-rank part as input data to a first SN-CNN discriminator, and input the sparse detail part as input data to a second SN-CNN discriminator; Optimizing the first SN-CNN discriminator and the second SN-CNN discriminator based on a local binary pattern algorithm and an improved Wasserstein distance algorithm; Outputting the target fused low-rank image based on the optimized first SN-CNN discriminator, wherein the target fused low-rank image can save temperature intensity information and detail information from the infrared image; The target fused sparse detail image is output based on the optimized second SN-CNN discriminator, wherein the target fused sparse detail image can preserve texture information and detail information from the visible light image.

9. The method for fusing infrared images and visible light images of photovoltaic modules according to claim 8, characterized in that: When optimizing the first SN-CNN discriminator and the second SN-CNN discriminator based on the local binary pattern algorithm and the improved Wasserstein distance algorithm, the method includes: The expression of the local binary pattern algorithm is as follows: Among them, LBP(x c ,y c ) is the central pixel, p is the number of neighboring pixels of the central pixel, i p is the neighborhood pixel intensity value of the pth central pixel, i c is the pixel intensity value of the center pixel, when i p -i c ≥0, then s(i p -i c )=1, when i p -i c <0, then s(i p -i c )=0; The expression of the improved Wasserstein distance algorithm is as follows: in, is the objective function of the generative adversarial network, E x□pr is the real data distribution p r The expectation is to calculate the expectation of all data x sampled from the real data distribution. D(x) is the output of the discriminator for the input x, which is used to determine whether the input x comes from real data or generated data. D is the discriminator, E z□pz To generate data distribution p z The expectation is to calculate the expectation of all noise z sampled from the generated data distribution. D(G(z)) is the output of the generator for the input noise z, that is, the generated data. G is the generator, μ is the penalty coefficient, For linear interpolation data expectations, is the change in x, P r and P z are the real data distribution and the generated data distribution respectively, The discriminator input Output.

10. A photovoltaic module infrared image and visible light image fusion device, applied to the photovoltaic module infrared image and visible light image fusion method as claimed in any one of claims 1 to 9, characterized in that: Image capture module, used to inspect and capture the entire area of ​​the photovoltaic station based on the pre-deployed drone equipped with thermal infrared camera and visible light camera, and obtain photovoltaic infrared images and photovoltaic visible light images of the photovoltaic power station strings; An image registration module is used to perform registration processing on the photovoltaic infrared image and the photovoltaic visible light image according to the edge rectangle feature method to obtain the photovoltaic dual light source image of the photovoltaic power station string; An image processing module, used to take the photovoltaic dual-light source image as input data, input it into a dual-discriminator generative adversarial network, and output a target fused low-rank image and a target fused sparse detail image based on the dual-discriminator generative adversarial network, and generate a target fused image of the photovoltaic power station string according to a weighted average fusion strategy; The image analysis module is used to annotate the fault type of the target fusion image based on a preset tool, and train a YOLO series target detection model according to the image data of the target fusion image, and detect the fault type and location of the target fusion image after the training is completed.

Images