A method and device for fusing infrared and visible light images of a photovoltaic module

Abstract

The application relates to the technical field of image data processing, and discloses a photovoltaic module infrared image and visible light image fusion method and device, which is based on a UAV carrying a thermal infrared camera and a visible light camera to shoot the whole area of a photovoltaic station, and obtains photovoltaic infrared images and photovoltaic visible light images; the photovoltaic infrared images and the photovoltaic visible light images are subjected to registration processing according to an edge rectangle feature method, and photovoltaic double-light-source images are obtained; the photovoltaic double-light-source images are input into a double-discriminator generative adversarial network, target fusion low-rank images and target fusion sparse detail images are output, and a target fusion image is generated according to a weighted average fusion strategy; the target fusion image is subjected to fault type labeling, and the fault type and position of the target fusion image are detected after training is completed according to a YOLO series target detection model; and the application can guarantee that the obtained target fusion image has clear texture features and temperature intensity information, and provides strong data support for subsequent fault detection.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and more specifically, to a method and apparatus for fusing infrared images and visible light images of photovoltaic modules. Background Technology

[0002] As the core power generation component of a photovoltaic (PV) power plant, the malfunction of photovoltaic (PV) modules directly affects the power generation and revenue of the entire plant, and may even impact its safety. Since PV modules in every PV power plant can malfunction, quickly identifying abnormal PV modules within an operating PV power plant is crucial for its operation and maintenance.

[0003] In recent years, with the rapid development of deep learning technology, an increasing number of related object detection algorithms have emerged. In existing technologies, based on whether or not a region proposal (RP) generation stage is used, deep learning-based detection algorithms can be divided into two categories: region proposal-based detection algorithms and non-region proposal-based detection algorithms. Region proposal-based detection algorithms first generate region proposals, then use a deep learning network for feature extraction to obtain the corresponding category and coordinate information. These algorithms have high detection accuracy but are relatively slow; major algorithms include SPP-NET and Faster R-CNN. Non-region proposal-based detection algorithms do not generate proposals; they directly use a deep learning network to predict the probability of an object existing at each location in the image. These algorithms have advantages such as high speed and strong generalization ability; major algorithms include SSD and the YOLO series. All of the above object detection algorithms only detect single images of photovoltaic modules and do not consider combining the temperature intensity information of infrared images with the detailed texture information of visible light images to perform fault detection on the fused image, thereby ensuring that abnormal defects in the modules are fully detected. Summary of the Invention

[0004] This invention provides a method and apparatus for fusing infrared and visible light images of photovoltaic modules, which solves the problems of insufficient fusion of infrared and visible light image information and low fault detection accuracy in the prior art.

[0005] To achieve the above objectives, the present invention provides a method for fusing infrared images and visible light images of photovoltaic modules, comprising:

[0006] Based on pre-deployed drones equipped with thermal infrared and visible light cameras, the entire area of ​​the photovoltaic power station is inspected and photographed to obtain photovoltaic infrared and visible light images of the photovoltaic power station strings;

[0007] According to the method of the edge rectangle feature, the photovoltaic infrared image and the photovoltaic visible light image are registered to obtain a photovoltaic dual light source image of the photovoltaic power station string;

[0008] The photovoltaic dual light source image is input into a dual discriminator generative adversarial network as input data, and target fusion low-rank images and target fusion sparse detail images are output based on the dual discriminator generative adversarial network, and a target fusion image of the photovoltaic power station string is generated according to a weighted average fusion strategy;

[0009] The target fusion image is labeled for fault types based on a preset tool, and a YOLO series target detection model is trained according to image data of the target fusion image, and the fault types and positions of the target fusion image are detected after the training is completed.

[0010] Further, when the photovoltaic infrared image and the photovoltaic visible light image are registered according to the method of the edge rectangle feature to obtain the photovoltaic dual light source image of the photovoltaic power station string, the method comprises:

[0011] The photovoltaic infrared image and the photovoltaic visible light image are respectively preprocessed, wherein the preprocessing comprises removing noise and interference;

[0012] First edge information and second edge information of the photovoltaic infrared image and the photovoltaic visible light image are extracted based on an edge detection algorithm, and first edge rectangles and second edge rectangles are generated according to the first edge information and the second edge information;

[0013] The first edge rectangles are converted to a frequency domain by Fourier transform, and corresponding first amplitude values in the frequency domain are extracted;

[0014] The second edge rectangles are converted to a frequency domain by Fourier transform, and corresponding second amplitude values in the frequency domain are extracted;

[0015] An edge matching degree coefficient of the first edge rectangles and the second edge rectangles is calculated according to the first amplitude values and the second amplitude values;

[0016] Whether the photovoltaic infrared image can be mapped to the photovoltaic visible light image is determined according to a relationship between the edge matching degree coefficient and a preset edge matching degree coefficient;

[0017] When the edge matching degree coefficient is greater than or equal to the preset edge matching degree coefficient, it is determined that the photovoltaic infrared image can be mapped to the photovoltaic visible light image to obtain the photovoltaic dual light source image of the photovoltaic power station string;

[0018] When the edge matching degree coefficient is less than the preset edge matching degree coefficient, it is determined that the photovoltaic infrared image cannot be mapped to the photovoltaic visible light image, and the photovoltaic infrared image and the photovoltaic visible light image are rephotographed until the edge matching degree coefficient is greater than or equal to the preset edge matching degree coefficient.

[0019] Further, in calculating the edge matching degree coefficient of the first edge rectangle and the second edge rectangle according to the first amplitude value and the second amplitude value, comprising:

[0020] A first amplitude value sequence is constructed according to all the first amplitude values, and a second amplitude value sequence is constructed according to all the second amplitude values;

[0021] A first sub-edge matching degree coefficient and a second sub-edge matching degree coefficient of the first edge rectangle and the second edge rectangle are calculated based on the first amplitude value sequence and the second amplitude value sequence;

[0022] The edge matching degree coefficient of the first edge rectangle and the second edge rectangle is calculated according to the first sub-edge matching degree coefficient and the second sub-edge matching degree coefficient.

[0023] Further, in calculating the first sub-edge matching degree coefficient of the first edge rectangle and the second edge rectangle based on the first amplitude value sequence and the second amplitude value sequence, comprising:

[0024] A first sequence mean corresponding to the first amplitude value sequence is calculated, and a second sequence mean corresponding to the second amplitude value sequence is calculated;

[0025] A sequence mean difference value of the first sequence mean and the second sequence mean is calculated;

[0026] The first amplitude value sequence and the second amplitude value sequence are compared one by one to determine a plurality of amplitude value difference values;

[0027] The maximum amplitude value difference value and the minimum amplitude value difference value are extracted from all the amplitude value difference values, and an extreme amplitude value difference value of the maximum amplitude value difference value and the minimum amplitude value difference value is calculated;

[0028] The first sub-edge matching degree coefficient is calculated according to the sequence mean difference value, the extreme amplitude value difference value, the maximum amplitude value difference value and the minimum amplitude value difference value.

[0029] Further, in calculating the first sub-edge matching degree coefficient according to the sequence mean difference value, the extreme amplitude value difference value, the maximum amplitude value difference value and the minimum amplitude value difference value, comprising:

[0030] The first sub-edge matching degree coefficient is calculated according to the following formula:

[0031]

[0032] wherein q is a first sub-edge matching degree coefficient, r1 is a sequence mean difference value, r2 is a maximum amplitude value difference value, r3 is a minimum amplitude value difference value, and r4 is a maximum amplitude value difference value.

[0033] Further, in calculating the second sub-edge matching degree coefficient of the first edge rectangle and the second edge rectangle based on the first amplitude value sequence and the second amplitude value sequence, comprising:

[0034] randomly matching the amplitude values in the first amplitude value sequence and the second amplitude value sequence to obtain a plurality of amplitude value random matching sets;

[0035] The second sub-edge matching degree coefficient is calculated according to the following formula:

[0036]

[0037] wherein t is a second sub-edge matching degree coefficient, n1 is the number of amplitude value random matching sets, a u is a larger amplitude value in the u-th amplitude value random matching set, d u is a smaller amplitude value in the u-th amplitude value random matching set, n2 is the number of first amplitude values in the first amplitude value sequence, g f is the f-th first amplitude value in the first amplitude value sequence, n3 is the number of second amplitude values in the second amplitude value sequence, and k h is the h-th second amplitude value in the second amplitude value sequence.

[0038] Further, in calculating the edge matching degree coefficient of the first edge rectangle and the second edge rectangle according to the first sub-edge matching degree coefficient and the second sub-edge matching degree coefficient, comprising:

[0039] configuring a first calculation coefficient for the first sub-edge matching degree coefficient, and configuring a second calculation coefficient for the first sub-edge matching degree coefficient;

[0040] The edge matching degree coefficient of the first edge rectangle and the second edge rectangle is calculated according to the following formula:

[0041] v = m1 x q + m2 x t.

[0042] wherein v is the edge matching degree coefficient of the first edge rectangle and the second edge rectangle, m3 is the first calculation coefficient, m2 is the second calculation coefficient, and m1 + m2 = 1, m1 > m2.

[0043] Further, when the photovoltaic dual light source image is input as input data into the dual discriminator generative adversarial network, and target fusion low-rank images and target fusion sparse detail images are output based on the dual discriminator generative adversarial network, and the target fusion images of the photovoltaic power station group string are generated according to a weighted average fusion strategy, the method comprises the following steps:

[0044] The photovoltaic dual light source image is decomposed into a low-rank part and a sparse detail part through a low-rank sparse decomposition method;

[0045] The low-rank part is input as input data into a first SN-CNN discriminator, and the sparse detail part is input as input data into a second SN-CNN discriminator;

[0046] The first SN-CNN discriminator and the second SN-CNN discriminator are optimized based on a local binary pattern algorithm and an improved Wasserstein distance algorithm;

[0047] The target fusion low-rank image is output based on the optimized first SN-CNN discriminator, wherein the target fusion low-rank image can save temperature intensity information and detail information from an infrared image;

[0048] The target fusion sparse detail image is output based on the optimized second SN-CNN discriminator, wherein the target fusion sparse detail image can save texture information and detail information from a visible light image.

[0049] Further, when the first SN-CNN discriminator and the second SN-CNN discriminator are optimized based on the local binary pattern algorithm and the improved Wasserstein distance algorithm, the method comprises the following steps:

[0050] The expression of the local binary pattern algorithm is as follows:

[0051]

[0052] Wherein, LBP(x c , y c ) is a center pixel, p is the number of neighborhood pixels of the center pixel, i p is the neighborhood pixel intensity value of the pth center pixel, i c is the pixel intensity value of the center pixel, when i p -i c ≥ 0, then s(i p -i c ) = 1, when i p -i c < 0, then s(i p -i c ) = 0.

[0053] The expression of the improved Wasserstein distance algorithm is as follows:

[0054]

[0055] wherein, is the objective function of the generative adversarial network, E x～pr is the expectation of the real data distribution p r , that is, the expectation calculation is performed on all data x sampled from the real data distribution, D(x) is the output of the discriminator to the input x, which is used to judge whether the input x is from the real data or the generated data, D is the discriminator, E z～pz is the expectation of the generated data distribution p z , that is, the expectation calculation is performed on all noise z sampled from the generated data distribution, D(G(z)) is the output of the generator to the input noise z, that is, the generated data, G is the generator, μ is a penalty coefficient, is the expectation of the linear interpolation data , is the change of x, P r and P z are the real data distribution and the generated data distribution respectively, is the output of the discriminator to the input .

[0056] In order to achieve the above purpose, the application also provides a photovoltaic module infrared image and visible light image fusion device, characterized in that,

[0057] An image shooting module is configured to shoot and capture the whole area of a photovoltaic power station by using a pre-deployed unmanned aerial vehicle carrying a thermal infrared camera and a visible light camera, so as to obtain photovoltaic infrared images and photovoltaic visible light images of a photovoltaic power station string.

[0058] An image registration module is configured to perform registration processing on the photovoltaic infrared images and the photovoltaic visible light images according to an edge rectangle feature method, so as to obtain photovoltaic dual-light-source images of the photovoltaic power station string.

[0059] An image processing module is configured to input the photovoltaic dual-light-source images as input data into a dual-discriminator generative adversarial network, and output target fusion low-rank images and target fusion sparse detail images based on the dual-discriminator generative adversarial network, so as to generate target fusion images of the photovoltaic power station string according to a weighted average fusion strategy.

[0060] An image analysis module is configured to label a fault type of the target fusion images based on a preset tool, and train a YOLO series target detection model according to image data of the target fusion images, so as to detect the fault type and position of the target fusion images after the training is completed.

[0061] The present application has the advantage of:

[0062] The application discloses a kind of infrared image and visible light image fusion method and device of photovoltaic module, based on unmanned aerial vehicle carries thermal infrared camera and visible light camera to the whole area of photovoltaic station is photographed, obtains photovoltaic infrared image and photovoltaic visible light image;According to edge rectangular feature method, photovoltaic infrared image and photovoltaic visible light image are registered and handled, and photovoltaic double light source image is obtained;Photovoltaic double light source image is input into double discriminator generative adversarial network, and target fusion low rank image and target fusion sparse detail image are output, and target fusion image is generated according to weighted average fusion strategy;Target fusion image is labeled with fault type, and according to YOLO series target detection model, the fault type and position of target fusion image are detected after training is completed, and the present application can guarantee that target fusion image obtained has clear texture feature and temperature intensity information, provides strong data support for subsequent fault detection. BRIEF DESCRIPTION OF DRAWINGS

[0063] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments with reference made to the accompanying drawings. The drawings are for purposes of illustration only and are not intended to be limiting in any respect. Like reference numerals have been used in the drawings to describe and refer to like parts throughout the drawings. In the drawings:

[0064] Figure 1 A flowchart of the fusion method of infrared image and visible light image of photovoltaic module in the embodiment of the present application is shown;

[0065] Figure 2 A structure diagram of the fusion device of infrared image and visible light image of photovoltaic module in the embodiment of the present application is shown;

[0066] Figure 3 An infrared and visible light image fusion algorithm framework of improved double discriminator generative adversarial network in the embodiment of the present application is shown;

[0067] Figure 4 A generator network structure in the embodiment of the present application is shown;

[0068] Figure 5 A discriminator network structure in the embodiment of the present application is shown. DETAILED DESCRIPTION

[0069] The specific embodiments of the present application are described in further detail below in conjunction with the drawings and examples. The following examples are used to illustrate the present application, but are not used to limit the scope of the present application.

[0070] In the description of the present application, it needs to be understood that the terms "center", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, which are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0071] The terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0072] In the description of the present application, it needs to be explained that, unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connection" should be understood in a broad sense, for example, it can be fixed connection, or detachable connection, or integral connection; it can be mechanical connection, or electrical connection; it can be direct connection, or indirect connection through intermediate medium, or internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0073] The following is a description of the preferred embodiments of the present application in conjunction with the drawings.

[0074] As Figure 1 shown, the embodiments of the present application disclose a method for fusing infrared image and visible light image of photovoltaic module, comprising:

[0075] S110: based on the pre-deployed unmanned aerial vehicle carrying thermal infrared camera and visible light camera, the whole area of photovoltaic power station is patrolled and shot, and the photovoltaic infrared image and photovoltaic visible light image of photovoltaic power station string are obtained;

[0076] In the present embodiment, the photovoltaic infrared image of photovoltaic power station string is shot based on the unmanned aerial vehicle carrying thermal infrared camera, and the photovoltaic visible light image of photovoltaic power station string is shot based on the visible light camera.

[0077] S120: according to the method of edge rectangular feature, the photovoltaic infrared image and photovoltaic visible light image are registered to obtain the photovoltaic dual light source image of photovoltaic power station string;

[0078] In some embodiments of the present application, when the photovoltaic infrared image and photovoltaic visible light image are registered according to the method of edge rectangular feature to obtain the photovoltaic dual light source image of photovoltaic power station string, it comprises:

[0079] respectively, wherein the preprocessing comprises removing noise and interference;

[0080] first edge information and second edge information of the photovoltaic infrared image and the photovoltaic visible light image are extracted based on an edge detection algorithm, and a first edge rectangle and a second edge rectangle are generated according to the first edge information and the second edge information;

[0081] the first edge rectangle is converted to a frequency domain by using Fourier transform, and a corresponding first amplitude value in the frequency domain is extracted;

[0082] the second edge rectangle is converted to a frequency domain by using Fourier transform, and a corresponding second amplitude value in the frequency domain is extracted;

[0083] an edge matching degree coefficient of the first edge rectangle and the second edge rectangle is calculated according to the first amplitude value and the second amplitude value;

[0084] whether the photovoltaic infrared image can be mapped to the photovoltaic visible light image is determined according to a relationship between the edge matching degree coefficient and a preset edge matching degree coefficient;

[0085] when the edge matching degree coefficient is greater than or equal to the preset edge matching degree coefficient, it is determined that the photovoltaic infrared image can be mapped to the photovoltaic visible light image, and a photovoltaic dual light source image of the photovoltaic power station string is obtained;

[0086] when the edge matching degree coefficient is less than the preset edge matching degree coefficient, it is determined that the photovoltaic infrared image cannot be mapped to the photovoltaic visible light image, and the photovoltaic infrared image and the photovoltaic visible light image are rephotographed until the edge matching degree coefficient is greater than or equal to the preset edge matching degree coefficient.

[0087] In the embodiment, the edge detection algorithm includes one or more of a Canny edge detection algorithm, a Sobel edge detection algorithm or a Laplacian edge detection algorithm.

[0088] In the embodiment, a morphological transformation method is used to connect edge points to form a closed edge rectangle.

[0089] In the embodiment, a reference point in the photovoltaic infrared image is mapped to the photovoltaic visible light image to complete image registration, and a photovoltaic dual light source image of the photovoltaic power station string is obtained.

[0090] The beneficial effects of the above technical solutions are: the application determines whether the photovoltaic infrared image can be mapped to the photovoltaic visible light image according to the relationship between the edge matching degree coefficient and the preset edge matching degree coefficient, ensures the determination accuracy of the photovoltaic dual light source image, and lays a foundation for subsequent image fusion.

[0091] In some embodiments of the application, when calculating the edge matching degree coefficient of the first edge rectangle and the second edge rectangle according to the first amplitude value and the second amplitude value, the following steps are included:

[0092] A first amplitude value sequence is constructed according to all the first amplitude values, and a second amplitude value sequence is constructed according to all the second amplitude values;

[0093] First and second sub-edge matching degree coefficients of the first edge rectangle and the second edge rectangle are calculated based on the first amplitude value sequence and the second amplitude value sequence;

[0094] The edge matching degree coefficient of the first edge rectangle and the second edge rectangle is calculated according to the first and second sub-edge matching degree coefficients.

[0095] In some embodiments of the application, when calculating the first sub-edge matching degree coefficient of the first edge rectangle and the second edge rectangle based on the first amplitude value sequence and the second amplitude value sequence, the following steps are included:

[0096] A first sequence mean corresponding to the first amplitude value sequence is calculated, and a second sequence mean corresponding to the second amplitude value sequence is calculated;

[0097] A sequence mean difference value of the first sequence mean and the second sequence mean is calculated;

[0098] The first amplitude value sequence and the second amplitude value sequence are compared one by one to determine a plurality of amplitude value difference values;

[0099] The maximum amplitude value difference value and the minimum amplitude value difference value are extracted from all the amplitude value difference values, and an extreme amplitude value difference value of the maximum amplitude value difference value and the minimum amplitude value difference value is calculated;

[0100] The first sub-edge matching degree coefficient is calculated according to the sequence mean difference value, the extreme amplitude value difference value, the maximum amplitude value difference value and the minimum amplitude value difference value.

[0101] In this embodiment, the calculation method of the sequence mean is not repeated.

[0102] In this embodiment, the sequence mean difference value is the absolute value of the difference between the first sequence mean and the second sequence mean.

[0103] The beneficial effects of the above technical solutions are: the first sub-edge matching degree coefficient is calculated according to the sequence mean difference value, the polar amplitude value difference value, the maximum amplitude value difference value and the minimum amplitude value difference value, so that the calculation accuracy of the first sub-edge matching degree coefficient is ensured, and a calculation basis for the calculation of the edge matching degree coefficient is provided.

[0104] In some embodiments of the present application, when the first sub-edge matching degree coefficient is calculated according to the sequence mean difference value, the polar amplitude value difference value, the maximum amplitude value difference value and the minimum amplitude value difference value, it includes:

[0105] The first sub-edge matching degree coefficient is calculated according to the following formula:

[0106]

[0107] Wherein, q is the first sub-edge matching degree coefficient, r1 is the sequence mean difference value, r2 is the polar amplitude value difference value, r3 is the maximum amplitude value difference value, and r4 is the minimum amplitude value difference value.

[0108] In some embodiments of the present application, when the second sub-edge matching degree coefficient of the first edge rectangle and the second edge rectangle is calculated based on the first amplitude value sequence and the second amplitude value sequence, it includes:

[0109] Randomly match the amplitude values in the first amplitude value sequence and the second amplitude value sequence to obtain a plurality of amplitude value random matching sets;

[0110] The second sub-edge matching degree coefficient is calculated according to the following formula:

[0111]

[0112] Wherein, t is the second sub-edge matching degree coefficient, n1 is the number of amplitude value random matching sets, a u is the larger amplitude value in the u-th amplitude value random matching set, d u is the smaller amplitude value in the u-th amplitude value random matching set, n2 is the number of first amplitude values in the first amplitude value sequence, g f is the f-th first amplitude value in the first amplitude value sequence, n3 is the number of second amplitude values in the second amplitude value sequence, and k h is the h-th second amplitude value in the second amplitude value sequence.

[0113] In this embodiment, the larger amplitude value in the u-th amplitude value random matching set can be a first amplitude value or a second amplitude value, and the larger amplitude value in the u-th amplitude value random matching set can be a first amplitude value or a second amplitude value, which can be compared according to actual conditions.

[0114] The beneficial effects of the above technical solutions are: the amplitude values in the first amplitude value sequence and the second amplitude value sequence are randomly matched two by two to obtain a plurality of amplitude value random matching sets, the calculation objectivity is ensured, and errors are avoided, and the second sub-edge matching degree coefficient is calculated to provide another calculation basis for the calculation of the edge matching degree coefficient.

[0115] In some embodiments of the present application, when calculating the edge matching degree coefficient of the first edge rectangle and the second edge rectangle according to the first sub-edge matching degree coefficient and the second sub-edge matching degree coefficient, it includes:

[0116] The first sub-edge matching degree coefficient is configured with a first calculation coefficient, and the first sub-edge matching degree coefficient is configured with a second calculation coefficient;

[0117] The edge matching degree coefficient of the first edge rectangle and the second edge rectangle is calculated according to the following formula:

[0118] v=m1xq+m2xt;

[0119] Wherein, v is the edge matching degree coefficient of the first edge rectangle and the second edge rectangle, m3 is the first calculation coefficient, m2 is the second calculation coefficient, and m1+m2=1, m1>m2.

[0120] The beneficial effects of the above technical solutions are: the edge matching degree coefficient of the first edge rectangle and the second edge rectangle is calculated according to the first sub-edge matching degree coefficient and the second sub-edge matching degree coefficient, which provides a registration basis for the registration processing of the photovoltaic infrared image and the photovoltaic visible light image, and ensures the registration processing precision and efficiency.

[0121] S130: input the photovoltaic dual light source image as input data into the dual discriminator generative adversarial network, and output target fusion low rank image and target fusion sparse detail image based on the dual discriminator generative adversarial network, and generate the target fusion image of the photovoltaic power station group string according to the weighted average fusion strategy;

[0122] In some embodiments of the present application, when the photovoltaic dual light source image is input as input data into the dual discriminator generative adversarial network, and the target fusion low rank image and the target fusion sparse detail image are output based on the dual discriminator generative adversarial network, and the target fusion image of the photovoltaic power station group string is generated according to the weighted average fusion strategy, it includes:

[0123] The photovoltaic dual light source image is decomposed into a low rank part and a sparse detail part by a low rank sparse decomposition method;

[0124] The low-rank part is input as input data to a first SN-CNN discriminator, and the sparse detail part is input as input data to a second SN-CNN discriminator;

[0125] The first SN-CNN discriminator and the second SN-CNN discriminator are optimized based on a local binary pattern algorithm and an improved Wasserstein distance algorithm;

[0126] The target fusion low-rank image is output based on the optimized first SN-CNN discriminator, wherein the target fusion low-rank image can save temperature intensity information and detail information from the infrared image;

[0127] The target fusion sparse detail image is output based on the optimized second SN-CNN discriminator, wherein the target fusion sparse detail image can save texture information and detail information from the visible light image.

[0128] In the embodiment, the Transformer network is used to replace the original GAN generator network structure, and a self-attention mechanism is added to enhance the correlation between each pixel, thereby improving the quality and visual effect of the fusion image and improving the accuracy of fault detection.

[0129] In the embodiment, in the GAN, the Transformer network with higher expression ability is used as the generator network, the low-rank part / sparse detail part of the infrared and visible light is connected in the channel direction, and is used as the input of the generator based on the Transformer network. The image generated by the improved generator can save part of the infrared intensity information and the background information of the visible light image to a certain extent. The spectral normalization layer is introduced in the present application, which effectively eliminates the adverse effects caused by singular sample data and accelerates the convergence speed. This improvement is very obvious because it not only satisfies the Lipschitz condition but also maintains the stability of the parameter matrix, which is not common in the previous GAN model. The spectral normalization convolutional neural network (SN-CNN) is used as the discriminator network, and the fusion image and the infrared image / visible light image are used as the input of the improved double discriminator (SN-CNN). When the two improved discriminators cannot distinguish whether the input image is a generated image or a source image, it is considered that the fusion image has saved the infrared intensity information and the detail information of the visible light. The present application can significantly improve the type and accuracy of photovoltaic module fault detection by fusing infrared and visible light images.

[0130] Compared with the CNN generator network in the traditional GAN, the Transformer can establish the global dependence of the image and obtain more global information of the image in the embodiment. Moreover, the structure of the Transformer as the generator can give the model stronger expression ability and generalization ability. The CBAM module as a lightweight universal module can be plugged and played, and it is placed after the feature extraction module. The introduction of the self-attention mechanism in the generator enhances the dependence between each pixel, so that the fused image can better preserve the information in the infrared image and the visible light image. In order to better adapt to the image fusion task, the model of the application adjusts the traditional structure, removes the decoder and only retains the encoder. Pruning destroys the structure of the weight matrix of the discriminator, so that the convergence speed of WGAN is slow. Therefore, the spectral normalization layer is introduced into the CNN discriminator network in the traditional GAN, and the weight matrix of each network is limited in a range, that is, by dividing each element in the parameter matrix by its spectral norm, the Lipschitz constant is 1. The SN-CNN discriminator can not only meet the Lipschitz condition, but also maintain the stability of the parameter matrix and speed up the training.

[0131] In the embodiment, the network includes a generator based on the Transformer and two discriminators based on the spectral normalization convolutional neural network (SN-CNN). This structure can not only meet the Lipschitz condition, but also maintain the stability of the parameter matrix and speed up the training process. The photovoltaic dual-light image is used as the input of the improved generator, and the image generated by the generator can preserve some infrared intensity information and background information of the visible light image to a certain extent. The ID2WGAN designs two discriminators based on the SN-CNN. The first discriminator aims to discriminate the generated image and the infrared image, so that the generated image can preserve the temperature intensity information and the detail information from the infrared image. The second discriminator aims to discriminate the generated image and the visible light image, so that the generated image can preserve sufficient texture information and detail information from the visible light image.

[0132] The above technical scheme has the beneficial effects that the application realizes photovoltaic module multi-type fault detection by combining low-rank sparse decomposition with an improved dual-discriminator generative adversarial network to fuse photovoltaic module dual-light source images, and no longer distinguishes infrared faults or visible light faults, effectively improving the fault detection accuracy. Replacing the generator network and the discriminator network of the traditional GAN makes the proposed image fusion method can preserve the target information in the infrared image and the background information in the visible light image at the same time, and selecting the loss function based on the local binary pattern can better preserve the texture information in the visible light image. The fusion effect is obviously better than the single-discriminator generative adversarial network image fusion result.

[0133] In some embodiments of the present application, when optimizing the first SN-CNN discriminator and the second SN-CNN discriminator based on the local binary pattern algorithm and the improved Wasserstein distance algorithm, the following are included:

[0134] The expression of the local binary pattern algorithm is as follows:

[0135]

[0136] wherein LBP(x c , y c ) is the center pixel, p is the number of neighborhood pixels of the center pixel, i p is the neighborhood pixel intensity value of the pth center pixel, i c is the pixel intensity value of the center pixel, when i p -i c ≥ 0, then s(i p -i c ) = 1, when i p -i c < 0, then s(i p -i c ) = 0.

[0137] The expression of the improved Wasserstein distance algorithm is as follows:

[0138]

[0139] wherein J is the objective function of the generative adversarial network, E x～pr is the expectation of the real data distribution p r , that is, the expectation calculation is performed on all data x sampled from the real data distribution, D(x) is the output of the discriminator to the input x, used to judge whether the input x is from the real data or the generated data, D is the discriminator, E z～pz is the expectation of the generated data distribution p z , that is, the expectation calculation is performed on all noise z sampled from the generated data distribution, D(G(z)) is the output of the generator to the input noise z, that is, the generated data, G is the generator, μ is the penalty coefficient, is the expectation of the linear interpolation data , is the change amount of x, P r and P z are the real data distribution and the generated data distribution respectively, is the output of the discriminator to the input .

[0140] S140: Based on the preset tool, the target fusion image is labeled with fault type, and a YOLO series target detection model is trained according to the image data of the target fusion image. After training, the fault type and position of the target fusion image are detected.

[0141] In this embodiment, the Labelimg labeling tool is used to label the target fusion image with fault type.

[0142] In this embodiment, during the labeling process, according to the preset fault type (such as crack, corrosion, wear, etc.), the corresponding area is framed on the image, and the corresponding label is assigned. After labeling, the label information is saved as a text file in YOLO format for subsequent training.

[0143] In this embodiment, the YOLO series target detection model is trained:

[0144] Data set division: divide the labeled image data set into training set and validation set. Generally, the training set contains about 70% of the data, which is used for model training; the validation set contains the remaining 30% of the data, which is used to evaluate the performance of the model.

[0145] Model configuration: select a suitable YOLO model (such as YOLOv7) as the base model according to the needs. Configure the parameters of the model, including input image size, learning rate, batch size, etc.

[0146] Model training: use the training set to train the model. During the training process, the model will learn how to extract features from the image and predict the fault type and position according to these features. After training, use the validation set to evaluate the performance of the model.

[0147] Model optimization: optimize the model according to the validation results. You can improve the accuracy and stability of the model by adjusting the model parameters, adding regularization terms, using more complex network structures, etc.

[0148] In this embodiment, the fault type and position detection:

[0149] Load the model: load the trained YOLO model into the target detection system.

[0150] Image preprocessing: preprocess the fusion image to be detected to meet the input requirements of the model.

[0151] Fault detection: use the model to perform forward propagation calculation on the image to be detected, and get the prediction result. The prediction result will include the type and position information of the fault.

[0152] Result visualization: visualize the prediction result on the image to help users intuitively understand the fault situation.

[0153] The technical scheme has the beneficial effects that the technical scheme improves the type and accuracy of photovoltaic module fault detection, and is suitable for different photovoltaic power station topography, whether it is a fish-light complementary photovoltaic power station or a farm-light complementary photovoltaic power station, the method can be reasonably applied, and the application provides strong support for the digitalization and intelligent development of photovoltaic power stations, and through the automatic image acquisition, fusion and fault detection process, the operation and maintenance efficiency of the photovoltaic power station is greatly improved.

[0154] In order to further illustrate the technical idea of the application, the technical scheme of the application will be described in combination with specific application scenarios.

[0155] Correspondingly, as shown in Figure 2 The application also provides an infrared image and visible light image fusion device of a photovoltaic module, characterized in that,

[0156] An image shooting module is configured to shoot and capture the whole area of the photovoltaic power station by using a pre-deployed unmanned aerial vehicle (UAV) carrying a thermal infrared camera and a visible light camera, and obtain photovoltaic infrared images and photovoltaic visible light images of photovoltaic strings of the photovoltaic power station.

[0157] An image registration module is configured to perform registration processing on the photovoltaic infrared images and the photovoltaic visible light images according to an edge rectangle feature method, and obtain photovoltaic dual-light-source images of the photovoltaic strings of the photovoltaic power station.

[0158] An image processing module is configured to input the photovoltaic dual-light-source images as input data into a dual-discriminator generative adversarial network, and output target fusion low-rank images and target fusion sparse detail images based on the dual-discriminator generative adversarial network, and generate target fusion images of the photovoltaic strings of the photovoltaic power station according to a weighted average fusion strategy.

[0159] An image analysis module is configured to label a fault type of the target fusion images based on a preset tool, and train a YOLO series target detection model based on image data of the target fusion images, and detect the fault type and position of the target fusion images after the training is completed.

[0160] In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0161] Although the present application has been described with reference to the above embodiments, various modifications can be made to the application and equivalents thereof without departing from the scope of the application. In particular, features of the disclosed embodiments can be used in any combination without departing from the scope of the application, and the description of the various embodiments does not imply that the combinations of features are not combinable unless the description states that a combination is not possible. The description of the various embodiments is not meant to limit the application but merely to provide examples of the application.

[0162] It is to be understood that the above description is intended to be illustrative and not restrictive. Many other embodiments will be apparent to those of skill in the art upon reading the above description. The scope of the application should therefore, be determined not with reference to the above description, but should instead be determined with reference to the appended claims, along with their full scope of equivalents. The disclosure of all articles and references referred to herein are incorporated by reference in their entirety.

Claims

1. A method for fusing infrared images and visible light images of a photovoltaic module, characterized in that, include: Based on pre-deployed drones equipped with thermal infrared and visible light cameras, the entire area of ​​the photovoltaic power station is inspected and photographed to obtain photovoltaic infrared and visible light images of the photovoltaic power station strings; The photovoltaic infrared image and photovoltaic visible light image are registered using the edge rectangle feature method to obtain the photovoltaic dual-source image of the photovoltaic power station string; The photovoltaic dual-source image is used as input data and fed into a dual-discriminator generator adversarial network. Based on the dual-discriminator generator adversarial network, a target fusion low-rank image and a target fusion sparse detail image are output. The target fusion image of the photovoltaic power station string is generated according to the weighted average fusion strategy. The target fused image is labeled with fault types using a preset tool, and a YOLO series target detection model is trained based on the image data of the target fused image. After training, the fault types and locations of the target fused image are detected. When registering the photovoltaic infrared image and the photovoltaic visible light image based on the edge rectangle feature method to obtain the photovoltaic dual-source image of the photovoltaic power station string, the process includes: The photovoltaic infrared image and the photovoltaic visible light image are preprocessed respectively, wherein the preprocessing includes noise and interference removal; First edge information and second edge information of the photovoltaic infrared image and the photovoltaic visible light image are extracted based on the edge detection algorithm, and first edge rectangle and second edge rectangle are generated based on the first edge information and the second edge information; The first edge rectangle is transformed to the frequency domain using Fourier transform, and the corresponding first amplitude value in the frequency domain is extracted. The second edge rectangle is transformed to the frequency domain using Fourier transform, and the corresponding second amplitude value in the frequency domain is extracted. Calculate the edge matching degree coefficient of the first edge rectangle and the second edge rectangle based on the first amplitude value and the second amplitude value; Based on the relationship between the edge matching degree coefficient and the preset edge matching degree coefficient, it is determined whether the photovoltaic infrared image can be mapped to the photovoltaic visible light image; When the edge matching degree coefficient is greater than or equal to the preset edge matching degree coefficient, it is determined that the photovoltaic infrared image can be mapped to the photovoltaic visible light image to obtain the photovoltaic dual-source image of the photovoltaic power station string; When the edge matching degree coefficient is less than the preset edge matching degree coefficient, it is determined that the photovoltaic infrared image cannot be mapped to the photovoltaic visible light image, and the photovoltaic infrared image and the photovoltaic visible light image are re-captured until the edge matching degree coefficient is greater than or equal to the preset edge matching degree coefficient. When using the photovoltaic dual-source image as input data, inputting it into a dual-discriminator generative adversarial network (GAN), and outputting a target fused low-rank image and a target fused sparse detail image based on the dual-discriminator GAN, and generating the target fused image of the photovoltaic power station string according to a weighted average fusion strategy, the process includes: The photovoltaic dual-light source image is decomposed into low-rank and sparse detail parts by low-rank sparse decomposition. The low-rank portion is used as input data and fed into the first SN-CNN discriminator, while the sparse detail portion is used as input data and fed into the second SN-CNN discriminator. The first SN-CNN discriminator and the second SN-CNN discriminator are optimized based on the local binary pattern algorithm and the improved Wasserstein distance algorithm; The target fusion low-rank image is output based on the optimized first SN-CNN discriminator, wherein the target fusion low-rank image preserves temperature intensity information and detail information from the infrared image; The target fused sparse detail image is output based on the optimized second SN-CNN discriminator, wherein the target fused sparse detail image preserves texture information and detail information from the visible light image.

2. The method for fusing infrared and visible light images of a photovoltaic module according to claim 1, characterized in that, Calculating the edge matching degree coefficient of the first edge rectangle and the second edge rectangle based on the first amplitude value and the second amplitude value includes: Construct a first amplitude value sequence based on all the first amplitude values, and construct a second amplitude value sequence based on all the second amplitude values; Calculate the first sub-edge matching degree coefficient and the second sub-edge matching degree coefficient of the first edge rectangle and the second edge rectangle based on the first amplitude value sequence and the second amplitude value sequence; The edge matching degree coefficients of the first edge rectangle and the second edge rectangle are calculated based on the first sub-edge matching degree coefficient and the second sub-edge matching degree coefficient.

3. The method for fusing infrared and visible light images of photovoltaic modules according to claim 2, characterized in that, When calculating the first sub-edge matching degree coefficient of the first edge rectangle and the second edge rectangle based on the first amplitude value sequence and the second amplitude value sequence, the following steps are included: Calculate the mean of the first sequence corresponding to the first amplitude value sequence, and calculate the mean of the second sequence corresponding to the second amplitude value sequence; Calculate the difference between the mean of the first sequence and the mean of the second sequence; The first amplitude value sequence and the second amplitude value sequence are compared one by one to determine multiple amplitude value differences; Extract the maximum amplitude difference and the minimum amplitude difference from all amplitude differences, and calculate the extreme amplitude difference between the maximum amplitude difference and the minimum amplitude difference; The first sub-edge matching degree coefficient is calculated based on the difference in the mean value, the difference in the extreme amplitude value, the difference in the maximum amplitude value, and the difference in the minimum amplitude value of the sequence.

4. The method for fusing infrared and visible light images of a photovoltaic module according to claim 3, characterized in that, When calculating the first sub-edge matching degree coefficient based on the difference in the sequence mean, the difference in extreme amplitude, the difference in the maximum amplitude, and the difference in the minimum amplitude, the following steps are included: The first sub-edge matching degree coefficient is calculated according to the following formula: ； Where q is the first sub-edge matching degree coefficient, r1 is the difference in sequence mean, r2 is the difference in extreme amplitude, r3 is the difference in maximum amplitude, and r4 is the difference in minimum amplitude.

5. The method for fusing infrared and visible light images of a photovoltaic module according to claim 4, characterized in that, When calculating the second sub-edge matching degree coefficient of the first edge rectangle and the second edge rectangle based on the first amplitude value sequence and the second amplitude value sequence, the following steps are included: Randomly match the amplitude values ​​in the first amplitude value sequence and the second amplitude value sequence pairwise to obtain multiple random amplitude value matching sets; The second sub-edge matching degree coefficient is calculated according to the following formula: ； Where t is the second sub-edge matching degree coefficient, n1 is the number of random matching sets of amplitude values, and a u For the u-th amplitude value, randomly match the larger amplitude value from the set, d u For the u-th amplitude value, randomly match the smaller amplitude value from the set, where n2 is the number of the first amplitude values ​​in the first amplitude value sequence, and g f Let n3 be the f-th first amplitude value in the first amplitude value sequence, and n3 be the number of second amplitude values ​​in the second amplitude value sequence. h It is the h-th second amplitude value in the second amplitude value sequence.

6. The method for fusing infrared and visible light images of a photovoltaic module according to claim 5, characterized in that, When calculating the edge matching degree coefficients of the first edge rectangle and the second edge rectangle based on the first sub-edge matching degree coefficient and the second sub-edge matching degree coefficient, the following steps are included: Configure a first calculation coefficient for the first sub-edge matching degree coefficient, and configure a second calculation coefficient for the first sub-edge matching degree coefficient; The edge matching coefficient of the first and second edge rectangles is calculated according to the following formula: ； Where v is the edge matching degree coefficient of the first edge rectangle and the second edge rectangle, m3 is the first calculation coefficient, m2 is the second calculation coefficient, and m1+m2=1, m1>m2.

7. The method for fusing infrared and visible light images of a photovoltaic module according to claim 1, characterized in that, When optimizing the first SN-CNN discriminator and the second SN-CNN discriminator based on the local binary pattern algorithm and the improved Wasserstein distance algorithm, the following steps are included: The expression for the local binary pattern algorithm is as follows: ； Among them, LBP(x c y c Let be the center pixel, p be the number of neighboring pixels of the center pixel, and i be the number of neighboring pixels. p Let i be the intensity value of the neighboring pixels of the p-th center pixel. c The pixel intensity value of the center pixel, when i p -i c When ≥0, then s(i) p -i c )=1, when i p -i c When < 0, then s(i p -i c ) = 0; The improved Wasserstein distance algorithm is expressed as follows: ； in, To generate the objective function of the adversarial network, E x～pr To determine the true data distribution p r The expectation, that is, the expectation calculation for all data x sampled from the real data distribution. D(x) is the output of the discriminator to the input x, used to determine whether the input x comes from real data or generated data. D is the discriminator, and E is the output of the discriminator. z～pz To generate data distribution p z The expectation is calculated over all noise z sampled from the generated data distribution, where D(G(z)) is the generator's output to the input noise z, i.e., the generated data, and G is the generator. The penalty coefficient is... For linear interpolation data Expectations Let P be the change in x. r and P z These represent the real data distribution and the generated data distribution, respectively, D( () is the discriminator for the input The output of .

8. A device for fusing infrared images and visible light images of a photovoltaic module, applied to the method for fusing infrared images and visible light images of a photovoltaic module as described in any one of claims 1-7, characterized in that, include: The image capture module is used to conduct inspection and photography of the entire photovoltaic power station area based on pre-deployed drones equipped with thermal infrared cameras and visible light cameras, and to acquire photovoltaic infrared images and photovoltaic visible light images of the photovoltaic power station strings; The image registration module is used to register the photovoltaic infrared image and the photovoltaic visible light image based on the edge rectangle feature method to obtain the photovoltaic dual-source image of the photovoltaic power station string; The image processing module is used to take the photovoltaic dual-source image as input data, input it into the dual discriminator to generate an adversarial network, and output the target fusion low-rank image and the target fusion sparse detail image based on the dual discriminator to generate the target fusion image of the photovoltaic power station string according to the weighted average fusion strategy. The image analysis module is used to label the target fused image with fault types based on preset tools, and to train a YOLO series target detection model based on the image data of the target fused image. After training, the model detects the fault types and locations of the target fused image.

Images