A photovoltaic power station unmanned inspection method and system based on multimodal fusion detection

Abstract

The present invention discloses an unmanned inspection method and system for photovoltaic power stations based on multimodal fusion detection, which relates to the field of intelligent operation and maintenance technology of photovoltaic power stations. The method mainly includes the following steps: (1) collecting multimodal data of photovoltaic components and performing preprocessing operations; (2) using pre-trained modal data markers to uniformly map heterogeneous data into discrete tags; (3) using training samples to train a multimodal fusion detection network based on the Transformer encoder-decoder architecture, adjusting network parameters, and obtaining a trained model; (4) using the trained model to output defect type, location bounding box, and severity score based on the acquired multimodal data or single-modal data of the photovoltaic components. The present invention solves the problems of high missed detection rate and low efficiency of multimodal fusion in traditional single-modal detection, and specifically proposes an inspection acquisition module, a data processing module, a multimodal fusion detection module, and a fault decision module, realizing full process automation from data acquisition to intelligent decision-making, significantly reducing the operation and maintenance cost of photovoltaic power stations and improving fault response efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of photovoltaic power station inspection, and in particular relates to an unmanned inspection method and system for photovoltaic power stations based on multimodal fusion detection. Background Art

[0002] As a vital component of clean energy, the operational efficiency of photovoltaic power plants is directly related to energy output and economic benefits. However, photovoltaic power plants typically occupy large areas and have a large number of components. Traditional manual inspection methods suffer from low efficiency, high costs, and a high rate of missed inspections.

[0003] Existing drone inspection technology primarily relies on single-modal data (such as visible light or infrared images), which cannot fully reflect the operating status of photovoltaic modules. Visible light images have difficulty identifying internal cracks or hot spots, infrared thermal imaging is easily affected by ambient temperature interference, and electrical performance data lacks spatial information. A single modality cannot fully capture complex fault characteristics, resulting in high rates of missed detection and false positives. Existing methods often use simple feature splicing or weighted averaging, failing to fully consider the semantic gap and spatiotemporal asynchrony of heterogeneous data (such as images and measurement time series). Cross-modal interaction mechanisms are weak, making it difficult to achieve global contextual awareness. Under extreme operating conditions such as sensor noise and missing data in some modalities (such as infrared failure at night), the performance of traditional models drops sharply, lacking effective fault tolerance and adaptability. Existing systems typically only output the fault type, lack localization and severity assessment, and rely on manual experience to formulate maintenance strategies, resulting in high operation and maintenance costs and significant response delays.

[0004] While previous research has attempted to introduce multimodal learning to address these issues, it is limited by technical bottlenecks such as the difficulty of aligning heterogeneous data, inefficient cross-modal interaction, and a single pre-training strategy. Therefore, there is an urgent need for an unmanned inspection method for photovoltaic power plants that can deeply integrate multimodal data, adaptively suppress noise interference, and achieve end-to-end detection and decision-making to improve fault identification accuracy and operation and maintenance efficiency. Summary of the Invention

[0005] In response to the shortcomings of the existing models, which are numerous and lacking in unity, the present invention provides an unmanned inspection method and system for photovoltaic power stations based on multimodal fusion detection. By fusing visible light images, infrared images, electroluminescence images and electrical performance measurement data, it realizes intelligent perception and precise detection of defects in photovoltaic components and equipment.

[0006] The present invention adopts an unmanned inspection method for photovoltaic power stations based on multimodal fusion detection, comprising the following steps:

[0007] 1) Acquire multimodal data of photovoltaic modules, including visible light images, infrared images, electroluminescence images, and real-time electrical performance measurement data, and perform preprocessing operations on the multimodal data;

[0008] 2) Establish data markers for each modality. The image modality marker is a pre-trained model obtained by training image samples using a vector quantization autoencoder. It is used to encode image blocks into discrete tokens. The measurement modality marker uses a pre-encoded vocabulary to convert electrical performance measurement data into time series tokens.

[0009] 3) Constructing a neural network for multimodal fusion detection, the neural network includes a learnable input embedding layer, a Transformer encoder layer, a cross-attention layer, a tag mask module, and a sequence decoding module; wherein the learnable input embedding layer is used to map discrete tags and temporal tags into a unified vector space and add position vectors to address the problem of heterogeneous data representation differences; the Transformer encoder layer fuses multimodal features through a self-attention mechanism to extract global context information; the cross-attention layer enables encoder and decoder feature interaction during the decoding phase to ensure that the generated content is consistent with the global context; the tag mask module is used to randomly mask some input tags during pre-training to drive the model to learn cross-modal reasoning capabilities; the sequence decoding module is used to generate discrete sequence outputs;

[0010] 4) Using the modal data markers in step 2), the multimodal data training samples are converted into corresponding markers, and then input into the multimodal fusion detection neural network to design a loss function for training, adjust the hyperparameters, and obtain a trained multimodal fusion detection model;

[0011] 5) The single modality or multimodal data to be detected is input into the trained multimodal fusion detection model, and the sequence decoding module of the model outputs the defect classification label, defect location and defect severity level.

[0012] Another object of the present invention is to provide an unmanned inspection system for photovoltaic power stations based on multimodal fusion detection for implementing the above method, comprising:

[0013] Inspection and acquisition module, equipped with a high-resolution visible light camera, infrared thermal imager, electroluminescent imaging device and voltage and current sensor, is used to collect visible light images, infrared images, electroluminescent images and real-time electrical performance measurement data of photovoltaic modules;

[0014] A data processing module, used for preprocessing the collected multimodal data;

[0015] An image modality tagger that encodes image data into discrete tags in chunks;

[0016] The measurement mode marker uses a pre-coded vocabulary to convert electrical performance measurement data into time series markers;

[0017] The multimodal fusion detection module is equipped with a trained multimodal fusion detection neural network, which analyzes the input single modality or multimodal data to identify the defect type, defect location and defect severity level of the photovoltaic module; the neural network includes a learnable input embedding layer, a Transformer encoder layer, a cross-attention layer, a label mask module and a sequence decoding module; among them, the learnable input embedding layer is used to map discrete labels and time series labels to a unified vector space and add position vectors to solve the problem of heterogeneous data representation differences; the Transformer encoder layer fuses multimodal features through a self-attention mechanism to extract global context information; the cross-attention layer realizes the interaction between encoder and decoder features in the decoding stage to ensure that the generated content is consistent with the global context; the label mask module is used to randomly mask part of the input labels in pre-training to drive the model to learn cross-modal reasoning capabilities; the sequence decoding module is used to generate discrete sequence outputs.

[0018] The beneficial effects of the present invention are:

[0019] 1) This paper proposes a unified tagging method for heterogeneous modalities. For the first time in the field of photovoltaic inspection, visible light, infrared, and electroluminescent images and electrical performance measurement data are mapped to a unified semantic space through a customized tagger, achieving cross-modal feature alignment and laying the foundation for fusion detection.

[0020] 2) This paper proposes a dynamic multimodal attention fusion mechanism, which is the first in the field of multimodal detection to generate adaptive attention weights based on cross-modal feature differences and task requirements. By strengthening the contribution of key modalities and suppressing noise interference, it effectively solves the problem of collaborative perception of heterogeneous data and significantly improves the detection robustness in complex scenarios.

[0021] 3) This paper develops a mask-driven cross-modal training strategy and introduces a labeling mask module. By randomly masking the multimodal input, the model is driven to learn robust features, enhancing the model's fault tolerance to partial modal loss (such as sensor failure) and ensuring detection stability under extreme working conditions.

[0022] 4) This paper develops an explainable fusion decision network, which dynamically generates modal fusion weights based on a gating mechanism and a causal reasoning model, and visually displays the contribution of each modality. This improves detection accuracy while meeting the explainability requirements of industrial detection scenarios and supports fault root cause tracing. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is the algorithm structure diagram of the multimodal fusion detection neural network;

[0024] Figure 2 This is a block diagram of the equipment composition of the unmanned inspection system for photovoltaic power stations based on multimodal fusion detection. DETAILED DESCRIPTION

[0025] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] Example 1, as Figure 1 As shown, the unmanned inspection method of a photovoltaic power station based on multimodal fusion detection of the present invention includes the following steps.

[0027] 1) Use high-resolution visible light cameras, infrared thermal imagers, and electroluminescent imaging devices to collect visible light images, infrared images, and electroluminescent images of photovoltaic modules. Collect the voltage and current data of the strings in real time on the inverter side of the photovoltaic power station, and perform denoising, enhancement, and cleaning operations on the multimodal data.

[0028] 2) Establish data markers for each modality. The image modality marker is a pre-trained model obtained by training image samples using a vector quantization autoencoder. It is used to encode image blocks into discrete tokens. The measurement modality marker uses a pre-encoded vocabulary to convert electrical performance measurement data into time series tokens.

[0029] Specifically, visible light, infrared and electroluminescence images are used to train a vector quantized variational autoencoder, and then the encoder and quantizer are used to convert the image into a discrete label vector.

[0030] More specifically, the image modality tagger first divides the input image into n local blocks in order, then maps each image block into a continuous feature vector through a variational autoencoder, and then uses a pre-trained codebook to replace the continuous features with the nearest vector in the codebook. The calculation formula is as shown in (1), and finally a discrete tag matrix is ​​generated;

[0031]

[0032] Among them, e j is the vector in the pre-trained codebook, k i is the codebook index corresponding to each image block.

[0033] The voltage and current data are converted into sequence tags using a word segmenter. The word segmenter covers common abnormal patterns (such as "voltage drop_5%" and "current oscillation_10Hz"). The classification and detection box sequences in the target are also standardized, that is, [c1, x1 min ,y1 min ,x1 max ,y1 max ,c2,x2 min ,y2 min ,x2 max ,y2 max ,…], are mapped to [1,n bins ] within the range.

[0034] 3) Establish a multimodal fusion detection model. The neural network includes a learnable input embedding layer, a Transformer encoder layer, a cross-attention layer, a tag mask module and a sequence decoding module. Among them, the learnable input embedding layer is used to map discrete tags and temporal tags to a unified vector space and add position vectors to solve the problem of heterogeneous data representation differences. The Transformer encoder layer fuses multimodal features through the self-attention mechanism to extract global context information. The cross-attention layer realizes the interaction between encoder and decoder features in the decoding stage to ensure that the generated content is consistent with the global context. The tag mask module is used to randomly mask part of the input tags during pre-training to drive the model to learn cross-modal reasoning capabilities. The sequence decoding module is used to generate discrete sequence outputs.

[0035] The learnable input embedding layer converts discrete tags into vectors. Each modality tag is converted into a unified vector through the learnable input embedding layer, and the position vector and modality embedding are added. The specific process is as follows:

[0036]

[0037] Where T vis 、T ir 、T el and T elect They are the marks of visible light image, infrared image, electroluminescence image and electrical energy measurement mode, E vis 、E ir 、E el and E elect are the position vectors corresponding to the visible light image, infrared image, electroluminescence image and electrical energy measurement mode, E mod represents modality embedding, is the output of the learnable input embedding layer.

[0038] For the position vector, the image modality is added with a two-dimensional position vector (i.e., two-dimensional sinusoidal encoding), and the measurement modality is added with a one-dimensional position vector, as shown in the following formula (3-4):

[0039]

[0040] Where i and j are the pixel numbers of the image token, k is the dimension index, d is the embedding dimension, and ind is the position number of the sequence token.

[0041] The token mask module defines a random mask matrix and multiplies the output of the learnable input embedding layer with the mask matrix to obtain the masked token.

[0042] The Transformer encoder layer encodes the vectors of each modality, and the cross-attention layer fuses the encoded features of each modality to calculate self-attention and cross-attention.

[0043] The masked markers output by the marker mask module pass through the Transformer encoder and are then aligned and fused in the cross-attention layer. The contribution of each modality is dynamically adjusted through the gating mechanism. The fusion process is shown in formula (5-6):

[0044]

[0045]

[0046] Among them F i and F j Represents the features of each modality after being encoded by the Transformer encoder, F ij represents the attention from modality i to modality j, F fused Represents the fused features, ω i 、ω j 、ω ij represents the attention score of the gating mechanism, d k Represents the zoom click factor.

[0047] The sequence decoding module is based on the prediction of the cross-attention output defect detection task requirements. Specifically, the sequence decoding module adopts an autoregressive Transformer decoder approach and uses attention masks to separate the decoder tokens of different modalities, ensuring that the decoder produces consistent output for each specific modality.

[0048] 4) Use the modal marker vectors obtained in step 2) to add modal embedding, and then input it into the neural network of multimodal fusion detection for training. The loss function is as follows. Adjust the weights to obtain a converged multimodal fusion detection model.

[0049]

[0050] Where α is the category balance factor, γ is the parameter for focusing on difficult and easy samples, and p t is the target category prediction probability, λ GioU and λ L1 are adjustment coefficients of loss weight, b pred is the predicted bounding box coordinate, b gt is the coordinate of the real bounding box, ρ is the Euclidean distance between the center point of the predicted box and the real box, and c is the distance between b pred with b gt The diagonal length of the minimum closed area of ​​the two boxes, IoU is b pred with b gtThe degree of overlap between the two frames; in this embodiment, α = 0.25, γ = 2, λ GioU =1,λ L1 =1.

[0051] 5) Input the single modality and multimodal data to be detected into the trained multimodal fusion detection model, and the model outputs the defect label (hidden crack, hot spot, stain, etc.), the bounding box coordinates (x min ,y min ,x max ,y max ) and severity (grades 1 to 5).

[0052] Embodiment 2: Corresponding to the aforementioned embodiment of an unmanned inspection method for a photovoltaic power station based on multimodal fusion detection, the present invention further provides an embodiment of an unmanned inspection system for a photovoltaic power station based on multimodal fusion detection. Figure 2 The block diagram of the unmanned inspection system for photovoltaic power stations based on multimodal fusion detection according to an embodiment is shown, and the device includes:

[0053] The inspection and acquisition module is used to capture visible defects such as surface contamination and cracks on the components, and can penetrate the packaging material to detect internal defects such as hidden cracks and broken grids. It also monitors abnormal heating such as component hot spots and diode failures in real time. In this embodiment, the inspection and acquisition module is equipped with a high-resolution visible light camera, an infrared thermal imager, an electroluminescent imaging device, and a voltage and current sensor to collect visible light images, infrared images, electroluminescent images, and real-time electrical performance measurement data of the photovoltaic components.

[0054] The data processing module uses the Retinex algorithm to enhance low-light electroluminescent images, applies non-local mean filtering to eliminate infrared image noise, and compensates for IV curve measurement deviation based on ambient temperature and humidity sensor data. The image mode marker is used to encode image data into discrete tags in blocks; the measurement mode marker uses a pre-coded vocabulary to convert electrical performance measurement data into time series tags.

[0055] The multimodal fusion detection module is equipped with a trained multimodal fusion detection neural network, which analyzes the input single modality or multimodal data to identify the defect type, defect location and defect severity level of the photovoltaic module; the neural network includes a learnable input embedding layer, a Transformer encoder layer, a cross-attention layer, a label mask module and a sequence decoding module; among them, the learnable input embedding layer is used to map discrete labels and time series labels to a unified vector space and add position vectors to solve the problem of heterogeneous data representation differences; the Transformer encoder layer fuses multimodal features through a self-attention mechanism to extract global context information; the cross-attention layer realizes the interaction between encoder and decoder features in the decoding stage to ensure that the generated content is consistent with the global context; the label mask module is used to randomly mask part of the input labels in pre-training to drive the model to learn cross-modal reasoning capabilities; the sequence decoding module is used to generate discrete sequence outputs. Based on the results of the above algorithm, a report is automatically generated, including the defect heat map distribution (including longitude and latitude coordinates), multimodal comparative diagnosis map (visible light / infrared / EL three-view), power generation loss prediction value (LSTM time series prediction ±5% error), and the priority evaluation matrix P, as shown in Formula (8). The mobile maintenance APP supports AR navigation, and maintenance personnel can locate defective components through spatial anchor points, thus forming an intelligent operation and maintenance closed-loop system.

[0056] P=0.3S loss +0.7C safety (8)

[0057] Among them S loss is the power generation loss, C safety It is the safety hazard level.

[0058] In Example 3, to further demonstrate the effectiveness of the present invention, we conducted an experimental validation experiment using a public ELPV dataset and datasets collected from ten PV power plants. The mixed dataset included 1020 sets of cascade-level visible light, infrared, and voltage and current data. To verify the effectiveness of the present invention, we designed multiple evaluation metrics, including objective measures such as accuracy (Acc), mean average precision (mAP), and recall.

[0059] Table 1 Experimental results

[0060] Acc(%) mAP (%) Recall (%) Visible light image defects 88.6 87.2 89.8 Infrared image defects 92.1 90.5 95.8 Inverter failure 79.5 82.3 85.7 total 86.7 86.7 90.4

[0061] Table 1 shows the results of the evaluation of the present invention, in which the defects that can be identified from each modal information are counted separately. It can be seen that the present method obtains relatively superior results, with the highest defect detection accuracy of infrared images and the highest recall rate among the overall indicators, which proves the accurate detection rate of the present method.

[0062] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A photovoltaic power station unmanned inspection method based on multimodal fusion detection, characterized in that: The following steps are involved: 1) Acquire multimodal data of photovoltaic modules, including visible light images, infrared images, electroluminescence images, and real-time electrical performance measurement data, and perform preprocessing operations on the multimodal data; 2) Establish data markers for each modality. The image modality marker is a pre-trained model obtained by training image samples using a vector quantization autoencoder, which is used to encode image blocks into discrete tokens. The measurement modality marker uses a pre-encoded vocabulary to convert electrical performance measurement data into time series tokens. 3) Constructing a neural network for multimodal fusion detection, the neural network includes a learnable input embedding layer, a Transformer encoder layer, a cross-attention layer, a tag mask module, and a sequence decoding module. The learnable input embedding layer is used to map discrete tags and temporal tags into a unified vector space and add position vectors to address the problem of heterogeneous data representation differences. The Transformer encoder layer fuses multimodal features through a self-attention mechanism to extract global context information. The cross-attention layer implements encoder and decoder feature interaction during the decoding phase to ensure that the generated content is consistent with the global context. The tag mask module is used to randomly mask the tags of some modalities during the training phase to drive the model to learn cross-modal reasoning capabilities. The sequence decoding module is used to generate discrete sequence outputs. 4) Using the modal data markers in step 2), the multimodal data training samples are converted into corresponding markers, and then input into the multimodal fusion detection neural network to design a loss function for training, adjust the hyperparameters, and obtain a trained multimodal fusion detection model; 5) The single modality or multimodal data to be detected is input into the trained multimodal fusion detection model, and the sequence decoding module of the model outputs the defect classification label, defect location and defect severity level.

2. The unmanned inspection method for photovoltaic power stations based on multimodal fusion detection according to claim 1 is characterized in that: In step 1), the preprocessing operation includes classification and denoising enhancement of multimodal data.

3. The unmanned inspection method for photovoltaic power stations based on multimodal fusion detection according to claim 1 is characterized in that: In step 2), the image modality tagger first divides the input image into n local blocks in order, and then maps each image block into a continuous feature vector through a variational autoencoder, and then uses a pre-trained codebook to replace the continuous features with the nearest vector in the codebook. The calculation formula is as shown in (1), and finally a discrete tag matrix is ​​generated; Among them, e j is the vector in the pre-trained codebook, k i The codebook index corresponding to each image block.

4. The unmanned inspection method for photovoltaic power stations based on multimodal fusion detection according to claim 1 is characterized in that In step 3), each modality tag is converted into a unified vector through a learnable input embedding layer, and the position vector and modality embedding are added. The specific process is as follows: Where T vis 、T ir 、T el and T elect They are the marks of visible light image, infrared image, electroluminescence image and electrical energy measurement mode, E vis 、E ir 、E el and E elect are the position vectors corresponding to the visible light image, infrared image, electroluminescence image and electrical energy measurement mode, E mod represents modality embedding, is the output of the learnable input embedding layer.

5. The unmanned inspection method for photovoltaic power stations based on multimodal fusion detection according to claim 4 is characterized in that In step 3), the tag mask module defines a random mask matrix and multiplies the output of the learnable input embedding layer with the mask matrix to obtain the masked tag; The masked markers output by the marker mask module pass through the Transformer encoder and are then aligned and fused in the cross-attention layer. The contribution of each modality is dynamically adjusted through the gating mechanism. The fusion process is shown in formula (3-4): Among them F i and F j Represents the features of each modality after being encoded by the Transformer encoder, F ij represents the attention from modality i to modality j, F fused Represents the fused features, ω i 、ω j 、ω ij represents the attention score of the gating mechanism, d k Represents the zoom click factor.

6. The unmanned inspection method for photovoltaic power stations based on multimodal fusion detection according to claim 5 is characterized in that In step 3), the sequence decoding module adopts an autoregressive Transformer decoder approach, using attention masks to separate the decoder tokens of different modalities, ensuring that the decoder produces consistent output for each specific modality.

7. The unmanned inspection method for photovoltaic power stations based on multimodal fusion detection according to claim 1 is characterized in that In step 4), the loss function is as shown in formula (5); Where α is the category balance factor, γ is the parameter for focusing on difficult and easy samples, and p t is the predicted probability of the target category, λ GioU and λ L1 are adjustment coefficients of loss weight, b pred is the predicted bounding box coordinate, b gt is the real bounding box coordinate, ρ is b pred with b gt The Euclidean distance of the center point, c is the distance including b pred with b gt The diagonal length of the minimum closed area of ​​the two boxes, IoU is b pred with b gt The degree of overlap between the two frames.

8. An unmanned inspection system for photovoltaic power stations based on multimodal fusion detection, characterized in that: include: Inspection and acquisition module, equipped with a high-resolution visible light camera, infrared thermal imager, electroluminescent imaging device and voltage and current sensor, is used to collect visible light images, infrared images, electroluminescent images and real-time electrical performance measurement data of photovoltaic modules; A data processing module, used for preprocessing the collected multimodal data; An image modality tagger that encodes image data into discrete tags in chunks; The measurement mode marker uses a pre-coded vocabulary to convert electrical performance measurement data into time series markers; The multimodal fusion detection module is equipped with a trained multimodal fusion detection neural network, which analyzes the input single modality or multimodal data to identify the defect type, defect location and defect severity level of the photovoltaic module; the neural network includes a learnable input embedding layer, a Transformer encoder layer, a cross-attention layer, a label mask module and a sequence decoding module; among them, the learnable input embedding layer is used to map discrete labels and time series labels to a unified vector space and add position vectors to solve the problem of heterogeneous data representation differences; the Transformer encoder layer fuses multimodal features through a self-attention mechanism to extract global context information; the cross-attention layer realizes the interaction between encoder and decoder features in the decoding stage to ensure that the generated content is consistent with the global context; the label mask module is used to randomly mask part of the input labels in pre-training to drive the model to learn cross-modal reasoning capabilities; the sequence decoding module is used to generate discrete sequence outputs.

Images