Defect segmentation method for reference photovoltaic module

Abstract

The invention relates to a reference photovoltaic module defect segmentation method, which specifically comprises the following steps of: firstly, loading an electroluminescent image and natural language description of an expected detection defect from an existing multi-mode photovoltaic defect data set, and respectively extracting image features and text features through a pre-trained image model and a pre-trained language model; then, designing a multi-modal fusion module, and effectively combining image and text features to form unified fusion feature representation; further, a multi-granularity perceptual grouping module is designed to handle defect segmentation requirements of different granularities, identifying single, multiple, and no-target defects that conform to the text. And decoding a target mask from the extracted image-text features, fusion features and perceptual grouping features through a text-guided decoder, thereby accurately segmenting the specified defect in the photovoltaic module battery image according to the input reference text. Different from a traditional photovoltaic defect segmentation task of uniformly segmenting all photovoltaic defects in the graph, the method can perform personalized detection on defects expected to be detected according to actual operation and maintenance requirements of the photovoltaic power station so as to support further operation and maintenance decisions of the photovoltaic power station. And the operation and maintenance work of the actual photovoltaic power station can be obviously improved in individuation and high efficiency.

Description

Technical Field

[0001] The present invention relates to a photovoltaic module defect detection method, and in particular to a reference photovoltaic module defect segmentation method, and belongs to the field of multimodal learning. Background Art

[0002] Solar energy is a promising renewable energy source with great potential in alleviating the global energy crisis. As the core of the solar energy system, the performance of photovoltaic modules is crucial to the efficiency and reliability of the photovoltaic system. However, photovoltaic modules are prone to various defects such as cracks, breaks and corrosion during manufacturing, transportation and operation. These defects can seriously reduce the performance and service life of the photovoltaic system. Therefore, timely and accurate detection and segmentation of these defects are essential to ensure the optimal performance, safety and life of the photovoltaic system.

[0003] At present, the research on photovoltaic module defect segmentation technology based on deep learning has been very extensive, and significant progress has been made in this field. Researchers have developed a variety of deep learning models that can better segment photovoltaic module defects. However, these models uniformly segment all defects in the battery image, and cannot perform refined detection of specific types of defects, and it is difficult to meet the personalized detection needs of specified defects. In the actual operation of photovoltaic power stations, different types of defects have different effects on the safety and power generation efficiency of photovoltaic modules. And due to the influence of factors such as the scale, location, environment, and weather of photovoltaic power stations, different types of defects may exist in different photovoltaic power stations, resulting in different operation and maintenance requirements for different photovoltaic power stations. Therefore, if accurate segmentation tasks are performed for specific defect types, it will help meet the personalized operation and maintenance needs of photovoltaic power stations.

[0004] However, there is currently no method for personalized segmentation of specified defects based on multimodal image and text datasets in the field of photovoltaic module defect detection. It is difficult to detect the specified type of defects that meet the personalized operation and maintenance needs of the current power station. Therefore, a new solution is urgently needed to solve the above problems. Summary of the invention

[0005] The present invention aims at the problems existing in the prior art and provides a reference photovoltaic module defect segmentation method. The scheme can detect defects in the image that are consistent with the reference text description according to the actual operation and maintenance needs of the photovoltaic power station, thereby realizing personalized operation and maintenance of the photovoltaic power station.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows: a reference photovoltaic module defect segmentation method based on text description:

[0007] S1. Build a deep learning model with the input of electroluminescent images of photovoltaic modules and text descriptions of defects to be detected. Through training, it can accurately segment the specified defects in the photovoltaic module cell images based on the input reference text.

[0008] S2: Load the electroluminescence image and the natural language description of the desired defect from the existing multimodal dataset.

[0009] The pre-trained image model and language model are used to extract high-level image features and high-level text features respectively;

[0010] S3: Design a multimodal fusion module to effectively combine image and text features to form a unified fusion feature representation;

[0011] S4. Design a multi-granularity-aware grouping module to handle defect segmentation requirements of different granularities and identify single, multiple, and non-target defects that match the text;

[0012] S5. Through a text-guided decoder, based on the extracted graphic features, fusion features and perception grouping features, it is determined whether the target exists and the target mask is decoded, so as to accurately segment the specified defects in the photovoltaic module cell image according to the input reference text.

[0013] Furthermore, in step S1, the input electroluminescent image is subjected to an image encoder to extract image features, and the reference text is subjected to a text encoder to extract text features. The extracted features are input into a multimodal fusion module for inter-modal alignment and fusion; the aligned fusion features are input into a multi-granularity perception grouping module to aggregate features of the same category from coarse to fine; and the output grouping features and fusion features are input into a text guidance decoder to generate a segmented image. The difference between the predicted value and the true value is calculated by cross entropy loss.

[0014] Furthermore, in step S2, for the input EL image, a pre-trained image model is used as an image encoder to extract high-level image features. For the information in the reference text, a pre-trained text model is used as a text encoder to extract high-level text features. Specifically, in the forward process, an image (I) of size H×W is first input into the image encoder, and a multi-scale feature map is generated through four stages: Input the reference text (T) into the text encoder to obtain the text feature f t .

[0015]

[0016] Among them, Encoder Image With Encoder Text are pre-trained image encoder and text encoder, respectively,1 is the number of channels for extracting image features, C 2 To extract the embedding dimension of text features, N t The maximum number of words in the input text.

[0017] Furthermore, in step S3, the input text feature F t After a multi-layer perceptron aligns the feature scale. The input single-scale image feature F i After feature reshaping, the feature scale is aligned through a linear layer to obtain a feature size of HW×C. Then, the cross attention of text features and image features and the self-attention of the image are performed respectively. The specific processing process can be defined by the following formula

[0018] F fusion =Conv 1×1 (Concat(MHCA(γ t ,γ i ),MHSA(γ i )))

[0019] γ i =Linear(Reshape(F i )),γ t =MLP(F t )

[0020] Among them, Conv 1×1 is a 1×1 convolution, Concat represents feature concatenation, MHCA and MHSA represent multi-head cross attention and multi-head self-attention operations respectively, and γ t ,γ i They represent the text features and image features after dimension alignment respectively.

[0021] Furthermore, in step S4, based on the learnable category query Q learnable Together with the cross-attention mechanism, we learn pixel representations of the same category in the image. Each query is equivalent to a category center, which can capture the common features of the category. Specifically, the image feature F at a certain scale of the input i And the fusion feature F fusion First, we further enhance it by using cross attention. The results are similar to those of the learnable category query Q learnable Generate group features through t times of cross attention

[0022]

[0023] Where MHCA stands for Multi-Head Cross Attention Operation, is the learnable category query at the jth stage.

[0024] Furthermore, in step S5, the text features, fusion features and grouping features are further fused through a cross attention mechanism through a text-guided decoder. The image features are pre-segmented through a pixel decoder to obtain pyramid decoding features. The fusion result is concatenated with the pre-segmentation result to obtain the pyramid decoding feature F decode , and finally generate the segmentation mask I through a convolution module mask , a multi-layer perceptron is used to generate target indications and determine whether there is a corresponding target.

[0025] In step S5, the segmentation mask and target indication are quantified by cross entropy to determine the difference between the model prediction and the true value. The model loss consists of two parts: mask segmentation loss and target existence loss. The two are the two outputs of the model. mask The cross entropy loss between the target indication and the true value. The final loss is the weighted sum of the two:

[0026]

[0027] Specifically, the weight values ​​α and β are set to 1 and 0.1 respectively.

[0028] Compared with the prior art, the present invention has the following advantages: by combining image and text data, the solution can more comprehensively understand and analyze the defects of photovoltaic modules. Images provide intuitive visual information, while text descriptions provide semantic information. In actual operation and maintenance, photovoltaic power stations can segment defects that have a greater impact on efficiency according to actual operation and maintenance needs and text descriptions, rather than uniformly segmenting all categories of defects in the image as in traditional defect detection. The constructed deep learning network is based on multimodal fusion and multi-granularity perception grouping, which can effectively segment the corresponding defects in the image according to multi-granularity text. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the overall process of the embodiment,

[0030] Figure 2 Schematic diagram of multimodal fusion, multi-granularity perceptual grouping and text-guided decoder of an embodiment. DETAILED DESCRIPTION

[0031] In order to deepen the understanding of the present invention, the present embodiment is described in detail below with reference to the accompanying drawings.

[0032] Example: See Figure 1 , a reference photovoltaic module defect segmentation method, comprising the following steps in sequence:

[0033] S1. Build a deep learning model with the input of electroluminescent images of photovoltaic modules and text descriptions of defects to be detected. Through training, it can accurately segment the specified defects in the photovoltaic module cell images based on the input reference text.

[0034] In this embodiment, the input electroluminescent image is extracted with an image encoder, and the reference text is extracted with a text encoder. The extracted features are input into the multimodal fusion module for inter-modal alignment and fusion; the aligned fusion features are input into the multi-granularity perception grouping module to aggregate features of the same category from coarse to fine; and the output grouping features and fusion features are input into the text guidance decoder to generate a segmented image. The difference between the predicted value and the true value is calculated by the cross entropy loss.

[0035] S2: Load the electroluminescent image and the natural language description of the desired defect from the existing multimodal dataset, and extract high-level image features and high-level text features through the pre-trained image model and language model respectively.

[0036] In this embodiment, for the input EL image, a pre-trained image model is used as an image encoder to extract high-level image features. Similarly, for the information in the reference text, a pre-trained text model is used as a text encoder to extract high-level text features. Specifically, in the forward process, an image (I) of size H×W is first input into the image encoder, and a multi-scale feature map is generated through four stages: Input the reference text (T) into the text encoder to obtain the text feature f t .

[0037]

[0038] Among them, Encoder Image With Encoder Text are pre-trained image encoder and text encoder, respectively, 1 is the number of channels for extracting image features, C 2 To extract the embedding dimension of text features, N t The maximum number of words in the input text.

[0039] S3: Design a multimodal fusion module to effectively combine image and text features to form a unified fusion feature representation;

[0040] In this embodiment, the input text feature F t After a multi-layer perceptron aligns the feature scale. The input single-scale image feature F iAfter feature reshaping, the feature scale is aligned through a linear layer to obtain a feature size of HW×C. Then, the cross attention of text features and image features and the self-attention of the image are performed respectively. The specific processing process can be defined by the following formula

[0041] F fusion =Conv 1×1 (Concat(MHCA(γ t ,γ i ),MHSA(γ i )))

[0042] γ i =Linear(Reshape(F i )),γ t =MLP(F t )

[0043] Among them, Conv 1×1 is a 1×1 convolution, Concat represents feature concatenation, MHCA and MHSA represent multi-head cross attention and multi-head self-attention operations respectively, and γ t ,γ i They represent the text features and image features after dimension alignment respectively.

[0044] S4. Design a multi-granularity-aware grouping module to handle defect segmentation requirements of different granularities and identify single, multiple, and non-target defects that match the text;

[0045] In this embodiment, based on the learnable category query Q learnable Together with the cross-attention mechanism, we learn pixel representations of the same category in the image. Each query is equivalent to a category center, which can capture the common features of the category. Specifically, the image feature F at a certain scale of the input i And the fusion feature F fusion First, we further enhance it by using cross attention. The results are similar to those of the learnable category query Q learnable Generate group features through t times of cross attention

[0046]

[0047] Where MHCA stands for Multi-Head Cross Attention Operation, is the learnable category query at the jth stage.

[0048] S5. A text-guided decoder is used to decode a target mask and a target indication for determining whether a target exists from the extracted graphic features, fusion features and perceptual grouping features.

[0049] In this embodiment, through a text-guided decoder, text features, fusion features and grouping features are further fused through a cross-attention mechanism. Image features are pre-segmented through a pixel decoder to obtain pyramid decoding features. The fusion result is concatenated with the pre-segmentation result to obtain the pyramid decoding feature F decode , and finally generate the segmentation mask I through a convolution module mask , a multi-layer perceptron is used to generate target indications and determine whether there is a corresponding target in the image.

[0050] In this embodiment, the segmentation mask and target indication are quantified by cross entropy to quantify the difference between the model prediction and the true value. The model loss consists of two parts: mask segmentation loss and target existence loss. The two are the two outputs of the model I mask The cross entropy loss between the target indication and the true value. The final loss of the model is the weighted sum of the two:

[0051]

[0052] Specifically, the weight values ​​α and β are set to 1 and 0.1 respectively.

[0053] It should be noted that the above embodiments are not intended to limit the protection scope of the present invention, and equivalent changes or substitutions made on the basis of the above technical solutions all fall within the protection scope of the claims of the present invention.

Claims

1. A reference photovoltaic module defect segmentation method, characterized in that: The method comprises the following steps: S1. Build a deep learning model with the input of electroluminescent images of photovoltaic modules and text descriptions of defects to be detected. Through training, it can accurately segment the specified defects in the photovoltaic module cell images based on the input reference text. S2: Load the electroluminescence image and the natural language description of the desired defect from the existing multimodal dataset. Through the pre-trained image model and language model, high-level image features and high-level text features are extracted respectively. S3: Design a multimodal fusion module to effectively combine image and text features to form a unified fusion feature representation. S4. Design a multi-granularity perception grouping module to handle defect segmentation requirements of different granularities and identify single, multiple, and non-target defects that match the text. S5. Through a text-guided decoder, based on the extracted graphic features, fusion features and perception grouping features, it is determined whether the target exists and the target mask is decoded, so as to accurately segment the specified defects in the photovoltaic module cell image according to the input reference text.

2. The reference photovoltaic module defect segmentation method according to claim 1, characterized in that: In step S1, a deep learning model is constructed, the input electroluminescent image is subjected to an image encoder to extract image features, and the reference text is subjected to a text encoder to extract text features, and the extracted features are input into a multimodal fusion module for inter-modal alignment and fusion; the aligned fusion features are input into a multi-granularity perception grouping module to aggregate features of the same category from coarse to fine; The output grouping features and fusion features are input into the text-guided decoder to generate segmented images, and the difference between the predicted value and the true value is calculated through the cross entropy loss.

3. The reference photovoltaic module defect segmentation method according to claim 2, characterized in that: In step S2, for the input electroluminescent image, a pre-trained image model is used as an image encoder to extract high-level image features, and for the information in the reference text, a pre-trained text model is used as a text encoder to extract high-level text features. Specifically, in the forward process, an image (I) of size H×W is first input into the image encoder, and a multi-scale feature map is generated through four stages: Input the reference text (T) into the text encoder to obtain the text feature f t , f t =Encoder Text (T), Among them, Encoder Image With Encoder Text are the pre-trained image encoder and text encoder, C1 is the number of channels for extracting image features, C2 is the embedding dimension for extracting text features, and N t The maximum number of words in the input text.

4. The reference photovoltaic module defect segmentation method according to claim 3, characterized in that: In step S3, the input text feature F t After a multi-layer perceptron aligns the feature scale, the input single-scale image feature F i After feature reshaping, the feature scale is aligned through a linear layer to obtain a feature size of HW×C. Then, the cross attention of text features and image features and the self-attention of the image are performed respectively. The specific processing process is defined by the following formula F fusion =Conv 1×1 (Concat(MHCA(γ t ,γ γ ),MHSA(γ i ))) c i =Linear(Reshape(F i )),c t =MLP(F t ) Among them Cony 1×1 is a 1×1 convolution, Concat represents feature concatenation, MHCA and MHSA represent multi-head cross attention and multi-head self-attention operations respectively, and γ t , γ i They represent the text features and image features after dimension alignment respectively.

5. The reference photovoltaic module defect segmentation method according to claim 4, characterized in that: In step S4, the fusion features obtained above are used to query Q based on the learnable category. learnable With the cross attention mechanism, we learn pixel representations belonging to the same category in the image. Each query is equivalent to a category center, which can capture the common features of the category. Specifically, the image feature F at a certain scale of the input i And the fusion feature F fusion First, we further enhance it through cross attention, and the result is consistent with the learnable category query Q learnable Generate grouping features f through t times of cross attention i GP : Where MHCA stands for Multi-Head Cross Attention Operation, is the learnable category query at the jth stage.

6. The reference photovoltaic module defect segmentation method according to claim 5, characterized in that: In step S5, the text features, fusion features and grouping features are further fused through the cross attention mechanism, and the image features are pre-segmented through a pixel decoder to obtain the pyramid decoding features. The fusion result is concatenated with the pre-segmentation result to obtain the pyramid decoding feature F decode , and finally generate the segmentation mask I through a convolution module mask , and at the same time, a multi-layer perceptron is used to generate target indications to determine whether there is a corresponding target in the image.

7. The reference photovoltaic module defect segmentation method according to claim 6, characterized in that: In step S5, the segmentation mask and target indication are quantified by cross entropy loss to quantify the difference between the model prediction and the true value. The loss of the model consists of two parts: mask segmentation loss and target existence loss, which are the two outputs of the model: I mask and target indication, and calculate the cross entropy loss between the two and the corresponding true value respectively. The final loss of the model is the weighted sum of the two: Among them, the weight values ​​α and β are set to 1 and 0.1 respectively.

Images