Photovoltaic module subfissure detection method and system based on feature fusion

Abstract

The invention belongs to the technical field of photovoltaic module detection, and provides a photovoltaic module subfissure detection method and system based on feature fusion, which combines two deep learning methods of a graph convolutional network and a convolutional neural network, effectively captures global structure features in an image through the graph convolutional network, and improves the detection accuracy. According to the method, the local features extracted by the convolutional neural network are combined, so that the model can pay attention to the global crack form and the local fine crack information at the same time, a weighted adjacent matrix based on texture similarity is further introduced, the connection relation between nodes is optimized, and the accurate capture capability of the model on cracks under the complex background is improved. According to the method, the precision and robustness of photovoltaic hidden crack detection can be remarkably improved, tiny and complex hidden cracks in a photovoltaic module can be more accurately recognized especially in scenes with complex backgrounds, tiny cracks and large illumination changes, and compared with the prior art, the method has higher real-time performance and adaptability and shows higher detection performance.

Description

Technical Field

[0001] This invention belongs to the field of photovoltaic module testing technology, specifically relating to a method and system for detecting microcracks in photovoltaic modules based on feature fusion. Background Technology

[0002] As the scale of photovoltaic power plants continues to expand, microcracks in photovoltaic modules are gradually becoming a significant factor affecting their operational efficiency. Microcracks typically occur during production, transportation, or installation, and their characteristics are extremely small and difficult to detect. Therefore, timely detection of microcracks in photovoltaic modules is crucial to ensuring the efficient operation of photovoltaic power plants and extending equipment lifespan.

[0003] Traditional methods for detecting hidden cracks mainly rely on manual inspection or basic image processing techniques. Manual inspection is not only inefficient and time-consuming, but also struggles to detect minute cracks due to the limited sensitivity of the human eye. Basic image processing techniques, such as edge detection and thresholding, while usable in some simple scenarios, have limited performance in complex textured backgrounds or environments with significant lighting variations. In recent years, the application of deep learning techniques, especially convolutional neural networks (CNNs) and graph convolutional networks (GCNs), has improved the accuracy and efficiency of crack detection; however, these methods still face challenges in handling complex backgrounds and detecting minute cracks.

[0004] Traditional CNN methods extract local features from images through convolutional layers and are widely used in image classification and object detection tasks. Additionally, some traditional image processing methods, such as edge detection and thresholding, are also widely used. While these methods can effectively handle the detection of some significant cracks, their main drawback is that their accuracy drops significantly when dealing with complex backgrounds, regions with similar textures, and small cracks. Furthermore, existing technologies generally suffer from the following problems: First, traditional CNN models cannot effectively identify the details and global structure of cracks in photovoltaic power plant images with complex textures and varying lighting conditions, resulting in low detection accuracy. Second, while YOLO has advantages in object detection, its localization accuracy is insufficient, especially in scenes where crack details are blurred. Finally, traditional image processing methods cannot adapt to nonlinear changes and complex crack patterns in images, resulting in insufficient generalization ability of the models. Summary of the Invention

[0005] The purpose of this invention is to solve the problem of microcrack detection in photovoltaic power plants, especially to address the problems of low crack detection accuracy, slow processing speed, and poor adaptability to complex backgrounds and changes in illumination in existing technologies, and to provide a method and system for microcrack detection in photovoltaic modules based on feature fusion.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] A method for detecting microcracks in photovoltaic modules based on feature fusion includes:

[0008] (1) Image acquisition and preprocessing: A 4K resolution camera combined with a drone and a fixed camera was used to acquire surface images of photovoltaic modules. The acquired images were then processed by grayscale conversion, Gaussian filtering for noise reduction, normalization and size adjustment.

[0009] (2) Image region segmentation and graph structure construction: The preprocessed image is divided into several regions, each region is regarded as a graph node, and an adjacency matrix based on spatial adjacency and texture similarity weighting is constructed. The feature information of the adjacency nodes in the adjacency matrix is ​​aggregated by the graph convolutional network to extract the global features of each region image.

[0010] (3) Local feature extraction and graph convolution feature fusion: Local features of each region of the image are extracted by convolutional neural network, and global features and local features are fused by an adaptive weighted fusion strategy.

[0011] (4) Hidden crack detection and localization: The probability of crack existence and the location coordinates of cracks in each region are output by parallel classification head and regression head, and the classification and localization tasks are optimized by joint loss function;

[0012] (5) Training and optimization: The Adam optimizer is used to optimize the deep learning model that combines graph convolutional network and convolutional neural network. The initial learning rate, batch size and training cycle are set. Data augmentation is used to improve the generalization ability. The model optimization process is monitored through the validation set.

[0013] (6) Model deployment and application: The trained deep learning model is converted into TensorFlow Lite format and deployed to the edge computing device. The edge computing device is connected to the photovoltaic power station monitoring system to collect images in real time and perform hidden crack detection.

[0014] Furthermore, in step (2), the adjacency matrix is ​​constructed in the following ways:

[0015] Spatial adjacency: Determines adjacent nodes i and j;

[0016] Texture similarity: The texture similarity between two nodes is measured by calculating the structural similarity index between node i and node j. The calculation formula is as follows:

[0017]

[0018] Among them, A ij SSIM(I) represents the weighted connection strength between node i and node j. i ,I j ) represents the structural similarity index between node i and node j, |x i -xj | 2 The feature difference between node i and node j.

[0019] Furthermore, in step (2), the operation of the graph convolutional network is represented as follows:

[0020]

[0021] Among them, H (l) H represents the feature matrix of the nodes in the l-th layer. (0) Features of the input image patch; It is a standardized adjacency matrix; W (l) σ is the weight matrix of the l-th layer; σ is the ReLU activation function.

[0022] Furthermore, in step (2), the feature information of each region includes the pixel value, texture information and edge information of that region.

[0023] Furthermore, in step (3), the local features include the edge information and texture information of the image. A standard 3×3 convolution kernel is used for feature extraction. After multiple convolution and pooling operations, the local features of each image region are finally obtained. The calculation formula is as follows:

[0024] F i =CNN(I i )

[0025] Among them, I i For the i-th image region, F i This is the local feature vector extracted by CNN.

[0026] Furthermore, in step (3), the fusion formula for the global features and local features is:

[0027] x i =α·H i +(1-α)·F i

[0028] Where, x i The fused features are represented by α, which is a hyperparameter learned through training to balance the influence of global and local information.

[0029] Furthermore, in step (4), the formula for calculating the probability of crack presence by the classification head is as follows:

[0030] P class =W class x i +b class

[0031] Among them, W class Let b be the weight matrix.class For bias terms;

[0032] The formula for calculating the crack location coordinates using the regression head is as follows:

[0033] P loc =W loc x i +b loc

[0034] Among them, W loc Let b be the weight matrix. loc This is a bias term.

[0035] Furthermore, in step (4), the joint loss function includes classification loss and localization loss. The classification loss uses binary cross-entropy loss, and the localization loss uses mean squared error. The joint loss function is defined as follows:

[0036] L=λ1·L class +λ2·L loc

[0037] Among them, L class For classification loss, L loc For localization loss, λ1 and λ2 are hyperparameters used to balance the weights of classification loss and localization loss.

[0038] Furthermore, in step (6), the specific steps for model deployment include:

[0039] (1) Save the model: Save the trained deep learning model as a TensorFlow SavedModel format;

[0040] (2) Model optimization: Convert the SavedModel format model to TensorFlow Lite (.tflite) format using the TensorFlow Lite conversion tool for lightweight inference on the Jetson Xavier NX device;

[0041] (3) Upload the model: Upload the TensorFlow Lite format model file and the required library dependencies to the Jetson Xavier NX device;

[0042] (4) Equipment configuration and connection: Configure the network connection between the equipment and the on-site monitoring camera of the photovoltaic power station, and transmit the image data collected by the camera to the equipment for inference in real time.

[0043] Another object of the present invention is to provide a photovoltaic module microcrack detection system based on feature fusion, comprising:

[0044] The image acquisition and preprocessing module is used to acquire surface images of photovoltaic modules using a 4K resolution camera combined with a drone and a fixed camera, and to perform grayscale conversion, Gaussian filtering for noise reduction, normalization and size adjustment on the acquired images.

[0045] The image region segmentation and graph structure construction module is used to divide the preprocessed image into several regions, each region is regarded as a graph node, and an adjacency matrix based on spatial adjacency and texture similarity weighting is constructed. The feature information of the adjacency nodes in the adjacency matrix is ​​aggregated by the graph convolutional network, and a deep learning model based on the fusion of graph convolutional network and convolutional neural network is constructed.

[0046] The local feature extraction and graph convolutional feature fusion module is used to extract local features of each region through a convolutional neural network, extract global features through a graph convolutional network, and use an adaptive weighted fusion strategy to weightedly fuse global features and local features.

[0047] The hidden crack detection and localization module is used to output the probability of crack presence and the location coordinates of cracks in each region through parallel classification and regression heads, and to optimize the classification and localization tasks using a joint loss function.

[0048] The training and optimization module is used to optimize deep learning models using the Adam optimizer, set the initial learning rate, batch size and training period, improve generalization ability through data augmentation, and monitor the model optimization process through the validation set.

[0049] The model deployment and application module is used to convert the trained deep learning model into TensorFlow Lite format and deploy it to edge computing devices. The edge computing devices are connected to the photovoltaic power station monitoring system to collect images in real time and perform hidden crack detection.

[0050] In combination with the above technical solutions, the beneficial effects of the present invention compared with the prior art are as follows:

[0051] Feature Fusion of Graph Convolutional Networks and Convolutional Neural Networks (CNNs): This invention innovatively combines two deep learning architectures, Graph Convolutional Networks (GCNs) and Convolutional Neural Networks (CNNs), to propose a novel feature fusion method. Specifically, GCNs are used to extract global structural features of images, while CNNs focus on extracting local detail features. Through this weighted fusion, this invention can fully utilize both global and local information in crack detection, improving detection accuracy. Unlike the static fusion methods commonly used in existing technologies, this invention employs an adaptive weighted fusion strategy, which can dynamically adjust the contribution ratio of global and local features, thereby optimizing feature representation, especially in complex backgrounds and noisy environments. Through the adaptive weighted fusion strategy, this invention not only improves the detection capability of micro-cracks but also enhances robustness to complex backgrounds and different lighting conditions. In scenarios with high noise or blurred crack details, dynamically adjusting the feature fusion ratio ensures that the model can more accurately capture crack details, thus providing high-accuracy and robust crack detection.

[0052] This invention optimizes the traditional adjacency matrix construction method in graph convolutional networks by introducing texture similarity to weight the adjacency matrix. Traditional GCN methods primarily rely on spatial adjacency relationships to construct connections between nodes. This invention, however, not only considers spatial adjacency relationships but also measures texture similarity by calculating the Structural Similarity Index (SSIM) between nodes, thereby optimizing the connection strength between nodes and ensuring that texture-similar regions have stronger connection weights. This innovation enables the model to pay more attention to texture-similar regions in images during crack detection, especially in the detection of complex backgrounds and micro-cracks, enhancing the complementarity of global and local information. Through texture similarity weighting, this invention significantly improves the accuracy of crack detection, particularly in complex backgrounds and texture-similar regions, enhancing the ability to identify crack details and improving the overall robustness of detection.

[0053] In summary, this invention combines two deep learning methods, Graph Convolutional Networks (GCN) and Convolutional Neural Networks (CNN), to solve the common problems of low accuracy, complex backgrounds, and difficulty in identifying crack details in photovoltaic hidden crack detection by innovatively fusing global and local features. Graph Convolutional Networks can efficiently capture the spatial relationships of cracks in an image when processing global structural features, while CNNs focus on local texture and edge features. By weightedly fusing the advantages of both, this invention can more accurately identify tiny and complex hidden cracks in photovoltaic modules. Furthermore, by innovatively introducing a texture similarity-weighted adjacency matrix, this invention not only improves the accuracy of crack detection but also adapts to image variations under different lighting conditions, enabling the system to maintain high robustness in complex environments. Compared to traditional image processing methods and single deep learning models, the feature fusion and graph structure learning method of this invention can better cope with the diversity and complexity of crack detection tasks, providing an efficient, accurate, and real-time solution. Attached Figure Description

[0054] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0055] Figure 1 This is a flowchart of the photovoltaic module microcrack detection method based on feature fusion provided in the embodiments of the present invention;

[0056] Figure 2 This is a model structure diagram provided in the embodiments of the present invention;

[0057] Figure 3 This is a schematic diagram of the structure of a photovoltaic module microcrack detection system based on feature fusion provided in an embodiment of the present invention. Detailed Implementation

[0058] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0059] Example 1:

[0060] like Figure 1 The image shows an embodiment of the photovoltaic module microcrack detection method based on feature fusion provided by the present invention, which specifically includes the following steps:

[0061] S1: Image acquisition and preprocessing: A 4K resolution camera combined with a drone and a fixed camera is used to acquire surface images of photovoltaic modules. The acquired images are then processed by grayscale conversion, Gaussian filtering for noise reduction, normalization, and size adjustment.

[0062] S2: Image region segmentation and graph structure construction: The preprocessed image is divided into several regions, each region is regarded as a graph node, and an adjacency matrix based on spatial adjacency and texture similarity weighting is constructed. The feature information of the adjacency nodes in the adjacency matrix is ​​aggregated by a graph convolutional network to extract the global features of each region image.

[0063] S3: Local feature extraction and graph convolution feature fusion: Local features of each region of the image are extracted through a convolutional neural network, and global features and local features are fused by an adaptive weighted fusion strategy.

[0064] S4: Hidden Crack Detection and Localization: Outputs the probability of crack presence and the location coordinates of cracks in each region through parallel classification and regression heads, and optimizes the classification and localization tasks using a joint loss function;

[0065] S5: Training and Optimization: The Adam optimizer is used to optimize the deep learning model that combines graph convolutional networks and convolutional neural networks. The initial learning rate, batch size and training cycle are set. Data augmentation is used to improve generalization ability. The model optimization process is monitored through the validation set.

[0066] S6: Model Deployment and Application: Convert the trained deep learning model into TensorFlow Lite format and deploy it to an edge computing device. The edge computing device is connected to the photovoltaic power station monitoring system to collect images in real time and perform hidden crack detection.

[0067] It should be noted that this invention employs Graph Convolutional Networks (GCNs) and Convolutional Neural Networks (CNNs) from deep learning methods as its foundational technologies. CNNs are one of the most widely used architectures in deep learning, particularly suitable for image classification and object detection tasks. CNNs extract local image features through convolutional layers, effectively capturing texture, edges, and other features in images, and perform well in simple crack detection tasks. However, CNNs primarily rely on fixed convolutional operations, making them less capable of recognizing the global structure of cracks or textured regions in images, especially when cracks are small or the background is complex; in such cases, CNNs may fail to accurately locate and classify cracks. Graph Convolutional Networks (GCNs), on the other hand, are deep learning models specifically designed for graph data, capable of feature learning through the structured information of graphs (such as adjacency relationships between nodes). GCNs can effectively capture global spatial relationships in images, making them particularly suitable for detecting complex spatial structures such as cracks in images. In photovoltaic hidden crack detection, GCNs can learn the global structure of cracks by aggregating information from adjacent nodes, improving the detection accuracy of cracks in complex backgrounds.

[0068] This invention innovatively combines Graph Convolutional Networks (GCNs) and Convolutional Neural Networks (CNNs), deeply fusing their features to overcome the limitations of traditional methods. Specifically, this invention effectively captures global structural features in images using GCNs and combines them with local features extracted by CNNs, enabling the model to simultaneously focus on both global crack morphology and local, subtle crack information. Furthermore, this invention introduces a weighted adjacency matrix based on texture similarity to optimize the connections between nodes, improving the model's ability to accurately capture cracks in complex backgrounds.

[0069] The construction of a deep learning model based on the fusion of graph convolutional networks and convolutional neural networks specifically includes the following:

[0070] First, the preprocessed image is divided into regions, which are divided into 16×16 regions. Each region is treated as a graph node, and initial features are extracted for it. These features include the pixel value, texture information and edge information of the region.

[0071] Secondly, an adjacency matrix is ​​constructed for the graph. This matrix not only considers the spatial adjacency relationships between nodes but also innovatively introduces texture similarity (measured by the structural similarity index SSIM) for weighting to optimize the connection strength between nodes. Based on this graph structure, a multi-layer graph convolutional network (GCN) is used to aggregate the feature information of adjacent nodes, thereby extracting the global structural features of the image. Simultaneously, the model also includes a parallel convolutional neural network (CNN) branch, which extracts local features (such as edge and texture information) for each image region through standard 3×3 convolutional kernels, multi-layer convolutions, and pooling operations.

[0072] Finally, the global features extracted by GCN and the local features extracted by CNN are effectively fused using an adaptive weighted fusion strategy. The fused features are then fed into parallel classification and regression heads, which output the probability of crack presence and the location coordinates of cracks in each region, respectively.

[0073] Through this innovative feature fusion method, the present invention can significantly improve the accuracy and robustness of photovoltaic microcrack detection, especially in complex backgrounds, microcracks and scenarios with large changes in illumination. It is more real-time and adaptable than existing technologies, and exhibits higher detection performance.

[0074] The steps of the embodiments of the present invention will be described in detail below with reference to specific examples.

[0075] Step 1: Image Acquisition and Preprocessing

[0076] 1. Image Acquisition

[0077] Image acquisition is fundamental to microcrack detection. To ensure high-quality crack detection, we used a 4K (3840×2160) resolution camera for image acquisition. The acquisition process combined drones and fixed cameras. Drones were used for large-scale inspections of the photovoltaic power plant, covering a wide area and ensuring that the surface of each photovoltaic module was clearly recorded. Fixed cameras were used to capture static images in specific environments (such as areas susceptible to human interference or areas inaccessible by drones) to ensure coverage of all areas.

[0078] High Dynamic Range (HDR) imaging technology is employed during image acquisition. This technology enables the camera to capture sufficient image details under various lighting conditions (such as day and night, or environments with strong light and low light). HDR technology maintains high visibility in both bright and shadow areas, reducing the impact of lighting variations on the detection results. Furthermore, the acquisition device features autofocus, exposure control, and high light sensitivity, ensuring high-quality, low-noise, and high-contrast images, guaranteeing clear capture of microcracks even in low-light environments.

[0079] 2. Image Preprocessing

[0080] The acquired images undergo the following preprocessing steps to ensure they are suitable for subsequent crack detection:

[0081] (1) Grayscale Conversion: Since color information has a relatively small impact on cracks, each RGB image is converted to a grayscale image, reducing unnecessary calculations. The conversion formula is as follows:

[0082] I gray =0.2989·R+0.5870·G+0.1140·B

[0083] Where R, G, and B represent the pixel values ​​of the red, green, and blue channels of the image, respectively.

[0084] (2) Denoising: The image is denoised using a 3×3 Gaussian filter with a standard deviation of 1.5 to reduce the impact of noise in the image. Gaussian filtering can smooth the image and remove random noise.

[0085] (3) Normalization: Normalize pixel values ​​to the range [0,1] to eliminate variations in lighting and differences between different images. The formula is:

[0086]

[0087] (4) Size adjustment: All images were uniformly adjusted to 256×256 pixels to meet the requirements of subsequent network input. This size ensures image detail while also providing good computational efficiency.

[0088] Step 2: Image Region Segmentation and Graph Neural Network Construction

[0089] This invention combines the advantages of Convolutional Neural Networks (CNNs) and Graph Convolutional Networks (GCNs) in the process of image region segmentation and Graph Neural Network (GNN) construction. However, it differs from existing technologies in that it innovatively improves the detection capability of micro-cracks in complex backgrounds by constructing an adjacency matrix with texture similarity weighting and using graph convolution for feature learning. Figure 2 This is a structural diagram of the model of the present invention.

[0090] 1. Image region segmentation

[0091] After image preprocessing, the image is divided into 16×16 regions, with each region considered a node in the image. The image size is 256×256, therefore the image is divided into 16×16=256 nodes, each representing a small region within the image. The features of each region include its pixel values, texture information, edge information, etc. Compared to existing technologies, this invention divides the image into smaller regions, allowing for more precise capture of local features in each region and providing higher detection accuracy, especially in the identification of fine cracks, avoiding the loss of detailed information due to larger region divisions in traditional methods.

[0092] 2. Adjacency Matrix Construction

[0093] Adjacency matrix This is used to represent the connections between nodes. The connections of each node depend not only on its spatial location but also on the texture similarity between nodes. This innovative improvement allows the model to better capture texture-similar regions during crack detection. The adjacency matrix is ​​constructed as follows:

[0094] (1) Spatial adjacency: If two nodes i and j are adjacent (i.e., they share an edge), then A ij =1, otherwise A ij =0

[0095] (2) Texture Similarity: The texture similarity between nodes i and j is measured by calculating the Structural Similarity Index (SSIM). If the texture similarity between two nodes is high, the connection weight between them is increased. The specific calculation formula is as follows:

[0096]

[0097] Among them, A ij A represents the weighted connection strength or affinity between node i and node j. ij The larger the value, the stronger the correlation between the two nodes (i.e., the corresponding image regions). SSIM(Ii ,I j ) is the structural similarity index between image patches i and j, |x i -x j | 2 This is to identify their feature differences. In this way, the model can focus on areas with similar textures during crack detection, thereby better capturing the details of the crack.

[0098] It should be noted that the spatial adjacency step first determines the topology of the graph, then determines whether node i and node j are physically adjacent. If they are adjacent, A... ij If the value is 1, then there is a "potential" connection between them; if they are not adjacent, then A ij If the value is 0, there is no connection between node i and node j.

[0099] The texture similarity calculation formula is for node pairs that have been determined to be spatially adjacent, i.e., A ij It applies to node pairs with a strength of 1. Instead of simply setting the connection strength to 1, it calculates a continuous weight value between 0 and 1 (or other reasonable range) based on the feature differences and texture similarity between the nodes. This weight value represents the "strength" or "importance" of the connection.

[0100] 3. Construction of Graph Convolutional Networks (GCNs)

[0101] This invention employs a multi-layer graph convolutional network (GCN) to extract global and local features of images. Unlike traditional convolutional neural network (CNN) methods, GCN learns the representation of each node by aggregating the feature information of neighboring nodes, which can more effectively capture the global structure of cracks and improve crack detection accuracy in complex backgrounds. Specifically, the graph convolution operation of GCN is as follows:

[0102]

[0103] Among them, H (l) H represents the feature matrix of the nodes in the l-th layer. (0) Features of the input image patch; It is a standardized adjacency matrix; W (l) σ is the weight matrix of the l-th layer; σ is the ReLU activation function. Multi-layer GCN is used for feature learning, enabling the model to capture the global structure of cracks in the image layer by layer, thus achieving more efficient crack detection. Each layer's graph convolution operation aggregates the node features of the previous layer with neighborhood features, enhancing the model's ability to learn spatial relationships within cracks.

[0104] This invention combines a texture similarity weighted adjacency matrix with a graph convolutional network, enabling the model to flexibly handle complex background information, micro-cracks, and various lighting changes. This significantly improves the crack detection accuracy and real-time performance, especially in high-noise environments or with similar background textures, where it exhibits better adaptability and robustness than traditional methods.

[0105] It's important to note that the key to fusing GCN and CNN lies in their complementary strengths. GCN, by processing graph structures, can capture the global spatial relationships and dependencies between different regions (nodes) in an image, which is crucial for identifying crack morphology spanning multiple regions. CNN, on the other hand, excels at extracting high-resolution texture, edge, and other detailed features from local image patches, which is indispensable for identifying micro-cracks and accurately locating crack boundaries. In this step, the deep learning model constructed in this embodiment of the invention is not simply a concatenation of GCN and CNN, but rather designed as a parallel processing architecture. This step primarily lays the foundation for the fusion model—namely, the construction of the graph structure and the definition of the GCN module. The detailed process and calculation formulas for specific local feature extraction (CNN part) and the final feature fusion operation are further elaborated in step three.

[0106] Step 3: Local Feature Extraction and Graph Convolution Feature Fusion

[0107] By combining the advantages of convolutional neural networks (CNN) and graph convolutional networks (GCN), this invention can simultaneously process the global structure and local details of images, especially in the detection of complex backgrounds and micro-cracks, showing significant improvement in accuracy and robustness compared to existing technologies.

[0108] 1. Local Feature Extraction

[0109] In addition to the global features extracted via graph convolutional networks, local features for each region are also extracted using convolutional neural networks (CNNs). Local features include edge and texture information of the image. Feature extraction is performed using standard 3×3 convolutional kernels, followed by multiple convolution and pooling operations, ultimately yielding the local features F for each image region. i The calculation formula is as follows:

[0110] F i =CNN(I i )

[0111] Among them, I i For the i-th image region, F iThese are the local feature vectors extracted using a CNN. To enhance the capture of crack details, the CNN is used not only to extract local texture information for each image patch but also to learn edge features in the image through multiple convolutional operations. In this way, the model can detect not only large-scale cracks but also tiny cracks against noisy or complex texture backgrounds.

[0112] 2. Fusion of graph convolution features and local features

[0113] To fully leverage the advantages of Graph Convolutional Networks (GCNs) and Convolutional Neural Networks (CNNs) in crack detection, this invention effectively fuses global features extracted by GCNs with local features extracted by CNNs. Unlike existing technologies, this invention employs a weighted fusion strategy to adaptively adjust the contributions of the two features, thereby optimizing crack detection results and maintaining high accuracy, especially in complex backgrounds.

[0114] Global features H extracted by graph convolution i Local features F extracted by CNN i Fusion is needed to combine the advantages of both methods in crack detection. The specific fusion method employs a weighted strategy, setting a weighting coefficient α to dynamically adjust the fusion ratio of the two features. The fusion formula is as follows:

[0115] x i =α·H i +(1-α)·F i

[0116] Where, x i The fused features are represented by α, a hyperparameter learned during training to balance the influence of global and local information, initialized to 0.5. By weighted feature fusion, the model can adaptively adjust the importance of global and local features according to the specific scenario. Especially when crack distribution is sparse or local image noise is high, the contribution of local features is enhanced, thereby improving sensitivity to fine cracks.

[0117] Step 4: Detection and Location of Hidden Cracks

[0118] 1. Classification and location of microcracks

[0119] In the image region after feature fusion, two parallel output heads are used to classify and locate cracks. The classification head is used to determine whether each image region contains cracks and outputs the probability of crack presence. The specific calculation formula is as follows:

[0120] P class =W class x i +b class

[0121] Among them W class Let b be the weight matrix. class This is the bias term. The specific calculation formula for the location of the crack (coordinates of the top left and bottom right corners of the rectangle) output by the regression head is as follows:

[0122] P loc =W loc x i +b loc

[0123] Among them W loc Let b be the weight matrix. loc This is a bias term.

[0124] 2. Definition of loss function

[0125] To optimize the model, a joint loss function is used, comprising classification loss and localization loss. Binary cross-entropy loss is used for classification loss, and mean squared error (MSE) is used for localization loss. The loss function is defined as follows:

[0126] L=λ1·L class +λ2·L loc

[0127] Among them, L class For classification loss, L loc For the localization loss, λ1 and λ2 are hyperparameters used to balance the weights of the classification and localization losses. In the regression head, this invention introduces a dynamic adjustment mechanism, allowing the network to automatically adjust the accuracy of crack location prediction based on the characteristics of the input image. For edge regions or complex textured regions, the model can enhance its focus on crack boundaries, thereby optimizing localization accuracy.

[0128] Step 5: Training and Optimization

[0129] 1. Hyperparameter settings

[0130] Training was performed using the Adam optimizer with an initial learning rate of 0.001, a batch size of 32, and 10 epochs. A gradually decreasing learning rate strategy was employed, reducing the learning rate to 0.5 times its original value every 5 epochs. Training stopped when the loss on the validation set did not decrease by more than 0.01 for five consecutive iterations.

[0131] 2. Data Augmentation

[0132] To improve the model's generalization ability during training, the following four data augmentation methods were used:

[0133] (1) Rotation: Randomly rotate the image by a maximum of 45 degrees;

[0134] (2) Translation: Randomly translate the image by a maximum of 10%;

[0135] (3) Cropping: Randomly crop 20% of the image area;

[0136] (4) Brightness variation: Randomly adjust the brightness and contrast of the image.

[0137] 3. Verification and Optimization

[0138] During training, after each epoch, the model's performance on the validation set is evaluated, monitoring precision, recall, and F1 score. Hyperparameters are then adjusted based on the validation set results to further optimize model performance.

[0139] Step Six: Deployment and Application

[0140] 1. Model Deployment

[0141] In this step, the deep learning model based on the fusion of Graph Convolutional Network (GCN) and Convolutional Neural Network (CNN) designed in Step 2 will be used. This model, through joint learning of the global structure and local detail features of images, can accurately detect hidden cracks in photovoltaic modules. The deployment goal is to apply this model to an automated hidden crack detection system for photovoltaic power plants, achieving real-time monitoring and crack assessment. The NVIDIA Jetson Xavier NX will be selected as the edge computing device, equipped with an NVIDIA Volta architecture GPU, featuring a 6-core ARM CPU, 384 CUDA cores, 64 Tensor cores, and 8GB of memory. This device supports GPU acceleration, enabling efficient execution of model inference tasks. The operating system used is Ubuntu 18.04LTS, utilizing CUDA 11.2 and cuDNN 8.0 for computational acceleration. The deep learning frameworks used are TensorFlow 2.4.1 and Keras 2.4.3, with Python version 3.8.

[0142] The deployment process is as follows:

[0143] (1) Save the model: Use the tf.saved_model.save() method to save the trained model as a TensorFlow SavedModel.

[0144] (2) Optimize the model: Convert the SavedModel format model to TensorFlow Lite (.tflite) format using the TensorFlow Lite conversion tool for lightweight inference on the Jetson Xavier NX device.

[0145] (3) Upload the model: Upload the TensorFlow Lite format model file and the required library dependencies to the Jetson Xavier NX device.

[0146] (4) Equipment configuration and connection: Configure the network connection between the equipment and the on-site monitoring camera of the photovoltaic power station to ensure that the image data collected by the camera can be transmitted to the equipment for inference in real time.

[0147] 2. Hidden Trace Detection

[0148] After deployment, the model will perform hidden crack detection according to the following process:

[0149] 1. Real-time Data Acquisition: The photovoltaic power station acquires images through fixed cameras or drones deployed on the photovoltaic modules. The cameras are set to 4K resolution (3840×2160) to capture high-definition images of the photovoltaic modules. Image data is captured and transmitted every second to the inference engine deployed on the Jetson Xavier NX device for processing.

[0150] 2. Image Preprocessing

[0151] The acquired images are processed by a real-time preprocessing module to ensure they can be successfully fed into the deep learning model for inference in step two. The specific preprocessing steps are as follows:

[0152] (1) Grayscale conversion: Each image is converted from RGB to grayscale to reduce unnecessary calculations. The formula is:

[0153] I gray =0.2989·R+0.5870·G+0.1140·B

[0154] Where R, G, and B represent the pixel values ​​of the red, green, and blue channels of the image, respectively.

[0155] (2) Denoising: Gaussian filtering (3×3 kernel, standard deviation 1.5) is applied to the image to reduce the impact of noise.

[0156] (3) Resizing and Normalization: Each image is resized to 256×256 pixels, and the pixel values ​​are normalized to the range [0,1]. The normalization formula is as follows:

[0157]

[0158] 3. Microcrack detection

[0159] The preprocessed image data will be fed to a model already deployed on NVIDIA Jetson Xavier NX for inference. Combining the global structure and local detail features of the image, crack classification and location prediction will be performed. The specific process is as follows:

[0160] (1) Image feature extraction: GCN captures the global crack structure by aggregating neighborhood information between nodes, while CNN extracts the local texture features of each image region. The two are combined through a weighted fusion strategy to obtain the final image features.

[0161] (2) Crack classification: The model outputs the probability value of whether each image region contains a crack.

[0162] (3) Crack location: The model returns the precise location of the crack (such as the coordinates of the upper left and lower right corners of the rectangle) and assesses the severity of the crack based on the size of the region.

[0163] This invention successfully improves the accuracy and robustness of microcrack detection in photovoltaic modules by employing an innovative image-to-graph structure, combined with a texture similarity-weighted adjacency matrix and a graph convolutional network (GCN). Through weighted fusion of CNN and GCN features, the model can flexibly combine global spatial information with local texture information, demonstrating particularly excellent performance in predicting crack details and locations. Furthermore, dynamic adjustment mechanisms and multi-layer feature fusion further enhance the model's adaptability and detection accuracy in complex environments. Using this method, photovoltaic power plants can detect microcracks in real time during operation, thereby effectively extending the lifespan of modules and reducing downtime due to failures.

[0164] Example 2:

[0165] like Figure 3 As shown in the figure, this embodiment of the invention provides a photovoltaic module microcrack detection system based on feature fusion, comprising:

[0166] The image acquisition and preprocessing module is used to acquire surface images of photovoltaic modules using a 4K resolution camera combined with a drone and a fixed camera, and to perform grayscale conversion, Gaussian filtering for noise reduction, normalization and size adjustment on the acquired images.

[0167] The image region segmentation and graph structure construction module is used to divide the preprocessed image into several regions, each region is regarded as a graph node, and an adjacency matrix based on spatial adjacency and texture similarity weighting is constructed. The feature information of the adjacency nodes in the adjacency matrix is ​​aggregated by the graph convolutional network, and a deep learning model based on the fusion of graph convolutional network and convolutional neural network is constructed.

[0168] The local feature extraction and graph convolutional feature fusion module is used to extract local features of each region through a convolutional neural network, extract global features through a graph convolutional network, and use an adaptive weighted fusion strategy to weightedly fuse global features and local features.

[0169] The hidden crack detection and localization module is used to output the probability of crack presence and the location coordinates of cracks in each region through parallel classification and regression heads, and to optimize the classification and localization tasks using a joint loss function.

[0170] The training and optimization module is used to optimize deep learning models using the Adam optimizer, set the initial learning rate, batch size and training period, improve generalization ability through data augmentation, and monitor the model optimization process through the validation set.

[0171] The model deployment and application module is used to convert the trained deep learning model into TensorFlow Lite format and deploy it to edge computing devices. The edge computing devices are connected to the photovoltaic power station monitoring system to collect images in real time and perform hidden crack detection.

[0172] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in the present invention, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0173] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0174] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting microcracks in photovoltaic modules based on feature fusion, characterized in that, The method includes: (1) Image acquisition and preprocessing: A 4K resolution camera combined with a drone and a fixed camera was used to acquire surface images of photovoltaic modules. The acquired images were then processed by grayscale conversion, Gaussian filtering for noise reduction, normalization and size adjustment. (2) Image region segmentation and graph structure construction: The preprocessed image is divided into several regions, each region is regarded as a graph node, and an adjacency matrix based on spatial adjacency and texture similarity weighting is constructed. The feature information of the adjacency nodes in the adjacency matrix is ​​aggregated by the graph convolutional network to extract the global features of each region image. (3) Local feature extraction and graph convolution feature fusion: Local features of each region of the image are extracted by convolutional neural network, and global features and local features are fused by an adaptive weighted fusion strategy. (4) Hidden crack detection and localization: The probability of crack existence and the location coordinates of cracks in each region are output by parallel classification head and regression head, and the classification and localization tasks are optimized by joint loss function; (5) Training and optimization: The Adam optimizer is used to optimize the deep learning model that combines graph convolutional network and convolutional neural network. The initial learning rate, batch size and training cycle are set. Data augmentation is used to improve the generalization ability. The model optimization process is monitored through the validation set. (6) Model deployment and application: The trained deep learning model is converted into TensorFlow Lite format and deployed to the edge computing device. The edge computing device is connected to the photovoltaic power station monitoring system to collect images in real time and perform hidden crack detection.

2. The method for detecting microcracks in photovoltaic modules based on feature fusion according to claim 1, characterized in that, In step (2), the adjacency matrix is ​​constructed in the following ways: Spatial adjacency: Determines adjacent nodes i and j; Texture similarity: The texture similarity between two nodes is measured by calculating the structural similarity index between node i and node j. The calculation formula is as follows: Among them, A ij SSIM(I) represents the weighted connection strength between node i and node j. i ,I j Let |x| be the structural similarity index between node i and node j. i -x j | 2 The feature difference between node i and node j.

3. The method for detecting microcracks in photovoltaic modules based on feature fusion according to claim 1, characterized in that, In step (2), the operation of the graph convolutional network is represented as follows: Among them, H (l) H represents the feature matrix of the nodes in the l-th layer. (0) Features of the input image patch; It is a standardized adjacency matrix; W (l) σ is the weight matrix of the l-th layer; σ is the ReLU activation function.

4. The method for detecting microcracks in photovoltaic modules based on feature fusion according to claim 1, characterized in that, In step (2), the feature information of each region includes the pixel value, texture information and edge information of that region.

5. The method for detecting microcracks in photovoltaic modules based on feature fusion according to claim 1, characterized in that, In step (3), the local features include the edge information and texture information of the image. A standard 3×3 convolution kernel is used for feature extraction. After multiple convolution and pooling operations, the local features of each image region are finally obtained. The calculation formula is as follows: F i =CNN(I i ) Among them, I i For the i-th image region, F i This is the local feature vector extracted by CNN.

6. The method for detecting microcracks in photovoltaic modules based on feature fusion according to claim 5, characterized in that, In step (3), the fusion formula for the global features and local features is: x i =α·H i +(1-α)·F i Where, x i The fused features are represented by α, which is a hyperparameter learned through training to balance the influence of global and local information.

7. The method for detecting microcracks in photovoltaic modules based on feature fusion according to claim 1, characterized in that, In step (4), the formula for calculating the probability of crack presence by the classification head is as follows: P class =W class x i +b class Among them, W class Let b be the weight matrix. class For bias terms; The formula for calculating the crack location coordinates using the regression head is as follows: P loc =W loc x i +b loc Among them, W loc Let b be the weight matrix. loc This is a bias term.

8. The method for detecting microcracks in photovoltaic modules based on feature fusion according to claim 1, characterized in that, In step (4), the joint loss function includes classification loss and localization loss. The classification loss uses binary cross-entropy loss, and the localization loss uses mean squared error. The joint loss function is defined as follows: L=λ1·L class +λ2·L loc Among them, L class For classification loss, L loc For localization loss, λ1 and λ2 are hyperparameters used to balance the weights of classification loss and localization loss.

9. The method for detecting microcracks in photovoltaic modules based on feature fusion according to claim 1, characterized in that, In step (6), the specific steps for model deployment include: (1) Save the model: Save the trained deep learning model as a TensorFlow SavedModel format; (2) Model optimization: Convert the SavedModel format model to TensorFlow Lite (.tflite) format using the TensorFlow Lite conversion tool for lightweight inference on the Jetson Xavier NX device; (3) Upload the model: Upload the TensorFlow Lite format model file and the required library dependencies to the JetsonXavier NX device; (4) Equipment configuration and connection: Configure the network connection between the equipment and the on-site monitoring camera of the photovoltaic power station, and transmit the image data collected by the camera to the equipment for inference in real time.

10. A photovoltaic module microcrack detection system based on feature fusion, characterized in that, The system includes: The image acquisition and preprocessing module is used to acquire surface images of photovoltaic modules using a 4K resolution camera combined with a drone and a fixed camera, and to perform grayscale conversion, Gaussian filtering for noise reduction, normalization and size adjustment on the acquired images. The image region segmentation and graph structure construction module is used to divide the preprocessed image into several regions, each region is regarded as a graph node, and an adjacency matrix based on spatial adjacency and texture similarity weighting is constructed. The feature information of the adjacency nodes in the adjacency matrix is ​​aggregated by the graph convolutional network, and a deep learning model based on the fusion of graph convolutional network and convolutional neural network is constructed. The local feature extraction and graph convolutional feature fusion module is used to extract local features of each region through a convolutional neural network, extract global features through a graph convolutional network, and use an adaptive weighted fusion strategy to weightedly fuse global features and local features. The hidden crack detection and localization module is used to output the probability of crack presence and the location coordinates of cracks in each region through parallel classification and regression heads, and to optimize the classification and localization tasks using a joint loss function. The training and optimization module is used to optimize deep learning models using the Adam optimizer, set the initial learning rate, batch size and training period, improve generalization ability through data augmentation, and monitor the model optimization process through the validation set. The model deployment and application module is used to convert the trained deep learning model into TensorFlow Lite format and deploy it to edge computing devices. The edge computing devices are connected to the photovoltaic power station monitoring system to collect images in real time and perform hidden crack detection.

Images