Photovoltaic module monitoring method and system based on machine vision

Abstract

The invention provides a photovoltaic module monitoring method and system based on machine vision, and the method comprises the steps: obtaining an infrared image and a visible light image of a to-be-monitored photovoltaic module, and forming multi-modal image data for analysis; inputting the multi-modal image data into a first neural network model, and positioning a photovoltaic module area in the image; extracting an image of the photovoltaic module area, inputting the image to a second neural network model, and outputting the type, position and confidence of the defect; wherein the second neural network model is a deep learning model based on a YOLOv5 architecture and embedded with a convolution attention module, and training is performed through a multi-modal data set, including hot spots, cracks, stains and internal subfissure defects; according to the method, the visual diagnosis report containing defect classification and positioning information is generated based on the type, the position and the confidence coefficient of the defect, and the accuracy and the practicability of photovoltaic module defect identification are effectively improved through combination of positioning of the first neural network model and identification of the second neural network model.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a method and system for monitoring photovoltaic modules based on machine vision. Background Technology

[0002] As a clean energy source, the efficiency of photovoltaic power generation's operation and maintenance directly impacts power generation revenue and system safety. Photovoltaic modules, operating in complex outdoor environments for extended periods, are prone to various defects such as hot spots, cracks, stains, and internal microcracks. Traditional operation and maintenance methods primarily rely on regular manual inspections, which are not only inefficient and costly but also heavily dependent on the experience of inspection personnel, leading to a high rate of missed detections, especially for internal microcracks that are difficult to detect with the naked eye.

[0003] In recent years, some automated inspection technologies based on image analysis have emerged. For example, some solutions use visible light cameras to capture images of components and then use traditional image processing algorithms (such as edge detection and thresholding) to identify obvious cracks or stains. However, these methods have poor robustness in complex and variable lighting environments and are extremely limited in their ability to identify defects with low contrast to the background (such as microcracks) and completely invisible internal cracks (such as microcracks inside solar cells), resulting in insufficient accuracy and practicality. Summary of the Invention

[0004] This invention provides a machine vision-based photovoltaic module monitoring method and system to address the shortcomings of poor accuracy and efficiency in identifying defects in photovoltaic modules in existing technologies.

[0005] In a first aspect, the present invention provides a photovoltaic module monitoring method based on machine vision, comprising:

[0006] Infrared and visible light images of the photovoltaic modules to be monitored are acquired to form multimodal image data for analysis;

[0007] The multimodal image data is input into the first neural network model to locate the photovoltaic module region in the image;

[0008] The image of the photovoltaic module region is extracted and input into the second neural network model, which outputs the type, location and confidence level of the defect; wherein, the second neural network model is a deep learning model based on the YOLOv5 architecture and embedding a convolutional attention module, and is trained by a multimodal dataset that fuses infrared and visible light features, the multimodal data including hot spots, cracks, stains and internal hidden cracks.

[0009] Based on the type, location, and confidence level of the defect, a visual diagnostic report containing defect classification and location information is generated.

[0010] According to a machine vision-based photovoltaic module monitoring method provided by the present invention, the step of inputting the multimodal image data into a first neural network model to locate the photovoltaic module region in the image includes:

[0011] The infrared image and the visible light image are stitched together along the channel dimension to form multi-channel input data;

[0012] The multi-channel input data is input into a first neural network model, which is a convolutional neural network based on an encoder-decoder structure.

[0013] The first neural network model outputs a pixel-level segmentation mask for each photovoltaic module region, and the bounding box coordinates of each module are determined based on the segmentation mask.

[0014] According to a machine vision-based photovoltaic module monitoring method provided by the present invention, before stitching the infrared image and the visible light image along the channel dimension, the method further includes:

[0015] Feature point matching and affine transformation are performed on the infrared image and the visible light image to complete image registration;

[0016] The registered bimodal images were then subjected to size standardization and grayscale normalization.

[0017] According to the machine vision-based photovoltaic module monitoring method provided by the present invention, the training process of the first neural network model includes:

[0018] Construct a dataset of photovoltaic module images that includes images taken from various scenes and angles;

[0019] A U-Net semantic segmentation model is constructed using ResNet as the backbone feature extraction network.

[0020] The U-Net structure semantic segmentation model is trained using the photovoltaic module image dataset until it can predict the binary segmentation mask of the module region, which is then used as the first neural network model.

[0021] According to a machine vision-based photovoltaic module monitoring method provided by the present invention, the step of extracting an image of the photovoltaic module region and inputting it into a second neural network model includes:

[0022] Infrared image blocks and visible light image blocks from the same photovoltaic module region are input into the infrared branch and visible light branch of the second neural network model, respectively.

[0023] High-dimensional features from the infrared and visible light branches are spliced ​​and fused together.

[0024] Based on the spliced ​​and fused features, the detection head of the second neural network model synchronously outputs the defect category, bounding box coordinates, and corresponding confidence level.

[0025] According to a machine vision-based photovoltaic module monitoring method provided by the present invention, the step of stitching and fusing high-dimensional features from the infrared and visible light branches includes:

[0026] Feature maps of different scales are extracted from the infrared and visible light branches respectively;

[0027] By utilizing a bidirectional feature pyramid network, feature maps of different scales are fused bidirectionally from top to bottom and from bottom to top, thereby enhancing the ability to represent features of defects of different sizes.

[0028] According to the photovoltaic module monitoring method based on machine vision provided by the present invention, after the detection head of the second neural network model synchronously outputs the defect category, bounding box coordinates and corresponding confidence score, the method further includes:

[0029] Based on the confidence level, all initially identified defect boxes are filtered out, and detection results with a confidence level lower than a preset threshold are removed.

[0030] Perform non-maximum suppression on the remaining defect boxes to eliminate duplicate selections of the same defect;

[0031] Based on the final defect bounding box coordinates, the defect region is cropped from the original image, and its pixel area is calculated as a quantitative assessment of the defect severity.

[0032] According to a machine vision-based photovoltaic module monitoring method provided by the present invention, the step of generating a visual diagnostic report containing defect classification and location information based on the type, location, and confidence level of the defect includes:

[0033] The type, location, and confidence level information of the defects are correlated and integrated with the corresponding original visible light images;

[0034] The predefined report template is invoked, and the associated and integrated data is automatically filled into the corresponding fields of the template to generate a structured document that integrates defect information and visualization elements as the visualization diagnostic report. The corresponding fields include at least defect type statistics, location distribution map and confidence assessment.

[0035] According to the photovoltaic module monitoring method based on machine vision provided by the present invention, after generating a structured document integrating defect information and visualization elements, the method further includes:

[0036] The generated visual diagnostic report will be automatically saved as an electronic file in the specified format, with a timestamp and a unique identifier attached.

[0037] The electronic file is sent to a remote operation and maintenance management platform or a designated user's terminal device via a preset communication interface.

[0038] Secondly, the present invention provides a photovoltaic module monitoring system based on machine vision, comprising:

[0039] The acquisition module is used to acquire infrared and visible light images of the photovoltaic module to be monitored, forming multimodal image data for analysis.

[0040] The positioning module is used to input the multimodal image data into the first neural network model to locate the photovoltaic module area in the image;

[0041] The identification module is used to extract images of the photovoltaic module area, input them into the second neural network model, and output the type, location, and confidence level of the defects. The second neural network model is a deep learning model based on the YOLOv5 architecture and embedding a convolutional attention module. It is trained by a multimodal dataset that integrates infrared and visible light features. The multimodal data includes hot spots, cracks, stains, and internal hidden cracks.

[0042] The reporting module is used to generate a visual diagnostic report containing defect classification and location information based on the type, location, and confidence level of the defect.

[0043] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the machine vision-based photovoltaic module monitoring method as described above.

[0044] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the machine vision-based photovoltaic module monitoring method as described above.

[0045] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the machine vision-based photovoltaic module monitoring method as described above.

[0046] Beneficial effects:

[0047] This invention discloses a machine vision-based photovoltaic module monitoring method and system, comprising: acquiring infrared and visible light images of the photovoltaic module to be monitored, forming multimodal image data for analysis; inputting the multimodal image data into a first neural network model to locate the photovoltaic module region in the image; extracting the image of the photovoltaic module region and inputting it into a second neural network model to output the type, location, and confidence level of defects; wherein, the second neural network model is a deep learning model based on the YOLOv5 architecture and embedding a convolutional attention module, trained by fusing infrared and visible light features into a multimodal dataset, the multimodal data including hot spots, cracks, stains, and internal hidden crack defects; and generating a visual diagnostic report containing defect classification and location information based on the defect type, location, and confidence level. By locating the photovoltaic module using the first neural network model and then identifying defects using the second neural network model to generate a visual diagnostic report, the accuracy and practicality of defect monitoring are effectively improved compared to manual inspection or a single image processing algorithm. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0049] Figure 1 This is a flowchart illustrating the photovoltaic module monitoring method based on machine vision provided in this embodiment;

[0050] Figure 2 This is a schematic diagram of the structure of the photovoltaic module monitoring system based on machine vision provided in this embodiment;

[0051] Figure 3 This is a schematic diagram of the structure of the electronic device provided in this embodiment. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0053] Figure 1 This is a flowchart illustrating the photovoltaic module monitoring method based on machine vision provided in this embodiment.

[0054] like Figure 1 As shown in the figure, the photovoltaic module monitoring method based on machine vision provided in this embodiment of the invention mainly includes the following steps:

[0055] 101. Acquire infrared and visible light images of the photovoltaic module to be monitored to form multimodal image data for analysis.

[0056] Specifically, infrared and visible light cameras are mounted on a photovoltaic power station inspection platform, such as a drone or a fixed bracket. The lens axes are adjusted to be parallel and the fields of view completely overlap, ensuring that a single frame image covers at least one photovoltaic module. Through on-site debugging, the synchronization of the dual camera triggers is ensured to avoid image misalignment caused by trigger delays.

[0057] Based on the illumination data fed back by the ambient light sensor, the camera parameters are automatically adjusted to avoid overexposure or underexposure of the image. The dual cameras are triggered to shoot synchronously according to the preset maintenance cycle, and the image resolution is set to adapt to the scene. The raw image data is stored in the local cache unit in real time.

[0058] Preprocessing is performed on the cached infrared and visible light images: Obvious feature points such as component border corners and junction box edges are extracted using a feature point extraction algorithm. Affine transformation is then used to complete image registration, ensuring that the component position deviation in the dual-modal images is controlled within an acceptable range. The registered images are then uniformly adjusted to the same size, and grayscale values ​​are normalized to eliminate brightness differences between the different modal images.

[0059] Therefore, the resulting multimodal images possess synchronicity and consistency, providing a data foundation for subsequent component localization and defect identification.

[0060] 102. Input the multimodal image data into the first neural network model to locate the photovoltaic module region in the image.

[0061] Specifically, the first step is to train the first neural network model. The process is as follows:

[0062] We collected images of photovoltaic modules covering different scenarios (such as sunny days, cloudy days, and backlighting in the morning and evening) and shooting angles to ensure that the samples covered various practical application situations. We manually annotated the outline of the photovoltaic modules in each image using professional annotation tools, generating pixel-level annotation masks. Module areas were marked as 1, and background areas were marked as 0, thus constructing a complete module localization dataset.

[0063] Using ResNet series networks as the backbone feature extraction network, a U-Net semantic segmentation model, i.e., the first neural network model, is constructed. The encoder part compresses image dimensions and extracts global features through convolution and pooling operations, focusing on the overall contour of components. The decoder part restores the feature map resolution through upsampling and skip connections, achieving pixel-level fine segmentation and ensuring accurate identification of component boundaries.

[0064] The dataset is divided into training, validation, and test sets according to a standard ratio. The cross-entropy loss function is used to calculate the error between the model's predicted mask and the manually labeled mask, quantifying the model's training bias. Iterative training is performed using a suitable optimizer and initial learning rate. After each training round, the segmentation accuracy is evaluated using the validation set. When the segmentation accuracy on the validation set reaches a stable state and meets the requirements of practical applications, training is stopped and the model parameters are saved to ensure reliable model performance.

[0065] The preprocessed infrared and visible light images are stitched together along the channel dimension to form multi-channel input data, which is then transmitted to the localization module of the pre-trained first neural network model to provide complete data for model inference. The first neural network model performs inference on the multi-channel data. The encoder part gradually compresses the image dimensions and extracts global features of the components through multiple convolutional and pooling operations, clarifying the approximate range of the components in the image. The decoder part gradually restores the feature map resolution through multiple upsampling and skip connections, finally outputting a binary segmentation mask of the component region, clearly distinguishing the components from the background. Edge detection and contour extraction algorithms are performed on the binary segmentation mask to obtain the minimum bounding rectangle of the component contour. The diagonal pixel coordinates of the rectangle, such as the upper left and lower right corners, are calculated, which are the bounding boxes of the photovoltaic component region, clarifying the specific location range of the component in the image.

[0066] The first neural network model enables automatic positioning of photovoltaic modules in complex backgrounds, effectively eliminating background interference from supports, sky, and ground, thus defining a precise and effective analysis range for subsequent defect identification and improving the regional positioning accuracy of photovoltaic modules.

[0067] 103. Extract images of the photovoltaic module area and input them into the second neural network model to output the type, location, and confidence level of defects. The second neural network model is a deep learning model based on the YOLOv5 architecture and embedding a convolutional attention module. It is trained by a multimodal dataset that integrates infrared and visible light features. The multimodal data includes hot spots, cracks, stains, and internal hidden crack defects.

[0068] Specifically, based on the component bounding box coordinates obtained from the localization, image blocks containing only a single photovoltaic module are cropped from the preprocessed infrared and visible light images, completely eliminating background areas, reducing the amount of invalid calculations in the model, and improving the efficiency of subsequent processing.

[0069] Infrared and visible light image patches of the same component are input into the infrared and visible light branches of the second neural network model, respectively. The infrared branch extracts temperature-related features of the component through multi-layer convolution operations, such as abnormal temperature features in hot spot areas and temperature gradient changes in internal hidden crack areas. The convolutional attention module automatically focuses on temperature-abnormal areas, suppressing feature interference from defect-free areas and enhancing the representation of defect features. The visible light branch extracts appearance-related features of the component through multi-layer convolution operations, such as linear gray-level changes in cracks and blocky gray-level distribution features of stains. The convolutional attention module automatically focuses on pixel gray-level abrupt change areas, enhancing the identification ability of weak defect features such as microcracks and avoiding missed detections.

[0070] Multi-scale feature maps corresponding to defects of different sizes are extracted from different convolutional layers of the infrared and visible light branches. Small-scale feature maps are used to focus on small defects such as microcracks, while large-scale feature maps are used to focus on larger defects such as hot spots, ensuring that all types of defects can be effectively captured.

[0071] A bidirectional feature pyramid network is employed to perform fusion operations on multi-scale feature maps: Large-scale feature maps are upsampled and concatenated with small-scale feature maps to supplement global location information for small defects, clarifying their relative positions within components and achieving top-down fusion. Conversely, small-scale feature maps are downsampled and concatenated with large-scale feature maps to supplement detailed contour information for large defects, accurately defining their boundaries and achieving bottom-up fusion.

[0072] The fused feature map is input into the detection head of the second neural network model, which includes a classification branch and a regression branch. The classification branch outputs the defect type and its corresponding confidence level. The defect types include hot spots, cracks, stains, and internal hidden cracks. The corresponding confidence level includes the reliability of the quantification result. The regression branch outputs the coordinates of the defect bounding box, including the specific location of the defect in the component.

[0073] Set a reasonable confidence threshold and remove detection results with confidence levels below the threshold to reduce invalid information caused by model misjudgment and ensure the reliability of the retained results. Perform conventional non-maximum suppression on the remaining defect bounding boxes to eliminate the situation where the same defect is repeatedly selected, ensuring that each defect corresponds to only one bounding box and simplifying the subsequent evaluation process.

[0074] Based on the optimized defect bounding box coordinates, the defect region is cropped from the original visible light image, and the number of pixels in the defect region is counted. The ratio of the number of pixels in the defect region to the total number of pixels in the component is used as a quantitative indicator of defect severity, and corresponding severity levels, such as mild, moderate, and severe, are classified according to actual operation and maintenance needs, providing an objective basis for determining operation and maintenance priorities.

[0075] By accurately identifying various defects in photovoltaic modules, especially internal microcracks that are difficult to detect with the naked eye, the accuracy of defect identification has been greatly improved. Furthermore, the quantitative assessment results provide a clear basis for operation and maintenance decisions, effectively solving the problems of difficulty in identifying low-contrast defects and the lack of quantitative standards for the severity of defects.

[0076] 104. Generate a visual diagnostic report containing defect classification and location information based on the type, location, and confidence level of the defect.

[0077] Specifically, the defect type, location (boundary box coordinates), confidence level, severity quantification value are associated with the corresponding original visible light image. Different colored bounding boxes are used to mark various defects in the image, such as different colors corresponding to different defects, which makes it easy to distinguish them intuitively. The defect type and confidence level are marked inside the box to achieve intuitive image-information correspondence.

[0078] The system calls a preset diagnostic report template, which includes core fields such as defect statistics, location distribution, confidence assessment, and severity classification. The system automatically fills the corresponding fields in the template with the associated and integrated data, generating a structured electronic document containing text descriptions and annotated images, i.e., a visual diagnostic report, ensuring that the report information is complete and logically clear.

[0079] The generated diagnostic report is timestamped and a unique identifier is added, and it is automatically saved locally to the edge computing device. Through a preset communication interface and network transmission protocol, the report is transmitted to a remote operation and maintenance management platform, or pushed to the terminal devices of operation and maintenance personnel (such as mobile phones and computers), ensuring that operation and maintenance personnel receive component defect information in a timely manner.

[0080] Using the method in this embodiment, the accuracy of identifying various defects is effectively improved in a test dataset covering multiple scenarios, especially the ability to identify internal hidden cracks, which is significantly enhanced, solving the problem that traditional methods struggle to detect internal hidden cracks. Furthermore, it can operate stably in complex outdoor environments, and the time consumption for single-component inspection, report generation, and transmission latency all meet actual operation and maintenance requirements. It supports various inspection carriers, such as drones and fixed supports, demonstrating high practical value. Compared to traditional manual inspection, inspection efficiency is significantly improved, the missed detection rate is greatly reduced, and labor costs and human error are minimized. Compared to single-modal inspection, the defect identification range is wider and the accuracy is higher, with particularly outstanding advantages in the detection of internal hidden cracks.

[0081] It is important to note that the first and second neural network models in this embodiment do not operate independently, but rather form a collaborative closed loop through data flow and feature transfer. Specifically, this is reflected in:

[0082] The first neural network's attention enhancement design embeds a spatial attention module in the encoder-decoder connection layer of the U-Net architecture. This module can automatically identify pixel differences between the outline of the photovoltaic module and background interference in the image (such as the straight line features of the module's border and the gradient features of the background sky). This improves the accuracy of the segmentation mask in recognizing the module's edges and solves the positioning deviation problem of traditional semantic segmentation models in scenarios where the module and support edges are adhered.

[0083] The second neural network features dynamic weight adjustment in its dual branches. During the feature extraction process of the infrared and visible light branches, dynamic weight coefficients are introduced. When temperature features are significant in the input image (such as hot spot areas), the weight of the infrared branch is automatically increased; when appearance features are significant (such as cracks), the weight of the visible light branch is automatically increased, thus avoiding misjudgment of defects caused by the dominance of single-modal features.

[0084] Figure 2 This is a schematic diagram of the structure of the photovoltaic module monitoring system based on machine vision provided in this embodiment.

[0085] like Figure 2 As shown in the figure, this embodiment provides a photovoltaic module monitoring system based on machine vision, including:

[0086] The acquisition module 201 is used to acquire infrared and visible light images of the photovoltaic module to be monitored, forming multimodal image data for analysis.

[0087] The positioning module 202 is used to input multimodal image data into the first neural network model to locate the photovoltaic module region in the image;

[0088] The identification module 203 is used to extract images of the photovoltaic module area, input them into the second neural network model, and output the type, location and confidence level of the defects. The second neural network model is a deep learning model based on the YOLOv5 architecture and embedding a convolutional attention module. It is trained by a multimodal dataset that integrates infrared and visible light features. The multimodal data includes hot spots, cracks, stains and internal hidden cracks.

[0089] Report module 204 is used to generate a visual diagnostic report containing defect classification and location information based on the type, location, and confidence level of the defect.

[0090] Figure 3 This is a schematic diagram of the structure of the electronic device provided in this embodiment.

[0091] like Figure 3As shown, the electronic device may include a processor 301, a communication interface 302, a memory 303, and a communication bus 304. The processor 301, communication interface 302, and memory 303 communicate with each other via the communication bus 304. The processor 301 can call logical instructions from the memory 303 to execute a machine vision-based photovoltaic module monitoring method.

[0092] Furthermore, the logical instructions in the aforementioned memory 303 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0093] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the machine vision-based photovoltaic module monitoring method provided by the above methods.

[0094] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the machine vision-based photovoltaic module monitoring method provided by the methods described above.

[0095] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0096] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A photovoltaic module monitoring method based on machine vision, characterized in that, include: Infrared and visible light images of the photovoltaic modules to be monitored are acquired to form multimodal image data for analysis; The multimodal image data is input into the first neural network model to locate the photovoltaic module region in the image; The image of the photovoltaic module region is extracted and input into the second neural network model, which outputs the type, location and confidence level of the defect; wherein, the second neural network model is a deep learning model based on the YOLOv5 architecture and embedding a convolutional attention module, and is trained by a multimodal dataset that fuses infrared and visible light features, the multimodal data including hot spots, cracks, stains and internal hidden cracks. Based on the type, location, and confidence level of the defect, a visual diagnostic report containing defect classification and location information is generated.

2. The photovoltaic module monitoring method based on machine vision according to claim 1, characterized in that, The step of inputting the multimodal image data into the first neural network model to locate the photovoltaic module region in the image includes: The infrared image and the visible light image are stitched together along the channel dimension to form multi-channel input data; The multi-channel input data is input into a first neural network model, which is a convolutional neural network based on an encoder-decoder structure. The first neural network model outputs a pixel-level segmentation mask for each photovoltaic module region, and the bounding box coordinates of each module are determined based on the segmentation mask.

3. The photovoltaic module monitoring method based on machine vision according to claim 2, characterized in that, Before stitching the infrared image and the visible light image together by channel dimension, the method further includes: Feature point matching and affine transformation are performed on the infrared image and the visible light image to complete image registration; The registered bimodal images were then subjected to size standardization and grayscale normalization.

4. The photovoltaic module monitoring method based on machine vision according to claim 2, characterized in that, The training process of the first neural network model includes: Construct a dataset of photovoltaic module images that includes images taken from various scenes and angles; A U-Net semantic segmentation model is constructed using ResNet as the backbone feature extraction network. The U-Net structure semantic segmentation model is trained using the photovoltaic module image dataset until it can predict the binary segmentation mask of the module region, which is then used as the first neural network model.

5. The photovoltaic module monitoring method based on machine vision according to claim 1, characterized in that, The step of extracting the image of the photovoltaic module region and inputting it into the second neural network model includes: Infrared image blocks and visible light image blocks from the same photovoltaic module region are input into the infrared branch and visible light branch of the second neural network model, respectively. High-dimensional features from the infrared and visible light branches are spliced ​​and fused together. Based on the spliced ​​and fused features, the detection head of the second neural network model synchronously outputs the defect category, bounding box coordinates, and corresponding confidence level.

6. The photovoltaic module monitoring method based on machine vision according to claim 5, characterized in that, The process of splicing and fusing high-dimensional features from the infrared and visible light branches includes: Feature maps of different scales are extracted from the infrared and visible light branches respectively; By utilizing a bidirectional feature pyramid network, feature maps of different scales are fused bidirectionally from top to bottom and from bottom to top, thereby enhancing the ability to represent features of defects of different sizes.

7. The photovoltaic module monitoring method based on machine vision according to claim 5, characterized in that, After the detection head of the second neural network model synchronously outputs the defect category, bounding box coordinates, and corresponding confidence score, the method further includes: Based on the confidence level, all initially identified defect boxes are filtered out, and detection results with a confidence level lower than a preset threshold are removed. Perform non-maximum suppression on the remaining defect boxes to eliminate duplicate selections of the same defect; Based on the final defect bounding box coordinates, the defect region is cropped from the original image, and its pixel area is calculated as a quantitative assessment of the defect severity.

8. The photovoltaic module monitoring method based on machine vision according to claim 1, characterized in that, The process of generating a visual diagnostic report containing defect classification and location information based on the type, location, and confidence level of the defect includes: The type, location, and confidence level information of the defects are correlated and integrated with the corresponding original visible light images; The predefined report template is invoked, and the associated and integrated data is automatically filled into the corresponding fields of the template to generate a structured document that integrates defect information and visualization elements as the visualization diagnostic report. The corresponding fields include at least defect type statistics, location distribution map and confidence assessment.

9. The photovoltaic module monitoring method based on machine vision according to claim 8, characterized in that, After generating the structured document integrating defect information and visualization elements, the process also includes: The generated visual diagnostic report will be automatically saved as an electronic file in the specified format, with a timestamp and a unique identifier attached. The electronic file is sent to a remote operation and maintenance management platform or a designated user's terminal device via a preset communication interface.

10. A photovoltaic module monitoring system based on machine vision, characterized in that, include: The acquisition module is used to acquire infrared and visible light images of the photovoltaic module to be monitored, forming multimodal image data for analysis. The positioning module is used to input the multimodal image data into the first neural network model to locate the photovoltaic module area in the image; The identification module is used to extract images of the photovoltaic module area, input them into the second neural network model, and output the type, location, and confidence level of the defects. The second neural network model is a deep learning model based on the YOLOv5 architecture and embedding a convolutional attention module. It is trained by a multimodal dataset that integrates infrared and visible light features. The multimodal data includes hot spots, cracks, stains, and internal hidden cracks. The reporting module is used to generate a visual diagnostic report containing defect classification and location information based on the type, location, and confidence level of the defect.

Images