A fault monitoring method and system based on combined analysis of thermal imaging and visible light images

Abstract

The application discloses a kind of based on thermal imaging and visible light image combined analysis's fault monitoring method and system, belong to the field of fault monitoring, including: to the fusion image using deep learning algorithm based on convolutional neural network, by constructing the multiscale convolutional neural network model including multiple different convolution kernel sizes Parallel convolution layer, learn different levels of fault feature representation, carry out the automatic extraction of fault feature, realize the identification of complex fault mode, obtain fault feature;Design acceleration strategy based on parallel computing, task decomposition is carried out to key tasks such as image registration, image fusion, fault feature extraction and fault location positioning, realize parallel computing using multi-core processor or graphics processing unit, improve the real-time performance of fault monitoring.

Description

Technical Field

[0001] This invention belongs to the field of fault monitoring, and in particular relates to a fault monitoring method and system based on joint analysis of thermal imaging and visible light images. Background Technology

[0002] Fault monitoring of photovoltaic equipment is a critical task. Traditional fault monitoring methods typically rely on a single source of information, such as thermal imaging or visible light images. However, a single source of information may not fully reflect the actual condition of the equipment, thus limiting the accuracy and reliability of fault monitoring.

[0003] In practical applications, the acquisition conditions of thermal imaging images and visible light images may differ, such as differences in shooting angle, distance, and lighting, leading to spatial misalignment and distortion between the two images. This poses a challenge to image registration. Designing efficient and robust image registration algorithms to adapt to complex industrial environments is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a fault monitoring method based on joint analysis of thermal imaging and visible light images, comprising:

[0005] Acquire thermal imaging images and visible light images of photovoltaic equipment, and register the thermal imaging images and visible light images to obtain registered thermal imaging images and visible light images;

[0006] The image fusion algorithm based on weighted fusion performs pixel-level weighted summation on the registered thermal imaging image and visible light image to obtain the fused image;

[0007] A multi-scale convolutional neural network model containing multiple parallel convolutional layers with different kernel sizes is constructed. The multi-scale convolutional neural network model is trained until it learns fault feature representations at different levels and then outputs them to obtain a fault pattern recognition model.

[0008] A fault location model based on region growth is constructed by iteratively expanding the neighborhood region and combining the similarity and connectivity of pixels within the neighborhood region.

[0009] The fused image is input into the fault pattern recognition model for calculation to obtain fault features, and the fault features are input into the fault location model to obtain the fault location.

[0010] Preferably, the process of obtaining the registered thermal imaging image and visible light image includes:

[0011] A thermal imaging image and a visible light image of a photovoltaic device are acquired. An image registration algorithm based on feature point matching is used to extract significant feature points from the thermal imaging image and the visible light image, establish the correspondence between the feature points, calculate the geometric transformation relationship between the thermal imaging image and the visible light image, and perform spatial alignment of the thermal imaging image and the visible light image based on the geometric transformation relationship to obtain the registered thermal imaging image and the visible light image.

[0012] Preferably, the process of obtaining the fused image includes:

[0013] Acquire registered thermal and visible light images;

[0014] The thermal imaging image is divided into blocks, and the average temperature gradient of each block is calculated to obtain a temperature gradient map.

[0015] The visible light image is divided into blocks, and the texture complexity of each block is calculated to obtain a texture complexity map.

[0016] Based on the temperature gradient map and texture complexity map, determine the thermal imaging weight coefficient and visible light weight coefficient for each block;

[0017] Multiply the thermal imaging block by the thermal imaging weighting coefficient, multiply the visible light block by the visible light weighting coefficient, and then add them together to obtain the fused block;

[0018] All the fusion blocks are stitched together to obtain a preliminary fused image. Local enhancement is then performed on the preliminary fused image to obtain the final fused image.

[0019] Preferably, the process of stitching together all the fusion blocks to obtain a preliminary fused image includes:

[0020] Obtain multiple image blocks to be fused, and determine the stitching position and order of each image block according to the preset stitching rules;

[0021] Given the determined stitching positions and order, an image stitching algorithm is used to stitch multiple images into a preliminary fused image.

[0022] An image fusion algorithm is used to process the preliminary fusion image to obtain the fused target image;

[0023] The target image after fusion is obtained. According to the preset quality assessment rules, the image quality assessment algorithm is used to assess the quality of the target image. If the quality assessment result meets the preset conditions, the target image is determined as the final fusion result image.

[0024] Preferably, the process of obtaining the fault mode recognition model includes:

[0025] Construct a multi-scale convolutional neural network model containing multiple parallel convolutional layers with different kernel sizes;

[0026] Max pooling is added to the output of each parallel convolutional layer in the multi-scale convolutional neural network model. The outputs of parallel convolution and pooling are concatenated and then flattened to generate an optimized multi-scale convolutional neural network model.

[0027] The optimized multi-scale convolutional neural network model is trained using historical datasets until it learns fault feature representations at different levels and outputs the result, thus obtaining a fault pattern recognition model.

[0028] Preferably, the process of constructing a fault location model based on region growing by iteratively expanding the neighborhood region and combining the similarity and connectivity of pixels within the neighborhood region includes:

[0029] The fault characteristic data and prior knowledge of photovoltaic equipment are obtained and used as input to the region growing algorithm. The initial seed point of the region growing algorithm is determined based on the fault characteristics, and the seed point is located inside the fault region.

[0030] Centered on the seed point, the neighborhood region is expanded iteratively. The selection of the neighborhood region is determined based on pixel similarity and region connectivity.

[0031] In each iteration of expansion, pixel data in the current neighborhood region is acquired, and its similarity with the seed point is calculated. If the similarity meets the preset threshold, the neighborhood region is merged into the current fault region, and its boundary pixels are used as new seed points to continue iterative expansion.

[0032] If the similarity does not meet the preset threshold or the neighboring regions are no longer connected, the iterative expansion of the current region is stopped, and the current fault region is marked as an independent fault location, thus completing the construction of the fault location model.

[0033] Preferably, the method further includes: designing an acceleration strategy based on parallel computing, decomposing the tasks of image registration, image fusion, fault feature extraction, and fault location localization into tasks, using multi-core processors or graphics processing units to achieve parallel computing, and introducing a fault interpretation module based on an attention mechanism, generating an attention weight map by calculating the spatial correlation between the fault feature map and the equipment component location map, highlighting key fault areas, and generating a fault interpretation report by combining the attention weight map and the fault features.

[0034] On the other hand, the present invention also provides a fault monitoring system based on joint analysis of thermal imaging and visible light images, comprising:

[0035] The data acquisition module is used to acquire thermal imaging images and visible light images of photovoltaic equipment, register the thermal imaging images and visible light images, and obtain registered thermal imaging images and visible light images.

[0036] The fusion module is used to perform pixel-level weighted summation on the registered thermal imaging image and visible light image based on a weighted fusion image fusion algorithm to obtain a fused image;

[0037] The identification model building module is used to construct a multi-scale convolutional neural network model containing multiple parallel convolutional layers with different kernel sizes, train the multi-scale convolutional neural network model until it learns fault feature representations at different levels and outputs them to obtain a fault pattern recognition model.

[0038] The localization construction module is used to construct a fault location localization model based on region growth by iteratively expanding the neighborhood region and combining the similarity and connectivity of pixels within the neighborhood region.

[0039] The calculation module is used to input the fused image into the fault pattern recognition model for calculation to obtain fault features, and input the fault features into the fault location model to obtain the fault location.

[0040] On the other hand, the present invention also provides an electronic device, including a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the fault monitoring method based on joint analysis of thermal imaging and visible light images.

[0041] On the other hand, the present invention also provides a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements a fault monitoring method based on joint analysis of thermal imaging and visible light images.

[0042] Compared with the prior art, the present invention has the following advantages and technical effects:

[0043] This invention achieves spatial alignment of two images through an image registration algorithm based on feature point matching. Then, a weighted fusion algorithm is employed, dynamically adjusting weights based on temperature gradient and texture complexity, to fuse the registered thermal and visible light images, resulting in a fused image that incorporates both temperature information and texture details. Next, a multi-scale convolutional neural network is used to automatically extract fault features from the fused image, and combined with prior knowledge, a region growing algorithm is used to locate the fault. This invention also introduces a parallel computing strategy and an attention-based fault interpretation module, improving the real-time performance of fault monitoring and generating an interpretation report containing key fault areas and causes. This method, through a combination of multimodal image fusion and deep learning, achieves accurate detection, location, and interpretation of photovoltaic equipment faults, providing strong support for equipment maintenance decisions and effectively improving the operational reliability and safety of photovoltaic equipment. Attached Figure Description

[0044] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0045] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of the system structure according to an embodiment of the present invention. Detailed Implementation

[0047] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0048] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0049] Example 1

[0050] like Figure 1-2 As shown, this embodiment provides a fault monitoring method based on joint analysis of thermal imaging and visible light images, including:

[0051] A thermal imaging image and a visible light image of a photovoltaic device are acquired. Addressing the spatial misalignment and deformation between the thermal and visible light images, an image registration algorithm based on feature point matching is employed. This algorithm extracts significant feature points from both the thermal and visible light images, establishes the correspondence between these feature points, calculates the geometric transformation relationship between the two images, and achieves spatial alignment between them, resulting in registered thermal and visible light images.

[0052] Specifically, thermal imaging and visible light image fusion technology for photovoltaic equipment is an important means of equipment diagnosis. First, thermal and visible light images of the equipment are acquired. These two images reflect the temperature distribution and actual appearance of the equipment, respectively. For example, imaging a running motor reveals the temperature distribution of different parts of the motor in the thermal image, while the visible light image shows its actual appearance. Next, salient feature points are extracted from both images. Feature points are points with unique properties in the image, such as edges and corners. Taking a motor image as an example, the outline of the motor casing and bolt positions can be extracted as feature points. Commonly used feature point extraction algorithms include SIFT and SURF. For thermal imaging images, areas with significant temperature gradient changes can be extracted as feature points; for visible light images, areas with significant brightness or color changes can be extracted. Establishing the correspondence between feature points is crucial for image alignment. By comparing the descriptions of feature points in the two images, corresponding feature point pairs can be found. For example, if a corner point of the motor casing can be identified in both images, a correspondence between these two points can be established. This step is typically achieved using algorithms such as nearest neighbor matching. Based on the corresponding feature points, the geometric transformation relationship between the two images can be calculated. This transformation relationship can typically be represented by a transformation matrix, encompassing transformations such as translation, rotation, and scaling. For example, if a thermal imaging image is rotated and scaled relative to a visible light image, the specific rotation angle and scaling ratio can be calculated. After obtaining the transformation relationship, the thermal imaging image can be spatially transformed to align it with the visible light image. This step is usually achieved using image resampling techniques. Specifically, each pixel in the thermal imaging image is mapped to a new position according to the calculated transformation relationship, thus aligning it with the visible light image. After image alignment, the next step is image fusion. The purpose of fusion is to combine thermal imaging information and visible light information to form an image containing more information.

[0053] Based on the complementary characteristics of the registered thermal imaging image and the visible light image, a weighted fusion-based image fusion algorithm is designed. By performing pixel-level weighted summation on the registered thermal imaging image and the visible light image, and dynamically adjusting the weight coefficients according to the temperature gradient and texture complexity of the local regions of the registered thermal imaging image and the visible light image, the fused image retains the temperature information of the registered thermal imaging image while enhancing the texture details of the registered visible light image, thus obtaining a fused image.

[0054] The specific steps are as follows: Step 1: Acquire the registered thermal imaging image and visible light image; Step 2: Divide the thermal imaging image into blocks, calculate the average temperature gradient of each block, and obtain a temperature gradient map; Step 3: Divide the visible light image into blocks, calculate the texture complexity of each block, and obtain a texture complexity map; Step 4: Determine the thermal imaging weight coefficient and visible light weight coefficient for each block based on the temperature gradient map and texture complexity map; Step 5: Perform weighted fusion on each block, that is, multiply the thermal imaging block by the thermal imaging weight coefficient, multiply the visible light block by the visible light weight coefficient, and then add them together to obtain a fused block; Step 6: Stitch together all fused blocks to obtain a preliminary fused image; Step 7: Perform local enhancement on the preliminary fused image to highlight texture details, and obtain the final fused image.

[0055] Specifically, acquiring registered thermal and visible light images is fundamental to the fusion process. Registration ensures spatial alignment of the two images, making the position of the same object consistent in both images. This step lays the foundation for subsequent analysis and fusion. Calculating the average temperature gradient in blocks of the thermal image can highlight areas with drastic temperature changes. For example, in photovoltaic equipment, a region with a significantly higher temperature gradient than its surroundings may indicate abnormal heating. Assuming the image is divided into a 10×10 grid, the temperature gradient is calculated within each grid. If the average temperature gradient of a grid is 2°C / pixel during normal operation, but is currently detected at 5°C / pixel, this area may require special attention. Calculating texture complexity in blocks of the visible light image helps identify detailed features on the equipment surface. Texture complexity can be obtained by calculating the gray-level co-occurrence matrix or local binary pattern (LBP) of local regions of the image. For example, bolts and seams on the equipment surface typically have high texture complexity, while smooth metal surfaces have lower texture complexity. Determining weighting coefficients based on temperature gradients and texture complexity is to highlight important information during fusion. In areas with high temperature gradients, thermal imaging should have a larger weight to highlight abnormal temperatures; in areas with high texture complexity, visible light should have a larger weight to preserve structural details. For example, for areas with high temperature gradients but simple textures, which may indicate large-area overheating, the thermal imaging weight can be set to 0.8, and the visible light weight to 0.2. Weighted fusion is a key step in combining thermal imaging and visible light information. For each block, the thermal imaging pixel value is multiplied by its weight, and the visible light pixel value is multiplied by its weight, then the results are added together. This preserves both temperature information and structural details. For example, if a block has a thermal imaging pixel value of 200 (8-bit grayscale) with a weight of 0.7, and a visible light pixel value of 150 with a weight of 0.3, then the fused pixel value is 200 × 0.7 + 150 × 0.3 = 185. Stitching the fused blocks together yields a preliminary fused image; this step integrates local information into a global image. However, direct stitching may result in obvious boundaries between blocks, affecting the visual effect. Therefore, methods such as weighted averaging or Poisson fusion can be used to smooth the transition between overlapping regions, making the transition between blocks more natural. Finally, local enhancement of the fused image can further improve image quality.

[0056] Multiple image blocks to be fused are obtained. Based on preset block stitching rules, the stitching position and order of each image block are determined. For the determined stitching position and order, an image stitching algorithm is used to stitch the multiple image blocks into a preliminary fused image. The preliminary fused image is then obtained, and processed using an image fusion algorithm according to preset image fusion rules to obtain the fused target image. The fused target image is then obtained, and its quality is evaluated using an image quality evaluation algorithm according to preset quality evaluation rules. If the quality evaluation result meets preset conditions, the target image is determined as the final fused result image. The final fused result image is obtained, and its enhancement is performed using an image enhancement algorithm according to preset image enhancement rules to obtain the enhanced fused result image. Finally, the enhanced fused result image is obtained, and its compression is performed using an image compression algorithm according to preset image compression rules to obtain the compressed fused result image.

[0057] The fused image is processed using a deep learning algorithm based on convolutional neural networks. By constructing a multi-scale convolutional neural network model containing multiple parallel convolutional layers with different kernel sizes, fault feature representations at different levels are learned, fault features are automatically extracted, and the identification of complex fault modes is achieved to obtain fault features.

[0058] The fused image is then acquired and processed using a deep learning algorithm based on convolutional neural networks (CNNs). This method automatically learns complex features in the image without requiring manual feature extractors, making it particularly suitable for identifying complex fault patterns. First, a multi-scale CNN model is constructed, containing multiple parallel convolutional layers with different kernel sizes. For example, a single parallel layer might contain 3x3, 5x5, and 7x7 kernels. This design allows the network to capture features at different scales simultaneously. For instance, a 3x3 kernel can capture detailed textures, while a 7x7 kernel can capture a wider range of structural information. The multi-scale CNN model learns fault feature representations at different levels. For example, in lower convolutional layers, the network might learn basic features such as edges and textures; in higher convolutional layers, it might learn more complex patterns, such as the specific shape of cracks or the distribution pattern of hot spots. These feature representations are crucial for identifying different fault types. Based on these learned features, the network can automatically extract fault features from the fused image. For example, if a hotspot exists in the fused image, representing an overheated area of ​​the device, the network will identify the specific temperature gradient of this area and the differences in surrounding texture. If a crack exists, the network will identify the texture interruption and edge features caused by the crack. By inputting the extracted fault features into a classifier, complex fault patterns can be identified. The classifier can be a support vector machine, random forest, or fully connected neural network, etc. For example, a well-trained classifier can distinguish between different types of cracks, such as fatigue cracks and stress cracks, or hotspots caused by different reasons, such as bearing failure and poor electrical connection. Based on the classifier's identification results, it is determined whether a fault exists in the fused image. If the classifier identifies a fault, it outputs the fault type. For example, it outputs "bearing overheating" or "circuit board crack." Different fault types correspond to different fault characteristics. For example, bearing overheating may manifest as an abnormally high temperature gradient in a specific area, while a circuit board crack may manifest as thin and discontinuous textures in the image. The final fault feature identification results are obtained. For example, if the output fault type is "bearing overheating," the corresponding fault features might be: in the thermal imaging image, the temperature of the bearing area is more than 15 degrees Celsius higher than the surrounding area, and the temperature gradient map shows a significant temperature abrupt change in this area; in the visible light image, slight discoloration or deformation may appear near the bearing. If the output fault type is "circuit board crack," the corresponding fault features might be: in the visible light image, a thin line approximately 5 mm long and less than 0.1 mm wide appears on the circuit board, and the texture on both sides of the line shows a significant difference; in the thermal imaging image, a localized temperature increase may occur near the crack because the crack hinders normal heat conduction.

[0059] Based on the extracted fault features, and combined with prior knowledge of photovoltaic equipment, a fault location algorithm based on region growing is designed. Using the fault features as seed points, the algorithm iteratively expands the neighborhood region and combines the similarity and connectivity of pixels within the neighborhood region to determine the boundary of the fault region, thereby achieving fault location and obtaining the fault location.

[0060] The algorithm acquires fault characteristic data and prior knowledge of photovoltaic equipment and uses them as input to a region growing algorithm. It determines initial seed points for the algorithm based on the fault characteristics, ensuring these seed points are located within the fault region. Using the seed point as the center, iteratively expands the neighboring region, selecting regions based on pixel similarity and region connectivity. In each iteration, it acquires pixel data within the current neighboring region and calculates its similarity to the seed point. If the similarity meets a preset threshold, the neighboring region is merged into the current fault region, and its boundary pixels are used as new seed points, continuing iterative expansion. If the similarity does not meet the preset threshold or the neighboring regions are no longer connected, the iterative expansion of the current region is stopped, and the current fault region is marked as an independent fault location. Steps 3-6 are repeated for all initial seed points until all seed points have been expanded, resulting in all fault locations and completing fault localization.

[0061] The core of region growing algorithms lies in iteratively expanding the neighborhood region. By continuously incorporating qualified neighboring pixels into the growing region, a complete fault region is eventually formed. The selection of the neighborhood region is typically based on two principles: pixel similarity and region connectivity. Pixel similarity refers to the degree of similarity between neighboring pixels and the seed point in terms of features (such as grayscale, color, texture, etc.), while region connectivity refers to the spatial adjacency between neighboring pixels and the seed point. For example, when processing an image of a defect on a metal surface, eight neighboring pixels whose grayscale values ​​differ from the seed point by no more than 10 can be selected as candidate regions. In each iterative expansion, the similarity between pixels within the current neighborhood region and the seed point needs to be calculated. The method for calculating similarity can be chosen based on the specific application scenario and data type. For example, for grayscale images, Euclidean distance or Manhattan distance can be used to measure the similarity between pixels; for color images, the similarity of color histograms can be used. Assuming the seed pixel has a grayscale value of 150, the neighboring pixel has a grayscale value of 145, and the preset similarity threshold is 8, then if we use the absolute value of the difference as the measure, the similarity between these two pixels is 5, which meets the preset threshold and can be considered similar. When the similarity between a neighboring pixel and the seed pixel meets the preset threshold, that neighboring region is merged into the current fault region, and its boundary pixels are marked as new seed pixels, serving as the starting point for the next iteration of expansion.

[0062] An acceleration strategy based on parallel computing is designed to decompose key tasks such as image registration, image fusion, fault feature extraction, and fault location, and to realize parallel computing using multi-core processors or graphics processing units to improve the real-time performance of fault monitoring.

[0063] By analyzing key tasks such as image registration, image fusion, fault feature extraction, and fault location, these tasks are decomposed into multiple subtasks, and a task dependency graph is established. Based on the task dependency graph, the execution order and parallelism of each subtask are determined to maximize parallel execution efficiency. Appropriate parallel computing modes, such as data parallelism and task parallelism, are selected based on the computational characteristics of each subtask, and task-to-hardware mapping is performed according to hardware resource characteristics, such as multi-core CPUs and GPUs. Multi-threaded programming models, such as OpenMP and TBB, are employed to achieve parallelization within subtasks, improving the execution efficiency of individual subtasks. Data communication and synchronization between subtasks are achieved through message passing interfaces (such as MPI) to ensure dependencies and data consistency. For data-intensive tasks, such as image registration and fusion, GPU-based parallel computing frameworks, such as CUDA and OpenCL, are used to fully utilize the massively parallel computing capabilities of GPUs and accelerate the computation process. For computationally intensive tasks, such as fault feature extraction and location, a parallel computing mode based on multi-core CPUs is adopted. By using techniques such as thread-level parallelism and instruction-level parallelism, the utilization and execution efficiency of a single-core CPU are improved, ultimately achieving real-time performance improvement in the fault monitoring process.

[0064] An attention-based fault interpretation module is introduced. By calculating the spatial correlation between the fault feature map and the equipment component location map, an attention weight map is generated to highlight key fault areas. Combining the attention weight map and the fault features, a fault interpretation report is generated to identify key fault locations and causes, providing a basis for maintenance decisions.

[0065] Based on the equipment component diagram, the location information of each component is obtained to construct an equipment component location map; monitoring data during equipment operation is acquired, and a fault feature map of the equipment is obtained through a fault diagnosis algorithm; an attention mechanism is used to calculate the spatial correlation between the fault feature map and the equipment component location map to obtain an attention weight map; based on the attention weight map, key fault areas are identified and key fault locations are highlighted; the key fault areas are fused with the fault feature map, and the specific fault causes of the key fault locations are determined through a fault cause analysis algorithm; based on the key fault locations and fault causes, a fault explanation report is generated, identifying the key fault information of the equipment; the fault explanation report is provided to maintenance personnel as a basis for making maintenance decisions and guiding equipment maintenance work.

[0066] On the other hand, this embodiment also provides a fault monitoring system based on joint analysis of thermal imaging and visible light images, including:

[0067] The data acquisition module is used to acquire thermal imaging images and visible light images of photovoltaic equipment, register the thermal imaging images and visible light images, and obtain registered thermal imaging images and visible light images.

[0068] The fusion module is used to perform pixel-level weighted summation on the registered thermal imaging image and visible light image based on a weighted fusion image fusion algorithm to obtain a fused image;

[0069] The identification model building module is used to construct a multi-scale convolutional neural network model containing multiple parallel convolutional layers with different kernel sizes, train the multi-scale convolutional neural network model until it learns fault feature representations at different levels and outputs them to obtain a fault pattern recognition model.

[0070] The localization construction module is used to construct a fault location localization model based on region growth by iteratively expanding the neighborhood region and combining the similarity and connectivity of pixels within the neighborhood region.

[0071] The calculation module is used to input the fused image into the fault pattern recognition model for calculation to obtain fault features, and input the fault features into the fault location model to obtain the fault location.

[0072] On the other hand, this embodiment also provides an electronic device, including a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the fault monitoring method based on joint analysis of thermal imaging and visible light images.

[0073] On the other hand, this embodiment also provides a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements a fault monitoring method based on joint analysis of thermal imaging and visible light images.

[0074] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A fault monitoring method based on joint analysis of thermal imaging and visible light images, characterized in that, include: Acquire thermal imaging images and visible light images of photovoltaic equipment, and register the thermal imaging images and visible light images to obtain registered thermal imaging images and visible light images; The image fusion algorithm based on weighted fusion performs pixel-level weighted summation on the registered thermal imaging image and visible light image to obtain the fused image; A multi-scale convolutional neural network model containing multiple parallel convolutional layers with different kernel sizes is constructed. The multi-scale convolutional neural network model is trained until it learns fault feature representations at different levels and then outputs them to obtain a fault pattern recognition model. A fault location model based on region growth is constructed by iteratively expanding the neighborhood region and combining the similarity and connectivity of pixels within the neighborhood region. The fused image is input into the fault pattern recognition model for calculation to obtain fault features, and the fault features are input into the fault location model to obtain the fault location.

2. The method according to claim 1, characterized in that, The process of obtaining registered thermal and visible light images includes: A thermal imaging image and a visible light image of a photovoltaic device are acquired. An image registration algorithm based on feature point matching is used to extract significant feature points from the thermal imaging image and the visible light image, establish the correspondence between the feature points, calculate the geometric transformation relationship between the thermal imaging image and the visible light image, and perform spatial alignment of the thermal imaging image and the visible light image based on the geometric transformation relationship to obtain the registered thermal imaging image and the visible light image.

3. The method according to claim 1, characterized in that, The process of obtaining the fused image includes: Acquire registered thermal and visible light images; The thermal imaging image is divided into blocks, and the average temperature gradient of each block is calculated to obtain a temperature gradient map. The visible light image is divided into blocks, and the texture complexity of each block is calculated to obtain a texture complexity map. Based on the temperature gradient map and texture complexity map, determine the thermal imaging weight coefficient and visible light weight coefficient for each block; Multiply the thermal imaging block by the thermal imaging weighting coefficient, multiply the visible light block by the visible light weighting coefficient, and then add them together to obtain the fused block; All the fusion blocks are stitched together to obtain a preliminary fused image. Local enhancement is then performed on the preliminary fused image to obtain the final fused image.

4. The method according to claim 3, characterized in that, The process of stitching together all the fusion blocks to obtain a preliminary fused image includes: Obtain multiple image blocks to be fused, and determine the stitching position and order of each image block according to the preset stitching rules; Given the determined stitching positions and order, an image stitching algorithm is used to stitch multiple images into a preliminary fused image. An image fusion algorithm is used to process the preliminary fusion image to obtain the fused target image; The target image after fusion is obtained. According to the preset quality assessment rules, the image quality assessment algorithm is used to assess the quality of the target image. If the quality assessment result meets the preset conditions, the target image is determined as the final fusion result image.

5. The method according to claim 1, characterized in that, The process of obtaining the fault mode recognition model includes: Construct a multi-scale convolutional neural network model containing multiple parallel convolutional layers with different kernel sizes; Max pooling is added to the output of each parallel convolutional layer in the multi-scale convolutional neural network model. The outputs of parallel convolution and pooling are concatenated and then flattened to generate an optimized multi-scale convolutional neural network model. The optimized multi-scale convolutional neural network model is trained using historical datasets until it learns fault feature representations at different levels and outputs the result, thus obtaining a fault pattern recognition model.

6. The method according to claim 1, characterized in that, The process of constructing a fault location model based on region growing by iteratively expanding the neighborhood region and combining the similarity and connectivity of pixels within the neighborhood region includes: The fault characteristic data and prior knowledge of photovoltaic equipment are obtained and used as input to the region growing algorithm. The initial seed point of the region growing algorithm is determined based on the fault characteristics, and the seed point is located inside the fault region. Centered on the seed point, the neighborhood region is expanded iteratively. The selection of the neighborhood region is determined based on pixel similarity and region connectivity. In each iteration of expansion, pixel data in the current neighborhood region is acquired, and its similarity with the seed point is calculated. If the similarity meets the preset threshold, the neighborhood region is merged into the current fault region, and its boundary pixels are used as new seed points to continue iterative expansion. If the similarity does not meet the preset threshold or the neighboring regions are no longer connected, the iterative expansion of the current region is stopped, and the current fault region is marked as an independent fault location, thus completing the construction of the fault location model.

7. The method according to claim 1, characterized in that, Also includes: An acceleration strategy based on parallel computing is designed, which decomposes the tasks of image registration, image fusion, fault feature extraction and fault location into tasks, and uses multi-core processors or graphics processing units to achieve parallel computing. At the same time, a fault interpretation module based on attention mechanism is introduced. By calculating the spatial correlation between the fault feature map and the equipment component location map, an attention weight map is generated to highlight key fault areas. Combining the attention weight map and the fault features, a fault interpretation report is generated.

8. A fault monitoring system based on joint analysis of thermal imaging and visible light images, characterized in that, include: The data acquisition module is used to acquire thermal imaging images and visible light images of photovoltaic equipment, register the thermal imaging images and visible light images, and obtain registered thermal imaging images and visible light images. The fusion module is used to perform pixel-level weighted summation on the registered thermal imaging image and visible light image based on a weighted fusion image fusion algorithm to obtain a fused image; The identification model building module is used to construct a multi-scale convolutional neural network model containing multiple parallel convolutional layers with different kernel sizes, train the multi-scale convolutional neural network model until it learns fault feature representations at different levels and outputs them to obtain a fault pattern recognition model. The localization construction module is used to construct a fault location localization model based on region growth by iteratively expanding the neighborhood region and combining the similarity and connectivity of pixels within the neighborhood region. The calculation module is used to input the fused image into the fault pattern recognition model for calculation to obtain fault features, and input the fault features into the fault location model to obtain the fault location.

9. An electronic device comprising a memory, a processor, and a computing program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1-7.

Images