Method and device for diagnosing equipment failure based on fusion of visible light and infrared thermal imaging

Abstract

The application provides a device fault diagnosis method and device based on visible light and infrared thermal imaging fusion, which comprises the following steps: performing space-time alignment on a visible light image sequence and a thermal radiation image sequence collected under the same field of view to obtain an aligned bimodal image pair; performing feature extraction and fusion based on the bimodal image pair to obtain target fusion features with enhanced and complementary characteristics; performing surface state analysis based on visible light branch information in the target fusion features to obtain a surface defect diagnosis result, and performing infrared thermal anomaly extraction based on infrared branch information in the target fusion features to obtain an anomaly diagnosis result; performing cross-modal joint reasoning based on the surface defect diagnosis result and the anomaly diagnosis result to obtain a comprehensive diagnosis result, and generating a structured diagnosis report based on the comprehensive diagnosis result. The application effectively solves the problems of diagnosis blind area, missed detection and misjudgment caused by single information in the prior art.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for diagnosing equipment faults based on the fusion of visible light and infrared thermal imaging. Background Technology

[0002] Currently, in drone inspections of power facilities and industrial equipment, equipment fault diagnosis generally relies on equipment condition analysis and diagnostic methods based on a single image source. This method uses a single type of image acquisition device (such as a visible light camera or an infrared thermal imager) mounted on the drone to acquire a single type of raw image data of the equipment surface, then processes this raw image data to finally output a diagnostic conclusion.

[0003] However, existing methods suffer from inherent limitations in complex real-world operating conditions. They suffer from a single dimension of diagnostic information and a lack of cross-verification between different dimensions, leading to high uncertainty in diagnostic results and difficulty in accurately identifying complex or potentially comprehensive faults. Specifically, while relying solely on visible light images can obtain high-resolution surface structural details, it cannot detect initial thermal anomalies such as overheating or poor contact within the equipment, easily overlooking latent faults that have not yet developed into external deformation. Relying solely on infrared thermal images can detect temperature anomalies, but due to the inherent characteristics of infrared thermal imaging technology, its low spatial resolution and lack of texture details make it difficult to accurately pinpoint the specific component and structural location of the fault, and it cannot distinguish between temperature rise caused by surface contamination and overheating caused by substantial internal damage. This diagnostic blind spot caused by a single information source often results in missed detections or misjudgments when facing complex faults that require simultaneous consideration of external deformation and internal thermal state, thus failing to provide sufficient and reliable basis for operational and maintenance decisions. Summary of the Invention

[0004] This invention provides a method and apparatus for equipment fault diagnosis based on the fusion of visible light and infrared thermal imaging. It solves the problems of diagnostic blind spots, missed detections and misjudgments caused by the single information in existing methods, and comprehensively improves the accuracy and reliability of equipment fault diagnosis. It can make accurate judgments on complex or potential comprehensive faults, and provides sufficient and reliable technical basis for equipment operation and maintenance decisions.

[0005] In a first aspect, the present invention provides a device fault diagnosis method based on the fusion of visible light and infrared thermal imaging, comprising: Spatiotemporal alignment of visible light image sequences and thermal radiation image sequences acquired under the same field of view yields aligned dual-modal image pairs. Based on the dual-modal image pairs, feature extraction and fusion are performed to obtain the target fusion features after feature enhancement and complementation; Surface state analysis is performed based on the visible light branch information in the target fusion features to obtain surface defect diagnosis results, and infrared thermal anomaly extraction is performed based on the infrared branch information in the target fusion features to obtain anomaly diagnosis results. Based on the surface defect diagnosis results and the anomaly diagnosis results, cross-modal joint reasoning is performed to obtain a comprehensive diagnosis result, and a structured diagnosis report is generated based on the comprehensive diagnosis result.

[0006] Secondly, the present invention also provides a device fault diagnosis apparatus based on the fusion of visible light and infrared thermal imaging, applied to the device fault diagnosis method based on the fusion of visible light and infrared thermal imaging as described in the first aspect; the device fault diagnosis apparatus based on the fusion of visible light and infrared thermal imaging includes: The spatiotemporal synchronization module is used to perform spatiotemporal alignment of visible light image sequences and thermal radiation image sequences acquired under the same field of view to obtain aligned dual-modal image pairs. The feature fusion module is used to extract and fuse features based on the dual-modal image pair to obtain the target fusion features after feature enhancement and complementation; The anomaly analysis and extraction module is used to perform surface state analysis based on the visible light branch information in the target fusion feature to obtain surface defect diagnosis results, and to extract infrared thermal anomalies based on the infrared branch information in the target fusion feature to obtain anomaly diagnosis results. The diagnostic reasoning and generation module is used to perform cross-modal joint reasoning based on the surface defect diagnosis results and the anomaly diagnosis results to obtain a comprehensive diagnostic result, and generate a structured diagnostic report based on the comprehensive diagnostic result.

[0007] Thirdly, the present invention also provides an electronic device, comprising: a memory for storing computer software programs; and a processor for reading and executing the computer software programs, thereby realizing the device fault diagnosis method based on visible light and infrared thermal imaging fusion as described above.

[0008] Fourthly, the present invention also provides a non-transitory computer-readable storage medium storing a computer software program, which, when executed by a processor, implements the device fault diagnosis method based on visible light and infrared thermal imaging fusion as described above.

[0009] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the device fault diagnosis method based on the fusion of visible light and infrared thermal imaging as described above.

[0010] The device fault diagnosis method based on visible light and infrared thermal imaging fusion provided in this invention aligns visible light image sequences and thermal radiation image sequences acquired under the same field of view in a spatiotemporal manner to obtain aligned dual-modal image pairs. This effectively solves the problem that existing single image sources cannot simultaneously acquire details of the device's surface structure and internal thermal state information. Based on these aligned dual-modal image pairs, feature extraction and fusion are performed to obtain enhanced and complementary target fusion features. This organically combines the high-resolution structural texture features of visible light images with the thermal anomaly features of infrared thermal images, overcoming the limitations of single-modal image features. Based on the visible light components in the target fusion features... By separately performing surface state analysis and infrared thermal anomaly extraction, surface defect diagnosis results and anomaly diagnosis results are obtained using surface branch information and infrared branch information, respectively. This allows the feature information of the two modes to give full play to their core advantages. Based on the two diagnosis results, a comprehensive diagnosis result is obtained and a structured diagnosis report is generated through cross-modal joint reasoning. This realizes the mutual verification and collaborative judgment of surface defect and thermal anomaly information, effectively solving the problems of diagnostic blind spots, missed detections and misjudgments caused by single information in existing methods. Ultimately, it comprehensively improves the accuracy and reliability of equipment fault diagnosis, and can make accurate judgments on complex or potential comprehensive faults, providing sufficient and reliable technical basis for equipment operation and maintenance decisions. Attached Figure Description

[0011] Figure 1 This is a schematic flowchart of a device fault diagnosis method based on the fusion of visible light and infrared thermal imaging provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the device fault diagnosis device based on the fusion of visible light and infrared thermal imaging provided in an embodiment of the present invention; Figure 3 An embodiment diagram of the electronic device provided in this invention; Figure 4 An embodiment diagram of a computer-readable storage medium provided in accordance with the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0014] See Figure 1 , Figure 1 This is a flowchart illustrating the device fault diagnosis method based on the fusion of visible light and infrared thermal imaging provided by the present invention. In this embodiment of the invention, the executing entity of the device fault diagnosis method based on the fusion of visible light and infrared thermal imaging is a fault diagnosis device. Therefore, the device fault diagnosis method based on the fusion of visible light and infrared thermal imaging includes: Step 10: Spatiotemporally align the visible light image sequence and the thermal radiation image sequence acquired under the same field of view to obtain an aligned dual-modal image pair.

[0015] Optionally, the fault diagnosis device uses a visible light camera and an infrared thermal imager mounted on a drone to collect visible light image sequences and thermal radiation image sequences of the same equipment area at the same inspection location and time. The visible light image sequence is a collection of multiple frames of visible light images continuously captured by the visible light camera; the thermal radiation image sequence is a collection of multiple frames of infrared thermal images continuously captured by the infrared thermal imager. Since each frame of the visible light image shows the structural details, color information, and appearance defects of the equipment surface, and each frame of the thermal radiation image shows the thermal radiation distribution on the equipment surface and inside, it is necessary to align each frame of the visible light image and each frame of the thermal radiation image in time and space to avoid image misalignment caused by shooting time differences and to eliminate visual deviations caused by factors such as differences in the installation position of the shooting equipment.

[0016] Specifically, spatiotemporal alignment includes alignment processing in the time dimension and alignment processing in the spatial dimension. The time alignment process specifically includes: reading the capture timestamp of each frame in the two image sequences, grouping visible light images and thermal radiation images with the same timestamp into a group to form an initial image pair; and for image frames with a timestamp deviation of no more than 100 milliseconds, supplementing the missing moment images through inter-frame interpolation, and then performing time alignment; and for image frames with a time deviation of more than 100 milliseconds, directly discarding them. After time alignment, spatial alignment is performed on each initial image pair. The specific alignment process includes: selecting feature reference points in the image. These feature reference points are fixed parts on the equipment surface that are easily identifiable and not easily deformed, including bolts, insulator edges, and equipment nameplate edges in power facilities, ensuring that the feature reference points can be clearly identified in both images, with a minimum of three. Then, the coordinates of the feature reference points in each frame of the visible light image and the corresponding thermal radiation image are extracted. The same feature reference points in the two images are matched to establish a coordinate mapping relationship. Based on the coordinate mapping relationship, the thermal radiation image is subjected to geometric transformation processing, including translation, scaling, and rotation, so that the coordinates of the feature reference points in the thermal radiation image are consistent with the corresponding feature reference points in the visible light image. Finally, after the geometric transformation is completed, the image edges are smoothed to eliminate jagged edges, blurring, and other distortions generated during the transformation process. After spatial alignment is completed, each set of matched and aligned visible light images is combined with thermal radiation images to obtain aligned bimodal image pairs. Each bimodal image pair contains a visible light image and a thermal radiation image corresponding to the same time, field of view, and spatial location.

[0017] In one embodiment, assuming a scenario of unmanned aerial vehicle (UAV) inspection of power facilities, the UAV is equipped with a visible light camera and an infrared thermal imager to inspect a transformer in a substation. The visible light camera captures images at a rate of 10 frames per second, obtaining a visible light image sequence containing 500 frames, each with a resolution of 1920×1080, showing structural details such as the transformer's casing, heat sink, and terminals. The infrared thermal imager also captures images at a rate of 10 frames per second, obtaining a thermal radiation image sequence containing 500 frames, each with a resolution of 640×480, showing the thermal radiation distribution of various parts of the transformer.

[0018] The fault diagnosis device reads the capture timestamp of each frame in the two image sequences, matches images with the same timestamp, and finds 480 sets of images with completely identical timestamps, forming 480 initial image pairs; another 20 sets of images have timestamp deviations between 50 and 80 milliseconds, and the missing images are supplemented by inter-frame interpolation to complete time alignment; there are no image frames with time deviations exceeding 100 milliseconds.

[0019] Subsequently, the fault diagnosis device selected three bolts on the transformer surface as feature reference points. These three bolts were clearly identifiable in both visible light and thermal radiation images. The coordinates of these three bolts were extracted from both images, a coordinate mapping relationship was established, and a geometric transformation was performed on the thermal radiation image to align the bolt coordinates in the thermal radiation image with the corresponding bolt coordinates in the visible light image. Edge smoothing was then performed to eliminate image distortion. Finally, 480 aligned dual-modal image pairs were obtained. Each dual-modal image pair contained a visible light image and a thermal radiation image of the corresponding part of the transformer, captured at the same time.

[0020] Step 20: Based on the dual-modal image pair, feature extraction and fusion are performed to obtain the target fusion feature after feature enhancement and complementation.

[0021] Optionally, the fault diagnosis device extracts features from the visible light image and thermal radiation image of each aligned bimodal image pair, and then fuses the extracted features to eliminate redundant information between the two modes, obtaining the target fused feature, as described in steps 201 to 205. This target fused feature simultaneously possesses details of the device's surface structure and internal thermal state information, specifically including visible light branch information and infrared branch information. The visible light branch information includes a spatial index mask of the defect region, a grayscale distribution map of the defect region, a local structural reference scale, and a fusion semantic classification label; the infrared branch information includes a spatial index mask of the thermal anomaly region, a temperature difference distribution map, a local structural reference scale, and a fusion semantic classification label. Furthermore, the local structural reference scale at both locations is consistent with the fusion semantic classification label.

[0022] Step 30: Surface state analysis is performed based on visible light branch information in the target fusion features to obtain surface defect diagnosis results, and infrared thermal anomaly extraction is performed based on infrared branch information in the target fusion features to obtain anomaly diagnosis results.

[0023] Optionally, the fault diagnosis device extracts visible light branch information and infrared branch information from the target fusion features based on the target fusion features. Based on the defect region spatial index mask, defect region grayscale distribution map, local structural reference scale, and fusion semantic classification label in the visible light branch information, it performs surface state analysis on the equipment to obtain surface defect diagnosis results, as detailed in steps 3011 to 3015. These surface defect diagnosis results include defect nature, severity level, and spatial location. This allows for the identification of the specific type of defect through defect nature, such as common surface damage, cracks, corrosion, dirt, and deformation in power facilities; the determination of the severity level based on the severity of the defect's impact on normal equipment operation, facilitating subsequent judgment on whether shutdown is necessary; and the determination of the specific location of the surface defect based on its spatial location.

[0024] Simultaneously, infrared thermal anomaly extraction is performed based on the spatial index mask of the thermal anomaly region, temperature difference distribution map, local structural reference scale benchmark, and fused semantic classification labels in the infrared branch information. This yields anomaly diagnosis results to determine whether the equipment exhibits thermal anomalies and their specific characteristics, as detailed in steps 3021 to 3026. These anomaly diagnosis results include the nature of the thermal anomaly, its severity level, peak temperature, and spatial location. This allows for the determination of the type of cause of the temperature anomaly through its nature, such as overheating due to poor contact, internal component wear, or surface contamination common in power facilities. The severity level also determines the degree of impact on normal equipment operation. The peak temperature determines the highest temperature value within the thermal anomaly region, and the spatial location determines the specific location of the thermal anomaly. Furthermore, the severity levels of both are consistent with the coordinate system.

[0025] Step 40: Perform cross-modal joint reasoning based on the surface defect diagnosis results and anomaly diagnosis results to obtain a comprehensive diagnosis result, and generate a structured diagnosis report based on the comprehensive diagnosis result.

[0026] Optionally, the fault diagnosis device performs cross-modal joint reasoning based on the nature, severity level, and spatial location of defects in the surface defect diagnosis results, and the nature, severity level, peak temperature, and spatial location of thermal anomalies in the anomaly diagnosis results, combined with a preset equipment fault diagnosis knowledge graph. Through cross-validation of bimodal information, a comprehensive diagnostic result is obtained, as detailed in steps 401 to 405. This comprehensive diagnostic result includes the nature of the fault, severity level, quantitative indicators, and tracing evidence to clarify whether the equipment has a fault, the specific cause of the fault, the degree of impact, and corresponding recommendations.

[0027] Optionally, after obtaining the comprehensive diagnostic results, the fault diagnosis device generates a structured diagnostic report based on these results according to a preset fixed format. This report includes at least basic inspection information, a comprehensive diagnostic conclusion, detailed fault information, and maintenance recommendations. The basic inspection information includes inspection time, name of the inspected equipment, inspection location, and UAV inspection parameters (shooting rate, inspection altitude, image resolution, etc.). The comprehensive diagnostic conclusion clarifies whether the equipment has a fault, the total number of faults, and the main fault types. The detailed fault information corresponds to each item in the comprehensive diagnostic results, including the fault nature (clarifying the specific type and core cause of the fault), the severity level (clarifying the severity of the fault (minor, moderate, severe), and the quantitative indicators (presenting all quantitative indicators in the comprehensive diagnostic results, specifying the specific value, unit, and meaning of each indicator, such as peak temperature of 75℃ and defect area of ​​5cm²). 2The fault is identified through various means, including a temperature difference of 20°C, spatial location (specifically, the location of the fault on the equipment surface), and tracing evidence presented in the comprehensive diagnostic results, clearly defining the type, source, and function of the evidence. Maintenance recommendations are provided based on the nature, severity, and quantitative indicators of the fault in the comprehensive diagnostic results, offering targeted handling measures, timelines, monitoring requirements, and tracing verification suggestions. For example, in this embodiment, for a "general-level, peak heat of 75°C contact overheating fault," it is recommended to "complete rust removal and tightening within 15 days, monitor temperature changes weekly (focusing on whether the quantitative indicators exceed the 80°C threshold), and retain monitoring data as supplementary evidence for subsequent tracing." Complete data is also provided, including aligned dual-modal image sequences, defect and thermal anomaly area marker images, and other data from the fault diagnosis process.

[0028] The embodiments of the present invention effectively solve the problems of diagnostic blind spots, missed detections and misjudgments caused by the limited information in existing methods. Ultimately, through the coordinated linkage of each step, the accuracy and reliability of equipment fault diagnosis are comprehensively improved. It can make accurate judgments on complex or potential comprehensive faults, and provide sufficient and reliable technical basis for equipment operation and maintenance decisions.

[0029] Optionally, the process of steps 201 to 205 includes: Step 201: Based on the visible light image sequence in the dual-modal image, perform gradient magnitude calculation and binarization to obtain a binarized edge mask, and perform morphological thinning based on the binarized edge mask to obtain a visible light structure skeleton line with a single pixel width.

[0030] Optionally, the fault diagnosis device, based on the visible light image sequence in the dual-modal image, sequentially performs gradient magnitude calculation and binarization processing on each frame of the visible light image to obtain a binarized edge mask corresponding to each frame. In this embodiment, during gradient calculation, a 3×3 neighborhood window is selected centered on each pixel. The gray-level difference between the pixel and each neighboring pixel is calculated. The sum of squares of all gray-level differences is then performed, and the square root of the sum of squares is taken to obtain the gradient magnitude of the pixel. This gradient magnitude reflects the degree of gray-level change at the pixel. The more drastic the gray-level change, the larger the gradient magnitude, corresponding to the edge region in the image; the more gradual the gray-level change, the smaller the gradient magnitude, corresponding to the non-edge region in the image.

[0031] After calculating the gradient magnitude, the gradient magnitude image of each visible light image frame is binarized. A fixed gradient magnitude threshold is set; pixels with gradient magnitudes greater than this threshold are identified as edge pixels and assigned a value of 255 (white); pixels with gradient magnitudes less than or equal to this threshold are identified as non-edge pixels and assigned a value of 0 (black), resulting in a binarized edge mask. The gradient magnitude threshold ranges from 50 to 80 and can be determined based on the gradient magnitude statistics of normal visible light images from the device.

[0032] Optionally, after obtaining the binarized edge mask, the fault diagnosis device performs morphological refinement processing on the binarized edge mask. While preserving the core features of the edge contour, it gradually removes redundant pixels in the edge region, ultimately obtaining a visible light structural skeleton line with a single pixel width. In this embodiment, for example, an iterative erosion algorithm is used for morphological refinement. In each iteration, only edge pixels that do not affect the continuity of the contour are removed, while core pixels that maintain the integrity of the edge contour are retained. The iteration process continues until the width of the edge region becomes one pixel, at which point the iteration stops. That is, the single-pixel-width visible light structural skeleton line is a line with a width of only one pixel, capable of completely reflecting the structural contour of the equipment surface and outlining the core morphology of key structures on the equipment surface (such as bolts, insulator edges, equipment housing contours, etc.).

[0033] Step 202: Extract high-temperature pixel set based on the thermal radiation image sequence in the dual-modal image combined with the preset high temperature threshold, and perform eight-neighbor connectivity labeling based on the high-temperature pixel set to obtain thermal anomaly connected region mask.

[0034] Optionally, the high-temperature threshold is a critical temperature value set based on the highest allowable temperature during normal equipment operation. It is used to distinguish between normal temperature pixels and high-temperature pixels. The specific threshold can be determined according to the equipment type, equipment operating parameters, and industry maintenance standards. For example, the high-temperature threshold for substation transformers is set to 65℃. The fault diagnosis device, based on the thermal radiation image sequence in the dual-modal image, performs temperature calibration on each frame of the thermal radiation image, converting the pixel grayscale values ​​in the thermal radiation image into corresponding actual temperature values. This temperature calibration is based on the calibration parameters of the infrared thermal imager. Subsequently, all pixels in each frame of the thermal radiation image whose actual temperature value is greater than the preset high-temperature threshold are extracted to form a high-temperature pixel set for that frame. This high-temperature pixel set contains the coordinates and temperature values ​​of all high-temperature pixels.

[0035] Optionally, after obtaining the high-temperature pixel set, the fault diagnosis device performs eight-neighbor connectivity marking based on this set. Taking each high-temperature pixel as the center, it detects its eight surrounding adjacent pixels (i.e., pixels in the eight directions: up, down, left, right, upper left, upper right, lower left, and lower right). If an adjacent pixel also belongs to the high-temperature pixel set, it is determined that the two pixels belong to the same connected region. All high-temperature pixels are grouped in this way, and interconnected high-temperature pixels are divided into a thermal anomaly connected region, ultimately obtaining a thermal anomaly connected region mask. This thermal anomaly connected region mask is a binary mask image used to mark all thermal anomaly connected regions in the thermal radiation image. Pixels belonging to thermal anomaly connected regions are assigned a value of 255 (white), and pixels not belonging to thermal anomaly connected regions are assigned a value of 0 (black). Each thermal anomaly connected region corresponds to an independent marker number.

[0036] Step 203: Based on the Euclidean distance transformation map of the visible light structural skeleton line, calculate the spatial distance from each pixel in the thermal anomaly connected region mask to the nearest structural skeleton line to obtain a thermal spatial distance map. Based on the thermal spatial distance map, remove isolated hot spots whose distance exceeds the preset component width threshold to obtain the geometrically corrected effective thermal area mask.

[0037] Optionally, the fault diagnosis device first performs Euclidean distance transformation processing based on the visible light structural skeleton line to form an Euclidean distance transformation map of the visible light structural skeleton line. This Euclidean distance transformation map is an image where the Euclidean distance value of each pixel is used as its grayscale value, and the grayscale value of each pixel corresponds to its spatial distance to the nearest structural skeleton line. Specifically, in the Euclidean distance calculation process, each pixel in the visible light structural skeleton line is used as a reference, and the straight-line distance between each pixel in the thermal anomaly connected region mask and the reference pixel is calculated. The calculation formula is: the Euclidean distance between two pixels is equal to the square of the difference in coordinates between the two pixels in the horizontal direction, plus the square of the difference in coordinates between the two pixels in the vertical direction, and then the square root of the sum of squares is taken. After calculating the Euclidean distance from each high-temperature pixel to all structural skeleton line pixels and forming the Euclidean distance transformation map, the smallest distance value is selected as the spatial distance from the high-temperature pixel to the nearest structural skeleton line. After calculating the spatial distance of all high-temperature pixels, the spatial distance value of each high-temperature pixel is used as the gray value of that pixel to generate a thermal spatial distance map. In the map, the gray value of each pixel in the thermal anomaly connected region corresponds to its spatial distance to the nearest structural skeleton line.

[0038] Optionally, the preset component width threshold is a distance threshold set based on the physical width of the actual components of the equipment, used to distinguish between thermal anomaly areas related to equipment components and isolated hot spots. This threshold can be determined according to the actual width of the equipment components; for example, the width of a transformer terminal is 2 cm, corresponding to a pixel width of 10 pixels in the image. Therefore, the preset component width threshold is set to the distance value corresponding to 10 pixels. Subsequently, the fault diagnosis device, based on the thermal spatial distance map and combined with the preset component width threshold, performs a process to eliminate isolated hot spots. The process is as follows: first, it detects the pixels within each thermal anomaly connected region in the thermal spatial distance map. If the spatial distance of all pixels within a thermal anomaly connected region exceeds the preset component width threshold, it indicates that the thermal anomaly area is unrelated to any equipment component, and it is identified as an isolated hot spot. The thermal anomaly connected region is then removed from the thermal anomaly connected region mask. If the spatial distance of at least one pixel within a thermal anomaly connected region is less than or equal to the preset component width threshold, it indicates that the thermal anomaly area is related to the equipment component and is retained. The final result is a geometrically corrected effective thermal mask, which is a binary mask image containing only the real thermal anomaly regions related to the device components.

[0039] Step 204: Using the effective hot area mask as the defined region of the visible light image sequence, the local texture complexity distribution map is determined based on the local Shannon entropy value calculated within the defined region, and a visible light surface defect sensitive mask is generated based on the high entropy region in the local texture complexity distribution map.

[0040] Optionally, the fault diagnosis device uses the effective thermal area mask and the visible light image sequence in the dual-modal image pair as a defined region of the visible light image sequence. This defined region is the area in the visible light image sequence that completely corresponds to the location of the white pixel area (the actual thermal anomaly area) in the effective thermal area mask. Then, for each frame of the visible light image sequence within this defined region, the local Shannon entropy value of each pixel is calculated. This local Shannon entropy value measures the texture complexity of the local image region; the more complex the texture, the higher the Shannon entropy value; the simpler the texture, the lower the Shannon entropy value. After calculation, the local Shannon entropy value of each pixel is used as its grayscale value to generate a local texture complexity distribution map. This distribution map reflects the texture complexity of each part within the defined region. In this embodiment, the calculation process is as follows: taking each pixel in the defined area as the center, and using the logarithm to the base 2, select a 5×5 local neighborhood window, calculate the probability of each gray value appearing in the neighborhood window, multiply the probability of each gray value by the logarithm of the probability, sum all the multiplication results, and finally take the opposite of the summation result to obtain the local Shannon entropy value of the pixel.

[0041] Optionally, the fault diagnosis device generates a visible light surface defect sensitive mask based on high-entropy regions in a local textured complex distribution map. These high-entropy regions are areas where the local Shannon entropy value is greater than a preset entropy threshold. The preset entropy threshold is determined based on statistical results of local Shannon entropy values ​​in normal areas of the device, and ranges from 2.0 to 3.0. The specific value can be adaptively adjusted according to the device type and surface material. Furthermore, this visible light surface defect sensitive mask is a binary mask image used to mark areas in the visible light image that may contain surface defects. Pixels in high-entropy regions (areas that may contain surface defects) are assigned a value of 255 (white), while pixels in low-entropy regions (normal areas) are assigned a value of 0 (black).

[0042] Step 205: Based on the visible light surface defect sensitive mask and the effective hot area mask, perform mutual exclusion decoupling fusion to obtain the target fusion feature.

[0043] Optionally, the fault diagnosis device performs mutual exclusion decoupling fusion based on the visible light surface defect sensitive mask and the effective hot area mask to obtain the target fusion features, as specifically in steps 2051 to 2054.

[0044] This invention combines the high-resolution structural texture advantage of visible light images with the thermal anomaly detection advantage of infrared thermal images, ensuring that the target fusion features contain both accurate details of equipment surface defects and reliable internal thermal anomaly information.

[0045] Optionally, the process of steps 2051 to 2054 includes: Step 2051: Perform logical operations based on the visible light surface defect sensitive mask and the effective thermal area mask to obtain the thermal-optical overlap region mask, the pure thermal anomaly region mask, and the pure surface defect region mask, respectively.

[0046] Optionally, the fault diagnosis device performs logical operations on two masks—a visible light surface defect-sensitive mask and an effective thermal area mask—to classify the target region into thermo-optical overlap region masks, pure thermal anomaly region masks, and pure surface defect region masks. Specifically, for generating the thermo-optical overlap region mask, a logical AND operation is performed on the visible light; that is, each pixel position in the image is traversed. If the pixel value of both the visible light surface defect-sensitive mask and the effective thermal area mask at that position is 255 (white / foreground), it is determined to be overlapped, and the pixel value at that position is assigned 255; otherwise, it is assigned 0 (black / background). This generates a thermo-optical overlap region mask that exhibits both surface defect features and thermal anomaly features. For generating pure thermal anomaly region masks, the goal is to extract regions that only exhibit thermal anomalies but lack visible light surface defect features. Therefore, a logical NOT operation is performed on the visible light surface defect-sensitive mask to obtain its complement (i.e., 255 in the original mask becomes 0, and 0 becomes 255). Then, a logical AND operation is performed between this complement and the effective thermal region mask, retaining pixels in the effective thermal region mask that are background (value 0) in the visible light surface defect-sensitive mask. This generates pure thermal anomaly region masks that only exhibit thermal anomalies and lack surface defect features. For generating pure surface defect region masks, the goal is to extract regions that only exhibit visible light surface defect features but lack thermal anomalies. Therefore, a logical NOT operation is performed on the effective thermal region mask to obtain its complement. Then, a logical AND operation is performed between this complement and the visible light surface defect-sensitive mask, retaining pixels in the visible light surface defect-sensitive mask that are background (value 0) in the effective thermal region mask. This generates pure surface defect region masks that only exhibit surface defects and lack thermal anomaly features.

[0047] Step 2052: Based on the thermo-optical overlap region mask, the pure thermal anomaly region mask, and the pure surface defect region mask, spatial mutual exclusion partitioning is performed to obtain three feature subspaces, and the corresponding bimodal original pixel intensity sequence is taken for each feature subspace.

[0048] Optionally, the fault diagnosis device performs spatially mutually exclusive partitioning based on the thermo-optical overlap region mask, the pure thermal anomaly region mask, and the pure surface defect region mask. The target regions corresponding to the three types of masks are divided into independent spatial regions, ensuring that the three regions do not overlap or intersect, and that the set of the three regions covers all target regions of the two original masks. Each independent spatial region is a feature subspace, resulting in three feature subspaces corresponding to the thermo-optical overlap feature subspace, the pure thermal anomaly feature subspace, and the pure surface defect feature subspace, respectively.

[0049] After completing the spatial mutual exclusion partitioning, the fault diagnosis device extracts the corresponding bimodal raw pixel intensity sequence for each feature subspace. This bimodal raw pixel intensity sequence is a sequence composed of the raw pixel intensity values ​​of the visible light image sequence and the thermal radiation image sequence in the aligned bimodal image pair obtained in step 10 within each feature subspace. Specifically, the visible light raw pixel intensity sequence corresponds to the grayscale value (0 to 255) of each pixel in the visible light image within that feature subspace; the thermal radiation raw pixel intensity sequence corresponds to the grayscale value (corresponding to the actual temperature value) of each pixel in the thermal radiation image within that feature subspace. The extraction process is as follows: first, determine the coordinates of all pixels in each feature subspace; then, based on the coordinates, extract the raw pixel intensity value of each pixel in each frame of the image from the visible light image sequence and the thermal radiation image sequence, respectively; and finally, assemble the visible light raw pixel intensity sequence and the thermal radiation raw pixel intensity sequence corresponding to that feature subspace according to the pixel coordinate order and the image frame order.

[0050] Step 2053: Perform extreme value normalization processing on the original pixel intensity sequence of the dual modes in each feature subspace to obtain the normalized modal data sequence, and perform zero value padding based on the dimensional difference of the modal feature sequence to obtain a dimension-aligned fixed-length normalized data array.

[0051] Optionally, the fault diagnosis device is based on the bimodal raw pixel intensity sequences corresponding to each of the three feature subspaces. Since the number of pixels differs in different feature subspaces, and the value ranges of the visible light raw pixel intensity sequences and the thermal radiation raw pixel intensity sequences differ, extreme value normalization processing is required for each bimodal raw pixel intensity sequence. This maps all data in the bimodal raw pixel intensity sequence of each feature subspace to a value range of 0 to 1. Finally, two sets of normalized modal data sequences (visible light mode and thermal radiation mode) are obtained for each feature subspace.

[0052] Optionally, the fault diagnosis device performs zero-value padding on the normalized modal data sequences of the three feature subspaces based on the dimensional differences of the modal feature sequences. The dimensional differences refer to the different lengths of the modal data sequences in different feature subspaces. Zero-value padding is performed by adding zero values ​​to the end of the shorter modal data sequences, using the longest dimension of the modal data sequences in the three feature subspaces as the standard, until the length of all modal data sequences is consistent with the longest dimension, thus obtaining a dimension-aligned fixed-length normalized data array.

[0053] Step 2054: Based on the preset logical gating mapping table, labels are assigned to each feature subspace to obtain the corresponding label features. The label features are then serially concatenated with a fixed-length standardized data array to obtain the target fusion features.

[0054] Optionally, the preset logic gating mapping table is a mapping relationship table pre-stored in the fault diagnosis device, used to associate feature subspace types with label features. This mapping table specifies a unique label feature corresponding to each feature subspace, which is a fixed-length binary sequence (with a value of 0 or 1). Therefore, based on the fixed-length standardized data arrays of the three feature subspaces, the fault diagnosis device retrieves the preset logic gating mapping table, assigns a label to each feature subspace based on this mapping table, and obtains the label feature corresponding to each feature subspace. Specifically, the mapping relationship of the logic gating mapping table includes: thermal-optical overlap feature subspace → label feature "01"; pure thermal anomaly feature subspace → label feature "10"; pure surface defect feature subspace → label feature "11".

[0055] Specifically, the labeling process is as follows: identify the type of each feature subspace (thermal-optical overlap, pure thermal anomaly, pure surface defect), find the corresponding label feature according to the logic gating mapping table, and assign the label feature to the feature subspace.

[0056] After tagging is completed, the fault diagnosis device serially concatenates the tag features corresponding to each feature subspace with the fixed-length standardized data array of that feature subspace. This involves sequentially connecting the tag features and the fixed-length standardized data array to form a longer continuous data sequence. For example, the tag features are concatenated first, followed by the visible light mode fixed-length standardized data array of that feature subspace, and then the thermal radiation mode fixed-length standardized data array of that feature subspace. Finally, the continuous data sequences of the three feature subspaces, after serial concatenation, are serially concatenated again in the order of thermal-optical overlap feature subspace, pure thermal anomaly feature subspace, and pure surface defect feature subspace, integrating them into a complete data sequence. This data sequence is the target fusion feature. Furthermore, within this target fusion feature, branching can be performed based on the tag features and data source channels. Information from visible light images used to characterize surface defects is identified as visible light branch information, and information from thermal radiation images used to characterize thermal anomalies is identified as infrared branch information. The label features for visible light branch information are the labels corresponding to the thermo-optical coincidence feature subspace and the pure surface defect feature subspace. The data comes from the standardized data obtained by normalizing the original pixel intensity sequence of the visible light image after extreme value normalization and zero value padding. The label features for infrared branch information are the labels corresponding to the thermo-optical coincidence feature subspace and the pure thermal anomaly region mask. The data comes from the standardized data obtained by normalizing the original pixel intensity sequence of the thermal radiation image after extreme value normalization and zero value padding.

[0057] Furthermore, the visible light branch information includes a spatial index mask of the defect region, a grayscale distribution map of the defect region, a local structural reference scale benchmark, and a fusion semantic classification label; while the infrared branch information includes a spatial index mask of the thermal anomaly region, a temperature difference distribution map, a local structural reference scale benchmark, and a fusion semantic classification label. The defect region spatial index mask consists of a set of spatial location indexes composed of a visible light surface defect sensitive mask, a thermo-optical overlap region mask, and a pure surface defect region mask, used to mark the pixel location of a suspected defect; the defect region grayscale distribution map refers to the normalized brightness distribution data of the visible light image within a defined area, presenting the grayscale distribution of the corresponding area of ​​the defect region spatial index mask, used to characterize the depth of surface defect texture; the thermal anomaly region spatial index mask consists of a set of spatial location indexes composed of an effective thermal area mask, a thermo-optical overlap region mask, and a pure thermal anomaly region mask, used to mark the pixel location of a suspected thermal anomaly; the temperature difference distribution map is the normalized infrared temperature difference distribution data within the effective thermal area, presenting the temperature difference distribution of the corresponding area of ​​the thermal anomaly region spatial index mask, used to characterize the intensity of the thermal anomaly; the local structure reference scale benchmark refers to the average pixel width of the visible light structural skeleton line corresponding to the defect region; and the fused semantic classification label is a subspace type label assigned by a logical gated mapping table, used to identify the modal type of surface defects or thermal anomalies.

[0058] The embodiments of the present invention achieve precise separation, standardized processing and efficient fusion of two mask features. It not only retains the correlation features of the thermal-optical overlap region, the thermal features of the pure thermal anomaly region, and the structural defect features of the pure surface defect region, but also ensures the standardization and distinguishability of the target fusion features through standardization and labeling, further enhancing the complementary enhancement effect of the dual-modal features.

[0059] Optionally, the process of steps 3011 to 3015 includes: Step 3011: Logical judgment is made based on the fusion semantic classification labels in the visible light branch information. When the label is external damage or severe composite fault, the corresponding defect area spatial index mask and defect area grayscale distribution map are retained. When the label is internal overheating or other non-surface defect labels, the corresponding defect area spatial index mask is set to an empty set and the corresponding defect area grayscale distribution map is set to background suppression state to obtain the effective surface defect mask and effective surface defect grayscale map.

[0060] Optionally, the fault diagnosis device retrieves visible light branch information from the target fusion features, and then uses this visible light branch information to retrieve fusion semantic classification labels, defect region spatial index masks, and defect region grayscale distribution maps. The fusion semantic classification labels include labels for external damage, severe composite faults, internal overheating, and other non-surface defects. Therefore, the fusion semantic classification labels are first logically judged. When a label is determined to be external damage or a severe composite fault, its corresponding suspected defect region contains surface defect components, so the corresponding defect region spatial index mask and defect region grayscale distribution map are directly retained. When a label is determined to be internal overheating or other non-surface defect labels, its corresponding suspected defect region does not contain surface defects, so the defect region spatial index mask is set to an empty set, and the defect region grayscale distribution map is set to a background suppression state (all pixel grayscale values ​​are adjusted to match the background region grayscale values). After processing, the retained defect region spatial index mask is determined as a valid surface defect mask, and the retained defect region grayscale distribution map is determined as a valid surface defect grayscale map.

[0061] Step 3012: Based on the pixel area and maximum linear span of each defect connected region in the effective surface defect mask, and combined with the reference value at the corresponding position in the local structure reference scale, determine the ratio of the defect pixel area to the square of the reference value, and the ratio of the maximum linear span of the defect to the reference value, to obtain the defect geometric parameter set.

[0062] Optionally, the fault diagnosis device extracts connected components of defects based on an effective surface defect mask. For example, it uses an eight-neighborhood connectivity labeling algorithm to analyze the connectivity of pixels in the effective surface defect mask, dividing interconnected pixels into a defect connected component. Then, it calculates the pixel area and maximum linear span of each defect connected component. The pixel area refers to the total number of pixels contained in each defect connected component, which can be determined by counting the number of pixels in each defect connected component. The maximum linear span refers to the maximum straight-line distance between any two pixels in each defect connected component, which can be calculated by calculating the Euclidean distance between all pixels in each defect connected component and selecting the maximum distance as the maximum linear span. Next, it retrieves the reference value at the corresponding position in the local structure reference scale, calculates the ratio of the defect pixel area to the square of the reference value, and the ratio of the maximum linear span of the defect to the reference value. These two ratios are then integrated across all defect connected regions to form a set of defect geometric parameters.

[0063] Step 3013: Based on the numerical value of the local structural reference scale, the size of the structural element is determined, and morphological opening operation is performed on the region defined by the effective surface defect mask in the effective surface defect grayscale image to reconstruct the local background estimation grayscale image. Based on the pixel-by-pixel difference between the original grayscale value in the effective surface defect grayscale image and the local background estimation grayscale image, a morphological grayscale residual image is obtained.

[0064] Optionally, the fault diagnosis device uses a local structural reference scale to determine the size of the structural elements required for morphological opening reconstruction, such as structural element size = local structural reference scale value ÷ 5 (rounded up). After determination, morphological opening reconstruction (i.e., first erosion and then dilation) is performed on the region defined by the effective surface defect mask in the effective surface defect grayscale image. The specific process is as follows: Erosion processing: Centered on each pixel, select a neighborhood window defined by the structuring element, calculate the minimum gray value of all pixels within the neighborhood window, and use the minimum value as the gray value of the current pixel in the eroded image; Dilation processing: Taking each pixel in the eroded image as the center, select a neighborhood window with the same structuring element, calculate the maximum gray value of all pixels in the neighborhood window, and use the maximum value as the gray value of the current pixel in the local background estimated gray map; The final result is a local background estimation grayscale image, which reflects the grayscale distribution of the background within a defined area. After generating the local background estimation grayscale image, the fault diagnosis device calculates the pixel-by-pixel difference between the original grayscale value in the effective surface defect grayscale image and the local background estimation grayscale image. The difference at each pixel is used as the grayscale value of that pixel, generating a morphological grayscale residual image (taking the absolute value of the difference between the original grayscale value and the estimated background value). The larger the residual in this morphological grayscale residual image, the more significant the grayscale difference between the area corresponding to that pixel and the background, and the more likely it is to be a real surface defect area.

[0065] Step 3014: Based on the mean and standard deviation in the morphological grayscale residual map, determine the surface micromorphology depth index, and construct a two-dimensional diagnostic mapping table based on the depth level classification of the surface micromorphology depth index and the relative size level classification of the defect geometric parameter set.

[0066] Optionally, the fault diagnosis device calculates the mean and standard deviation of the effective region (i.e., the region defined by the effective surface defect mask) in the morphological grayscale residual image. The mean is calculated by summing the grayscale residual values ​​of all pixels within the effective region and dividing by the total number of pixels in the effective region. The standard deviation is calculated as follows: first, the difference between the grayscale residual value of each pixel within the effective region and the mean is calculated; then, each difference is squared; the sum of all squared differences is calculated; this sum is divided by the total number of pixels in the effective region to obtain the average of the squared differences; finally, the square root of this average is taken to obtain the standard deviation. After obtaining the mean and standard deviation, the mean and standard deviation are added together and then divided by 2 to obtain the surface micro-morphology depth index. This index accurately reflects the micro-depth characteristics of surface defects; the larger the index value, the deeper the micro-depth of the surface defect and the more severe the defect.

[0067] Optionally, the fault diagnosis device classifies the depth of surface defects based on the obtained surface micro-morphology depth index, dividing it into different levels to quantify the depth of surface defects. The classification criteria are determined based on the equipment surface material and maintenance standards. For example, the classification criteria are: a depth index less than or equal to 50 is classified as a light depth level; a depth index greater than 50 and less than or equal to 100 is classified as a medium depth level; and a depth index greater than 100 is classified as a heavy depth level. Simultaneously, a relative size level is classified based on the defect geometric parameter set. This is based on the ratio of defect pixel area and the ratio of the maximum linear span of the defect in the defect geometric parameter set, comprehensively classifying the relative size level of the defect. The classification criteria are: both ratios are less than or equal to 0.5, classified as a small size level; both ratios are greater than 0.5 and less than or equal to 1.0, classified as a medium size level; and at least one of the two ratios is greater than 1.0, classified as a large size level. After the division is completed, a mapping table is constructed with the surface micromorphology depth level as the vertical dimension and the relative size level of the defect as the horizontal dimension to establish a preliminary correspondence between the level combination and the defect category, thus obtaining the two-dimensional diagnostic mapping table.

[0068] Step 3015: Based on the two-dimensional logical diagnosis mapping table and the preset two-dimensional logical mapping rules for defects, a combined mapping is performed to obtain the target defect category. The target defect category is then integrated with the geometric centroid coordinates calculated by the effective surface defect mask to generate a surface defect diagnosis result that includes the defect nature, severity level, and spatial location.

[0069] Optionally, the preset two-dimensional logical mapping rules for defects refer to the rules pre-stored in the fault diagnosis device, used to standardize the combination mapping process of the two-dimensional diagnostic mapping table, and clearly define the target defect category and severity level corresponding to the combination of depth level and size level. Based on the two-dimensional diagnostic mapping table and the depth level and relative size level of each defect connected region, the fault diagnosis device calls the preset two-dimensional logical mapping rules for combination mapping. For each defect connected region, its depth level and relative size level are substituted into the two-dimensional diagnostic mapping table to determine the target defect category and severity level. Then, the geometric centroid coordinates of each defect connected region are calculated, including horizontal and vertical coordinates, where the horizontal coordinate = the average of the horizontal coordinates of all pixels; the vertical coordinate = the average of the vertical coordinates of all pixels. The target defect category (defect nature), severity level, and geometric centroid coordinates (spatial location) are integrated to generate the surface defect diagnosis result.

[0070] The embodiments of the present invention fully leverage the high-resolution structural texture advantage of visible light branch information in target fusion features, accurately capture the geometric and grayscale features of surface defects, effectively solve the problems of large noise interference, inaccurate defect judgment, and ambiguous spatial positioning in single surface detection methods, and ensure the accuracy and reliability of surface defect diagnosis results.

[0071] Optionally, the process of steps 3021 to 3026 includes: Step 3021: Based on the fusion semantic classification label in the infrared branch information, perform logical judgment. When the label is internal overheating or severe composite fault, retain the corresponding thermal anomaly area spatial index mask and temperature difference distribution map. When the label is identified as other non-thermal anomaly labels, set the corresponding thermal anomaly area spatial index mask to an empty set and set the corresponding temperature difference distribution map to a background suppression state to obtain the effective thermal anomaly mask and effective thermal anomaly intensity map.

[0072] Optionally, the fault diagnosis device retrieves infrared branch information from the target fusion features. This infrared branch information includes fusion semantic classification labels, a spatial index mask for thermal anomaly regions, and a temperature difference distribution map. Logical judgment is performed based on the fusion semantic classification labels in the infrared branch information. When the label indicates internal overheating or a severe composite fault, the corresponding spatial index mask for thermal anomaly regions and the temperature difference distribution map are directly retained without any modification. When the label indicates other non-thermal anomaly labels, the corresponding spatial index mask for thermal anomaly regions is set to an empty set (a set containing no pixel coordinate indices), and the corresponding temperature difference distribution map is set to a background suppression state (the grayscale values ​​of all pixels in the temperature difference distribution map are adjusted to match the grayscale values ​​of the background region, corresponding to a temperature difference of 0). After completing the logical judgment and corresponding processing, the retained spatial index mask for thermal anomaly regions is determined as the effective thermal anomaly mask, and the retained temperature difference distribution map is determined as the effective thermal anomaly intensity map.

[0073] Step 3022: Based on the pixel area and maximum linear span of each thermal anomaly connected region in the effective thermal anomaly mask, and combined with the reference value at the corresponding position in the local structural reference scale, determine the ratio of the thermal anomaly pixel area to the square of the reference value, and the ratio of the maximum linear span of the thermal anomaly to the reference value, to obtain the thermal diffusion geometric parameter set.

[0074] Optionally, the fault diagnosis device, based on an effective thermal anomaly mask, extracts thermal anomaly connected components from the mask. Using the same eight-neighbor connectivity labeling algorithm, it analyzes the connectivity of pixels in the mask, dividing interconnected pixels into thermal anomaly connected components. Then, it calculates the pixel area and maximum linear span of each thermal anomaly connected component. The calculation method is the same as that for the defect connected component in step 3012. Next, it retrieves a local structural reference scale. For each thermal anomaly connected component, based on the reference value at the corresponding position in the local structural reference scale, it calculates the ratio of the thermal anomaly pixel area to the square of the reference value, and the ratio of the maximum linear span of the thermal anomaly to the reference value. The two ratios corresponding to each thermal anomaly connected component are integrated to form the thermal diffusion geometric parameters of that component. The thermal diffusion geometric parameters of all thermal anomaly connected components form a thermal diffusion geometric parameter set.

[0075] Step 3023: Based on the region defined by the effective thermal anomaly mask in the effective thermal anomaly intensity map, determine the peak pixel of the maximum normalized thermal radiation intensity value, and calculate the normalized eccentricity of the peak pixel relative to the geometric centroid of the thermal anomaly connected domain.

[0076] Optionally, the fault diagnosis device, based on the effective thermal anomaly intensity map and the effective thermal anomaly mask, determines the area defined by the effective thermal anomaly mask. According to the grayscale value of each pixel in the effective thermal anomaly intensity map corresponding to the temperature difference of each pixel in the thermal anomaly area, the device performs thermal radiation intensity normalization processing on all pixels within the area defined by the effective thermal anomaly mask to obtain a normalized thermal radiation intensity value for each pixel. The processing procedure is as follows: find the maximum and minimum temperature difference values ​​within the area defined by the effective thermal anomaly intensity map, calculate the difference between the maximum and minimum temperature difference values ​​(range). If the range is not 0, subtract the minimum temperature difference value from the temperature difference value of each pixel, and then divide by the range to obtain the normalized thermal radiation intensity value of that pixel; if the range is 0, assign a value of 0.5 to the normalized thermal radiation intensity value of all pixels in that area. Next, the peak pixel with the maximum normalized thermal radiation intensity value is determined. This peak pixel is the pixel with the largest normalized thermal radiation intensity value within the defined area of ​​the effective thermal anomaly intensity map. If there are multiple pixels with the same normalized thermal radiation intensity value and all of them are the maximum value, then the pixel whose coordinates are closest to the center of the thermal anomaly connected domain is selected.

[0077] Subsequently, the fault diagnosis device calculates the geometric centroid coordinates of each thermal anomaly connected domain. First, it statistically analyzes the horizontal and vertical coordinates of all pixels within each thermal anomaly connected domain. The average of all horizontal coordinates is used as the horizontal coordinate of the geometric centroid, and the average of all vertical coordinates is used as the vertical coordinate. Then, based on the coordinates of the peak pixel and the geometric centroid coordinates of the thermal anomaly connected domain, it calculates the normalized eccentricity of the peak pixel relative to the geometric centroid of the thermal anomaly connected domain. The calculation process is as follows: first, the straight-line distance (Euclidean distance) between the peak pixel and the geometric centroid is calculated; then, the maximum linear span of the thermal anomaly connected domain is calculated; finally, the straight-line distance between the peak pixel and the geometric centroid is divided by the maximum linear span of the thermal anomaly connected domain to obtain the normalized eccentricity. The normalized eccentricity ranges from 0 to 1. A value of 0 indicates that the peak pixel coincides with the geometric centroid, and the closer the value is to 1, the greater the deviation.

[0078] Step 3024: Calculate the average decay rate of thermal radiation intensity on both sides along the direction of the maximum linear span of the thermal anomaly, and determine the absolute value of the rate difference between the two sides as the thermal gradient asymmetry index.

[0079] Optionally, the fault diagnosis device, based on the maximum linear span of each thermal anomaly connected domain, as well as the coordinates of the peak pixel and the normalized thermal radiation intensity value, first determines the direction of the maximum linear span of each thermal anomaly connected domain. This direction is the direction connecting the two pixels at the two ends of the maximum straight-line distance within the thermal anomaly connected domain. It determines the pixel coordinates at both ends of the maximum linear span, connects the two points to form a straight line, and the extension direction of this straight line is the direction of the maximum linear span. Subsequently, using the peak pixel as the dividing point, the direction of the maximum linear span is divided into two sides: the first side refers to the side where the peak pixel extends towards one end of the maximum linear span; the second side refers to the side where the peak pixel extends towards the other end.

[0080] Next, calculate the average decay rate of thermal radiation intensity on both sides. The calculation process for the average decay rate on the first side is as follows: along the first side of the maximum linear span, starting from the peak pixel, select all pixels in that direction sequentially, record the straight-line distance between each pixel and the peak pixel, and the normalized thermal radiation intensity value of each pixel; calculate the difference in normalized thermal radiation intensity between each pixel and the peak pixel (peak pixel intensity value minus the pixel's intensity value), and then divide it by the straight-line distance between the pixel and the peak pixel to obtain the instantaneous decay rate of that pixel; calculate the average of the instantaneous decay rates of all pixels, which is the average decay rate of thermal radiation intensity on the first side. The calculation process for the average decay rate on the second side is as follows: using the same calculation method as the first side, along the second side of the maximum linear span, starting from the peak pixel, select all pixels in that direction sequentially, calculate the instantaneous decay rate of each pixel, and then take the average of all instantaneous decay rates, which is the average decay rate of thermal radiation intensity on the second side.

[0081] After calculating the average decay rate on both sides, the absolute value of the rate difference between the two sides is calculated, and this absolute value is determined as the thermal gradient asymmetry index. When the index value is 0, it indicates that the decay rates on both sides are the same.

[0082] Step 3025: Based on the thermal diffusion range classification of the thermal diffusion geometric parameter set, and combined with the heat source distribution pattern classification based on the normalized eccentricity and thermal gradient asymmetry index, a two-dimensional thermal anomaly mapping table is constructed.

[0083] Optionally, the fault diagnosis device, based on a set of thermal diffusion geometric parameters, first classifies the thermal diffusion range into levels. Based on the ratio of the thermal anomaly pixel area and the ratio of the maximum linear span of the thermal anomaly in the set of thermal diffusion geometric parameters, it comprehensively classifies the diffusion range level of the thermal anomaly, thereby quantifying the diffusion size and impact range of the thermal anomaly. The classification criteria are determined based on equipment thermal anomaly maintenance standards and equipment component dimensions. For example, the classification criteria might be: when both ratios are less than or equal to 0.5, it is classified as a small-range diffusion level; when both ratios are greater than 0.5 and less than or equal to 1.0, it is classified as a medium-range diffusion level; and when at least one of the two ratios is greater than 1.0, it is classified as a large-range diffusion level.

[0084] Simultaneously, the distribution morphology of heat sources is classified according to the normalized eccentricity and thermal gradient asymmetry index. Specifically, the uniformity of heat source distribution within the thermal anomaly area is comprehensively assessed based on the values ​​of the normalized eccentricity and thermal gradient asymmetry index, thus quantifying the heat source distribution morphology. For example, the classification criteria are as follows: when the normalized eccentricity is less than or equal to 0.3 and the thermal gradient asymmetry index is less than or equal to 0.1, it is classified as a uniform distribution (uniform heat source distribution with no significant shift or asymmetry); when the normalized eccentricity is greater than 0.3 and less than or equal to 0.6, or the thermal gradient asymmetry index is greater than 0.1 and less than or equal to 0.2, it is classified as a moderately uniform distribution (basically uniform heat source distribution with slight shifts or asymmetry); when the normalized eccentricity is greater than 0.6, or the thermal gradient asymmetry index is greater than 0.2, it is classified as an uneven distribution (severely shifted heat source distribution with significant asymmetry).

[0085] After completing the classification of heat diffusion range and heat source distribution pattern, a preliminary correspondence between the level combination and the heat anomaly category is established with the heat diffusion range level as the vertical dimension and the heat source distribution pattern level as the horizontal dimension, thus obtaining a two-dimensional heat anomaly mapping table.

[0086] Step 3026: Based on the two-dimensional thermal anomaly mapping table and the preset two-dimensional logical mapping rules for thermal anomalies, a combined mapping is performed to obtain the target thermal anomaly category. The target thermal anomaly category is then integrated with the geometric centroid coordinates calculated by the effective thermal anomaly mask and the maximum peak intensity in the temperature difference distribution map to generate an anomaly diagnosis result that includes the nature of the thermal anomaly, its severity level, peak temperature, and spatial location.

[0087] Optionally, the preset two-dimensional logical mapping rule for thermal anomalies refers to the rule pre-stored in the fault diagnosis device to standardize the combination mapping process of the two-dimensional thermal anomaly mapping table. This rule clarifies the specific target thermal anomaly category corresponding to the combination of the thermal diffusion range level and the heat source distribution pattern level, as well as the criteria for determining the severity level of the thermal anomaly. Based on the two-dimensional thermal anomaly mapping table and the thermal diffusion range level and heat source distribution pattern level of each thermal anomaly connected domain, the fault diagnosis device substitutes the thermal diffusion range level and heat source distribution pattern level of each thermal anomaly connected domain into the two-dimensional thermal anomaly mapping table, and performs combination queries and mapping in conjunction with the preset two-dimensional logical mapping rule for thermal anomalies to determine the target thermal anomaly category and severity level corresponding to the thermal anomaly connected domain.

[0088] After determining the target thermal anomaly category and severity level, the fault diagnosis device again obtains the geometric centroid coordinates of each thermal anomaly connected domain in the effective thermal anomaly mask, and extracts the maximum peak intensity of each thermal anomaly connected domain (the maximum temperature difference value within the thermal anomaly connected domain) from the effective thermal anomaly intensity map.

[0089] The target thermal anomaly category (thermal anomaly nature), severity level, maximum peak intensity (peak heat), and geometric centroid coordinates (spatial location) corresponding to each thermal anomaly connected domain are integrated to generate anomaly diagnosis results.

[0090] The embodiments of the present invention fully leverage the advantages of the thermal anomaly features of infrared branch information in target fusion features, accurately capture the core features of thermal anomaly diffusion range, thermal intensity, and heat source distribution, and effectively solve the problems of fuzzy thermal anomaly localization, inaccurate intensity quantification, and unclear category determination in single infrared detection methods, ensuring the accuracy and reliability of anomaly diagnosis results.

[0091] Optionally, the processes of steps 401 to 405 include: Step 401: Based on the nature, severity level, and spatial location of the defects in the surface defect diagnosis results, and the nature, severity level, peak heat, and spatial location of the thermal anomalies in the anomaly diagnosis results, respectively construct surface defect instances and thermal anomaly instances.

[0092] Optionally, the fault diagnosis device retrieves the defect nature, severity level, and spatial location from the surface defect diagnosis results. For each surface defect in the surface defect diagnosis results, it extracts its corresponding defect nature, severity level, and spatial location, and associates and encapsulates these three core pieces of information to form a surface defect instance. Each surface defect instance uniquely corresponds to one surface defect. Simultaneously, it retrieves the thermal anomaly nature, severity level, peak temperature, and spatial location from the anomaly diagnosis results. For each thermal anomaly in the anomaly diagnosis results, it extracts its corresponding thermal anomaly nature, severity level, peak temperature, and spatial location, and associates and encapsulates these four core pieces of information to form a thermal anomaly instance. Similarly, each thermal anomaly instance corresponds to one thermal anomaly.

[0093] Step 402: Inject corresponding data attributes based on surface defect instances and thermal anomaly instances to obtain an initial fact set containing all objects to be diagnosed and their complete attribute information.

[0094] Optionally, the fault diagnosis device, based on surface defect instances and thermal anomaly instances, first acquires the data attributes corresponding to each surface defect instance and thermal anomaly instance. These data attributes include instance identifier, associated equipment components, detection time, and detection accuracy. The instance identifier is a unique identifier assigned to each surface defect instance and thermal anomaly instance, and is sequentially assigned using Arabic numerals. The associated equipment components are specific equipment components corresponding to each instance, such as the No. 3 terminal block on the high-voltage side of a transformer, or the internal windings of a transformer. These can be determined based on the spatial coordinates of the instance, combined with a preset range of equipment component coordinates. That is, it determines which equipment component's coordinate range the instance's spatial coordinates fall within; this equipment component is the associated equipment component for that instance. The detection time is the specific time at which the diagnostic result corresponding to the instance is acquired, consistent with the acquisition time of the visible light image sequence and the thermal radiation image sequence. The detection accuracy is the detection accuracy of the diagnostic result corresponding to the instance, consistent with the preset detection accuracy of the fault diagnosis device, such as a positional accuracy of 0.1 pixels or a temperature accuracy of 0.1℃.

[0095] Specifically, for each surface defect instance, four data attributes are injected: instance identifier, associated equipment component, detection time, and detection accuracy. These attributes are then integrated with the instance's original defect nature, severity level, and spatial location. Similarly, for each thermal anomaly instance, these four data attributes are injected and integrated with the instance's original thermal anomaly nature, severity level, peak temperature, and spatial location. After injecting the data attributes for all instances, all complete surface defect instances and thermal anomaly instances are combined to form an initial fact set. The initial fact set contains all objects to be diagnosed (surface defects and thermal anomalies) and their complete attribute information.

[0096] Step 403: Based on the centroid coordinate data attributes of each instance in the initial fact group, map the topology nodes of the equipment components in the pre-built equipment fault diagnosis knowledge graph to obtain the component node set. Then, based on the component node set, calculate the shortest electrical path length between the component associated with the surface defect instance and the component associated with the thermal anomaly instance to obtain the electrical path length value.

[0097] Optionally, the pre-built equipment fault diagnosis knowledge graph refers to a pre-stored knowledge network used to describe the topological relationships, fault types, and association rules of various equipment components. This knowledge graph includes equipment component topological nodes and the electrical connection relationships between nodes. Equipment component topological nodes refer to nodes in the knowledge graph used to represent specific components of the equipment. Each node corresponds to one equipment component, and each node contains the coordinate range information of the component (i.e., the range of horizontal and vertical coordinate values ​​of the component in the detection image) and electrical connection information (i.e., the electrical connection relationship between the component and other equipment components). The electrical connection relationship refers to the actual electrical connection path between various equipment components, including the connection order and path length (in pixels, calculated based on the equipment image resolution), used to calculate the electrical path length between different components.

[0098] The fault diagnosis device extracts the centroid coordinate data attribute of each instance in the initial fact set based on the initial fact set. It then matches the centroid coordinate data attribute of each instance with the coordinate range information of all equipment component topology nodes in the pre-built equipment fault diagnosis knowledge graph to determine which equipment component topology node the centroid coordinate falls within. The equipment component topology node is the mapping node corresponding to the instance. After the mapping is completed, the equipment component topology nodes corresponding to all instances are integrated, and duplicate nodes are removed to obtain the component node set.

[0099] Optionally, the fault diagnosis device pairs the equipment component topology nodes associated with each surface defect instance with the equipment component topology nodes associated with each thermal anomaly instance, based on the component node set, to form node pairing groups; then, for each node pairing group, it extracts all electrical connection paths between the two nodes and the length of each path from the equipment fault diagnosis knowledge graph; and filters out the shortest path among all paths, the length of which is the shortest electrical path length value corresponding to the node pairing group.

[0100] Step 404: Based on the comparison results of the electrical path length value and the preset electrical coupling threshold, construct the electrical space coexistence object attribute between the two instances to obtain the cross-modal association set, and construct the local reasoning subgraph based on the cross-modal association set and the location object attribute of each instance and its component node.

[0101] Optionally, the preset electrical coupling threshold is a pre-defined critical length value used to determine whether there is an electrical coupling relationship between two related components. This threshold is determined based on the electrical structure, component layout, and fault diagnosis experience of the equipment, and the preset electrical coupling threshold varies for different equipment types. For example, the preset electrical coupling threshold for a transformer is 60 pixels.

[0102] The fault diagnosis device, based on the electrical path length value, calls a preset electrical coupling threshold and compares the electrical path length value with the preset electrical coupling threshold. Based on the comparison result, it constructs an electrical spatial coexistence object attribute between the two instances. The construction rule for the electrical spatial coexistence object attribute is as follows: when the electrical path length value is less than or equal to the preset electrical coupling threshold, a tightly coupled electrical spatial coexistence object attribute is constructed; when the electrical path length value is greater than the preset electrical coupling threshold, a loosely coupled electrical spatial coexistence object attribute is constructed. After completing the construction of the electrical spatial coexistence object attribute, the association relationships between all surface defect instances and thermal anomaly instances are integrated to obtain a cross-modal association relationship set. This cross-modal association relationship set contains cross-modal association information between all surface defect instances and thermal anomaly instances, with each association corresponding to a pair of surface defect instances and thermal anomaly instances.

[0103] Optionally, the fault diagnosis device constructs a local inference subgraph based on the cross-modal association set and the location object attribute of each instance and its corresponding component node. The location object attribute describes the positional relationship between an instance and its corresponding equipment component topology node, determined by the node mapping result in step 403. Each instance corresponds to a location object attribute to determine the equipment component topology node to which the instance belongs. Specifically, the construction process of the local inference subgraph includes: using the component node set as the base node, treating each equipment component topology node as a node of the subgraph; treating each association in the cross-modal association set as an edge of the subgraph, with the two ends of the edge connecting the surface defect instance associated component node and the thermal anomaly instance associated component node corresponding to that association, respectively, and the edge attribute being the electrical space coexistence object attribute corresponding to that association; simultaneously, treating each instance as an affiliated node of the subgraph, connecting it to the corresponding equipment component topology node through the location object attribute, forming an association structure of instance-location-component node-association-component node-location-instance, ultimately constructing the local inference subgraph.

[0104] Step 405: Based on the local reasoning subgraph and the preset SWRL rule set, reasoning analysis is performed to obtain a comprehensive diagnostic result.

[0105] Optionally, the preset SWRL rule set refers to a pre-stored set of rules used for cross-modal joint reasoning. This rule set is formulated based on equipment fault diagnosis experience, equipment electrical principles, and bimodal diagnostic characteristics, and covers rules for the association between surface defects and thermal anomalies, fault level determination rules, and comprehensive fault type determination rules. The fault diagnosis device performs reasoning analysis based on the local reasoning subgraph combined with the preset SWRL rule set to obtain a comprehensive diagnostic result, as detailed in steps 4051 to 4054.

[0106] The embodiments of the present invention integrate surface defect information of visible light mode and thermal anomaly information of infrared mode, realizing mutual verification and collaborative judgment of the two diagnostic results. This solves the problems of diagnostic blind spots, missed detections and misjudgments caused by the one-sidedness of single-mode diagnostic information, makes up for the limitations of single-mode diagnosis, and enables the comprehensive diagnostic results to present the equipment fault status in a comprehensive and accurate manner.

[0107] Optionally, the processes of steps 4051 to 4054 include: Step 4051: Based on the local reasoning subgraph and the preset SWRL rule set, perform forward chain reasoning to obtain a mechanism inference type fault set containing fault mechanism attributes, and determine the implicit causal relationship in the local reasoning subgraph based on the mechanism inference type fault set to form an enhanced local subgraph.

[0108] Optionally, the fault diagnosis device performs forward chain reasoning based on the local reasoning subgraph and the SWRL rule set. That is, starting from the node information and edge attribute information in the local reasoning subgraph, it deduces step-by-step according to the preset SWRL rule set, following the logic of fact → rule matching → new fact generation, until no new facts can be derived. This results in a mechanism-inference type fault set containing fault mechanism attributes. The fault mechanism attributes are properties used to describe the inherent physical and electrical mechanisms of the fault occurrence, i.e., the root cause and development law of the fault. For example, external damage leading to increased contact resistance, which in turn causes increased contact resistance in local overheating, is a fault mechanism attribute. This attribute is pre-stored in the preset SWRL rule set and corresponds one-to-one with different fault types.

[0109] The specific reasoning process includes: taking the known information (instance attributes, component node associations, and electrical space coexistence object attributes) in the local reasoning subgraph as initial facts, matching it with the preset SWRL rule set, extracting the fault mechanism attributes contained in the rule for successfully matched rules, associating them with the corresponding fault instances and fault types, and integrating them to form a mechanism inference type fault set containing fault types, fault mechanism attributes, and potential association clues.

[0110] Optionally, the fault diagnosis device extracts the mechanistic attributes of each fault from the mechanistic inference fault set, analyzes the logical relationships between the mechanistic attributes of different faults (surface defect instances and thermal anomaly instances), and determines whether the mechanistic attribute of one fault leads to the occurrence of another fault. If a clear logical relationship exists (e.g., the mechanistic attribute of surface defects is "poor contact," and the mechanistic attribute of thermal anomalies is "local overheating," then "poor contact" will lead to "local overheating"), then an implicit causal relationship is determined between the two, and the causal direction is determined (surface defect is the cause, and thermal anomaly is the effect, or vice versa). Finally, the fault mechanism attributes in the mechanistic inference fault set are injected into the corresponding fault instances and component nodes, and the determined implicit causal relationships are added as new edges to the local inference subgraph, forming an enhanced local subgraph containing the original nodes, original relationships, newly added fault mechanism attributes, and implicit causal relationships.

[0111] Step 4052: Perform a logical consistency check on the description logic inference engine based on the enhanced local subgraph and the knowledge graph. When a logical contradiction is detected, resolve the conflict in the enhanced local subgraph based on the preset evidence priority adjudication rules to obtain a logically consistent modified subgraph.

[0112] Optionally, the descriptive logic inference engine is a module in the knowledge graph used to check knowledge consistency and detect logical contradictions. Based on the descriptive logic rules of the knowledge graph, it can check the node information, attribute information, and relationships (including implicit causal relationships) in the enhanced local subgraph to determine whether there is any contradictory or conflicting information. Specifically, the fault diagnosis device performs a logical consistency check based on the enhanced local subgraph and the descriptive logic inference engine of the knowledge graph. That is, all information (nodes, instances, attributes, and relationships) in the enhanced local subgraph is input into the descriptive logic inference engine. The inference engine verifies the rationality of each piece of information and the logical relationships between different pieces of information based on the descriptive logic rules in the knowledge graph. If no logical contradictions are detected, the enhanced local subgraph is directly used as a logically consistent corrected subgraph. Among them, logical contradictions include attribute contradictions, causal contradictions, and contradictions between attributes and relationships. Attribute contradictions refer to the simultaneous existence of mutually opposing attributes in the same instance or node; causal contradictions refer to the conflict between implicit causal relationships and fault mechanism rules in the knowledge graph; and contradictions between attributes and relationships refer to the conflict between the attributes of an instance and the attributes of associated component nodes.

[0113] When a logical contradiction is detected, the contradiction type, the instances and attributes involved are recorded. Based on a preset evidence priority adjudication rule, the enhanced local subgraph is conflict-resolved. This evidence priority adjudication rule refers to the handling of logically mutually exclusive cases in the physical state descriptions of visible light and infrared modal diagnostic results when knowledge graph reasoning detects logical contradictions (i.e., conflicting diagnostic results from different modalities). A hierarchical, progressive judgment logic is employed, specifically including: The first step is to determine whether the conflicting parties constitute absolute mutual exclusion in physical state (such as "structural integrity" and "structural fracture"). If they do not constitute mutual exclusion, and the conflicting content is a coexisting state (such as "surface corrosion" and "local temperature rise"), then it is determined to be bimodal complementary information. No conflict resolution is performed, and the two facts are directly retained and "accompaniment relationship" or "causal relationship edge" is added to the subgraph to continue subsequent reasoning. If they constitute mutual exclusion (such as "structural integrity" and "structural fracture", "no thermal anomaly" and "severe overheating"), then it is determined to be a logical contradiction that must be resolved, and the second step, confidence determination, is initiated.

[0114] The second step is to calculate the comprehensive confidence score for the visible light mode facts and infrared mode facts corresponding to the mutual exclusion conflicts. The score is obtained by weighted summation of three factors: Mechanism fit weight: The basic weight is assigned based on the mapping relationship between fault type and modal sensitivity (e.g., internal faults have high infrared weight, and surface faults have high visible light weight). Quantification of significance weight: The weight is assigned based on the prominence of the modal feature value relative to the background, including the texture entropy value and geometric size significance of visible light defects, as well as the temperature rise amplitude and thermal gradient intensity of infrared thermal anomalies. The higher the value exceeds the threshold, the higher the weight. Environmental robustness weights: Dynamically adjust the interference levels of different modes based on the current inspection environment. Reduce the visible light weight when there is strong light or shadow occlusion, and reduce the infrared weight when there is high temperature background, strong airflow, or strong reflection.

[0115] The third step involves comparing the overall confidence scores of the two conflicting facts. The diagnosis result of the high-confidence modality is established as the final principal fact and written into the core node attribute of the logically consistent modified subgraph. The diagnosis result of the low-confidence modality is marked as "questionable observation" or "minor reference," and attached to the same node as an auxiliary annotation attribute, without participating in subsequent main logical reasoning. Through the above hierarchical judgment and factual adjudication, all logical contradictions are eliminated, ultimately resulting in a logically consistent modified subgraph with unique facts, consistent attributes, and self-consistent relationships.

[0116] Step 4053: Based on the logically consistent modified subgraph, starting from the fault individual, traverse upwards along the attributes of the objects to which it belongs to to obtain the system-level path associated with the fault. Based on the system-level path, the mechanism type of the fault individual, and the system impact value determined by the preset risk propagation weight table, combined with the fault individual's own severity level attribute, determine the target severity level.

[0117] Optionally, the fault diagnosis device, based on a logically consistent modified subgraph, first identifies the individual faults, i.e., all surface defect instances and thermal anomaly instances in the graph. Each individual fault corresponds to an independent fault and includes its own severity level attribute (mild, moderate, severe) and mechanism type attribute (e.g., increased contact resistance, decreased insulation performance, etc.). Then, starting with each individual fault, the device's component topology node is found based on the "located in object" attribute. Next, the parent component node of that component topology node is found based on the "belongs to object" attribute. This process is repeated upwards until a system-level node is reached. The fault individuals, component topology nodes, parent component nodes, and system-level nodes traversed during the traversal are arranged sequentially to form the system-level path corresponding to that individual fault. The "located in object" attribute describes the positional relationship between the fault individual and its component topology node; the "belongs to object" attribute describes the hierarchical relationship between the component topology node and its parent components and system-level nodes. A system-level node refers to the highest-level node in the equipment fault diagnosis knowledge graph, with each device corresponding to a unique system-level node.

[0118] Optionally, the preset risk propagation weight table is a weight table used to quantify the impact of a fault on the system. It includes the weight values ​​of nodes at different levels in the system hierarchy path and the weight values ​​of different fault mechanism types. The weight values ​​range from 0 to 1. The larger the weight value, the greater the impact of the fault on the system and the stronger the risk propagation capability.

[0119] The fault diagnosis device determines the system impact value based on the system hierarchical path, the mechanism type of the fault individual, and a preset risk propagation weight table. The process is as follows: extract the system hierarchical path corresponding to the fault individual, determine the weight value of each node in the path, calculate the average value of the weight values ​​of all nodes, extract the mechanism type of the fault individual, determine its corresponding weight value, and multiply the average node weight value by the mechanism type weight value to obtain the system impact value of the fault individual. The system impact value ranges from 0 to 1. The larger the value, the greater the impact of the fault on the system. Next, considering both the individual severity level of the fault and the system impact value, the system impact value is first determined based on three preset threshold values: mild (system impact value 0 to 0.3), moderate (system impact value 0.3 to 0.7), and severe (system impact value 0.7 to 1.0). This system impact value is then compared with the individual severity level of the fault. If the system impact value corresponds to a higher level than the individual severity level, the target severity level is the level corresponding to the system impact value. If the system impact value corresponds to a lower level than or equal to the individual severity level, the target severity level is the individual severity level of the fault. This determines the final fault severity level.

[0120] Step 4054: Extract and integrate based on the logically consistent modified subgraph, target severity level and preset SPARQL query statement to obtain a comprehensive diagnostic result including fault nature, severity level, quantitative indicators and source tracing evidence.

[0121] Optionally, the SPARQL query statement is a pre-written query statement used to extract specific fault information from the logically consistent correction subgraph, and to perform precise queries for the various information required for the comprehensive diagnostic results. The fault diagnosis device, based on a logically consistent modified subgraph, target severity level, and preset SPARQL query statements, inputs the preset SPARQL query statements into the logically consistent modified subgraph and queries each individual fault for its fault nature, spatial coordinates, peak heat (thermal anomaly instance), fault mechanism attributes, implicit causal relationships, and system hierarchical path. It then extracts the target severity level, system impact value, and electrical path length value obtained in step 403 for each individual fault. All extracted information is then integrated for each individual fault to ensure a one-to-one correspondence between all information for each fault. This integration forms a complete comprehensive diagnostic result, which includes fault nature, severity level, quantitative indicators, and source tracing evidence. The integration of source tracing evidence must include key information about the fault diagnosis process, namely, the key basis for "fault detection → instance construction → attribute injection → node mapping → association reasoning → mechanism derivation → conflict resolution → level determination," to ensure that each fault determination is supported by clear evidence.

[0122] The embodiments of the present invention further deepen the integration of surface defects and thermal anomaly information, and solve problems such as one-sided diagnostic information of single mode, unclear causes of faults, inaccurate severity level determination, and conflicting results of different modes, thereby improving the accuracy, reliability and practicality of comprehensive diagnostic results.

[0123] Furthermore, the device fault diagnosis device based on visible light and infrared thermal imaging fusion provided by the present invention will be described below. The device fault diagnosis device based on visible light and infrared thermal imaging fusion described below and the device fault diagnosis method based on visible light and infrared thermal imaging fusion described above can be referred to in correspondence with each other.

[0124] Optional, refer to Figure 2 , Figure 2 This is a schematic diagram of the device fault diagnosis device based on visible light and infrared thermal imaging fusion provided by the present invention. The device fault diagnosis device based on visible light and infrared thermal imaging fusion includes: The spatiotemporal synchronization module 210 is used to perform spatiotemporal alignment of visible light image sequences and thermal radiation image sequences acquired under the same field of view to obtain aligned dual-modal image pairs. The feature fusion module 220 is used to extract and fuse features based on dual-modal image pairs to obtain target fusion features after feature enhancement and complementation; The anomaly analysis and extraction module 230 is used to perform surface state analysis based on visible light branch information in the target fusion features to obtain surface defect diagnosis results, and to extract infrared thermal anomalies based on infrared branch information in the target fusion features to obtain anomaly diagnosis results. The diagnostic reasoning and generation module 240 is used to perform cross-modal joint reasoning based on surface defect diagnosis results and anomaly diagnosis results to obtain comprehensive diagnostic results, and generate a structured diagnostic report based on the comprehensive diagnostic results.

[0125] The embodiments of the present invention effectively solve the problems of diagnostic blind spots, missed detections and misjudgments caused by the limited information in existing methods. Ultimately, through the coordinated linkage of each step, the accuracy and reliability of equipment fault diagnosis are comprehensively improved. It can make accurate judgments on complex or potential comprehensive faults, and provide sufficient and reliable technical basis for equipment operation and maintenance decisions.

[0126] Please see Figure 3 , Figure 3 An embodiment diagram of an electronic device provided in accordance with the present invention. For example... Figure 3 As shown, an electronic device 300 provided in this embodiment of the invention includes a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor 320. When the processor 320 executes the computer program 311, it implements the following steps 10 to 40.

[0127] Please see Figure 4 , Figure 4 An embodiment diagram of a computer-readable storage medium provided in accordance with an embodiment of the present invention is shown. Figure 4 As shown, this embodiment provides a computer-readable storage medium 400, on which a computer program 311 is stored. When the computer program 311 is executed by a processor, it performs the following steps 10 to 40.

[0128] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the device fault diagnosis method based on visible light and infrared thermal imaging fusion provided by the above methods, which includes steps 10 to 40.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A device fault diagnosis method based on the fusion of visible light and infrared thermal imaging, applied to an intelligent inspection system for unmanned aerial vehicles (UAVs), characterized in that, include: Spatiotemporal alignment of visible light image sequences and thermal radiation image sequences acquired under the same field of view yields aligned dual-modal image pairs. Based on the dual-modal image pairs, feature extraction and fusion are performed to obtain the target fusion features after feature enhancement and complementation; Surface state analysis is performed based on the visible light branch information in the target fusion features to obtain surface defect diagnosis results, and infrared thermal anomaly extraction is performed based on the infrared branch information in the target fusion features to obtain anomaly diagnosis results. Based on the surface defect diagnosis results and the anomaly diagnosis results, cross-modal joint reasoning is performed to obtain a comprehensive diagnosis result, and a structured diagnosis report is generated based on the comprehensive diagnosis result.

2. The equipment fault diagnosis method based on the fusion of visible light and infrared thermal imaging according to claim 1, characterized in that, The feature extraction and fusion are performed based on the dual-modal image pairs. The target fusion features obtained after feature enhancement and complementation include: Gradient magnitude calculation and binarization are performed on the visible light image sequence in the dual-modal image to obtain a binarized edge mask. Morphological thinning is then performed on the binarized edge mask to obtain a visible light structural skeleton line with a single pixel width. Based on the thermal radiation image sequence in the dual-modal image and a preset high temperature threshold, a high temperature pixel set is obtained, and an eight-neighbor connectivity label is performed on the high temperature pixel set to obtain a thermal anomaly connectivity region mask. Based on the Euclidean distance transformation map of the visible light structural skeleton line, the spatial distance from each pixel in the thermal anomaly connected region mask to the nearest structural skeleton line is calculated to obtain a thermal spatial distance map. Based on the thermal spatial distance map, isolated hot spots with a distance exceeding a preset component width threshold are removed to obtain a geometrically corrected effective thermal area mask. Using the effective hot area mask as the defined region of the visible light image sequence, a local texture complexity distribution map is determined based on the local Shannon entropy value calculated within the defined region, and a visible light surface defect sensitive mask is generated based on the high entropy region in the local texture complexity distribution map. The target fusion feature is obtained by mutually decoupling and fusing the visible light surface defect sensitive mask and the effective hot zone mask.

3. The equipment fault diagnosis method based on the fusion of visible light and infrared thermal imaging according to claim 2, characterized in that, The method of obtaining the target fusion feature by mutually exclusive decoupling and fusion of the visible light surface defect sensitive mask and the effective hot zone mask includes: Logical operations are performed on the visible light surface defect sensitive mask and the effective thermal area mask to obtain the thermal-optical overlap region mask, the pure thermal anomaly region mask, and the pure surface defect region mask, respectively. Based on the thermo-optical overlap region mask, the pure thermal anomaly region mask, and the pure surface defect region mask, spatial mutual exclusion partitioning is performed to obtain three feature subspaces, and the corresponding dual-modal original pixel intensity sequence is taken for each feature subspace. The extreme value normalization process is performed on the original pixel intensity sequence of the dual modality in each feature subspace to obtain the normalized modal data sequence. Zero values ​​are padded based on the dimensional difference of the modal feature sequence to obtain a dimension-aligned fixed-length normalized data array. Labels are assigned to each feature subspace based on a preset logical gating mapping table to obtain corresponding label features. The label features are then serially concatenated with the fixed-length standardized data array to obtain the target fusion features.

4. The equipment fault diagnosis method based on the fusion of visible light and infrared thermal imaging according to claim 1, characterized in that, The visible light branch information includes a spatial index mask of the defect region, a grayscale distribution map of the defect region, a local structural reference scale benchmark, and a fused semantic classification label. The surface state analysis based on the visible light branching information in the target fusion features to obtain surface defect diagnosis results includes: Logical judgment is made based on the fusion semantic classification labels in the visible light branch information. When the label is external damage or severe composite fault, the corresponding defect area spatial index mask and defect area grayscale distribution map are retained. When the label is internal overheating or other non-surface defect labels, the corresponding defect area spatial index mask is set to an empty set and the corresponding defect area grayscale distribution map is set to background suppression state to obtain an effective surface defect mask and an effective surface defect grayscale map. Based on the pixel area and maximum linear span of each defect connected region in the effective surface defect mask, and combined with the reference value at the corresponding position in the local structure reference scale, the ratio of the defect pixel area to the square of the reference value and the ratio of the maximum linear span of the defect to the reference value are determined to obtain the defect geometric parameter set. Based on the numerical value of the local structural reference scale, the size of the structural element is determined, and morphological opening operation is performed on the region defined by the effective surface defect mask in the effective surface defect grayscale image to reconstruct the local background estimation grayscale image. Based on the pixel-by-pixel difference between the original grayscale value in the effective surface defect grayscale image and the local background estimation grayscale image, a morphological grayscale residual image is obtained. Based on the mean and standard deviation in the morphological grayscale residual map, the surface micromorphology depth index is determined, and a two-dimensional diagnostic mapping table is constructed based on the depth level classification of the surface micromorphology depth index and the relative size level classification of the defect geometric parameter set. Based on the two-dimensional logical diagnostic mapping table and the preset two-dimensional logical mapping rules for defects, a combined mapping is performed to obtain the target defect category. The target defect category is then integrated with the geometric centroid coordinates calculated by the effective surface defect mask to generate the surface defect diagnostic result that includes the defect nature, severity level, and spatial location.

5. The equipment fault diagnosis method based on the fusion of visible light and infrared thermal imaging according to claim 1, characterized in that, The infrared branch information includes a spatial index mask of thermal anomaly regions, a temperature difference distribution map, a local structural reference scale benchmark, and fused semantic classification labels; Infrared thermal anomaly extraction is performed based on the infrared branch information in the target fusion features to obtain anomaly diagnosis results, including... Logical judgment is made based on the fusion semantic classification labels in the infrared branch information. When the label is internal overheating or severe composite fault, the corresponding thermal anomaly area spatial index mask and temperature difference distribution map are retained. When the label is identified as other non-thermal anomaly labels, the corresponding thermal anomaly area spatial index mask is set to an empty set and the corresponding temperature difference distribution map is set to background suppression state to obtain the effective thermal anomaly mask and effective thermal anomaly intensity map. Based on the pixel area and maximum linear span of each thermal anomaly connected region in the effective thermal anomaly mask, and combined with the reference value at the corresponding position in the local structure reference scale, the ratio of the thermal anomaly pixel area to the square of the reference value, and the ratio of the maximum linear span of the thermal anomaly to the reference value are determined to obtain the thermal diffusion geometric parameter set. Based on the region defined by the effective thermal anomaly mask in the effective thermal anomaly intensity map, the peak pixel of the maximum normalized thermal radiation intensity value is determined, and the normalized eccentricity of the peak pixel relative to the geometric centroid of the thermal anomaly connected domain is calculated. The average decay rate of thermal radiation intensity on both sides is calculated along the direction of the maximum linear span of the thermal anomaly, and the absolute value of the difference between the two sides is determined as the thermal gradient asymmetry index. Based on the thermal diffusion range classification of the thermal diffusion geometric parameter set, and combined with the heat source distribution pattern classification based on the normalized eccentricity and thermal gradient asymmetry index, a two-dimensional thermal anomaly mapping table is constructed. Based on the two-dimensional thermal anomaly mapping table and the preset two-dimensional logical mapping rules for thermal anomalies, a target thermal anomaly category is obtained. The target thermal anomaly category is then integrated with the geometric centroid coordinates calculated by the effective thermal anomaly mask and the maximum peak intensity in the temperature difference distribution map to generate the anomaly diagnosis result, which includes the nature of the thermal anomaly, severity level, peak temperature, and spatial location.

6. The equipment fault diagnosis method based on the fusion of visible light and infrared thermal imaging according to claim 1, characterized in that, The surface defect diagnosis results include the defect nature, severity level, and spatial location; the anomaly diagnosis results include the thermal anomaly nature, severity level, peak temperature, and spatial location. The comprehensive diagnostic result obtained by performing cross-modal joint reasoning based on the surface defect diagnosis result and the anomaly diagnosis result includes: Based on the defect nature, severity level, and spatial location in the surface defect diagnosis results, and the thermal anomaly nature, severity level, peak heat, and spatial location in the anomaly diagnosis results, surface defect instances and thermal anomaly instances are constructed respectively. Based on the surface defect instance and the thermal anomaly instance, corresponding data attributes are injected to obtain an initial fact set containing all objects to be diagnosed and their complete attribute information. Based on the centroid coordinate data attributes of each instance in the initial fact set, the topological nodes of the equipment components are mapped in the pre-constructed equipment fault diagnosis knowledge graph to obtain a set of component nodes. Based on the set of component nodes, the shortest electrical path length between the component associated with the surface defect instance and the component associated with the thermal anomaly instance is calculated to obtain the electrical path length value. Based on the comparison result between the electrical path length value and the preset electrical coupling threshold, the electrical space coexistence object attribute between the two instances is constructed to obtain the cross-modal association relationship set. Based on the cross-modal association relationship set and the location object attribute of each instance and its component node, a local reasoning subgraph is constructed. The comprehensive diagnostic result is obtained by performing reasoning analysis based on the local reasoning subgraph combined with the preset SWRL rule set.

7. The equipment fault diagnosis method based on the fusion of visible light and infrared thermal imaging according to claim 6, characterized in that, The comprehensive diagnostic result is obtained by performing reasoning analysis based on the local reasoning subgraph combined with a preset SWRL rule set, including: Based on the local reasoning subgraph and the preset SWRL rule set, forward chain reasoning is performed to obtain a mechanism inference type fault set containing fault mechanism attributes, and the implicit causal relationship in the local reasoning subgraph is determined based on the mechanism inference type fault set. Logical consistency checks are performed on the enhanced local subgraph and the descriptive logic inference engine of the knowledge graph. When a logical contradiction is detected, the enhanced local subgraph is conflict-resolved based on the preset evidence priority adjudication rules to obtain a logically consistent modified subgraph. Based on the logically consistent modified subgraph, starting from the fault individual, the system-level path associated with the fault is obtained by traversing upwards along the attributes of the objects to which the fault belongs to the system-level nodes. Based on the system-level path, the mechanism type of the fault individual, and the system impact value determined by the preset risk propagation weight table, the target severity level is determined in combination with the fault individual's own severity level attribute. Based on the logically consistent modified subgraph, the target severity level, and the preset SPARQL query statement, the comprehensive diagnostic result containing the fault nature, severity level, quantitative indicators, and source tracing evidence is obtained.

8. A device for diagnosing equipment faults based on the fusion of visible light and infrared thermal imaging, characterized in that, The device fault diagnosis method based on visible light and infrared thermal imaging fusion as described in any one of claims 1 to 7; the device fault diagnosis device based on visible light and infrared thermal imaging fusion includes: The spatiotemporal synchronization module is used to perform spatiotemporal alignment of visible light image sequences and thermal radiation image sequences acquired under the same field of view to obtain aligned dual-modal image pairs. The feature fusion module is used to extract and fuse features based on the dual-modal image pair to obtain the target fusion features after feature enhancement and complementation; The anomaly analysis and extraction module is used to perform surface state analysis based on the visible light branch information in the target fusion feature to obtain surface defect diagnosis results, and to extract infrared thermal anomalies based on the infrared branch information in the target fusion feature to obtain anomaly diagnosis results. The diagnostic reasoning and generation module is used to perform cross-modal joint reasoning based on the surface defect diagnosis results and the anomaly diagnosis results to obtain a comprehensive diagnostic result, and generate a structured diagnostic report based on the comprehensive diagnostic result.

9. An electronic device, characterized in that, include: Memory, used to store computer software programs; A processor is configured to read and execute the computer software program, wherein when the processor executes the computer software program, it implements the device fault diagnosis method based on the fusion of visible light and infrared thermal imaging as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that, The storage medium stores a computer software program, which, when executed by a processor, implements the device fault diagnosis method based on the fusion of visible light and infrared thermal imaging as described in any one of claims 1 to 7.

Images