Power equipment defect automatic identification and electricity inspection method based on multi-modal visual data

Abstract

The present application relates to the technical field of power equipment defect detection, in particular to a power equipment defect automatic identification electricity inspection method based on multi-modal visual data, aiming at solving the problem that in the power equipment defect detection, the visible light and infrared images cannot be accurately matched due to the time and space differences in collection, affecting the recognition accuracy of the deep learning model, the present application collects two types of images, aligns the time stamps through a synchronous trigger module, extracts shared physical feature points as a reference, calculates the spatial deviation vector and the time deviation, constructs an optimized affine transformation matrix to realize pixel-level calibration, improves the efficiency through parallel processing pipeline and reverse mapping method, optimizes local alignment through image pyramid registration and non-rigid deformation correction, and generates a dual-modal defect feature map through weighted fusion, finally automatically identifies defects through a pre-trained deep learning model, improving the detection accuracy and reliability.

Description

Technical Field

[0001] This invention relates to the field of power equipment defect detection technology, and more specifically, to a method for automatic identification of power equipment defects based on multimodal visual data for power inspection. Background Technology

[0002] Power equipment defect detection is an important technology, specifically applied to the automatic defect identification stage of power equipment inspection. Its core principle is to improve defect identification accuracy by precisely aligning dual-modal visual data, thus meeting the core requirement of power inspection for efficient and accurate equipment defect detection. In power equipment defect detection, visible light images can clearly present structural and textural defects, while infrared images can capture defects with abnormal thermal distribution. The combination of the two can comprehensively reflect the equipment status. However, due to the spatiotemporal differences in the acquisition equipment, the position and shape of the same defect cannot be precisely matched in the two modalities, thus affecting the accurate identification of defects by deep learning models. To solve this technical problem, we provide an automatic power equipment defect identification method based on multimodal visual data for power inspection. Summary of the Invention

[0003] The purpose of this invention is to provide an automatic identification and power inspection method for defects in power equipment based on multimodal visual data, so as to solve the problems mentioned in the background art.

[0004] To achieve the above objectives, one objective of this invention is to provide an automatic power equipment defect identification and inspection method based on multimodal visual data, comprising the following steps:

[0005] S1. Collect multimodal visual data of power equipment, including visible light images and infrared images, and ensure that the acquisition equipment achieves timestamp alignment through the synchronization trigger module;

[0006] S2. Establish a calibration mechanism for intramodal pixel coordinates and intermodal spatiotemporal mapping, including detecting shared physical feature points in visible light and infrared images as reference points. The reference points are inherent identifiable structural points on the surface of power equipment, automatically extracted by a corner detection algorithm. The spatial deviation vector of the pixel coordinates of the reference points in the visible light image and the pixel coordinates in the infrared image is calculated. Simultaneously, the time deviation is calculated based on the difference in acquisition timestamps. A mapping function is constructed. The mapping function maps the pixel coordinates of the visible light image to the spatial position of the infrared image and compensates for the time deviation using a linear interpolation method. Specifically, a two-dimensional transformation matrix is ​​generated. The two-dimensional transformation matrix is ​​adjusted based on the spatial deviation vector and the time deviation and is used to perform offset correction on each pixel coordinate. Finally, the mapping function is applied to perform real-time calibration on all pixels in the visible light and infrared images, and the calibrated multimodal dataset is output.

[0007] S3. Based on the calibration results, the visible light image and infrared image are aligned to match the position and shape of the same defect in the two modalities. The calibrated multimodal dataset is then used to automatically identify power equipment defects through a pre-trained deep learning model.

[0008] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0009] This invention achieves timestamp alignment between visible light and infrared images through a synchronous triggering module, then extracts shared physical feature points through multi-scale sliding windows and bidirectional feature matching, and accurately selects benchmark points by combining corner detection and thermal gradient analysis. A total correction vector is generated through spatial deviation vector and temporal deviation compensation, and an optimized affine transformation matrix is ​​constructed to achieve pixel-level spatiotemporal calibration, effectively solving the spatiotemporal misalignment problem of two types of modal data. Parallel processing pipeline improves processing efficiency, reverse mapping method avoids pixel gaps, image pyramid registration and non-rigid deformation correction further optimize the local alignment effect, and weighted fusion algorithm allows defects to simultaneously present clear texture and thermal distribution characteristics. The calibrated multimodal dataset provides accurate and comprehensive feature input for pre-trained deep learning models, improving the accuracy and reliability of power equipment defect identification. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating the overall workflow of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] Please see Figure 1 As shown, this embodiment provides a method for automatic identification of electrical equipment defects based on multimodal visual data during power consumption inspection, including the following steps:

[0013] S1. Collect multimodal visual data of power equipment, including visible light images and infrared images, and ensure that the acquisition equipment achieves timestamp alignment through the synchronization trigger module;

[0014] S2. Establish a calibration mechanism for intramodal pixel coordinates and intermodal spatiotemporal mapping. This includes detecting shared physical feature points in visible light and infrared images as reference points. The reference points are inherent identifiable structural points on the surface of power equipment, which are automatically extracted using a corner detection algorithm. The spatial deviation vector between the pixel coordinates of the reference points in the visible light image and the pixel coordinates in the infrared image is calculated. Simultaneously, the time deviation is calculated based on the difference in acquisition timestamps. A mapping function is constructed, which maps the pixel coordinates of the visible light image to the spatial position of the infrared image and compensates for the time deviation using a linear interpolation method. Specifically, a two-dimensional transformation matrix is ​​generated. The two-dimensional transformation matrix is ​​adjusted based on the spatial deviation vector and the time deviation and is used to perform offset correction on each pixel coordinate. Finally, the mapping function is applied to perform real-time calibration on all pixels in the visible light and infrared images, and the calibrated multimodal dataset is output.

[0015] S3. Based on the calibration results, the visible light image and infrared image are aligned to match the position and shape of the same defect in the two modalities. The calibrated multimodal dataset is then used to automatically identify power equipment defects through a pre-trained deep learning model.

[0016] Detecting shared physical feature points as reference points in visible light and infrared images specifically includes:

[0017] By traversing the surface area of ​​power equipment through a multi-scale sliding window, the metal clips of insulator strings and the vertices of bolts and nuts in the visible light image are identified as candidate points. At the same time, the corresponding heat conduction abrupt change points are captured in the same spatial area of ​​the infrared image. The candidate points and the corresponding heat conduction abrupt change points are matched bidirectionally, and the paired points with a spatial distance of less than a preset threshold are selected as the reference points for shared physical feature points.

[0018] For visible light images, an improved Harris corner response function is used to calculate the pixel gradient covariance matrix. When both eigenvalues ​​of the pixel gradient covariance matrix are greater than the dynamic threshold, it is determined to be a structural corner. For infrared images, an adaptive thermal gradient operator is used to detect extreme points of temperature change rate along the normal of the device surface. The similarity between visible light corner points and infrared extreme points is calculated by a normalized cross-correlation algorithm, and the top few pairs of points with the highest similarity are retained as reference points.

[0019] Establish a first pixel coordinate system with the upper left corner of the visible light image as the origin and a second pixel coordinate system with the upper left corner of the infrared image as the origin. Subtract the horizontal and vertical coordinates of the same reference point in the two coordinate systems to obtain the horizontal and vertical deviation components of the point. Iterate through all reference points to generate a set of deviation vectors. After removing outlier vectors through a random sampling consensus algorithm, calculate the average deviation vector.

[0020] Read the time difference between visible light and infrared acquisition recorded by the synchronous trigger module, convert the time difference into spatial displacement compensation based on the device motion speed model, and superimpose the average deviation vector and the spatial displacement compensation to generate a total correction vector. Use the total correction vector as a parameter to construct an affine transformation function as a mapping function.

[0021] The visible light image is divided into several grid cells. An affine transformation function is applied with the grid vertices as control points. For any pixel point inside the grid, bilinear interpolation is performed according to its position weights relative to the four vertices to output the mapped coordinates of the point in the infrared image space. At the same time, the time deviation compensation is incorporated into the coordinate offset.

[0022] The total correction vector is decomposed into translation matrix components, and rotation matrix components are generated by combining the estimated equipment rotation angle. The translation matrix and rotation matrix are multiplied to generate an affine transformation matrix with six parameters. The matrix parameters are optimized by least squares to minimize the reprojection error of all reference points.

[0023] The original coordinates of each pixel in the visible light image are used to form a homogeneous coordinate vector. This vector is then multiplied by an affine transformation matrix. The resulting non-homogeneous coordinates are used as the corrected coordinates. The inverse mapping method is then used to traverse each grid position in the infrared image and interpolate the pixel values ​​from the original visible light image.

[0024] A parallel processing pipeline containing visible light image data stream channels and infrared image data stream channels is constructed. In the visible light channel, a geometrically corrected image is generated using the inverse mapping method. In the infrared channel, nearest neighbor sampling is used to maintain the original thermal distribution. The images output from the two channels are aligned by pixel coordinates and merged into a multimodal data frame. A timestamp and device location label are added to form a dataset.

[0025] Multimodal data frames are input into the image pyramid registration module, and the mutual information entropy of visible light and infrared images is calculated at multiple scales. The local deformation field is optimized by gradient descent method, and non-rigid deformation correction is performed on the infrared image. Finally, a weighted fusion algorithm is used to overlay the visible light texture and the infrared thermal map to generate a defect feature map with matching position and shape.

[0026] Further explanation is needed: after completing the multimodal visual data acquisition and timestamp alignment, the primary task in establishing a calibration mechanism for intermodal spatiotemporal mapping is to extract shared physical feature points common to both visible light and infrared images as reference points. These reference points must simultaneously possess clear identifiability and spatial correspondence in both modalities to ensure the accuracy of subsequent calibration. The specific implementation process is as follows:

[0027] First, the surface area of ​​the power equipment is traversed using a multi-scale sliding window. This involves setting three sets of rectangular windows of different sizes: a small scale for small structures like bolts and nuts, a medium scale for insulator string metal clips, and a large scale for composite structures. The window step size is set to 1 / 3 of the corresponding window's side length, ensuring full coverage while avoiding redundant calculations. The surface area of ​​the power equipment refers to the effective area in the image containing core components such as insulators, towers, and bolts. After cropping the image to remove background interference, the traversal begins from the top left corner, sliding horizontally row by row and vertically column by column to ensure no potential structural points are missed. During the traversal, insulator string metal clips and bolt / nut vertices are identified as candidate points in the visible light image. Visible light images, acquired by a visible light camera, clearly show the surface texture and structural morphology of the power equipment; their core advantage is the ability to accurately capture the geometric contours of metal components. The metal clip of the insulator string is the metal fixing structure at the connection point of the insulator string. The apex of the bolt and nut is the apex of the edge of the bolt head. Both have significant differences in gray scale and sharp edges, making them naturally identifiable structural points. The specific identification process is as follows:

[0028] For the visible light image region covered by the sliding window, the pixel grayscale gradient and edge intensity are calculated. When a continuous high grayscale gradient point is detected within the region (consistent with the grayscale difference characteristics between the metal part and the surrounding insulating material), and its shape conforms to the geometric contour of the buckle or nut apex (such as the rectangular contour of the buckle, the acute angle feature of the nut apex), the center coordinates or vertex coordinates of the region are marked as candidate points. Simultaneously, the grayscale feature vector and local edge histogram of the candidate point are recorded to provide a basis for subsequent matching. While identifying candidate points in the visible light image, corresponding abrupt changes in heat conduction are captured within the same spatial region in the infrared image. The infrared image is a... Images captured by infrared thermal imagers, reflecting the surface temperature distribution of power equipment, reveal abrupt temperature changes due to differences in thermal conductivity between metal components and insulating materials. This characteristic is crucial for cross-modal matching. The same spatial region refers to the infrared image region that corresponds one-to-one with the sliding window position of the visible light image. Time stamp alignment ensures initial spatial correspondence. Abrupt thermal conductivity changes occur at pixels in the infrared image where the temperature gradient changes dramatically. These points correspond to the interface between metal and non-metal on the power equipment surface or critical structural points of metal components. The thermal conductivity efficiency at these locations differs significantly from the surrounding area, resulting in pronounced temperature abrupt changes. The specific capture process is as follows:

[0029] For the spatial region in the infrared image corresponding to the visible light candidate point, an adaptive thermal gradient operator is used to calculate the pixel temperature change rate. When the temperature change rate exceeds a dynamically set gradient threshold (adjusted in real time according to the overall temperature distribution of the image), the pixel is marked as a thermal conduction abrupt change point. Simultaneously, its temperature value, temperature gradient direction, and other characteristic information are recorded to form an infrared feature set corresponding to the visible light candidate point. Then, bidirectional feature matching is performed between the candidate point and the corresponding thermal conduction abrupt change point. Bidirectional feature matching means both forward matching of the feature vector of the visible light candidate point with the feature vector of the infrared thermal conduction abrupt change point and reverse verification of the feature vector of the infrared thermal conduction abrupt change point with the feature vector of the visible light candidate point, ensuring the mutual exclusivity and accuracy of the matching relationship and avoiding mismatches caused by one-way matching. The specific matching process is as follows:

[0030] First, local features (gray-level co-occurrence matrix, edge orientation histogram) of visible light candidate points and local features (temperature distribution histogram, thermal gradient vector) of infrared thermal conduction abrupt change points are extracted. The similarity between the two types of features is calculated using a normalized cross-correlation algorithm (values ​​range from 0 to 1, with values ​​closer to 1 indicating higher feature fit). Using the visible light candidate points as a benchmark, the top three thermal conduction abrupt change points with the highest similarity within the corresponding infrared spatial region are selected as candidate matching objects. Then, using these three infrared abrupt change points as a benchmark, a reverse similarity verification is performed with the visible light candidate points, retaining only the bidirectional similarity. The point pairs with the highest similarity ranking form the initial matching results, ensuring that each pair of matching points is the most suitable feature point for each of the two modalities. Finally, pairs of points with a spatial distance less than a preset threshold are selected. The spatial distance refers to the Euclidean distance between the pixel coordinates of the visible light candidate points (based on the first pixel coordinate system with the upper left corner of the visible light image as the origin) and the pixel coordinates of the infrared thermal conduction abrupt change points (based on the second pixel coordinate system with the upper left corner of the infrared image as the origin) after preliminary coordinate mapping (converted according to the image resolution ratio), quantifying the degree of spatial correspondence of feature points in the two modalities. The preset threshold is a critical value calibrated based on the acquisition resolution of the multimodal image, the device structure size, and the acquisition distance (for example, calibrated to 3 pixel units for a 1080P resolution image), ensuring that only point pairs with corresponding spatial heights are retained. The specific selection process is as follows:

[0031] For each pair of points that are successfully matched bidirectionally, the spatial distance after mapping is calculated. If the distance is less than a preset threshold, it means that the point pair corresponds to the same physical structure point on the power equipment and is retained as a valid pairing point. If the distance exceeds the threshold, it is judged as a mismatch and is removed. Finally, all valid pairing points that pass the screening are the shared physical feature point reference points that have both cross-modal recognition and spatial correspondence, providing a reliable core basis for subsequent calculation of spatial deviation vector and construction of mapping function.

[0032] After initially identifying candidate points in the visible light image and abrupt changes in infrared thermal conduction using a multi-scale sliding window, further refined screening and cross-modal similarity matching of feature points in the two modalities are needed to improve the accuracy and reliability of the benchmark points. The core of this process is to use a corner detection algorithm to lock structural features in the visible light image, capture temperature abrupt change features in the infrared image through thermal gradient analysis, and then achieve precise pairing through similarity calculation. The specific implementation process is as follows:

[0033] The entire refined processing flow revolves around the logic of visible light structural corner point extraction—infrared temperature extreme point detection—cross-modal similarity matching—high-quality point pair selection. First, an improved Harris corner response function is used to calculate the pixel gradient covariance matrix for the visible light image. The improved Harris corner response function, based on the traditional Harris algorithm, optimizes the Gaussian smoothing kernel size and response function coefficients for the grayscale distribution characteristics of metal structural points in power equipment, enhancing the sensitivity to sharp edges and vertices while reducing background noise interference. The pixel gradient covariance matrix is ​​a two-dimensional matrix describing the statistical correlation characteristics of pixel gradients in the x and y axes. Its core function is to quantify the distribution pattern of gradients around a pixel, thereby determining whether the pixel is a corner point. The specific calculation process is as follows:

[0034] First, the visible light image is converted to grayscale. The Sobel operator is used to calculate the gradient values ​​of each pixel in the x-direction (horizontal) and y-direction (vertical) to obtain a gradient map. Then, a preset Gaussian smoothing kernel (calibrated to 3×3 size) is used to smooth the three sets of data: the square of the gradient in the x-direction, the square of the gradient in the y-direction, and the product of the x and y gradients, reducing the influence of isolated noise points. Finally, a 3×3 neighborhood window is taken centered on each pixel, and the three smoothed data sets within the window are summed to obtain the gradient covariance matrix of that pixel. The two elements of the matrix correspond to the sum of the squares of the gradients in the x and y directions, respectively, and the intersection element corresponds to the sum of the squares of the gradients in the x and y directions. After calculating the pixel gradient covariance matrix by summing the gradient products in the y-direction, the pixel is determined to be a structural corner point when both eigenvalues ​​of the matrix are greater than the dynamic threshold. The eigenvalues ​​are two values ​​obtained after eigenvalue decomposition of the gradient covariance matrix; their magnitude directly reflects the distribution characteristics of the gradient around the pixel. When both eigenvalues ​​are large, it indicates significant gradient changes in multiple directions around the pixel, meaning the point is a sharp structural corner point. If one eigenvalue is large and the other small, it may be an edge point; if both are small, it is a point in a flat region. The dynamic threshold is a critical value dynamically calculated based on the overall gradient distribution of the current visible light image, rather than a fixed value. The specific calibration logic is as follows:

[0035] The median of the eigenvalues ​​of the gradient covariance matrix of all pixels in the image is used. 1.5 times this median is then used as a dynamic threshold, which is automatically recalculated for each image processed to ensure that the threshold adapts to the gradient differences under different lighting conditions. The specific determination process is as follows:

[0036] For each pixel, the gradient covariance matrix is ​​decomposed to obtain two eigenvalues. If both eigenvalues ​​are greater than the dynamic threshold of the current image, and their ratio is within a preset reasonable range (to avoid misjudging edge points), the pixel is marked as a structural corner point. Simultaneously, its pixel coordinates and local gradient features are recorded as the core basis for subsequent matching. While extracting visible light structural corner points, an adaptive thermal gradient operator is applied to the infrared image to detect extreme points of temperature change rate along the normal to the device surface. The adaptive thermal gradient operator automatically adjusts the gradient calculation sensitivity based on the local temperature distribution of the infrared image, reducing sensitivity when local temperature differences are large. To avoid saturation, sensitivity is increased to capture subtle abrupt changes when temperature differences are small, ensuring accurate detection of abrupt changes across different temperature ranges. The equipment surface normal refers to the direction perpendicular to the surface of the electrical equipment. Detecting the temperature change rate along this direction more accurately reflects the differences in heat conduction between the equipment surface and its interior or at the interface of different materials, avoiding the omission of abrupt changes caused by detection along the surface tangent. Extreme points of the temperature change rate refer to pixels where the temperature change rate along the equipment surface normal reaches a local maximum or minimum value. These points correspond to the locations of abrupt changes in heat conduction in infrared images and have spatial correspondence with structural corner points in visible light images. The specific detection process is as follows:

[0037] First, the power equipment region in the infrared image is extracted using image segmentation technology. Based on the 3D model of the equipment or edge fitting results, the surface normal corresponding to each pixel is determined. Then, the temperature change rate along the normal of each pixel is calculated using an adaptive thermal gradient operator, which is the ratio of the temperature difference between adjacent pixels in the normal direction to the pixel distance. All pixels are traversed, and the pixels with the largest and smallest temperature change rates within their local neighborhoods (3×3 windows) are marked as extreme points of temperature change rate. Their temperature value, change rate direction, and other feature information are recorded. After extracting the two types of feature points, the similarity between the visible light corner points and the infrared extreme points is calculated using a normalized cross-correlation algorithm. The normalized cross-correlation algorithm is a similarity calculation method that eliminates the influence of brightness scale differences and mean values. Its core advantage is that it can accurately measure the similarity between two local feature windows under conditions of illumination changes and grayscale shifts. The calculation result (cross-correlation coefficient) ranges from -1 to 1. The closer to 1, the higher the similarity between the two feature windows; the closer to -1, the greater the difference. The specific calculation process is as follows:

[0038] For each visible light structural corner point, a fixed-size local feature window (calibrated to 11×11 pixels, which contains sufficient structural information while avoiding excessive background) is taken centered on it, and the grayscale matrix within the window is extracted. For each infrared temperature change rate extreme point, a local feature window of the same size is also taken centered on it, and the temperature matrix within the window is extracted. The matrices of the two windows are normalized by subtracting their respective means and dividing by the standard deviation to eliminate the influence of brightness and temperature scales. Then, the cross-correlation coefficient between the two normalized matrices is calculated. This coefficient represents the similarity between the visible light corner point and the infrared extreme point. The higher the coefficient value, the more likely the corresponding physical locations are the same structural point. Finally, the top few pairs of points with the highest similarity are retained as reference points. The top few pairs are pre-calibrated based on the structural complexity and calibration accuracy requirements of the power equipment (e.g., 20 pairs, which ensures the reliability of the subsequent mapping function construction while avoiding excessive redundant points that increase computational load). The core is to select high-quality point pairs with the highest cross-modal matching degree and eliminate mismatched points with low similarity. The specific selection process is as follows:

[0039] First, all possible pairings of visible light structural corner points and infrared temperature change rate extreme points are established, and the similarity of each pair is calculated. Then, all pairs are sorted from high to low similarity. The top-ranked pairs are selected, and the spatial position rationality of each pair is further checked, eliminating abnormal pairs with obvious spatial mismatches. Finally, the remaining pairs are the shared physical feature point reference points with both high similarity and spatial rationality, providing accurate and reliable core data support for subsequent spatial deviation vector calculation and mapping function construction.

[0040] After determining the shared physical feature point reference points, in order to accurately quantify the spatial position differences of the reference points in visible light images and infrared images, it is necessary to first establish a unified logical pixel coordinate system, and then obtain a reliable average deviation vector through deviation calculation and outlier removal, providing core data support for the subsequent construction of mapping functions. The specific implementation process is as follows:

[0041] First, a first pixel coordinate system with the top-left corner of the visible light image as its origin and a second pixel coordinate system with the top-left corner of the infrared image as its origin are established. The first pixel coordinate system is a two-dimensional planar coordinate system used to locate the pixel position in the visible light image. Its origin is set at the top-left corner vertex of the pixel in the top-left corner of the image. The positive x-axis is horizontally to the right (corresponding to the direction of increasing column number in the image), and the positive y-axis is vertically downward (corresponding to the direction of increasing row number in the image). The position of each pixel is uniquely identified by the coordinate value (x1, y1), where x1 represents the number of horizontal pixels (column index) from the origin, and y1 represents the number of vertical pixels (row index). The second pixel coordinate system is used to locate the pixels in the infrared image. It adopts the same definition rules as the first pixel coordinate system. The origin is also at the top-left corner of the image, and the coordinate axis direction and coordinate value meaning are consistent with the first pixel coordinate system to ensure that the position measurement logic of the two types of images is the same. The specific establishment process is as follows:

[0042] The system automatically identifies the boundary contours of the visible light image, locks the top-left vertex as the origin of the first pixel coordinate system, and assigns continuously increasing x1 and y1 values ​​to each pixel in the image in a left-to-right and top-to-bottom order (e.g., the coordinates of the first pixel in the top-left corner are (0, 0), the adjacent pixel on the right is (1, 0), and the adjacent pixel below is (0, 1)). The infrared image is processed with the same logic to establish a second pixel coordinate system, so that each pixel in the infrared image can accurately represent its spatial position through the (x2, y2) coordinates, laying a unified benchmark for subsequent cross-modal deviation calculation. After the coordinate system is established, the horizontal and vertical coordinates of the same benchmark point in the two coordinate systems are subtracted respectively. The same benchmark point is the shared physical feature point selected above, which corresponds to a unique (x1, y1) coordinate in the visible light image and a unique matching (x2, y2) coordinate in the infrared image. These two sets of coordinates belong to the first and second pixel coordinate systems respectively and point to the same physical structure point on the power equipment. The x-axis and y-axis coordinates refer to the x-axis (horizontal direction) and y-axis (vertical direction) coordinates in the coordinate system. x1 and x2 are the x-axis values, reflecting the horizontal position; y1 and y2 are the y-axis values, reflecting the vertical position. The specific difference calculation process is as follows: The system extracts two sets of coordinates for the same reference point from the feature point storage module, one by one, according to the reference point number. First, it retrieves the x-coordinate x2 of the second pixel coordinate system and the x-coordinate x1 of the first pixel coordinate system, calculating the difference between x2 and x1. Then, it retrieves the y-coordinate y2 of the second pixel coordinate system and the y-coordinate y1 of the first pixel coordinate system, calculating the difference between y2 and y1. By performing these two differences, the system directly obtains the positional offset values ​​of the same reference point in the horizontal and vertical directions. Through the above subtraction process, the horizontal and vertical deviation components of the point are obtained. The horizontal deviation component is the result of the difference in the horizontal coordinates, and its value (positive or negative) indicates the direction of the shift (positive value indicates that the reference point in the infrared image shifts to the right relative to the visible light image, and negative value indicates that it shifts to the left). The absolute value represents the number of pixels shifted in the horizontal direction. The vertical deviation component is the result of the difference in the vertical coordinates, and its value (positive value indicates downward shift, and negative value indicates upward shift) corresponds to the vertical direction of the shift. The absolute value is the number of pixels shifted in the vertical direction. The specific process is as follows:

[0043] The calculation result of x2-x1 is directly defined as the horizontal deviation component of the reference point, and the result of y2-y1 is defined as the vertical deviation component. For example, if the coordinates of a reference point in the first coordinate system are (100, 200) and the coordinates in the second coordinate system are (103, 198), then the horizontal deviation component is 3 (3 pixels to the right) and the vertical deviation component is -2 (2 pixels upward). The two components together completely describe the spatial offset of the reference point in the cross-modal image. Then, all reference points are traversed to generate a set of deviation vectors. After removing outliers using a random sampling consensus algorithm, the average deviation vector is calculated. Traversing all reference points means processing all shared physical feature point reference points one by one in sequence to ensure that the deviation data of any valid reference point is not missed. The deviation vector is a two-dimensional vector formed by combining the horizontal and vertical deviation components of each reference point in a fixed order of horizontal first and vertical last (e.g., (3, -2)). It can intuitively and completely represent the cross-modal spatial offset of a single reference point. The set of deviation vectors is a dataset composed of the deviation vectors of all reference points, which centrally reflects the overall offset trend of the two types of images. The specific generation process is as follows:

[0044] Following the selection order of the reference points, the horizontal and vertical deviation components of each reference point are extracted sequentially, combined into a two-dimensional deviation vector, and stored in a preset set container. After all reference points have been processed, the set of two-dimensional vectors stored in the set container constitutes the deviation vector set. The random sampling consensus algorithm is a robust data filtering algorithm. Its core function is to separate valid data (inliers) that conform to the overall pattern and outliers (outliers) that deviate from the pattern from a set containing noisy and mismatched data, thus avoiding interference from outliers in the overall results. Outliers are deviation vectors in the deviation vector set that significantly deviate from the offset trend of most vectors. These vectors are usually caused by reference point mismatch, image noise, or local interference. If they are not removed, they will seriously affect the accuracy of the average deviation. The specific removal process is as follows:

[0045] The algorithm parameters are preset, including the number of iterations (calibrated to 100) and the inlier threshold (i.e., the allowable deviation range between the vector and the assumed model, calibrated to 2 pixels). After the algorithm starts, three vectors are randomly selected from the deviation vector set in each iteration to construct an initial offset model. Then, the deviation of all vectors in the set from the model is calculated, and vectors with deviations less than the inlier threshold are marked as inliers. The iteration process is repeated, and the optimal model with the most inliers is recorded. All inlier vectors corresponding to this model are determined as effective deviation vectors. The remaining outlier vectors are removed, and finally, the average deviation vector is calculated. The average deviation vector is the statistical average of the effective deviation vectors and can represent the overall average spatial offset between the visible light image and the infrared image, providing a core basis for subsequent calibration. The specific calculation process is as follows:

[0046] For the set of effective deviation vectors after removing outliers, the sum of the horizontal and vertical deviation components of all vectors is calculated. Then, the two sums are divided by the number of effective vectors to obtain the average horizontal and vertical deviations. These two averages are combined in the order of horizontal first and vertical last to form the final average deviation vector. This vector comprehensively reflects the overall spatial offset pattern of the two modal images. For example, if the sum of the horizontal components of the effective vectors is 300 and the number is 100, the average horizontal deviation is 3. If the sum of the vertical components is -200 and the number is 100, the average vertical deviation is -2. The final average deviation vector is (3, -2), which provides a reliable spatial offset basis for the subsequent generation of the total correction vector by fusing temporal deviations.

[0047] After obtaining the average deviation vector reflecting the spatial offset pattern, since there may be a slight time difference between the acquisition of visible light and infrared images, and the movement of the acquisition equipment may cause this time difference to be converted into additional spatial deviation, it is necessary to fuse the spatial compensation amount corresponding to the time deviation to generate a more comprehensive total correction vector, and then construct an accurate mapping function. The specific implementation process is as follows:

[0048] First, the time difference between visible light and infrared image acquisition recorded by the synchronization trigger module is read. The synchronization trigger module is a control module used to coordinate the acquisition timing of the visible light camera and the infrared thermal imager. Its core function is to ensure that the two types of images are acquired at the same time as much as possible, while accurately recording the actual time difference between the two acquisitions. This module has a built-in high-precision clock to ensure the accuracy of the time recording. The time difference between visible light and infrared image acquisition refers to the difference between the acquisition time of the visible light image and the acquisition time of the infrared image in the same physical scene. A positive value indicates that the infrared image acquisition lags behind the visible light image acquisition, while a negative value indicates that the infrared image acquisition is ahead. The specific reading process is as follows:

[0049] Based on the unique frame number, visible light images are paired with their corresponding infrared images (based on scene association and trigger signal association at the time of acquisition). Then, the two sets of acquisition timestamps (visible light acquisition timestamp and infrared acquisition timestamp) corresponding to the paired frame are retrieved from the storage unit of the synchronization trigger module. The difference between the two sets of timestamps is calculated, which is the acquisition time difference value of the paired frame. This ensures that each paired image can obtain the corresponding time difference data, providing a basis for subsequent compensation. After obtaining the acquisition time difference value, the time difference is converted into a spatial displacement compensation amount according to the device motion speed model. The device motion speed model is a pre-established mathematical model describing the motion state of the acquisition device. This model is calibrated based on the motion characteristics of the acquisition device and includes two core parameters: horizontal velocity and vertical velocity. These parameters can be updated in real time through the device's GPS and inertial measurement unit (IMU) to ensure that the model reflects the actual motion state of the device. The spatial displacement compensation amount refers to the offset of the target position in the image caused by the movement of the acquisition device during the acquisition time difference. It also includes horizontal displacement compensation components and vertical displacement compensation components, used to compensate for the additional spatial deviation caused by the time difference. The specific conversion process is as follows:

[0050] The horizontal and vertical velocities at the current acquisition time are extracted from the device motion speed model. The acquisition time difference is multiplied by the horizontal and vertical velocities respectively to obtain the spatial displacement compensation amount in the horizontal and vertical directions. For example, if the acquisition time difference is 10 milliseconds, the horizontal velocity is 0.3 pixels / millisecond, and the vertical velocity is 0.2 pixels / millisecond, then the horizontal displacement compensation amount is 3 pixels and the vertical displacement compensation amount is 2 pixels. The two are combined to form a two-dimensional spatial displacement compensation vector, which fully represents the spatial offset caused by the time difference. The average deviation vector and the spatial displacement compensation are then superimposed to generate the total correction vector. Vector superposition means adding the corresponding components of the two two-dimensional vectors respectively, that is, adding the horizontal component of the average deviation vector to the horizontal component of the spatial displacement compensation, and adding the vertical component of the average deviation vector to the vertical component of the spatial displacement compensation. The resulting vector is the total correction vector. The average deviation vector is the vector calculated using the reference point, which reflects the inherent spatial offset of the two types of images. The spatial displacement compensation is the additional offset vector caused by the time difference. The core logic of superposition is to comprehensively consider the motion deviation caused by the inherent spatial offset and the time difference, ensuring that the total correction vector can fully cover all major offset sources of cross-modal images. The specific superposition process is as follows:

[0051] Extract the horizontal and vertical deviation components of the average deviation vector, and then extract the horizontal and vertical components of the spatial displacement compensation. Add the two horizontal components to obtain the horizontal component of the total correction vector; add the two vertical components to obtain the vertical component of the total correction vector. For example, if the average deviation vector is (3, -2) and the spatial displacement compensation vector is (2, 1), then the horizontal component of the total correction vector is 3+2=5, and the vertical component is -2+1=-1. The final total correction vector is (5, -1). This vector integrates all key offset factors and provides the core basis for subsequent coordinate correction. Finally, an affine transformation function is constructed using the total correction vector as a parameter, which serves as the mapping function. An affine transformation function is a linear transformation function that preserves the parallelism of the image and can perform geometric transformations such as translation, rotation, and scaling. Here, it is mainly used to map the pixel coordinates of the visible light image to the spatial position of the infrared image, while compensating for time deviations. Its core parameters are jointly determined by the total correction vector and the estimated device rotation angle. The mapping function acts as a bridge connecting the pixel coordinates of the visible light image and the pixel coordinates of the infrared image. Given any pixel coordinates from the visible light image, this function can output their corresponding calibrated coordinates in the infrared image. The specific construction process is as follows:

[0052] First, the horizontal and vertical components of the total correction vector are used as translation parameters of the affine transformation function and directly incorporated into the translation term of the function to ensure that the overall offset is offset. Then, the rotation parameters of the affine transformation are determined by combining the estimated rotation angle of the device (provided by the device IMU, reflecting the device's attitude rotation during acquisition). If the rotation angle is small (usually the device attitude is stable in inspection scenarios and the rotation angle can be ignored), the translation parameters are used as the main parameters. Finally, the translation and rotation parameters are substituted into the preset affine transformation function template to generate a complete affine transformation function. This function is the final mapping function, which can map any pixel coordinate in the visible light image to the corresponding spatial position in the infrared image through linear transformation, while completing dual compensation for time and spatial deviations, laying the foundation for subsequent real-time calibration of all pixels.

[0053] After constructing the affine transformation function that can comprehensively compensate for spatial and temporal deviations, in order to achieve accurate mapping of all pixels in the visible light image to the infrared image space, ensuring mapping efficiency while avoiding accuracy fluctuations caused by direct transformation of individual pixels, it is necessary to use a combination of grid partitioning and interpolation to calibrate the pixel coordinates step by step. The specific implementation process is as follows:

[0054] First, the visible light image is divided into several grid cells. An affine transformation function is applied using the grid vertices as control points. A grid cell is a rectangular area formed by uniformly dividing the visible light image into fixed sizes. The size is experimentally calibrated to 16×16 pixels (balancing interpolation accuracy and computational efficiency; too small a size will increase the computational load, while too large a size may lead to insufficient fitting of local deformations). The division starts from the upper left corner of the image and is arranged sequentially in horizontal and vertical order. Areas with edges smaller than one grid size are used to form a complete grid with the actual remaining pixels, ensuring that the entire image is covered without omissions. The grid vertices are the four corner points of each grid cell. The core reason for using these vertices as control points is that their coordinates can be directly calculated to obtain the accurate infrared image coordinates through the affine transformation function, and they can provide a reliable reference for the interpolation of pixels within the grid. Applying the affine transformation function means substituting the original pixel coordinates of each control point (belonging to the first pixel coordinate system) into the affine transformation function constructed above, and obtaining the target coordinates of the vertex in the infrared image space (second pixel coordinate system) through function operation. The specific process is as follows: The system extracts the coordinates of the four vertices of each grid according to the grid number. For example, the coordinates of the top left vertex of a certain grid are (x0, y0), the top right vertex is (x1, y0), the bottom left vertex is (x0, y1), and the bottom right vertex is (x1, y1). These four coordinates are input into the affine transformation function respectively, and the corresponding coordinates of each vertex in the infrared image are calculated in sequence as (x0', y0'), (x1', y0'), (x0', y1'), and (x1', y1'). These transformed vertex coordinates become the core basis for subsequent interpolation calculations. For any pixel point inside the grid, bilinear interpolation is performed according to its position weight relative to the four vertices. Any pixel point inside the grid refers to all pixels located inside the grid cell except for the grid vertices. These pixels cannot directly obtain accurate coordinates through affine transformation, and interpolation estimation is required with the help of the transformation results of surrounding control points. Position weights are weighted coefficients calculated based on the spatial distances from a pixel to its four vertices. The closer a pixel is to a vertex, the greater its weight. The total weight is always 1, ensuring that the interpolation result is a reasonable fusion of control point coordinates. Bilinear interpolation is a method that achieves interpolation in two-dimensional space through two linear interpolations (first horizontal then vertical, or vice versa). It can effectively fit the coordinate change trend within the grid and avoid coordinate mapping distortion. The specific process is as follows:

[0055] First, determine the relative position of the pixel to be interpolated within the current grid. Calculate the ratio of the horizontal distance from the pixel to the left edge of the grid to the grid width (denoted as u), and the ratio of the vertical distance from the pixel to the top edge of the grid to the grid height (denoted as v). The values ​​of u and v are both between 0 and 1. Then, perform horizontal interpolation by calculating the horizontal interpolation coordinates of the pixel at the top edge (connecting the top left and top right vertices) and the bottom edge (connecting the bottom left and bottom right vertices). The top edge interpolation coordinates are obtained by multiplying the transformed coordinates of the top left vertex by (1-u) and the transformed coordinates of the top right vertex by u, and then adding them together. The bottom edge interpolation coordinates are obtained by multiplying the transformed coordinates of the bottom left vertex by (1-u) and the transformed coordinates of the bottom right vertex by u. Multiply by u and add them together to get the result; finally, perform vertical interpolation, multiply the horizontal interpolation coordinates of the upper and lower edges by (1-v) and v respectively, and add them together to obtain the preliminary mapping coordinates of the pixels inside the grid. The entire process ensures that the coordinate changes are consistent with the spatial trend within the grid through weight allocation. Through the above bilinear interpolation calculation, the mapping coordinates of the point in the infrared image space are output. The mapping coordinates in the infrared image space refer to the precise position coordinates of the pixels inside the visible light image after interpolation calculation in the infrared image (second pixel coordinate system). These coordinates correspond one-to-one with the pixel grid of the infrared image, ensuring that the same physical structure point is aligned in the two modal images. The specific output process is as follows:

[0056] The initial mapped coordinates obtained by bilinear interpolation are numerically normalized (retaining integer pixel coordinates, or retaining one decimal place according to sub-pixel accuracy requirements), and abnormal coordinates exceeding the infrared image boundary are removed (if the calculation result exceeds the width and height range of the infrared image, it is mapped to the image edge pixels according to the nearest principle). Then, the normalized coordinates are used as the final mapped coordinates of the pixel and stored in the coordinate mapping table, corresponding one-to-one with the original coordinates of the pixel, providing data support for subsequent image alignment. While outputting the mapped coordinates, the time deviation compensation is incorporated into the coordinate offset. The time deviation compensation is the spatial displacement compensation vector obtained earlier based on the difference between the acquisition time of visible light and infrared images and combined with the device motion speed model. It includes compensation components in both horizontal and vertical directions and is used to offset the additional spatial deviation caused by device motion due to asynchronous acquisition. The coordinate offset is the offset value of the mapped coordinates relative to the original coordinates, that is, the difference between the mapped coordinates and the original coordinates, which includes the compensation effect of spatial deviation. The specific integration process is as follows:

[0057] The horizontal and vertical components of the time deviation compensation are extracted and superimposed on the horizontal and vertical offsets of the mapped coordinates, respectively. That is, the horizontal component of the mapped coordinates is added to the horizontal component of the time deviation compensation, and the vertical component of the mapped coordinates is added to the vertical component of the time deviation compensation. For example, if the horizontal offset of a pixel's mapped coordinates is 5 pixels and the vertical offset is -2 pixels, and the horizontal component of the time deviation compensation is 1 pixel and the vertical component is 0.5 pixels, then the combined horizontal offset is 6 pixels and the vertical offset is -1.5 pixels. The corresponding mapped coordinates are also adjusted accordingly, ultimately achieving dual compensation for spatial and temporal deviations, ensuring that the mapped coordinates accurately reflect the actual position of the same physical point in the infrared image.

[0058] After initially constructing the affine transformation function using the total correction vector as parameters, to further improve the accuracy of coordinate mapping, especially to address image shifts caused by slight rotation of the acquisition device, the total correction vector needs to be decomposed into pure translation components and independent rotation components. Through matrix fusion and parameter optimization, an affine transformation matrix that better fits the actual scene is generated. The specific implementation process is as follows:

[0059] First, the total correction vector is decomposed into translation matrix components. The total correction vector is a two-dimensional vector that integrates spatial offset and temporal difference compensation, containing combined horizontal and vertical offsets. Its core function is to provide the basis for positional correction. The translation matrix components are two-dimensional matrices containing only translation transformation information. Translation transformation refers to moving all pixels in the image a fixed distance along the horizontal or vertical direction without changing the image's shape, size, or orientation. The parameters of this matrix are directly determined by the components of the total correction vector. The specific decomposition process is as follows:

[0060] The horizontal and vertical offset components are extracted from the total correction vector and used as the core parameters of the translation matrix to construct a standard two-dimensional translation matrix. Except for the parameters representing horizontal and vertical translation, the matrix maintains the inherent structure of the identity matrix, ensuring that it can only perform pure translation transformations without introducing any additional deformations such as rotation or scaling, accurately preserving the translation information in the total correction vector. Simultaneously, the rotation matrix components are generated by combining the estimated device rotation angle. The estimated device rotation angle is the attitude parameter obtained in real-time by the inertial measurement unit (IMU) on the acquisition device, reflecting the rotation angle (in degrees) of the device relative to the horizontal reference plane during image acquisition. This value is usually small (the device attitude is relatively stable in inspection scenarios), but it has a significant impact on coordinate mapping accuracy. The rotation matrix components are two-dimensional matrices used to describe image rotation transformations. Rotation transformation refers to rotating the image around a center point (here, the image center is assumed to be the default) by a certain angle, without changing the image's shape and size, only adjusting its orientation in the plane. The specific generation process is as follows:

[0061] First, the estimated device rotation angle is converted into a numerical form for adaptation matrix calculation. Then, a standard two-dimensional rotation matrix is ​​constructed based on this rotation angle. The elements of the matrix consist of trigonometric function values ​​for that angle. Through the combination of these elements, it is ensured that when this matrix is ​​applied to pixel coordinates, it can achieve precise rotation of the pixel around the image center, and the rotation direction is opposite to the actual rotation direction of the device, thus offsetting the image shift caused by the device rotation. Subsequently, the translation matrix and the rotation matrix are multiplied. Matrix multiplication is a linear algebraic operation that follows the rules of matrix operations. Here, it specifically refers to the multiplication of two two-dimensional matrices, and its core function is to integrate the effects of translation and rotation transformations. By combining them into a single matrix, subsequent coordinate mapping can achieve both transformations simultaneously with only one matrix operation, improving processing efficiency. The specific operation process is as follows: the order of matrix multiplication is clearly defined (rotation first, then translation, i.e., translation matrix multiplied by rotation matrix), ensuring that the rotation transformation first offsets the offset caused by the device posture, and then the translation transformation compensates for the inherent spatial deviation; according to the matrix multiplication rules, each row element of the translation matrix is ​​multiplied by each column element of the rotation matrix respectively, and then summed to obtain the fused matrix elements, finally generating a composite transformation matrix that simultaneously contains rotation and translation information. This matrix inherits the orientation correction capability of the rotation matrix and the position compensation capability of the translation matrix. Through the matrix multiplication operation described above, an affine transformation matrix containing six parameters is generated. The affine transformation matrix is ​​a 3×3 homogeneous matrix used to implement affine transformations (homogeneous matrices can uniformly handle various transformations such as translation and rotation without separate calculations). Its six parameters correspond to: horizontal scaling factor, horizontal and vertical shearing factors, vertical scaling factor, horizontal translation factor, and vertical translation factor. The scaling factor is set to 1 by default (does not change the image size), and the shearing factor is set to 0 by default (no shearing deformation). The core effective parameters are the rotation-derived correlation parameters and the translation parameters provided by the translation matrix components, which together constitute a complete six-parameter matrix structure. The specific generation process is as follows: The composite transformation matrix obtained by matrix multiplication is expanded into a 3×3 homogeneous matrix form. The third row and third column elements required for homogeneous coordinates are added (maintaining the standard structure of the homogeneous matrix), ultimately forming an affine transformation matrix containing six core parameters. This matrix can be directly used for affine transformation calculations of pixel coordinates. Finally, the matrix parameters are optimized using the least squares method to minimize the reprojection error of all reference points. The least squares method is a mathematical optimization method used to find the optimal fit of data. Its core idea is to adjust the parameters to minimize the sum of squared errors of all data points, ensuring that the optimized parameters best fit the actual data patterns. The reprojection error refers to the deviation between the projected coordinates (i.e., the calibrated coordinates) obtained after substituting the original pixel coordinates of the reference point into the affine transformation matrix and the true target coordinates of the reference point (the corresponding coordinates in the infrared image determined earlier). This error directly reflects the mapping accuracy of the affine transformation matrix. The specific optimization process is as follows:

[0062] An optimization model is constructed with the six parameters of the affine transformation matrix as variables and the sum of squared reprojection errors of all reference points as the objective function. This model is then solved using the least squares method, which involves calculating the partial derivatives of the objective function with respect to each parameter, setting these partial derivatives to zero, obtaining a system of equations, and solving them to find the parameter values ​​that minimize the objective function. This process is iterated until the sum of squared reprojection errors converges to a preset minimum range (calibrated to pixel-level error). The affine transformation matrix parameters at this point are considered optimal, ensuring that applying this matrix to all pixels minimizes coordinate mapping deviations and provides a crucial guarantee for subsequent accurate pixel coordinate calibration.

[0063] After obtaining the accurate six-parameter affine transformation matrix through least squares optimization, this matrix needs to be applied to all pixels of the visible light image to achieve accurate mapping from the original coordinates to the infrared image spatial correction coordinates. Meanwhile, to avoid pixel gaps that may occur during forward mapping, a reverse mapping method is used to ensure that each grid cell in the infrared image obtains a reliable pixel value. The specific implementation process is as follows:

[0064] First, the original coordinates of each pixel in the visible light image are used to construct a homogeneous coordinate vector. The original coordinates are the two-dimensional position coordinates (x, y) of each pixel in the first pixel coordinate system of the visible light image, directly representing the inherent position of the pixel in the image without any geometric transformation. The homogeneous coordinate vector is a standardized representation that extends the two-dimensional coordinates into a three-dimensional vector. By adding a fixed component (default value is 1) after the two-dimensional coordinates, various geometric transformations such as translation and rotation can be implemented through a unified matrix multiplication, eliminating the need to design separate operation logic for different transformations. This greatly simplifies the processing flow of multiple transformation superposition. The specific construction process is as follows:

[0065] Traverse all pixels of the visible light image from left to right and top to bottom, extracting the original two-dimensional coordinates (x, y) of each pixel. Add a component 1 to these coordinates to form a three-dimensional homogeneous coordinate vector of the form (x, y, 1). For example, the homogeneous coordinate vector corresponding to the original coordinates (50, 80) is (50, 80, 1). This ensures that the coordinates of each pixel are compatible with the subsequent 3×3 affine transformation matrix. After constructing the homogeneous coordinate vector, perform matrix multiplication with the affine transformation matrix. The affine transformation matrix is ​​a 3×3 homogeneous matrix obtained after parameter optimization, containing core transformation parameters such as translation and rotation. It can achieve linear transformation and translation compensation of coordinates in one operation and is the core computational carrier connecting the visible light pixel coordinates and the infrared pixel coordinates. Matrix multiplication is a linear algebraic operation following the rules of homogeneous matrix operations. Its core function is to fuse the positional information in the homogeneous coordinate vector with the transformation information in the affine transformation matrix to generate a new homogeneous vector containing the transformed coordinates. The specific operation process is as follows:

[0066] The homogeneous coordinate vector of each pixel is used as a column vector and multiplied by a 3×3 affine transformation matrix. Following matrix multiplication rules, each row element of the transformation matrix is ​​multiplied by the corresponding element of the homogeneous coordinate vector, and the products of each row are summed. This results in a new homogeneous coordinate vector consisting of three elements. This vector incorporates all pre-defined geometric transformation information and directly corresponds to the transformed position of the pixel in the infrared image space. After the matrix multiplication is complete, the non-homogeneous coordinates of the result are used as the corrected coordinates. The non-homogeneous coordinates are the representation of the homogeneous coordinate vector as two-dimensional pixel coordinates, obtained by dividing the first two components of the homogeneous coordinate vector by the third component (i.e., the extended fixed component), and directly correspond to the actual pixel position in the image. The corrected coordinates are the final position coordinates of the pixel in the infrared image (second pixel coordinate system) after the affine transformation, and are the core basis for achieving pixel-level alignment between the two types of images. The specific process is as follows:

[0067] After matrix multiplication, the three components (x'', y'', w) of the new homogeneous coordinate vector are extracted, where w is the third component (usually close to 1, as there is no scaling during the transformation). The first two components, x'' and y'', are divided by w to obtain the two-dimensional non-homogeneous coordinates (x'' / w, y'' / w). These coordinates are the corrected coordinates of the pixel in the infrared image space. For example, the non-homogeneous coordinates corresponding to the new homogeneous coordinate vector (150, 240, 1) are (150, 240), directly representing the precise aligned position of the pixel in the infrared image. Finally, the inverse mapping method is used to traverse each grid position of the infrared image and interpolate the pixel value from the original visible light image. The inverse mapping method is a method of extracting pixel values ​​from the target image (infrared image). The reverse mapping method, which traces the source image (visible light image) back to its corresponding pixels, effectively avoids pixel overlap or gaps compared to forward mapping (from visible light to infrared). It ensures that each grid cell in the infrared image obtains a unique and continuous pixel value. Each grid cell position corresponds to each pixel position in the infrared image, and each grid cell has unique two-dimensional coordinates (u, v). The traversal process must cover the entire effective area of ​​the infrared image without omissions or repetitions. Interpolation is used to obtain pixel values ​​when the coordinates of the source image being traced back are non-integer. It calculates the target grid pixel value using the grayscale values ​​of surrounding integer coordinate pixels, ensuring a smooth transition of pixel values ​​and avoiding jagged edges or color block distortion. The specific process is as follows:

[0068] First, all grid positions in the infrared image are traversed sequentially, and the coordinates (u, v) of each grid are extracted. These coordinates are then used to construct a homogeneous vector (u, v, 1) after correction. This vector is multiplied by the inverse of the affine transformation matrix to obtain the original coordinates (x', y') of the grid in the visible light image. If (x', y') are integer coordinates, the pixel value of that coordinate in the visible light image is directly extracted as the pixel value of that grid in the infrared image. If (x', y') are non-integer coordinates, bilinear interpolation is used to extract the pixel values ​​of the four adjacent integer coordinates around (x', y'). A weighted average is calculated based on the positional weights of (x', y') relative to the four adjacent points, and this average is used as the pixel value of the corresponding grid in the infrared image. This process ensures complete filling of all grid pixel values ​​in the infrared image, guaranteeing accurate alignment of the two types of images at the pixel level.

[0069] After completing the reverse mapping correction of visible light pixel coordinates and the filling of infrared image raster pixel values, in order to improve data processing efficiency and ensure the synchronization and integrity of the two modal data, a parallel processing pipeline needs to be constructed to perform synchronous optimization processing on the two types of images. Then, a standardized multimodal dataset is generated by pixel-level alignment and merging. The specific implementation process is as follows:

[0070] First, a parallel processing pipeline is constructed, comprising a visible light image data stream channel and an infrared image data stream channel. A parallel processing pipeline refers to a hardware or software architecture consisting of two independent and synchronously operating data stream processing channels. The two channels share a clock synchronization signal, simultaneously receiving their respective raw image data and executing corresponding processing flows. This avoids the latency caused by single-channel serial processing and ensures consistent processing progress for both modalities. The visible light image data stream channel is a dedicated logical channel for processing visible light images, responsible for performing optimization operations such as geometric correction. The infrared image data stream channel is dedicated to processing infrared images, with its core task being to achieve synchronous processing while preserving the original thermal distribution characteristics. The specific construction process is as follows:

[0071] The data distribution module routes the timestamp-aligned visible light and infrared images to two independent channels. Each channel has a dedicated buffer unit, processing unit, and storage unit. A synchronization control module monitors the processing progress of both channels in real time. Once one channel completes processing the current frame, it waits for the other channel to complete synchronously before proceeding to the next frame, ensuring the timing consistency of the data frames between the two channels and laying a synchronous foundation for subsequent merging. In the visible light channel, a geometrically corrected image is generated using a reverse mapping method. This reverse mapping method, previously described as tracing the corresponding pixels in the visible light image from the infrared image grid, is used here for geometric correction of the visible light image. Its core purpose is to eliminate image geometric distortion caused by changes in the attitude and spatial offset of the acquisition device. The geometrically corrected image is a visible light image whose geometry is precisely matched to the infrared image after coordinate mapping and pixel value redistribution. Its pixel positions are spatially aligned with the infrared image, avoiding the influence of geometric deviations in the original visible light image on defect localization. The specific process is as follows:

[0072] After receiving the original visible light image, the visible light channel calls the optimized affine transformation matrix and iterates through each pixel position of the target geometric correction image according to the reverse mapping logic. It calculates the corresponding coordinates in the original visible light image, obtains the pixel value of that coordinate using bilinear interpolation, and fills it into the target position. The geometric correction of the entire image is completed pixel by pixel. During the correction process, the validity of the pixel coordinates is simultaneously verified to ensure there are no out-of-bounds or missing pixels. Finally, a geometrically regular visible light corrected image aligned with the infrared image space is output. Meanwhile, in the infrared channel, nearest neighbor sampling is used to preserve the original thermal distribution. Nearest neighbor sampling is a simple and efficient sampling method. Its core is to map the target pixel coordinates to the original infrared image and directly select the nearest integer coordinate pixel value as the target pixel value, without performing complex weighted calculations. The original thermal distribution is the core feature reflecting the surface temperature distribution of power equipment in the infrared image, including key information such as temperature gradient and hotspot locations. This information is crucial for defect identification and must be preserved to the greatest extent possible. The specific process is as follows:

[0073] After receiving the original infrared image, the infrared channel does not require complex geometric transformations (because the visible light image has already been mapped and aligned to the infrared image space). It only needs to adjust the resolution of the infrared image according to the synchronous rhythm of the parallel pipeline (if the resolution of the visible light corrected image is inconsistent with that of the infrared image, it is scaled according to the visible light resolution). During the scaling process, the nearest neighbor sampling method is used to traverse each grid position of the adjusted infrared image, calculate its corresponding coordinates in the original infrared image, and select the temperature value of the nearest integer coordinate as the pixel value of the grid. This ensures that the temperature distribution trend of the scaled image is completely consistent with the original image, without any additional thermal distribution distortion. After completing the independent processing of the two channels, the images output by the two channels are aligned and merged into a multimodal data frame. Aligning and merging by pixel coordinates means that for the same pixel coordinate (u, v), the gray value (texture information) of the geometrically corrected image output by the visible light channel at that coordinate is associated and stored with the temperature value (thermal information) of the image output by the infrared channel at that coordinate, forming a single data unit containing dual-modal information. Multimodal data frames are standardized data structures that integrate visible light texture features and infrared thermal distribution features. Each data frame corresponds to complete bimodal information from the same acquisition time and the same physical scene, facilitating the simultaneous extraction of both types of features by subsequent models. The specific merging process is as follows:

[0074] The output images of the two channels are traversed sequentially by pixel coordinates (from left to right, from top to bottom). For each coordinate (u, v), the grayscale value G of the visible light corrected image and the temperature value T of the infrared image are extracted. (u, v, G, T) are stored as a set of associated data. After all pixels have been processed, this associated data is encapsulated according to the image frame format to form a single multimodal data frame. This ensures that the bimodal information of each pixel within the frame corresponds one-to-one, without misalignment or missing information. Finally, a timestamp and device location tag are added to the multimodal data frame to form a dataset. The timestamp is the acquisition time corresponding to the data frame, recorded by the synchronization trigger module, ensuring the time sequence of the data frame is traceable. The device location tag is the spatial location information (such as latitude, longitude, and altitude) provided by the GPS or inertial measurement unit (IMU) on the acquisition device when the image frame is acquired. It is used to mark the spatial location of the power equipment where the defect is located. The specific process is as follows:

[0075] The acquisition timestamp corresponding to the multimodal data frame is retrieved from the synchronization trigger module, and the corresponding spatial location information is retrieved from the device positioning module. Both are appended as metadata to the header of the multimodal data frame. All multimodal data frames with appended metadata are stored in the specified storage medium in the order of acquisition time to form a structured multimodal dataset. The dataset is stored in a standard format and supports retrieval by timestamp and device location label, which facilitates subsequent reading by the pre-trained deep learning model and defect identification.

[0076] After generating multimodal data frames with additional timestamps and device location tags through a parallel processing pipeline, in order to further eliminate local subtle deformation deviations between the two modal images and achieve accurate matching of defect location and shape, image pyramid registration technology is needed to optimize the alignment effect. Then, weighted fusion is used to integrate the dual-modal features to generate a feature map that can intuitively reflect defect information. The specific implementation process is as follows:

[0077] First, the multimodal data frame is input into the image pyramid registration module. The mutual information entropy of the visible light and infrared images is calculated at multiple scales. The image pyramid registration module is a functional module that achieves accurate multi-scale registration based on the image pyramid structure. The image pyramid is a set of images of different resolutions generated by continuously downsampling (reducing the size by a fixed ratio) the original image, with the resolution gradually decreasing from the bottom to the top. This structure allows registration to quickly find a rough alignment position on the low-resolution image first, and then refine the adjustment on the high-resolution image, balancing registration efficiency and accuracy. Mutual information entropy is a quantitative indicator that measures the correlation of pixel grayscale distribution between two modal images, reflecting the degree of information sharing between the visible light and infrared images. The higher the mutual information entropy value, the better the alignment effect between the two images and the stronger the correspondence of defect positions. The specific process is as follows:

[0078] First, image pyramids (calibrated to 4 layers, with the bottom layer representing the original resolution and each layer downsampling by 2) are constructed for the visible light corrected image and infrared image in the multimodal data frame. Starting from the top layer of the pyramid (lowest resolution), the mutual information entropy of the two images at the current scale is calculated. The mutual information entropy value is derived by statistically analyzing the joint probability distribution and edge probability distribution of the corresponding pixel gray values ​​of the two images. Subsequently, the process is progressively moved down to the high-resolution image. Using the registration result of the previous layer as the initial position, the mutual information entropy is recalculated and the alignment position is fine-tuned to ensure that each layer can be further optimized based on the accuracy of the previous layer. Finally, a highly reliable mutual information entropy distribution is obtained on the bottom original resolution image, providing a basis for subsequent deformation optimization. Based on the multi-scale mutual information entropy distribution, the local deformation field is optimized using the gradient descent method. Gradient descent is an iterative optimization algorithm whose core is to gradually adjust the parameters along the negative gradient direction of the mutual information entropy loss function to minimize the loss and achieve the optimal solution of the objective function. Here, it is used to optimize the local deformation field parameters and improve the registration accuracy. The local deformation field is a continuous vector field describing the local pixel offset of an infrared image. Each pixel corresponds to a tiny offset vector, which characterizes the local deformation deviation of that pixel relative to its corresponding position in the visible light image. Unlike global affine transformation, the local deformation field can adapt to non-uniform deformations in different regions of the image (such as local image distortion caused by the curvature of the device surface). The specific optimization process is as follows:

[0079] With the goal of maximizing the mutual information entropy after multi-scale registration, a loss function is constructed (the loss value is the difference between the maximum mutual information entropy and the current mutual information entropy). The offset vector of the local deformation field is used as the optimization parameter. The gradient of the loss function with respect to each offset vector is calculated (i.e., the gradient direction is the direction in which the loss value decreases the fastest). According to a preset learning rate (calibrated to 0.01 to control the magnitude of each parameter adjustment and avoid overshoot or slow convergence), the offset vector of each pixel is updated along the negative gradient direction. This process is repeated iteratively until the loss value is less than a preset threshold (i.e., the mutual information entropy no longer increases significantly). The local deformation field obtained at this point is the optimal deformation field, which can accurately describe the deformation deviation of each local region in the infrared image. After optimizing the local deformation field, non-rigid deformation correction is performed on the infrared image. Non-rigid deformation correction is a correction method relative to rigid transformations (translation, rotation, etc., which do not change the internal structure of the image). It allows local regions of the image to undergo stretching, twisting, and other deformations. The core is to adjust the position of the infrared image pixels according to the optimal local deformation field, so that the local structure of the infrared image is completely matched with the visible light image, eliminating the alignment deviation caused by non-uniform deformation. The specific process is as follows:

[0080] Each pixel in the infrared image is traversed, and the offset vector corresponding to the pixel is obtained according to the optimal local deformation field. The original coordinates of the pixel are superimposed with the offset vector to obtain the target coordinates of the pixel after correction. If the target coordinates are not integers, the corresponding temperature value is obtained from the original infrared image using bilinear interpolation and filled into the target coordinate position. After completing the deformation adjustment of all local areas pixel by pixel, the corrected infrared image is output. At this time, the image is not only globally aligned with the visible light image, but also accurately matched in local details, ensuring that the position and shape of the same defect are completely consistent in the two modes. Finally, a weighted fusion algorithm is used to overlay the visible light texture and the infrared thermal image to generate a defect feature map with matching position and shape. The weighted fusion algorithm is a fusion method that linearly combines the pixel values ​​of the two modal images according to preset weights, highlighting the advantages of each. The advantage of the visible light image is its clear texture, which can accurately present the geometric shape of the defect. The advantage of the infrared image is its intuitive thermal distribution, which can highlight the location of overheating defects. Through weight allocation, the fused image can have both advantages at the same time. The visible light texture is the grayscale information in the visible light corrected image that reflects the surface structure and texture details of the power equipment. The infrared thermal image is the infrared image after non-rigid deformation correction, and its pixel values ​​correspond to the surface temperature of the equipment. It is usually presented in pseudo-color. The specific fusion process is as follows:

[0081] Preset fusion weights (experimentally calibrated, visible light texture weight is 0.4, and infrared thermal image weight is 0.6, preserving texture details while highlighting temperature features); traverse each pixel of the two modal images, multiply the grayscale value of the visible light image by the corresponding weight, and multiply the temperature value of the infrared thermal image (normalized to the 0-255 grayscale range) by the corresponding weight, and add the two to obtain the fused pixel value; encapsulate the fused values ​​of all pixels in image format to generate a defect feature map. In this feature map, the geometric shape of the defect is clearly outlined by the visible light texture, and the temperature anomaly of the defect is intuitively presented by the infrared thermal image. Moreover, the position and shape of the same defect in the image are completely consistent, providing accurate and comprehensive feature input for subsequent pre-trained deep learning models to automatically identify defects in power equipment.

[0082] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for automatic identification of electrical equipment defects based on multimodal visual data during power equipment inspection, characterized in that: Includes the following steps: S1. Collect multimodal visual data of power equipment, including visible light images and infrared images, and ensure that the acquisition equipment achieves timestamp alignment through the synchronization trigger module; S2. Establish a calibration mechanism for intramodal pixel coordinates and intermodal spatiotemporal mapping, including detecting shared physical feature points in visible light and infrared images as reference points. The reference points are inherent identifiable structural points on the surface of power equipment, automatically extracted by a corner detection algorithm. The spatial deviation vector of the pixel coordinates of the reference points in the visible light image and the pixel coordinates in the infrared image is calculated. Simultaneously, the time deviation is calculated based on the difference in acquisition timestamps. A mapping function is constructed. The mapping function maps the pixel coordinates of the visible light image to the spatial position of the infrared image and compensates for the time deviation using a linear interpolation method. Specifically, a two-dimensional transformation matrix is ​​generated. The two-dimensional transformation matrix is ​​adjusted based on the spatial deviation vector and the time deviation and is used to perform offset correction on each pixel coordinate. Finally, the mapping function is applied to perform real-time calibration on all pixels in the visible light and infrared images, and the calibrated multimodal dataset is output. S3. Based on the calibration results, the visible light image and infrared image are aligned to match the position and shape of the same defect in the two modalities. The calibrated multimodal dataset is then used to automatically identify power equipment defects through a pre-trained deep learning model.

2. The method for automatic identification of electrical equipment defects based on multimodal visual data for power inspection according to claim 1, characterized in that: Detecting shared physical feature points as reference points in visible light and infrared images specifically includes: By traversing the surface area of ​​power equipment through a multi-scale sliding window, the metal clips of insulator strings and the vertices of bolts and nuts in the visible light image are identified as candidate points. At the same time, the corresponding heat conduction abrupt change points are captured in the same spatial area of ​​the infrared image. The candidate points and the corresponding heat conduction abrupt change points are matched bidirectionally, and the paired points with a spatial distance of less than a preset threshold are selected as the reference points for shared physical feature points.

3. The method for automatic identification of electrical equipment defects based on multimodal visual data for power inspection according to claim 2, characterized in that: For visible light images, an improved Harris corner response function is used to calculate the pixel gradient covariance matrix. When both eigenvalues ​​of the pixel gradient covariance matrix are greater than the dynamic threshold, it is determined to be a structural corner. For infrared images, an adaptive thermal gradient operator is used to detect extreme points of temperature change rate along the normal of the device surface. The similarity between visible light corner points and infrared extreme points is calculated by a normalized cross-correlation algorithm, and the top few pairs of points with the highest similarity are retained as reference points.

4. The method for automatic identification of electrical equipment defects based on multimodal visual data for power inspection according to claim 3, characterized in that: Establish a first pixel coordinate system with the upper left corner of the visible light image as the origin and a second pixel coordinate system with the upper left corner of the infrared image as the origin. Subtract the horizontal and vertical coordinates of the same reference point in the two coordinate systems to obtain the horizontal and vertical deviation components of the point. Iterate through all reference points to generate a set of deviation vectors. After removing outlier vectors through a random sampling consensus algorithm, calculate the average deviation vector.

5. The method for automatic identification of electrical equipment defects based on multimodal visual data for power inspection according to claim 4, characterized in that: Read the time difference between visible light and infrared acquisition recorded by the synchronous trigger module, convert the time difference into spatial displacement compensation based on the device motion speed model, and superimpose the average deviation vector and the spatial displacement compensation to generate a total correction vector. Use the total correction vector as a parameter to construct an affine transformation function as a mapping function.

6. The method for automatic identification of electrical equipment defects based on multimodal visual data for power inspection according to claim 5, characterized in that: The visible light image is divided into several grid cells. The affine transformation function is applied with the grid vertices as control points. For any pixel point inside the grid, bilinear interpolation is performed according to its position weights relative to the four vertices to output the mapping coordinates of the point in the infrared image space. At the same time, the time deviation compensation is incorporated into the coordinate offset.

7. The method for automatic identification of electrical equipment defects based on multimodal visual data for power inspection according to claim 6, characterized in that: The total correction vector is decomposed into translation matrix components, and rotation matrix components are generated by combining the estimated equipment rotation angle. The translation matrix and rotation matrix are multiplied to generate an affine transformation matrix with six parameters. The matrix parameters are optimized by least squares to minimize the reprojection error of all reference points.

8. The method for automatic identification of electrical equipment defects based on multimodal visual data for power inspection according to claim 7, characterized in that: The original coordinates of each pixel in the visible light image are used to form a homogeneous coordinate vector, which is then multiplied by the affine transformation matrix. The non-homogeneous coordinates of the result are used as the corrected coordinates. The inverse mapping method is used to traverse each grid position of the infrared image and interpolate the pixel values ​​from the original visible light image.

9. The method for automatic identification of electrical equipment defects based on multimodal visual data for power inspection according to claim 8, characterized in that: A parallel processing pipeline containing visible light image data stream channels and infrared image data stream channels is constructed. In the visible light channel, a geometrically corrected image is generated using the reverse mapping method. In the infrared channel, nearest neighbor sampling is used to maintain the original thermal distribution. The images output from the two channels are aligned by pixel coordinates and merged into a multimodal data frame. A timestamp and device location label are added to form a dataset.

10. The method for automatic identification of electrical equipment defects based on multimodal visual data for power inspection according to claim 9, characterized in that: Multimodal data frames are input into the image pyramid registration module, and the mutual information entropy of visible light and infrared images is calculated at multiple scales. The local deformation field is optimized by gradient descent method, and non-rigid deformation correction is performed on the infrared image. Finally, a weighted fusion algorithm is used to overlay the visible light texture and the infrared thermal map to generate a defect feature map with matching position and shape.

Images