A method for detecting fires in outdoor oil-immersed transformers based on binocular 3D vision

Abstract

This invention discloses a method for detecting fires in outdoor oil-immersed transformers based on binocular 3D vision. Based on the imaging effect of an infrared thermal imager, the geometric imaging model of the infrared thermal imager is treated as a pinhole camera model, and a pinhole camera model is established. A checkerboard calibration plate is fabricated. Using external transmission infrared radiation, the specially made checkerboard calibration plate is attached to the surface of the heat source. Multiple infrared images of the calibration plate from different angles are acquired, and then calibration is performed using a camera calibration tool provided by OpenCV. The intersection center of the horizontal and vertical bars of a ladder is used as a feature reference point for calibrating the extrinsic parameters of the infrared thermal imager. The EPnP algorithm and ICP-based 3D-3D pose estimation are used to calculate the camera pose. After registration of the infrared and visible light images, abnormal high-temperature areas and target points are extracted. This invention solves the problem of large errors in single-band infrared fire detection methods due to interference from outdoor sunlight, humidity, and wind speed, and improves the accuracy of calibration.

Description

Technical Field

[0001] This invention belongs to the field of transformer technology, specifically relating to a fire detection method for outdoor oil-immersed transformers based on binocular three-dimensional vision. Background Technology

[0002] The main causes of fires and explosions in operating oil-immersed power transformers are: insulation damage, poor contact, lightning overvoltage, load short circuit, transformer overheating, and ignition by external sources. Internal electrical faults are the primary cause of transformer fires. Due to the cooling system, the main characteristic of an electrical fault inside the transformer is a rapid increase in the temperature of the transformer's outer surface. Therefore, transformer fire detection primarily monitors the outer surface temperature, using heat-sensing detectors, such as oil-point heat detectors, flame detectors, and cable-type linear heat detectors. These are installed on the transformer surface in a contact manner and have drawbacks including slow response speed, a single response threshold, weak electromagnetic interference resistance, susceptibility to external environmental interference, inability to accurately determine the location of the fire source, and impact on transformer operation and maintenance. Summary of the Invention

[0003] This invention relates to a binocular 3D vision fusion fire detection device for outdoor transformers. It solves the problem of using visual methods to detect temperature anomalies and locate the spatial position of abnormal areas on irregular transformer surfaces in complex outdoor environments. Outdoors, this invention fuses the thermal radiation image of the transformer with a high-sensitivity visible light video image, and through 3D reconstruction, it determines the location of the fire source while providing more comprehensive and clearer monitoring information. This enables early identification and warning of potential fire hazards in outdoor transformers, allowing for timely detection of fire sources to prevent accidents from occurring or escalating.

[0004] To achieve early identification and warning of fire hazards in outdoor transformers, the key to a fire detection system is to detect temperature anomalies and locate the spatial position of abnormal areas on the irregular surfaces of oil-immersed transformers, focusing on the transformer body, oil tank, bushing riser, and radiator in complex outdoor environments. The technical solution adopted in this invention is: a method for fire detection of outdoor oil-immersed transformers based on binocular three-dimensional vision, comprising the following steps:

[0005] Step S1: Model building: Based on the imaging effect of the infrared thermal imager, the geometric imaging model of the infrared thermal imager is regarded as a pinhole camera model, and a pinhole camera model is built.

[0006] Step S2, Intrinsic Parameter Calibration: Cut out white squares from the black and white checkerboard calibration board. Adhere the cut squares to the black squares using insulating foam adhesive to completely cover them, thus obtaining the checkerboard calibration board. Using external transmissive infrared radiation, attach the specially designed checkerboard calibration board to the surface of a heat source. The different insulating capabilities of the treated and untreated squares in the checkerboard calibration board result in different surface temperatures, which are clearly displayed in the infrared image. Acquire multiple infrared images of the calibration board from different angles, and then use the camera calibration tool provided by OpenCV for calibration.

[0007] Step S3, Acquisition of feature reference point coordinates: The intersection center of the ladder's horizontal and vertical bars is used as the feature reference point to calibrate the external parameters of the infrared thermal imager;

[0008] Step S4: Calculate the camera pose using the EPnP algorithm and ICP-based 3D-3D pose estimation: First, estimate the camera pose using the EPnP algorithm, then construct a problem to minimize the reprojection error to adjust the estimated value; after obtaining the pose of the feature reference points in the camera coordinate system, solve the ICP problem using linear algebra.

[0009] Step S5, Registration of Infrared and Visible Images: Depth information is acquired using a depth camera, and then the depth information corresponding to the infrared image is indirectly obtained by using the matching relationship between the infrared image and the depth camera image.

[0010] Step S6, Extraction of abnormal high temperature areas and target points: After recovering the three-dimensional spatial points corresponding to the infrared image pixels, if abnormal high temperature points are detected, their spatial locations are extracted.

[0011] Further optimization, the calibration steps in step S2 are as follows: Initialization, allocating storage space for spatial coordinates and pixel coordinates of corner points; reading a calibration board image and extracting corner points; determining whether corner point extraction was successful; if not, directly proceeding to determine whether all calibration images have been read; if so, calculating the sub-pixel coordinates of corner points, drawing the corner points, and then storing the corner point coordinates, followed by determining whether all calibration images have been read; if all calibration images have not been read, returning to "reading a calibration board image and extracting corner points"; if all calibration images have been read, calibrating the infrared thermal imager and outputting the results.

[0012] Further optimization involves step S4, where multiple control points are selected using principal component analysis based on feature reference points whose coordinates are known in the world coordinate system. The feature reference points are then represented using a weighted sum of the control points. The same representation is performed in the camera coordinate system, with the same weight allocation as in the world coordinate system. Based on the obtained weight allocation, the infrared thermal imager's intrinsic parameters, and the coordinates of the two-dimensional points in the image, the position of each virtual point in the camera coordinate system is calculated, thus obtaining the coordinates of the reference points in the camera coordinate system.

[0013] Further optimization involves using three-dimensional voxels of a certain size to divide the spatial region where the transformer is located, forming a voxel array, and then counting the abnormal high-temperature points exceeding the threshold in each voxel to determine whether the region is an abnormal high-temperature region.

[0014] Further optimization involves step S6, where the transformer is first enclosed by a cuboid box, and then the cuboid is divided into voxel arrays using small square voxels with side length l. After obtaining the voxel array, the specific steps for anomaly region division and target point determination are as follows:

[0015] (1) For any point p i Calculate the voxel lattice containing the point, and denote its voxel lattice as S. j The members of this voxel object include all spatial points it contains, all high-temperature anomaly spatial points, the highest temperature, and regional target points.

[0016] (2) For any point p i If its temperature is greater than the set threshold T s If it is, then add it to the anomalous space point member of its corresponding voxel lattice.

[0017] (3) For any single prime lattice S j If the number of outliers exceeds the set threshold N, the region is determined to be an outlier region. The target point of the region is the centroid of all outliers, and the highest temperature of the region is the temperature of the highest temperature point in the region.

[0018] The beneficial effects of this invention are as follows: It changes the current traditional method of directly contacting the transformer surface to detect fires in transformers, overcoming the shortcomings of traditional contact fire detectors such as slow response speed, single response threshold, weak resistance to electromagnetic interference, susceptibility to external environmental interference, inability to accurately determine the location of the fire source, and impact on transformer operation and maintenance. It also solves the problem of large errors in single-band infrared fire detection methods due to interference from outdoor sunlight, humidity, and wind speed. The invention uses a checkerboard calibration device with heating and light emission devices to calibrate infrared images, improving calibration accuracy and enabling precise determination of the fire source location. This is of great significance for the detection and early warning of transformer fire hazards in outdoor environments. Attached Figure Description

[0019] Figure 1 This is a flowchart of the present invention.

[0020] Figure 2 This is a schematic diagram of coordinate projection.

[0021] Figure 3 This is a flowchart of the internal parameter calibration process. Detailed Implementation

[0022] The present invention will now be described in further detail with reference to the accompanying drawings.

[0023] Reference Figure 1 A method for detecting fires in outdoor oil-immersed transformers based on binocular three-dimensional vision includes the following steps:

[0024] Step S1: Establish the model;

[0025] The geometric imaging model of an infrared thermal imager projects external three-dimensional points onto the internal imaging plane of the imager, constituting the imager's intrinsic parameters. Therefore, the form of the imager's intrinsic parameters depends on the selection of the geometric imaging model. Based on the imaging effect of the infrared thermal imager, the geometric imaging model is considered as a pinhole camera model. A pinhole camera model is established, and then the intrinsic parameters are calibrated based on this model. In the pinhole camera model, camera imaging is simplified to pinhole imaging; however, for ease of processing, the imaging plane is often moved to the front of the camera during mathematical processing.

[0026] Let the coordinates of point P in the world coordinate system and the camera coordinate system be P1, P2, P3, P4, P5, P6, P7, P8, P9, P1, P1, P1, P2, P1, P2, P1, P2, P1, P2, P3 ... w =[X w ,Y w Z w ] T and P c = [X c ,Y c Z c ] T Then the coordinates P in the world coordinate system w coordinates P in the camera coordinate system c The conversion formula is as follows:

[0027] P c =RP w +t (1)

[0028] Where R is the rotation matrix of the third-order orthogonal unit, and t is the translation vector.

[0029] Consider the projection of point P onto the center of a plane, where the projection plane is located at z = f (where f is the focal length in mm), such as... Figure 2As shown. p = [x, y] T P is the coordinate of the projection plane. c =[X c ,Y c Z c ] T These are coordinates in the camera coordinate system, which can be obtained from similarity relationships:

[0030]

[0031] Rewrite the above equation in matrix form using homogeneous coordinates:

[0032]

[0033] Let the pixel coordinates of point P be [μ,ν] T ,[μ0,ν0] T Let be the pixel coordinates of the camera center (optical center), a and b be the scaling factors from the image plane to the pixel plane along the x and y axes, respectively, and γ be the non-perpendicularity factor between the μ and ν axes in the pixel coordinate system. Then the relationship between the pixel coordinates and the image coordinates is:

[0034]

[0035] Substituting equations (3) and (4) into equation (1), we get:

[0036]

[0037] Let α = af, let β = bf, then the above equation can be simplified to:

[0038]

[0039] In equation (6): K is the intrinsic parameter matrix of the camera; D is the extrinsic parameter matrix of the camera.

[0040] Step S2: Internal parameter calibration;

[0041] Based on a black and white checkerboard calibration board, 28mm squares of white squares are cut out. These 28mm squares are then glued to the black squares using heat-insulating foam adhesive to completely cover them, thus obtaining the specially designed checkerboard calibration board of this embodiment. Using an external transmissive infrared radiation method, the specially designed checkerboard calibration board is attached to the surface of a heat source. The different heat insulation capabilities of the treated and untreated squares in the specially designed checkerboard calibration board result in different surface temperatures, which clearly displays the checkerboard pattern in the infrared image. Multiple infrared images of the calibration board are acquired from different angles, and then calibration is performed using the camera calibration tool provided by OpenCV. Figure 3As shown, the calibration steps are as follows: Initialization, allocate storage space for spatial coordinates and pixel coordinates of corner points; read a calibration board image and extract corner points; determine whether corner points have been successfully extracted; if not, proceed directly to determine whether all calibration images have been read; if so, calculate the sub-pixel coordinates of corner points, draw the corner points, and then store the corner point coordinates, then determine whether all calibration images have been read; if all calibration images have not been read, return to "read a calibration board image and extract corner points"; if all calibration images have been read, calibrate the infrared thermal imager and output the results.

[0042] Let P be the world homogeneous coordinate of the m-th point on the calibration plate. m =[X,Y,Z,1] T The homogeneous pixel coordinates of the corresponding two-dimensional camera plane are p m =[μ,ν,1] T Based on the pinhole camera model, we have:

[0043] sp m =K[R t]P m (7)

[0044] Where s is a non-zero scale factor, K is the camera's intrinsic parameter matrix, R is the rotation matrix of the third-order orthogonal unit, and t is the translation vector.

[0045] If we consider the chessboard plane as a plane with z = 0 in the world coordinate system, then we can obtain:

[0046]

[0047] [r1 r2 r3 t] is the column vector expansion of the matrix [R t];

[0048] In equation (8): H is the homography matrix, which, when expanded as column vectors, gives:

[0049] H=[h1 h2 h3]=λK[r1 r2 t] (9)

[0050] [h1 h2 h3] is the column vector expansion of vector [H];

[0051] In equation (9): λ is an arbitrary proportionality coefficient. From the orthogonality property of the rotation matrix R of the third-order orthogonal unit, we know that r1 and r2 are orthogonal, thus the constraint equations for the intrinsic parameters can be obtained:

[0052]

[0053] For ease of calculation, the matrix is ​​defined as follows:

[0054] in

[0055]

[0056] As we can see, B is a symmetric matrix with 6 valid elements. Let b be a vector composed of these valid elements. m for:

[0057] b m =[B 11 B 12 B 22 B 13 B 23 B 33 ] T (12)

[0058] It can be deduced that:

[0059]

[0060] In equation (13):

[0061]

[0062] [h i1 ,h i2 ,h i3 ]、[h j1 ,h j2 ,h j3 ] is the row vector expansion of vector [H].

[0063] The constraint equations can then be reformulated as:

[0064]

[0065] Assuming that n images from different angles are acquired, all intrinsic parameter constraint equations can be written as a large system of linear equations:

[0066] Vb m =0 (16)

[0067] In equation (16): V is a 2n×6 matrix. When n≥3, we can obtain b m The unique solution is usually found by Singular Value Decomposition (SVD), and then the intrinsic parameter matrix K can be obtained.

[0068] Step S3: Acquisition of feature reference point coordinates;

[0069] The extrinsic parameter calibration of an infrared thermal imager involves determining the imager's pose in the world coordinate system, which is related to the selection of the world coordinate system and the imager's own pose. Since infrared thermal imagers cannot obtain depth information, determining the camera's pose requires knowing the 3D spatial coordinates and corresponding 2D image coordinates of at least three reference points. Then, a Perspective-n-Point (PnP) problem is constructed and solved. The ladder grid on the transformer has a relatively regular shape, making it easy to measure its spatial position and relatively clear in the infrared image. Therefore, the intersection centers of the horizontal and vertical bars of the ladder are used as feature reference points, a total of 10, for the calibration of the infrared thermal imager's extrinsic parameters.

[0070] Step S4: Calculate the camera pose using the EPnP algorithm and ICP-based 3D-3D pose estimation.

[0071] First, the camera pose is estimated using the Efficient Perspective-n-Point (EPnP) method, and then the estimated value is adjusted by constructing a problem to minimize the reprojection error.

[0072] The core idea of ​​the EPnP algorithm is to represent the 3D coordinates of a feature reference point using a linear combination of multiple virtual control points. Using the feature reference point whose coordinates are known in the world coordinate system, four control points are selected through principal component analysis, and the feature reference point is represented as a weighted sum of these four control points. The same representation is performed in the camera coordinate system, with the same weight allocation for the feature reference point as in the world coordinate system. Then, based on the obtained weight allocation, the infrared thermal imager's intrinsic parameters, and the coordinates of the 2D points in the image, the position of each virtual point in the camera coordinate system is calculated, thus yielding the coordinates of the reference point in the camera coordinate system.

[0073] Let the coordinates of the feature reference point in the world coordinate system and the camera coordinate system be respectively. and The coordinates of the four control points in the world coordinate system and the camera coordinate system are as follows: and The reference points for each feature can then be represented as follows:

[0074]

[0075] In the formula, α ij Each marker point corresponds to 4 weighting coefficients a ij (j = 1, 2, 3, 4) and the sum is 1.

[0076] Assuming the extrinsic parameter of the infrared thermal imager is [Rt], then:

[0077]

[0078] Since the feature reference point can be represented as a weighted sum of control points, we can further obtain:

[0079]

[0080] Substituting equation (18) into equation (19), we get:

[0081]

[0082] As can be seen from the above formula, for a certain spatial point, the weights of each control point are the same in both coordinate systems. After solving for the coordinates of the control point in the camera coordinate system, the external parameters of the thermal imager can be further obtained.

[0083] Let u i (i = 1, ..., n) = [μ i ,ν i ] T Reference point P i The projection points of (i=1,…,n) onto the image plane (the xy plane under z=1) can be obtained from the camera's projection model:

[0084]

[0085] Two constraint equations can be obtained:

[0086]

[0087] In the above formula, besides the coordinates of the control points One point is unknown, while the rest are known. Combining the constraint equations corresponding to all points yields a linear equation:

[0088] Mc x =0 (23)

[0089] In equation (23): M represents a 2n*12 matrix, And x is in the right null space of M, therefore:

[0090]

[0091] In equation (24): N is M T The dimension of the M-kernel space, v i β is the right singular vector of M, with a singular value of 0. i Since these are undetermined coefficients, for the i-th control point:

[0092]

[0093] In equation (25): For the feature vector vk The i-th 3×1 sub-vector.

[0094] get Then the coordinates of the reference point can be calculated according to equation (20). Then, the camera pose is solved using the ICP method.

[0095] Once the poses of the feature reference points in the camera coordinate system are obtained, the pose estimation problem is transformed into a pose estimation problem based on a set of matched 3D points, which can typically be solved using ICP (Integrated Point Coding). There are two different methods for ICP: linear and nonlinear. When the 3D point pair matching is known, the nonlinear method can also yield an analytical solution, eliminating the need for iterative optimization. Therefore, a linear algebraic approach is used for ICP solving, with the following steps:

[0096] (1) Calculate the centroid of the reference point in the world coordinate system. And the centroid-free coordinate matrix A:

[0097]

[0098] (2) Calculate the centroid of the reference point in the camera coordinate system. And the centroid-free coordinate matrix B:

[0099]

[0100] (3) Define matrix H = B T A, and calculate the SVD decomposition of H: H = U∑V T .

[0101] (4) Calculate the rotation matrix R in the pose of the infrared thermal imager: R = UV T If |R|<0, then R[2,:]=-R[2,:].

[0102] (5) Calculate the translation vector t in the pose:

[0103] Step S5: Registration of infrared and visible light images:

[0104] After knowing the camera's intrinsic and extrinsic parameters, a depth camera is used to acquire depth information. Then, the depth information corresponding to the infrared image is indirectly obtained by using the matching relationship between the infrared image and the depth camera image.

[0105] Suppose that in the coordinate system of the infrared camera, the spatial coordinates of a point P are P[X,Y,Z]. T If the pixel in the infrared image is p1 and the pixel in the visible light image is p2, then the pixel positions of the two pixels are:

[0106]

[0107] In equation (29): K1 is the intrinsic parameter of the infrared thermal imager; K2 is the intrinsic parameter of the visible light camera; R,t is the relative pose relationship between the thermal imager coordinate system and the visible light coordinate system. Using homogeneous coordinates, the above equation can be rewritten as:

[0108]

[0109] Let x1 = K1 -1 p1, x2 = K2 -1 Substituting into equation (30), we get:

[0110] x2=Rx1+t (31)

[0111] Multiply both sides of the above equation by t, and then multiply by the left side. We can obtain:

[0112]

[0113] Substituting p1 and p2 again, we have:

[0114]

[0115] Let E = t∧R, which is called the essential matrix, and the above equation (33) is called the epipolar constraint. It can be seen that K1 and K2 are known. If E is solved, the pixel coordinates of the corresponding point in another image can be solved based on the pixel coordinates of a point in one image, thus realizing the registration between images.

[0116] Step S6, Extraction of abnormal high temperature areas and target points: After recovering the three-dimensional spatial points corresponding to the pixels in the infrared image, if abnormal high temperature points are detected, their spatial locations can be extracted.

[0117] In reality, if a transformer exhibits abnormal temperature, the high-temperature area extracted from the infrared image is likely to be an irregular shape, making it difficult to effectively and uniformly segment the abnormal area in three-dimensional space based on image-based methods. Therefore, this invention utilizes three-dimensional voxels of a certain size to segment the space containing the transformer, forming a voxel array. Then, it counts the abnormally high-temperature points exceeding a threshold within each voxel to determine whether the area is indeed an abnormally high-temperature region. This enables early identification and warning of potential fire hazards from outdoor transformers, allowing for timely detection of fire sources to prevent accidents from occurring or escalating.

[0118] First, the transformer is enclosed in a rectangular box. Then, the rectangular box is divided into voxel arrays using small square voxels with side length l. After obtaining the voxel array, the specific steps for anomaly region division and target point determination are as follows:

[0119] (1) For any point p i Calculate the voxel lattice containing the point, and denote its voxel lattice as S. j The members of this voxel object include all spatial points it contains, all high-temperature anomaly spatial points, the highest temperature, and regional target points.

[0120] (2) For any point p i If its temperature is greater than the set threshold T s If it is, then add it to the anomalous space point member of its corresponding voxel lattice.

[0121] (3) For any single prime lattice S j If the number of outliers exceeds the set threshold N, the region is determined to be an outlier region. The target point of the region is the centroid of all outliers, and the highest temperature of the region is the temperature of the highest temperature point in the region.

[0122] Example 1

[0123] Step S1: Model building: Treat the geometric imaging model of the infrared thermal imager as a pinhole camera model, build the pinhole camera model, and then calibrate the intrinsic parameters based on the pinhole camera model.

[0124] Step S2, Internal Parameter Calibration

[0125] When calibrating visible light cameras, a printed black and white checkerboard pattern is usually sufficient. However, for infrared thermal imaging cameras, ordinary black and white checkerboard calibration boards do not exhibit thermal radiation differences between the squares, resulting in a lack of checkerboard texture information in infrared images, making them unsuitable as calibration boards for infrared thermal imagers. Therefore, this embodiment improves upon Zhang's calibration method by fabricating a calibration board that displays a clear and distinct checkerboard texture in infrared images.

[0126] A standard black and white checkerboard calibration board has each checkerboard square measuring 28 millimeters in length and width. The specially designed checkerboard calibration board in this embodiment is created by cutting 28-millimeter squares of white squares from the standard black and white checkerboard calibration board. These 28-millimeter squares are then glued to the black squares using heat-insulating foam adhesive, completely covering them. Using an external transmissive infrared radiation method, the specially designed checkerboard calibration board is attached to a heat source surface. The different heat insulation capabilities of the treated and untreated squares in the specially designed checkerboard calibration board result in different surface temperatures, thus clearly displaying the checkerboard squares in the infrared image.

[0127] Multiple infrared images of the calibration board were acquired from different angles, and then calibration was performed using the camera calibration tool provided by OpenCV, with reference to... Figure 3The calibration steps are as follows: Initialization, allocate storage space for spatial coordinates and pixel coordinates of corner points; read a calibration board image and extract corner points; determine whether the corner points have been successfully extracted. If not, proceed directly to determine whether all calibration images have been read; if so, calculate the sub-pixel coordinates of the corner points, draw the corner points, and then store the corner point coordinates. After that, determine whether all calibration images have been read; if all calibration images have not been read, return to "read a calibration board image and extract corner points"; if all calibration images have been read, calibrate the infrared thermal imager and output the results.

[0128] The calibration result of the intrinsic parameter matrix is ​​shown in equation (34), and its corresponding back projection error is 0.456.

[0129]

[0130] 2. Calibration of external parameters of infrared thermal imager

[0131] Step S3: Acquisition of Feature Reference Point Coordinates: To perform extrinsic parameter calibration, it is necessary to extract the pixel coordinates and 3D coordinates in space of the intersection center of the ladder's horizontal and vertical bars. First, extract the pixel coordinates. Use the mouse to select 2D points near the reference point, then find nearby corner points and subpixelate them. Use the centroid coordinates of all corner points as the pixel coordinates of the actual feature reference points. The resulting pixel coordinates of the 10 reference points are: (705, 510), (706, 573), (707, 632), (707, 695), (707, 757), (763, 511), (762, 570), (763, 631), (763, 693), (763, 755).

[0132] The spatial coordinates of the feature reference points were obtained through actual measurements. Their coordinates are (2824, 2484, 2638), (2799, 2484, 2240), (2774, 2484, 1842), (2749, 2484, 1444), (2724, 2484, 1046), (2824, 2133, 2638), (2799, 2133, 2240), (2774, 2133, 1842), (2749, 2133, 1444), and (2724, 2133, 1046), in millimeters (mm).

[0133] Step S4: Calculate the camera pose using the EPnP algorithm based on ICP 3D-3D pose estimation;

[0134] Using the solvePnP tool in OpenCV, the extrinsic parameters were calibrated using the EPnP method, and the camera extrinsic parameters are as follows:

[0135]

[0136] t cw = [3900.93 3497.12 6020.03] T

[0137] The coordinates of the camera origin in the world coordinate system are:

[0138] t wc =-R cw -1 ·t cw = [-5589.39 2853.46 4230.19] T

[0139] Step S5: Registration of infrared and visible light images

[0140] Due to the significant differences between infrared and visible light images, including inconsistent grayscale and texture features, automatic feature reference point extraction and registration yields poor results. Therefore, a manual method is used for feature reference point extraction and matching. Obvious corner points are extracted from both the infrared image and the visible light image obtained from the depth camera as feature reference points and then paired.

[0141] In equation (33), let Solving for F allows us to complete the mapping transformation between image pixels. Next, we will solve for F using the eight-point method.

[0142] Consider a pair of matching points with homogeneous pixel coordinates p1 = [u1, v1, 1]. T p2 = [u2, v2, 1] T According to the epipolar constraint, we have:

[0143]

[0144] Based on the epipolar constraints of all matching points, the following system of linear equations can be obtained:

[0145]

[0146] This system of linear equations contains epipolar constraints between eight pairs of paired feature points, and the solution f lies in the null space of the coefficient matrix. Substituting the pixel coordinates of each feature reference point, the F matrix can be obtained as follows:

[0147]

[0148] After obtaining F, the relative pose relationships R and t between the infrared thermal imager and the depth camera can also be obtained using the SVD decomposition method. Since... Therefore, for any point p1 in the infrared image, the corresponding point p2 in the visible light image can be obtained. Based on the depth d of point p2, the coordinates of its corresponding spatial point under the depth camera can be obtained. Then, based on the relative pose between the infrared thermal imager and the depth camera and the extrinsic parameters of the infrared thermal imager, the coordinates of the point in the world coordinate system can be recovered.

[0149] Step S6: Extraction of abnormally high temperature areas and target points

[0150] First, the transformer is enclosed in a rectangular box. Then, the rectangular box is divided into voxel arrays using small square voxels with side length l. After obtaining the voxel array, the specific steps for anomaly region division and target point determination are as follows:

[0151] (1) For any point p i Calculate the voxel lattice containing the point, and denote its voxel lattice as S. j The members of this voxel object include all spatial points it contains, all high-temperature anomaly spatial points, the highest temperature, and regional target points.

[0152] (2) For any point p i If its temperature is greater than the set threshold T s If it is, then add it to the anomalous space point member of its corresponding voxel lattice.

[0153] (3) For any single prime lattice S j If the number of outliers exceeds the set threshold N, the region is determined to be an outlier region. The target point of the region is the centroid of all outliers, and the highest temperature of the region is the temperature of the highest temperature point in the region.

[0154] By observing abnormal areas, we can achieve early identification and warning of potential fire hazards in outdoor transformers, and promptly detect fire sources to prevent accidents from occurring or escalating.

[0155] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting fire of outdoor oil-immersed transformer based on binocular three-dimensional vision, characterized in that, Comprise the following steps: Step S1, model establishment: according to the imaging effect of the infrared thermal imager, the geometric imaging model of the infrared thermal imager is regarded as a pinhole camera model, and the pinhole camera model is established; Step S2, internal parameter calibration: on the basis of the black and white checkerboard calibration plate, the white square block is cut, the cut square block is adhered to the black square by the adiabatic foam to completely cover the black square, and the checkerboard calibration plate is obtained; by using the method of additional transmission type infrared radiation, the specially made checkerboard calibration plate is attached to the heat source surface, the checkerboard calibration plate is different in heat insulation capacity, which causes the temperature of the surface to be different, and the checkerboard is obviously displayed in the infrared image; collect multiple infrared images of the calibration plate under different angles, and then calibrate the camera by using the camera calibration tool provided by OpenCV; Step S3, collection of feature reference point coordinates: the intersection center of the ladder horizontal rod and the vertical rod is used as the feature reference point to calibrate the external parameter of the infrared thermal imager; Step S4, camera pose is calculated by using EPnP algorithm and 3D-3D pose estimation based on ICP: first, the camera pose is estimated by using EPnP algorithm, then the estimated value is adjusted by constructing the minimum reprojection error problem; after obtaining the pose of the feature reference point in the camera coordinate system, the ICP is solved by using linear algebra, Step S5, registration of infrared image and visible light image: the depth information is obtained by using the depth camera, and then the corresponding depth information of the infrared image is indirectly obtained by using the matching relationship between the infrared image and the depth camera image; Step S6, extraction of abnormally high temperature area and target point: after the three-dimensional space points corresponding to the infrared image pixel points are restored, if an abnormally high temperature point is monitored, the spatial position thereof is extracted; In step S6, firstly, a rectangular box is used to enclose the transformer, and then a box with a side length of... The cuboid is divided into small square voxels to obtain a voxel array; then, the abnormally high temperature points exceeding the threshold in each voxel are counted to determine whether the region is an abnormally high temperature region. The specific steps for dividing the abnormally high temperature region and determining the target points of the region are as follows: (1) For any point , calculate the voxel grid where the point is located, and record it as , the voxel grid object has all the spatial points it contains, all high temperature anomaly spatial points, the highest temperature, and the regional target point; (2) For any point , if its temperature is greater than the set threshold , it is added to the high temperature anomaly space point member of its corresponding voxel grid; (3) For any voxel grid , if the number of high temperature anomaly spatial points in it is greater than the set threshold , it is determined that the region is an abnormal high temperature region, the region target point is the centroid of all high temperature anomaly spatial points, and the region highest temperature is the temperature of the highest temperature spatial point.

2. The outdoor oil-immersed transformer fire detection method based on binocular three-dimensional vision according to claim 1, characterized in that, The calibration step of step S2 is as follows: initialization, assigning storage space for spatial coordinates and pixel coordinates of the corner point; reading a calibration plate image, extracting a corner point; judging whether the corner point is successfully extracted, if not, directly entering the judgment whether all calibration images are read; if yes, calculating the sub-pixel coordinates of the corner point, drawing the corner point, and then storing the corner point coordinates, and then judging whether all calibration images are read; if all calibration images are not read, return to "read a calibration plate image, extract a corner point", if all calibration images are read, calibrate the infrared thermal imager, and output the result.

3. The outdoor oil-immersed transformer fire detection method based on binocular three-dimensional vision according to claim 1, characterized in that, In step S4, the feature reference point with known coordinates in the world coordinate system is selected by principal component analysis method, and the feature reference point is represented by a plurality of control points in a weighted form; in the camera coordinate system, a plurality of control points are also selected by principal component analysis method, and the feature reference point is represented by a plurality of control points in a weighted form, and the weight distribution corresponding to the feature reference point is the same as that in the world coordinate system; then, according to the obtained weight distribution, the internal parameter of the infrared thermal imager and the coordinates of the two-dimensional points in the image, the positions of the virtual points in the camera coordinate system are calculated, and then the coordinates of the reference point in the camera coordinate system are obtained.

4. The outdoor oil-immersed transformer fire detection method based on binocular three-dimensional vision according to claim 1, characterized in that, In step S1, let the coordinates of point P in the world coordinate system and the camera coordinate system be and respectively, then the conversion formula from the coordinates in the world coordinate system to the coordinates in the camera coordinate system is as follows: （1） ； Wherein R is a three-order orthogonal unit rotation matrix, and t is a translation vector; Consider the central projection of point P onto a plane, the projection plane being at the position z = f, f being the focal length, is the projection plane coordinate, is the coordinate in the camera coordinate system, from the similarity relation: （2）； The above formula is rewritten in the form of matrix by using homogeneous coordinates: （3） ； Let the pixel coordinate of point P be , is the pixel coordinate of the camera center, a and b are the scale factors of the image plane to the pixel plane in the x and y directions, respectively, and γ is the non-perpendicular factor of the μ and v axes in the pixel coordinate system. The relationship between the pixel coordinate and the image coordinate is （4） ； Substitute formula (3) and formula (4) into formula (1), we get: （5）； Let α=af, let β=bf, then the above formula is simplified as: （6） ； In formula (6): is an intrinsic matrix of the camera; is an extrinsic matrix of the camera.

5. The outdoor oil-immersed transformer fire detection method based on binocular three-dimensional vision according to claim 4, characterized in that, In step S2, the world homogeneous coordinates of the mth point on the calibration board are denoted as The pixel homogeneous coordinates of the mth point on the two-dimensional camera plane corresponding thereto are denoted as According to the pinhole camera model, the following equation is obtained: （7）； wherein, is a non-zero scale factor, is an intrinsic matrix of the camera, is a rotation matrix of a third order orthogonal unit, is a translation vector; The chessboard plane is regarded as the z=0 plane in the world coordinate system, then we get: （8） ； is a column vector expansion of the matrix ] In formula (8), is a homography matrix, which is expanded as a column vector: （9） ； is the column vector expansion of the vector ] In formula (9), λ is an arbitrary proportional coefficient; because the rotation matrix R of the third-order orthogonal unit is orthogonal, thus and are orthogonal, the constraint equation of the internal reference is obtained: （10）； For the convenience of calculation, the following matrix definitions are made: wherein = (11); and (11). is a symmetric matrix with 6 effective elements, the vector of effective elements is defined as : （12）； The derivation is as follows: （13） ； In formula (13): （14） ； is a row vector expansion of the vector ] The constraint equation is re-expressed as: =0 （15）； Assuming the collected data includes Images from different angles are used to compose all the intrinsic parameter constraint equations into a large system of linear equations: （16）； In equation (16): It is The matrix; when At that time, The unique solution is obtained by using singular value decomposition to obtain the intrinsic parameter matrix. .

6. The outdoor oil-immersed transformer fire detection method based on binocular three-dimensional vision according to claim 5, characterized in that, In step S4, let the coordinates of the feature reference points in the world coordinate system and the camera coordinate system be and respectively, and let the coordinates of the 4 control points in the world coordinate system and the camera coordinate system be and respectively, then each feature reference point is represented as follows: （17）； In the formula, α ij 4 weighting coefficients corresponding to each mark point (j = 1, 2, 3, 4) and Σαj= 1. Assume that the extrinsic parameters of the infrared thermal imager are Then, we have: （18）； Since the feature reference point is expressed as a weighted sum of the control points, we further get: （19）； Substitute formula (18) into formula (19), we get: （20）； According to the above formula, for a certain space point, the weight corresponding to each control point is the same in two coordinate systems. After solving the coordinates of the control points in the camera coordinate system, we can further solve the external parameters of the thermal imager; Let is the reference point the projection point on the xy-plane at z = 1, then from the camera's projection model we get: （21）； Two constraint equations are obtained: （22） ； In the above formula, the coordinates of the control points are unknown, and the rest are known. Combining all the constraint equations corresponding to the points, we obtain the linear equation: （23） ； In formula (23): M represents a 2n*12 matrix, ; and In the right null space of the matrix （24）； In formula (24), N is the dimension of the kernel space, is the right singular vector of the matrix is an undetermined coefficient, so for the th control point, we have: (25)； In formula (25): is the eigenvector is the first 3 x 1 subvector of Obtained After that, the coordinates of the reference point are calculated according to formula (20) Then the camera pose is solved by using the ICP method.

7. The outdoor oil-immersed transformer fire detection method based on binocular three-dimensional vision according to claim 6, characterized in that, In step S4, the ICP is solved by using linear algebra, and the steps are as follows: (1) Calculate the reference point centroid in the world coordinate system and the decentered coordinate matrix : (27)； (2) Calculate the reference point centroid in the camera coordinate system and the de-centred coordinate matrix : (28)； (3) Define the matrix and compute the SVD decomposition of ;​ (4) Calculate the rotation matrix in the pose of the infrared thermal imager : ; if , correct the matrix R by negating its third column element; (5) calculating the translation vector in the pose : .

8. The outdoor oil-immersed transformer fire detection method based on binocular three-dimensional vision according to claim 7, characterized in that, In step S5, assuming that the spatial coordinates of a point P in the infrared camera coordinate system are the pixel point of P in the infrared image is the pixel point of P in the visible light image is the pixel positions of the two pixel points are: (29)； In formula (29): is the internal parameter of the infrared thermal imager; is the internal parameter of the visible light camera; is the relative pose relationship between the thermal imager coordinate system and the visible light coordinate system; the above formula is rewritten as: (30)； Let , , into equation (30) gives （31）； Take the exterior product of both sides of the above equation with and then left multiply to get: （32）； re-substituting and have: （33）； Let It is called essential matrix, and the above equation (33) is called epipolar constraint. and are known, if then the pixel coordinates of the corresponding point in the other image can be solved according to the pixel coordinates of a point in one image, and the registration between images is realized.

Images