A method and device for generating a power grid inspection image under a cross-view angle, a terminal device and a computer readable storage medium

Abstract

The application discloses a kind of generation methods, device, terminal equipment and computer readable storage medium of power grid inspection image under cross view angle, belong to image data processing technical field.Method includes: obtaining power grid inspection image sequence and its attitude parameter and positioning data;According to attitude parameter and positioning data, camera relative pose matrix is solved, three-dimensional reconstruction is generated three-dimensional point cloud;According to virtual camera pose, two-dimensional background grid is projected and constructed by three-dimensional point cloud;Three-dimensional normal vector is extracted, and physical light direction and observation direction vector are combined to calculate physical light and shade change feature;Original texture pixel and physical light and shade change feature are fused and filled to two-dimensional background grid, and power grid inspection image under cross view angle is generated.The application solves the problem that existing two-dimensional image expansion method is difficult to restore real three-dimensional space perspective change and is easy to produce light and shadow physical distortion and direction conflict, leading to inaccurate generation of cross view angle power grid inspection image.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to a method, apparatus, terminal device, and computer-readable storage medium for generating power grid inspection images from multiple perspectives. Background Technology

[0002] With the development of drone inspection and artificial intelligence technology, computer vision-based defect recognition models have been widely used in the safety monitoring of power grid equipment. The effective amplification of high-quality, multi-view defect image samples will directly affect the training accuracy and generalization ability of intelligent recognition models.

[0003] While existing 2D image augmentation methods can quickly generate samples from new perspectives, they struggle to reproduce the deep background perspective changes caused by the drone's movement in real 3D space when processing complex UAV power grid inspection images. Furthermore, they are prone to physically distorting and directionally conflicting the light and shadow on defect surfaces, leading to severe perspective artifacts in the generated images. This results in geometric inconsistencies and optical distortions in the generated cross-perspective power grid inspection images. Therefore, improving the accuracy of cross-perspective generation of power grid inspection images has become a pressing technical problem in this field. Summary of the Invention

[0004] This invention provides a method, apparatus, terminal device, and computer-readable storage medium for generating power grid inspection images from multiple perspectives. It can solve the problem that existing two-dimensional image augmentation methods are difficult to restore the true three-dimensional spatial perspective changes and are prone to light and shadow physical distortion and directional conflicts, resulting in inaccurate generation of power grid inspection images from multiple perspectives.

[0005] An embodiment of the present invention provides a method for generating power grid inspection images from multiple perspectives, comprising: acquiring a sequence of power grid inspection images containing a target object captured by an image acquisition device mounted on a drone, and simultaneously acquiring attitude parameters and positioning data corresponding to the power grid inspection image sequence collected by an attitude sensor and a satellite positioning receiver; calculating the camera relative pose matrix of adjacent frames in the power grid inspection image sequence based on the attitude parameters and the positioning data, and performing three-dimensional reconstruction of the power grid inspection image sequence in combination with the camera relative pose matrix to generate a three-dimensional point cloud containing the target object; acquiring a set virtual camera pose, projecting the three-dimensional point cloud onto a two-dimensional virtual view plane based on the virtual camera pose, and constructing a two-dimensional background mesh corresponding to the target object; extracting the three-dimensional normal vector of the target object surface from the three-dimensional point cloud, and calculating the physical brightness and darkness variation characteristics of the target object under the two-dimensional virtual view plane based on the three-dimensional normal vector, the acquired physical environment lighting direction vector, and the observation direction vector determined by the virtual camera pose; The original texture pixels of the target object are extracted from the power grid inspection image sequence. The original texture pixels are fused with the physical brightness and darkness variation features to generate mapped texture pixels. The mapped texture pixels are then filled into the two-dimensional background grid to generate a power grid inspection image from a cross-view perspective.

[0006] Furthermore, based on the attitude parameters and the positioning data, the camera relative pose matrix of adjacent frames in the power grid inspection image sequence is calculated, including: converting the attitude parameters into an initial rotation matrix in the world coordinate system and converting the positioning data into an initial translation vector in the world coordinate system; calculating the initial rotation difference matrix and the initial translation difference vector between adjacent frames based on the initial rotation matrix and the initial translation vector, and concatenating them to form an initial relative pose matrix; performing singular value decomposition on the rotation components in the initial relative pose matrix to obtain a left singular matrix, a singular value diagonal matrix, and a right singular matrix; The modified singular value matrix is ​​obtained by forcibly setting all non-zero diagonal elements of the singular value diagonal matrix to 1. The matrix is ​​then reconstructed by combining the left singular matrix and the right singular matrix to obtain the orthogonal rotation matrix. Based on the velocity vector provided by the positioning data and the time difference between the corresponding frames, the inertial prediction translation vector is calculated, and the candidate translation vector derived from the orthogonal rotation matrix is ​​weighted and averaged with the inertial prediction translation vector to obtain the fused translation vector; the camera relative pose matrix is ​​generated by combining the orthogonal rotation matrix and the fused translation vector.

[0007] Further, the candidate translation vector derived from the orthogonal rotation matrix is ​​weighted and averaged with the inertial prediction translation vector to obtain a fused translation vector, including: extracting the three-dimensional position accuracy factor from the positioning data; calculating the inertial prediction weight based on the three-dimensional position accuracy factor and a preset normalization coefficient; calculating the difference between a constant 1 and the inertial prediction weight, and using the difference as a candidate translation weight; weighting the candidate translation weight with the candidate translation vector to obtain a first weighted translation vector; weighting the inertial prediction weight with the inertial prediction translation vector to obtain a second weighted translation vector; and combining the first weighted translation vector with the second weighted translation vector to obtain the fused translation vector.

[0008] Furthermore, the power grid inspection image sequence is reconstructed in three dimensions using the camera relative pose matrix to generate a three-dimensional point cloud containing the target object. This includes: extracting pixel-level dense optical flow correspondences between adjacent frames in the power grid inspection image sequence; performing triangulation calculations using the camera relative pose matrix and the pixel-level dense optical flow correspondences to obtain an initial three-dimensional point cloud; back-projecting the initial three-dimensional point cloud onto the image plane based on the camera relative pose matrix to generate back-projected two-dimensional coordinates, and calculating the coordinate residual distance between the back-projected two-dimensional coordinates and the actual two-dimensional coordinates obtained through the pixel-level dense optical flow correspondences; removing outliers whose coordinate residual distance is greater than a preset threshold based on the coordinate residual distance to obtain a filtered three-dimensional point cloud; obtaining the spatial region of interest (ROI) of the target object in the power grid inspection image sequence using a target detection algorithm, saving the three-dimensional data points of the filtered three-dimensional point cloud located within the ROI, and performing density clustering processing on the saved three-dimensional data points to generate a three-dimensional point cloud containing the target object.

[0009] Further, based on the virtual camera pose, the 3D point cloud is projected onto a 2D virtual view plane to construct a 2D background mesh corresponding to the target object, including: projecting the 3D point cloud onto the 2D virtual view plane based on the virtual camera pose to obtain a set of 2D projection coordinate points; performing Deloni triangulation on the set of 2D projection coordinate points to generate an initial planar topology composed of multiple triangular facets; reading the 3D spatial direct adjacency relationship of each point in the set of 2D projection coordinate points in the 3D point cloud; traversing each triangular facet in the initial planar topology and checking whether the two 2D projection coordinate points connected by the three edges of each triangular facet have a 3D spatial direct adjacency relationship in the 3D point cloud; if the two endpoints of any edge do not have a direct adjacency relationship in 3D space, the corresponding triangular facet is determined to be a degenerate facet, and the triangular facet corresponding to the degenerate facet is removed from the initial planar topology, and all the remaining valid triangular facets constitute the 2D background mesh.

[0010] Furthermore, the three-dimensional normal vector of the target object surface is extracted from the three-dimensional point cloud, and the physical brightness and darkness variation characteristics of the target object under the two-dimensional virtual view plane are calculated based on the three-dimensional normal vector, the obtained physical environment lighting direction vector, and the observation direction vector determined by the virtual camera pose. This includes: taking each three-dimensional data point in the three-dimensional point cloud as the center, searching for a set of adjacent three-dimensional data points within a preset search radius. Calculate the decentralized coordinate matrix and covariance matrix of the adjacent 3D data point set, and perform eigenvalue decomposition on the covariance matrix, taking the normalized eigenvector corresponding to the smallest eigenvalue as the 3D normal vector; parse the exchangeable image file format information of the power grid inspection image sequence, extract the shooting timestamp and geographical location data from the exchangeable image file format information, and calculate the solar azimuth angle and solar zenith angle based on the shooting timestamp and geographical location data, and determine the physical environment illumination direction vector based on the solar azimuth angle and solar zenith angle; extract the spatial coordinates of the virtual camera optical center from the virtual camera pose, and construct a direction vector with the virtual camera optical center as the starting point and the corresponding 3D data point as the ending point, taking the direction vector as the observation direction vector; substitute the 3D normal vector, the physical environment illumination direction vector, and the observation direction vector into the half-range vector illumination model to calculate the reflection intensity value of the target object, and take the reflection intensity value as the physical brightness change feature.

[0011] Furthermore, the original texture pixels are fused with the physical brightness variation features to generate mapped texture pixels, and the mapped texture pixels are filled into the two-dimensional background grid to generate a power grid inspection image from a cross-view perspective. This includes: multiplying the color channel values ​​of the original texture pixels by the reflection intensity values ​​corresponding to the physical brightness variation features to obtain multiple fusion channel values, and recombinating the multiple fusion channel values ​​into initial fusion pixel values; performing a color truncation operation on the initial fusion pixel values, forcing values ​​exceeding the upper limit of the color space to the maximum value of the color space boundary, thereby generating the mapped texture pixels; using a bilinear interpolation algorithm, filling the mapped texture pixels into the grid vertices of the two-dimensional background grid that coincide with the two-dimensional projection coordinates, and performing rasterization rendering on the two-dimensional background grid after pixel filling to obtain a two-dimensional pixel array, and outputting the two-dimensional pixel array as the power grid inspection image from a cross-view perspective.

[0012] Another embodiment of the present invention provides a device for generating power grid inspection images from multiple perspectives, comprising: a data acquisition module, a three-dimensional reconstruction module, a mesh construction module, a feature calculation module, and an image generation module; The data acquisition module is used to acquire a sequence of power grid inspection images containing target objects taken by an image acquisition device mounted on a drone, and simultaneously acquire attitude parameters and positioning data corresponding to the power grid inspection image sequence collected by an attitude sensor and a satellite positioning receiver. The 3D reconstruction module is used to calculate the camera relative pose matrix of adjacent frames in the power grid inspection image sequence based on the pose parameters and the positioning data, and to perform 3D reconstruction of the power grid inspection image sequence in combination with the camera relative pose matrix to generate a 3D point cloud containing the target object. The mesh construction module is used to obtain the set virtual camera pose, and project the three-dimensional point cloud onto the two-dimensional virtual view plane according to the virtual camera pose to construct a two-dimensional background mesh corresponding to the target object. The feature calculation module is used to extract the three-dimensional normal vector of the surface of the target object from the three-dimensional point cloud, and calculate the physical brightness and darkness variation features of the target object under the two-dimensional virtual view plane based on the three-dimensional normal vector, the obtained physical environment lighting direction vector and the observation direction vector determined by the virtual camera pose. The image generation module is used to extract the original texture pixels of the target object from the power grid inspection image sequence, fuse the original texture pixels with the physical brightness and darkness variation features to generate mapped texture pixels, and fill the mapped texture pixels into the two-dimensional background grid to generate a power grid inspection image from a cross-view perspective.

[0013] Another embodiment of the present invention provides a terminal device, including: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements a method for generating power grid inspection images from a cross-view perspective according to the present invention.

[0014] Another embodiment of the present invention provides a computer-readable storage medium, including: a stored computer program, wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the method for generating power grid inspection images from a cross-view perspective according to the present invention.

[0015] The embodiments of the present invention have the following beneficial effects: This invention provides a method for generating power grid inspection images from multiple perspectives. The method involves acquiring a sequence of power grid inspection images containing a target object, captured by an image acquisition device mounted on a drone, and simultaneously acquiring attitude parameters and positioning data corresponding to the power grid inspection image sequence from an attitude sensor and a satellite positioning receiver. Based on the attitude parameters and positioning data, the method calculates the camera relative pose matrix of adjacent frames in the power grid inspection image sequence, and uses the camera relative pose matrix to perform 3D reconstruction of the power grid inspection image sequence, generating a 3D point cloud containing the target object. Furthermore, a set virtual camera pose is obtained, and the 3D point cloud is projected onto a 2D image based on the virtual camera pose. A virtual view plane is used to construct a two-dimensional background mesh corresponding to the target object. The three-dimensional normal vector of the target object's surface is extracted from the three-dimensional point cloud. Based on the three-dimensional normal vector, the obtained physical environment lighting direction vector, and the observation direction vector determined by the virtual camera pose, the physical brightness and darkness variation characteristics of the target object under the two-dimensional virtual view plane are calculated. Finally, the original texture pixels of the target object are extracted from the power grid inspection image sequence. The original texture pixels are fused with the physical brightness and darkness variation characteristics to generate mapped texture pixels, which are then filled into the two-dimensional background mesh, ultimately generating a power grid inspection image from a cross-view perspective.

[0016] By adopting the above technical solution, this invention addresses the shortcomings of existing two-dimensional image augmentation methods, which struggle to restore realistic three-dimensional spatial perspective changes and are prone to light and shadow distortions and directional conflicts. It introduces a three-dimensional reconstruction mechanism assisted by multi-source sensor data and a virtual perspective mesh projection mechanism to restore the spatial geometry of the real physical world at the algorithm level, eliminating background perspective distortion artifacts during cross-view generation. Simultaneously, based on the joint computational mechanism of three-dimensional normal vectors extracted from three-dimensional point clouds and the physical lighting direction and viewing direction, the reconstruction of the target object's defect texture is transformed from passive two-dimensional plane pixel translation to directional physical rendering fusion that conforms to the laws of real three-dimensional optical reflection. This avoids physical direction conflicts in the shadows on the defect surface, achieving high-fidelity restoration of geometric perspective and optical characteristics in cross-view images and improving the accuracy of cross-view generation of power grid inspection images. Attached Figure Description

[0017] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a method for generating power grid inspection images from multiple perspectives, according to an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the apparatus structure for a method of generating power grid inspection images from multiple perspectives, provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0022] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0023] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0024] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0025] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0026] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0027] To address the problem that existing two-dimensional image augmentation methods struggle to accurately reproduce true three-dimensional spatial perspective changes and are prone to producing physical distortions of light and shadow and directional conflicts, leading to inaccurate generation of cross-view power grid inspection images, an embodiment of the present invention provides a method for generating cross-view power grid inspection images, comprising: Step S1: Acquire a sequence of power grid inspection images containing the target object captured by the image acquisition device mounted on the UAV, and simultaneously acquire the attitude parameters and positioning data corresponding to the power grid inspection image sequence collected by the attitude sensor and the satellite positioning receiver; In a preferred embodiment, acquiring a power grid inspection image sequence containing the target object captured by an image acquisition device mounted on a UAV, and simultaneously acquiring attitude parameters and positioning data corresponding to the power grid inspection image sequence collected by an attitude sensor and a satellite positioning receiver, specifically includes: continuously capturing and extracting the power grid inspection image sequence using the image acquisition device at a preset acquisition frame rate; acquiring the attitude parameters and positioning data characterizing the current spatial physical state of the UAV in real time through the attitude sensor and the satellite positioning receiver; and synchronously extracting multi-source data in the time domain dimension from the power grid inspection image sequence, the attitude parameters, and the positioning data based on a hardware pulse and timestamp joint alignment mechanism.

[0028] Preferably, the power grid inspection image sequence containing the target object is obtained by the following steps using an image acquisition device mounted on a drone: The drone flies along a preset transmission line inspection route, and the onboard image acquisition device (specifically, an imaging payload such as a visible light high-definition mirrorless camera, industrial camera, or multispectral camera) continuously captures images at a fixed preset frame rate (e.g., a fixed frequency value within the range of 24 to 60 frames per second). The obtained power grid inspection image sequence consists of three consecutively acquired images, defined chronologically as the preceding frame, the middle frame, and the subsequent frame. The power grid inspection image sequence contains the target object, which specifically refers to key equipment and its local defects on distribution or transmission towers, including but not limited to common power grid equipment defects such as insulator damage, broken strands in the line, corrosion of hardware, displacement of vibration dampers, or missing tower bolts. During this acquisition process, the hardware imaging parameters (such as camera focal length, aperture size, exposure time, and white balance) of three consecutive frames are forced to remain consistent to avoid drastic changes in pixel brightness caused by automatic exposure, thereby ensuring the consistency of image photometric and geometric scale in subsequent extraction of pixel-level dense optical flow and 3D reconstruction.

[0029] Preferably, the attitude parameters and positioning data corresponding to the power grid inspection image sequence are obtained through the following steps: The attitude sensor (specifically, its structure includes an onboard inertial measurement unit (IMU) and a gimbal multi-axis angle encoder) collects the attitude parameters of the UAV and camera gimbal in real time. To facilitate the subsequent construction of a three-dimensional rotation matrix, the attitude parameters are represented in Euler angles, specifically including the roll angle (ϕ), pitch angle (θ), and yaw angle (ψ) of the gimbal in the world coordinate system. Simultaneously, the satellite positioning receiver (preferably an RTK-GNSS positioning module based on real-time dynamic carrier phase differential technology) collects the positioning data of the UAV in space in real time. The positioning data represents the absolute three-dimensional spatial position of the UAV in a standard geodetic coordinate system (such as the WGS84 geographic coordinate system), specifically including longitude (Lon), latitude (Lat), and altitude (Altitude). Alt). Simultaneously, the positioning data also extracts and retains the UAV's three-dimensional velocity vector at the corresponding moment (for subsequent inertial prediction calculations) and the three-dimensional position dilution of precision (PDOP, used to quantify the impact of the current satellite spatial geometry distribution on positioning errors; a smaller PDOP value indicates higher current positioning accuracy).

[0030] Preferably, the synchronous acquisition of multi-source data is achieved through the following steps based on a hardware pulse and timestamp joint alignment mechanism: In the multi-source heterogeneous sensor fusion algorithm, to ensure the physical rigor of the subsequent motion recovery structure (SfM) calculation, it is essential to ensure strict alignment of visual data and sensor motion state data on the time axis. This embodiment specifically employs a method combining hardware pulse triggering (PPS triggering) and software timestamp interpolation. Specifically, the data output frequency of the attitude sensor and satellite positioning receiver is configured to be equal to the preset acquisition frame rate, or an integer multiple of the preset acquisition frame rate. At the instant the image acquisition device shutter opens and triggers image exposure, the system hardware cascade outputs a high-level pulse signal, synchronously triggering the attitude sensor and satellite positioning receiver to latch the current transient data, and assigning a system timestamp based on the same unified global clock to each frame of image and the attitude parameters and positioning data extracted at this time. If the actual sampling time of different sensors has a small sub-millisecond time deviation due to hardware bus transmission delay, the timestamp timg at the center of the image shutter exposure is used as the absolute reference; in the time series of the attitude parameters, the spherical linear interpolation algorithm (Slerp) is used to calculate the time sequence of the attitude parameters. The compensated attitude angle at time timg is calculated. Within the time series of the positioning data, compensated latitude, longitude, and altitude coordinates precisely corresponding to time timg are calculated using a Newton polynomial or linear interpolation algorithm. Finally, a strictly aligned multi-source data tuple combination is generated, meaning each frame uniquely corresponds to a set of gimbal attitude parameters and three-dimensional spatial coordinates accurate to the same timestamp, thus transforming the asynchronous multi-source data stream into a synchronous input array.

[0031] In this embodiment, the system-level initial pose drift error caused by the time asynchrony between camera exposure delay, gimbal high-frequency rotation, and UAV flight is effectively eliminated through the strict timestamp synchronization mechanism for multi-source heterogeneous sensors and the hardware-software collaborative compensation processing described above. By accurately acquiring three consecutive frames containing the target defect region, as well as Euler angles and high-precision 3D positioning coordinates that are forcibly aligned in the spatiotemporal physical dimension, not only is strong consistency of data association between the visual stream and the sensor stream ensured, providing a time-difference-free underlying physical input for subsequent accurate calculation of the rigid body relative motion matrix between adjacent frames, but the robustness of the overall system in data acquisition under complex outdoor electromagnetic interference environments and UAV high-speed cruise jitter conditions is also significantly improved. This lays a reliable source data foundation for the subsequent construction of a high geometrically realistic 3D point cloud of defects.

[0032] Step S2: Based on the attitude parameters and the positioning data, calculate the camera relative pose matrix of adjacent frames in the power grid inspection image sequence, and combine the camera relative pose matrix to perform three-dimensional reconstruction of the power grid inspection image sequence to generate a three-dimensional point cloud containing the target object. In a preferred embodiment, calculating the camera relative pose matrix of adjacent frames in the power grid inspection image sequence based on the attitude parameters and the positioning data includes: converting the attitude parameters into an initial rotation matrix in the world coordinate system and converting the positioning data into an initial translation vector in the world coordinate system; calculating an initial rotation difference matrix and an initial translation difference vector between adjacent frames based on the initial rotation matrix and the initial translation vector, and concatenating them to form an initial relative pose matrix; performing singular value decomposition on the rotation components in the initial relative pose matrix to obtain a left singular matrix, a singular value diagonal matrix, and a right singular matrix. The modified singular value matrix is ​​obtained by forcibly setting all non-zero diagonal elements of the singular value diagonal matrix to 1. The matrix is ​​then reconstructed by combining the left singular matrix and the right singular matrix to obtain the orthogonal rotation matrix. Based on the velocity vector provided by the positioning data and the time difference between the corresponding frames, the inertial prediction translation vector is calculated, and the candidate translation vector derived from the orthogonal rotation matrix is ​​weighted and averaged with the inertial prediction translation vector to obtain the fused translation vector; the camera relative pose matrix is ​​generated by combining the orthogonal rotation matrix and the fused translation vector.

[0033] Preferably, the fused translation vector is obtained by weighted averaging of the candidate translation vector derived from the orthogonal rotation matrix and the inertial prediction translation vector, including: extracting the three-dimensional position accuracy factor from the positioning data; calculating the inertial prediction weight based on the three-dimensional position accuracy factor and a preset normalization coefficient; calculating the difference between a constant 1 and the inertial prediction weight, and using the difference as a candidate translation weight; weighting the candidate translation weight with the candidate translation vector to obtain a first weighted translation vector; weighting the inertial prediction weight with the inertial prediction translation vector to obtain a second weighted translation vector; and combining the first weighted translation vector with the second weighted translation vector to obtain the fused translation vector.

[0034] Preferably, the method of performing three-dimensional reconstruction of the power grid inspection image sequence by combining the camera relative pose matrix to generate a three-dimensional point cloud containing the target object includes: extracting the pixel-level dense optical flow correspondence between adjacent frames in the power grid inspection image sequence; performing triangulation calculation using the camera relative pose matrix and the pixel-level dense optical flow correspondence to obtain an initial three-dimensional point cloud; back-projecting the initial three-dimensional point cloud onto the image plane according to the camera relative pose matrix to generate back-projected two-dimensional coordinates, and calculating the coordinate residual distance between the back-projected two-dimensional coordinates and the actual two-dimensional coordinates obtained through the pixel-level dense optical flow correspondence; removing outliers whose coordinate residual distance is greater than a preset threshold according to the coordinate residual distance to obtain a filtered three-dimensional point cloud; obtaining the spatial region of interest of the target object in the power grid inspection image sequence through a target detection algorithm, saving the three-dimensional data points of the filtered three-dimensional point cloud located in the spatial region of interest, performing density clustering processing on the saved three-dimensional data points, and generating a three-dimensional point cloud containing the target object.

[0035] Preferably, the attitude parameters are converted into an initial rotation matrix in the world coordinate system through the following steps, and the positioning data is converted into an initial translation vector in the world coordinate system: To unify the spatial reference of multi-source data, this embodiment pre-establishes the world coordinate system as the Northeast-Sky (ENU) Cartesian coordinate system, i.e., the X-axis points due east, the Y-axis points due north, and the Z-axis points to the zenith. For the attitude parameters (including the gimbal roll angle ϕ, pitch angle θ, and yaw angle ψ), the initial rotation matrix is ​​constructed using the ZYX Euler angle rotation sequence. Specifically, the rotation matrix around the Z-axis is calculated respectively. Rotation matrix around the Y-axis and the rotation matrix around the X-axis Multiply the three together (i.e.) The output three-dimensional orthogonal matrix is ​​the initial rotation matrix in the world coordinate system. For the positioning data (including longitude, latitude, and altitude), the Gauss-Kruger map projection algorithm is used to convert the geodetic coordinates on the Earth ellipsoid into spatial rectangular coordinates (X, Y, Z) in the northeast celestial rectangular coordinate system. This three-dimensional spatial rectangular coordinate is then constructed into a three-dimensional column vector, which is the initial translation vector in the world coordinate system.

[0036] Preferably, the initial rotation difference matrix and initial translation difference vector between adjacent frames are calculated through the following steps, and the initial relative pose matrix is ​​generated: The initial rotation matrix and initial translation vector corresponding to the middle frame image in the power grid inspection image sequence are used as the reference coordinate system for relative motion. For the preceding or subsequent frame (defined as the current frame), extract the transpose of the initial rotation matrix corresponding to the intermediate frame; perform matrix multiplication on the transpose matrix and the initial rotation matrix corresponding to the current frame to obtain the initial rotation difference matrix ΔR describing the angle transformation between the two frames; simultaneously, calculate the three-dimensional vector difference between the initial translation vector of the current frame and the initial translation vector of the intermediate frame, and multiply the transpose matrix and the three-dimensional vector difference to obtain the initial translation difference vector Δt describing the positional displacement between the two frames; finally, concatenate the initial rotation difference matrix ΔR and the initial translation difference vector Δt using a spatial transformation matrix (i.e., augment the 3×3 ΔR and the 3×1 Δt into a 3×4 matrix) to output the initial relative pose matrix.

[0037] Preferably, the rotation components in the initial relative pose matrix are subjected to singular value decomposition (SVD) to reconstruct an orthogonal rotation matrix through the following steps: Due to wind disturbances and sensor measurement errors during UAV flight, the extracted initial rotation difference matrix often does not fully satisfy the orthogonality of a rigid body rotation matrix (i.e., it contains non-rigid deformation errors). Therefore, the 3×3 rotation components in the initial relative pose matrix are extracted, and singular value decomposition (SVD) is performed on them, decomposing them into a left singular orthogonal matrix U, a diagonal matrix Σ containing singular values, and the transpose of a right singular orthogonal matrix. Subsequently, all non-zero diagonal elements in the singular value diagonal matrix Σ are forcibly modified to the value 1, thereby transforming it into an identity matrix (i.e., a modified singular value matrix); the transposes of the left singular orthogonal matrix U and the right singular orthogonal matrix are then... Matrix reconstruction is performed by direct multiplication.

[0038] Furthermore, the determinant of the reconstructed matrix is ​​calculated. If the determinant value is -1 (indicating that a mirror reflection that does not conform to the laws of physics has occurred), the last column of the right singular orthogonal matrix V is multiplied by -1 and then multiplied again to ensure that the determinant value is positive 1. The output matrix that strictly satisfies the rigid body orthogonal constraint is the orthogonal rotation matrix, thereby eliminating non-rigid distortion in pose transformation in the mathematical dimension.

[0039] Preferably, the inertial prediction translation vector and the corresponding weighting coefficients are calculated through the following steps to obtain the fused translation vector: First, extract the instantaneous UAV three-dimensional velocity vector synchronously recorded in the positioning data. And calculate the time difference between the system timestamps of two adjacent image frames. The three-dimensional velocity vector The time difference Perform scalar multiplication to calculate the inertial predicted translation vector tpred based on the principles of physical kinematics; simultaneously, multiply the transpose of the reconstructed orthogonal rotation matrix by the initial translation difference vector to calculate the candidate translation vector derived from the purely visual perspective. To achieve optimal fusion of the two, the three-dimensional position accuracy factor (PDOP), which characterizes the quality of satellite spatial geometric distribution, is extracted from the positioning data. A weighting allocation formula is then constructed. Where β is a preset normalization coefficient (preferably set to 0.2 to 0.5 based on empirical values); the inertial prediction weight is calculated using this formula. When the PDOP value is smaller (i.e., the satellite signal quality is better and the positioning is more accurate), the calculated inertial prediction weight is larger. Next, the difference between the constant 1 and the inertial prediction weight (i.e., 1 − 1) is calculated. ), and use it as the candidate translation weight. ; the candidate translation weights With the candidate translation vector Perform a scalar-vector multiplication and weighting to obtain the first weighted translation vector; then apply the inertial prediction weights... With the inertial prediction translation vector Perform multiplication and weighting to obtain the second weighted translation vector; then perform vector summation on the two (i.e., ... The fused translation vector is then output. Subsequently, the orthogonal rotation matrix and the fused translation vector are concatenated and augmented again to generate the camera relative pose matrix that eliminates non-rigid translation deviations and has been corrected by multi-source fusion.

[0040] Preferably, the following steps are used to perform triangulation calculations and point cloud filtering and clustering using the camera relative pose matrix: For the power grid inspection image sequence, using the intermediate frame image as the reference frame, an image block matching algorithm (such as calculating the normalized cross-correlation coefficient NCC) is used to extract matching points pixel by pixel in the search window of adjacent frames, generating the pixel-level dense optical flow correspondence representing the pixel coordinate offset. Combining the camera intrinsic parameter matrix obtained by the image acquisition device before calibration and the calculated camera relative pose matrix, a projection matrix between the reference frame and adjacent frames is constructed. For each pair of two-dimensional pixels with a pixel-level dense optical flow correspondence, a linear overdetermined system of equations based on the projection matrix is ​​constructed. The singular value decomposition method is used to solve the system of equations to calculate the corresponding homogeneous coordinates in three-dimensional space, thereby transforming the two-dimensional pixels into three-dimensional coordinates and obtaining the initial three-dimensional point cloud. To filter out erroneous 3D points caused by optical flow mismatch, the camera relative pose matrix and camera intrinsic parameter matrix are used to reproject each 3D coordinate point in the initial 3D point cloud onto the corresponding 2D image plane to obtain the reprojected 2D coordinates. The Euclidean distance (i.e., coordinate residual distance) between the reprojected 2D coordinates and the actual 2D coordinates obtained by the previous optical flow matching is calculated. A preset threshold is defined as the maximum allowable reprojection error pixel value (e.g., set to a threshold of 1 to 3 pixels). 3D data points with coordinate residual distances greater than this preset threshold are forcibly removed, completing the initial filtering under epipolar geometric constraints, and obtaining the filtered 3D point cloud. Subsequently, the reference frame image is input into a pre-trained convolutional neural network object detection model (such as the YOLO series algorithms), which outputs the minimum bounding rectangle enclosing the target object. This minimum bounding rectangle is expanded spatially by a preset multiple (e.g., expanded outward by 1.5 times the length and width parameters) and projected into 3D space to form the region of interest (ROI). Useless background points outside the region of interest are removed, and density-based spatial clustering algorithms (such as the DBSCAN algorithm, based on a preset neighborhood search radius and a minimum number of core points) are performed on the remaining 3D data points within the region. Through clustering, low-density discrete floating noise points are separated and removed, ultimately outputting a high-purity 3D point cloud containing only the target object and closely connected background physical structures.

[0041] In this embodiment, the aforementioned anti-interference reconstruction mechanism for low-level pose calculation and multi-level noise reduction mechanism for 3D reconstruction achieve significant technical effects. First, in the pose calculation stage, instead of relying on a single geometric calculation or pure hardware measurement, an SVD orthogonal constraint and an inertial velocity fusion algorithm based on the PDOP quality factor are introduced. This reduces the sensor non-rigid deformation error caused by strong wind disturbance and complex electromagnetic interference of the UAV from a mathematical perspective, achieving high-precision restoration of the real rigid body motion trajectory. Second, in the point cloud generation stage, through the joint collaborative filtering of epipolar residual constraints, target detection ROI space clipping, and density clustering, optical flow mismatch noise and discrete suspended objects caused by natural environment (such as sudden changes in light, background leaves, etc.) are eliminated. This ensures that the reconstructed 3D point cloud has extremely high spatial geometric fidelity and topological purity, providing a reliable underlying data foundation for the subsequent accurate construction of the background mesh and the extraction of real physical illumination normal vectors.

[0042] Step S3: Obtain the set virtual camera pose, and project the three-dimensional point cloud onto the two-dimensional virtual view plane according to the virtual camera pose to construct a two-dimensional background mesh corresponding to the target object; In a preferred embodiment, projecting the 3D point cloud onto a 2D virtual view plane based on the virtual camera pose to construct a 2D background mesh corresponding to the target object includes: projecting the 3D point cloud onto the 2D virtual view plane based on the virtual camera pose to obtain a set of 2D projection coordinate points; performing Deloni triangulation on the set of 2D projection coordinate points to generate an initial planar topology composed of multiple triangular facets; reading the 3D spatial direct adjacency relationship of each point in the set of 2D projection coordinate points in the 3D point cloud; traversing each triangular facet in the initial planar topology and checking whether there is a 3D spatial direct adjacency relationship between the two 2D projection coordinate points connected by the three edges of each triangular facet in the 3D point cloud; if there is no direct adjacency relationship between the two endpoints of any edge in 3D space, the corresponding triangular facet is determined to be a degenerate facet, and the triangular facet corresponding to the degenerate facet is removed from the initial planar topology, and all the remaining valid triangular facets constitute the 2D background mesh.

[0043] Preferably, the three-dimensional point cloud is projected onto the two-dimensional virtual view plane based on the virtual camera pose to obtain a set of two-dimensional projection coordinate points: The virtual camera pose is a virtual observation target pose pre-set in three-dimensional space for generating cross-view power grid inspection images, specifically embodied in a virtual rotation matrix and a virtual translation vector in the world coordinate system. First, the camera intrinsic parameter matrix of the original image acquisition device is extracted, and the virtual rotation matrix, the virtual translation vector, and the camera intrinsic parameter matrix are combined to construct a virtual perspective projection matrix describing the imaging geometry model of the virtual camera. Subsequently, the spatial three-dimensional coordinates of each three-dimensional data point in the three-dimensional point cloud are extracted, converted into homogeneous coordinate form, and matrix multiplication is performed between the homogeneous coordinates and the virtual perspective projection matrix to calculate the two-dimensional homogeneous projection coordinates of the corresponding point on the two-dimensional virtual view plane. Furthermore, the two-dimensional homogeneous projected coordinates are converted into actual two-dimensional pixel coordinates through homogeneous division, and all converted two-dimensional pixel coordinates are collected to obtain the set of two-dimensional projected coordinate points.

[0044] Preferably, the initial planar topology is generated by performing Deloni triangulation on the set of two-dimensional projected coordinate points through the following steps: To establish a geometric representation of surface continuity between discrete pixels, the coordinates of each two-dimensional pixel in the set of two-dimensional projected coordinate points are used as discrete vertex inputs, and the Deloni triangulation algorithm is used to perform meshing connections. During this process, the algorithm determines vertex connections based on the Deloni empty circle property (i.e., the circumcircle of any triangle formed does not contain any other discrete vertices in the set) and the principle of maximizing the minimum angle. Through this operation, the originally isolated discrete vertices are connected to each other on the two-dimensional virtual view plane, generating the initial planar topology composed of multiple seamlessly connected and non-overlapping triangular facets, providing a basic two-dimensional spatial skeleton for subsequent texture pixel filling.

[0045] Preferably, the direct adjacency relationship in 3D space for each point in the set of 2D projected coordinate points is read in the 3D point cloud through the following steps: Since the depth dimension information collapses during the projection of 3D data points onto the 2D plane, to restore the true physical space constraints, for any 2D projected coordinate point in the set of 2D projected coordinate points, the index mapping matrix established during the projection calculation stage is used to trace back and lock its unique corresponding 3D original data point in the 3D point cloud. In the original physical coordinate system of the 3D point cloud, with the 3D original data point as the center, the K-nearest neighbor (KNN) algorithm or a spherical neighborhood search algorithm based on a preset physical distance threshold is used to calculate the Euclidean distance between the 3D original data point and the surrounding 3D data points. 3D data points that satisfy the distance constraint condition (i.e., the Euclidean distance is within a set range representing the continuity of the object surface) are determined to have the direct adjacency relationship in 3D space with the center point, and the adjacency results corresponding to all points are stored as an adjacency relationship mapping table.

[0046] Preferably, the initial planar topology is traversed and degenerate faces are removed to form the two-dimensional background mesh through the following steps: Each triangular facet in the initial planar topology is traversed sequentially; for the currently traversed triangular facet, the three edges constituting the geometric contour of the triangular facet are extracted, and the two-dimensional projection coordinates of the two ends of each edge are obtained respectively. The adjacency mapping table generated above is called to check the two original three-dimensional data points corresponding to the two two-dimensional projection coordinate points associated with each edge in the physical three-dimensional space, and to verify whether the two have a direct adjacency relationship in the three-dimensional space. If it is detected that the two endpoints of any edge in the triangular facet do not have a direct adjacency relationship in the real three-dimensional space (for example, due to the occlusion and overlap of the viewing angle, the foreground insulator data points that are far apart in space and the background data points behind are abnormally close on the two-dimensional projection plane and are incorrectly connected by the Deloni algorithm), then the edge is determined to be a topologically abnormal edge, and the triangular facet containing the topologically abnormal edge is marked as the degenerate facet. The regions marked as degenerate faces are physically disconnected and removed from the initial planar topology. After the traversal and inspection process is fully completed, all valid triangular faces that satisfy the three-dimensional spatial physical continuity constraints after spatial logic filtering are extracted. These valid triangular face sets are then jointly constructed to output the two-dimensional background mesh.

[0047] In this embodiment, through the aforementioned virtual projection mechanism for 3D point clouds and the mesh topology correction process based on physical 3D adjacency constraints, the 3D point cloud is dimensionality-reduced and projected, and Deloni subdivision is implemented, establishing a meshed transition from discrete spatial points to a continuous imaging plane. Furthermore, addressing the spatial clipping phenomenon easily caused by traditional triangulation relying solely on 2D coordinates (i.e., forcibly stitching unrelated occlusions with the background in 3D depth, leading to structural errors in subsequent image generation), this solution introduces a cross-verification mechanism based on underlying 3D physical adjacency relationships. By forcibly enforcing 3D-2D cross-dimensional connectivity constraints during the patch retention stage, it precisely cuts off and eliminates degenerate patches caused by visual overlap, maintaining the geometric consistency between the constructed mesh and the physical boundaries of the real power grid equipment. This invention not only endows the 2D background mesh with strict spatial physical correctness attributes but also eliminates structural interference noise in subsequent optical rendering and texture interpolation processes, providing a solid underlying geometric architecture support for high-fidelity output of cross-view power grid inspection images.

[0048] Step S4: Extract the three-dimensional normal vector of the target object surface from the three-dimensional point cloud, and calculate the physical brightness and darkness variation characteristics of the target object under the two-dimensional virtual view plane based on the three-dimensional normal vector, the obtained physical environment lighting direction vector and the observation direction vector determined by the virtual camera pose. In a preferred embodiment, the three-dimensional normal vector of the target object surface is extracted from the three-dimensional point cloud, and the physical brightness and darkness variation characteristics of the target object under the two-dimensional virtual view plane are calculated based on the three-dimensional normal vector, the obtained physical environment lighting direction vector and the observation direction vector determined by the virtual camera pose. This includes: taking each three-dimensional data point in the three-dimensional point cloud as the center, searching for a set of adjacent three-dimensional data points within a preset search radius. Calculate the decentralized coordinate matrix and covariance matrix of the adjacent 3D data point set, and perform eigenvalue decomposition on the covariance matrix, taking the normalized eigenvector corresponding to the smallest eigenvalue as the 3D normal vector; parse the exchangeable image file format information of the power grid inspection image sequence, extract the shooting timestamp and geographical location data from the exchangeable image file format information, and calculate the solar azimuth angle and solar zenith angle based on the shooting timestamp and geographical location data, and determine the physical environment illumination direction vector based on the solar azimuth angle and solar zenith angle; extract the spatial coordinates of the virtual camera optical center from the virtual camera pose, and construct a direction vector with the virtual camera optical center as the starting point and the corresponding 3D data point as the ending point, taking the direction vector as the observation direction vector; substitute the 3D normal vector, the physical environment illumination direction vector, and the observation direction vector into the half-range vector illumination model to calculate the reflection intensity value of the target object, and take the reflection intensity value as the physical brightness change feature.

[0049] Preferably, the following steps are performed to find a set of adjacent 3D data points within a preset search radius, centered on each 3D data point in the 3D point cloud: Since the 3D point cloud is composed of a massive number of discrete spatial coordinate points, in order to accurately characterize the local microscopic geometric morphology of the target object (such as insulators, hardware, etc.), a 3D spherical search space is established with each 3D data point in the 3D point cloud as its center. The preset search radius is a physical distance threshold set a priori based on the spatial resolution of the 3D point cloud and the geometric scale of the target object. To improve search efficiency, a spatial index data structure based on a KD-tree (K-Dimensional Tree) or an octree is constructed. Using this spatial index data structure, all valid 3D data points falling within the 3D spherical search space are quickly traversed and extracted. These 3D data points are then merged and output as the set of adjacent 3D data points corresponding to the current center point, thereby providing sufficient geometric constraint points for fitting the local surface.

[0050] Preferably, the decentralized coordinate matrix and covariance matrix of the adjacent 3D data point set are calculated through the following steps, and the 3D normal vector is obtained by eigenvalue decomposition: First, the mean vector of the 3D spatial coordinates of all points in the adjacent 3D data point set (i.e., the geometric centroid of the local region) is calculated; the mean vector is subtracted from the coordinate vector of each 3D data point in the set to eliminate the translation offset of the absolute spatial position, thereby obtaining the decentralized coordinate matrix. Subsequently, the decentralized coordinate matrix is ​​multiplied by its own transpose matrix and divided by the number of degrees of freedom of the points to calculate the 3×3 dimensional covariance matrix representing the variance characteristics of the local point cloud distribution. Further, the covariance matrix is ​​subjected to eigenvalue decomposition using the principal component analysis (PCA) algorithm to solve for three non-negative eigenvalues ​​and three orthogonal eigenvectors corresponding to the three eigenvalues. According to the geometrical properties of principal component analysis, the eigenvalue with the smallest value represents the direction with the smallest variance of the 3D data point distribution, that is, the normal direction perpendicular to the local micro-tangent plane. Therefore, the feature vector corresponding to the minimum feature value is extracted, and vector normalization is performed on it (i.e., the magnitude of the vector is scaled to 1). To ensure the global consistency of the normal vector direction (to avoid the normal vector pointing inwards), the normalized feature vector is checked by performing a dot product with the viewpoint direction vector. If the dot product is negative, the feature vector is reversed. Finally, the vector after adjusting the direction and normalizing is used as the three-dimensional normal vector of the surface where the current three-dimensional data point is located.

[0051] Preferably, the physical environment illumination direction vector is determined by parsing the power grid inspection image sequence and calculating the solar angle through the following steps: To restore the true physical light and shadow distribution, the exchangeable image file format information (EXIF metadata) of each image file in the power grid inspection image sequence is read. The shooting timestamp (accurate to the second in Coordinated Universal Time, UTC) at the instant of image exposure is accurately parsed from the exchangeable image file format information, along with the geographical location data (including longitude, latitude, and altitude) written by the airborne GNSS module. The extracted shooting timestamp and geographical location data are input into an astronomical solar position algorithm. Based on the Earth's rotation, orbital parameters, and local meridian offset, the relative position angle of the sun in the sky dome at the corresponding geographical coordinates at the shooting time is calculated. Specifically, the output is the solar azimuth angle (describing the direction of orientation on the horizontal plane) and the solar zenith angle (describing the angle of altitude in the vertical direction). Using the conversion formula from spherical coordinates to spatial rectangular coordinates, the solar azimuth angle and the solar zenith angle are converted into three-dimensional unit vectors in the world coordinate system. This unit vector precisely points to the spatial orientation of the only main light source (the sun) in the actual shooting scene, and is thus determined as the physical environment illumination direction vector.

[0052] Preferably, the coordinates are extracted from the virtual camera pose and the observation direction vector is constructed through the following steps: The virtual camera pose includes rotation and translation parameters for simulating new perspective observation. By extracting the translation parameters, the absolute spatial position of the virtual camera in the world coordinate system is obtained, and this position is used as the spatial coordinates of the virtual camera's optical center. For the 3D data points currently involved in the lighting calculation, the spatial coordinates of the virtual camera's optical center are used as the starting vector coordinates, and the spatial 3D coordinates of the 3D data points are used as the ending vector coordinates. A vector subtraction operation is performed between the two (i.e., the starting coordinates are subtracted from the ending coordinates, so that the vector direction points from the observed point to the observer), and the magnitude of the difference vector is normalized. The generated 3D unit direction vector is defined as the observation direction vector. This vector adaptively deflects according to the spatial distribution of different 3D data points, thereby accurately reflecting the perspective observation angle of the virtual camera for different parts of the target object's surface.

[0053] Preferably, the reflection intensity value is calculated by substituting it into the half-range vector lighting model through the following steps, so as to serve as the physical brightness change feature: obtain the three-dimensional normal vector (denoted as N), the physical environment lighting direction vector (denoted as L), and the observation direction vector (denoted as V) calculated by the above steps.

[0054] First, the ambient lighting direction vector L and the observation direction vector V are added together, and the result is normalized to calculate the half-range vector (denoted as H) representing the direction of the bisector of the angle between the ray and the line of sight. Then, the half-range vector lighting model (Blinn-Phong Reflection Model) is constructed and substituted into this model. This model defines light reflection as the linear superposition of ambient light reflection, diffuse reflection, and specular highlight reflection, and its mathematical expression is: ; in, For ambient light intensity, The intensity of the main light source; , , These are the ambient light reflectance, diffuse reflectance, and specular reflectance coefficients preset based on the materials of the power grid equipment (such as ceramic insulators, metal fittings, etc.); α is the preset specular index coefficient characterizing surface smoothness; (N⋅L) and (N⋅H) represent the dot product operation between vectors, and the dot product result is truncated to not less than zero. Substituting each vector parameter into this mathematical expression and performing a summation calculation, the resulting comprehensive output value I is the reflection intensity value of the current 3D data point under specific lighting and observation conditions. This reflection intensity value accurately quantifies the optical distribution law from light to dark on the object surface, recording it as the physical light and dark variation characteristic accompanying the data point.

[0055] In this embodiment, significant technical effects are achieved through the joint calculation mechanism of principal component normal vector extraction based on local geometry, solar position inversion combined with real EXIF ​​data, and half-range vector illumination model. Traditional two-dimensional image augmentation methods, lacking depth-dimensional optical analysis, often result in shadow adhesion or specular highlight directions violating physical laws when the viewing angle changes. This solution maps the position of the main light source (the solar angle accurately inverted using timestamps and latitude / longitude) that actually occurs in the physical world into a three-dimensional virtual space and uses strict geometric topological relationships to obtain the micro-tangent plane normal vector. By forcibly incorporating the virtual camera's observation direction and half-range vector into the parameter execution flow, the system can adaptively calculate the brightness and darkness intensity that conforms to the real optical reflection theorem. Through this explicit light and shadow reconstruction mechanism, not only are accurate light-dark boundaries and specular highlights given to the target objects in the generated image, but the logical contradictions in light and shadow caused by drastic changes in viewing angle are also eliminated. This provides an effective core computational foundation for outputting cross-view power grid inspection images with high optical fidelity and physical consistency in the subsequent fusion stage.

[0056] Step S5: Extract the original texture pixels of the target object from the power grid inspection image sequence, fuse the original texture pixels with the physical brightness and darkness variation features to generate mapped texture pixels, and fill the mapped texture pixels into the two-dimensional background grid to generate a power grid inspection image from a cross-view perspective.

[0057] In a preferred embodiment, the original texture pixels are fused with the physical brightness variation features to generate mapped texture pixels, and the mapped texture pixels are filled into the two-dimensional background grid to generate a power grid inspection image from a cross-view perspective. This includes: multiplying the color channel values ​​of the original texture pixels by the reflection intensity values ​​corresponding to the physical brightness variation features to obtain multiple fusion channel values; recombinating the multiple fusion channel values ​​into initial fusion pixel values; performing a color truncation operation on the initial fusion pixel values, forcing values ​​exceeding the upper limit of the color space to the maximum boundary value of the color space, thereby generating the mapped texture pixels; using a bilinear interpolation algorithm, filling the mapped texture pixels into the grid vertices of the two-dimensional background grid that coincide with the two-dimensional projection coordinates, and performing rasterization rendering on the two-dimensional background grid after pixel filling to obtain a two-dimensional pixel array; and outputting the two-dimensional pixel array as the power grid inspection image from a cross-view perspective.

[0058] Preferably, the following steps are used to multiply the color channel values ​​of the original texture pixel by the reflection intensity value corresponding to the physical brightness variation feature, and then recombine them into the initial fused pixel value: First, based on the index relationship of the two-dimensional projection coordinate points in the three-dimensional point cloud, the original texture pixel at the corresponding position of the target object in the reference frame image is traced back and extracted. The three independent color channel values ​​of the original texture pixel in a standard digital image color space (such as RGB color space) are extracted, specifically the red channel value, green channel value, and blue channel value. Then, the single scalar corresponding to the physical brightness variation feature, i.e., the reflection intensity value, is obtained from the output of the half-path vector lighting model. The extracted red channel value, green channel value, and blue channel value are multiplied by the reflection intensity value to calculate multiple fused channel values ​​modulated by physical illumination (i.e., new red, green, and blue channel values ​​that have undergone brightness variation due to physical illumination). Finally, the multiple fused channel values ​​are cascaded and assembled according to the channel arrangement order of the original color space, and recombined into a three-dimensional pixel vector containing multi-channel color information. This three-dimensional pixel vector is then output as the initial fused pixel value.

[0059] Preferably, the initial fused pixel values ​​are subjected to a color truncation operation to generate the mapped texture pixels through the following steps: Since specular highlight reflection and directional diffuse reflection components are superimposed in the lighting model, the values ​​of each fused channel in the initial fused pixel values ​​may exceed the physical representation limit of conventional digital image storage formats in the mathematical domain (e.g., numerical overflow due to highlight accumulation, resulting in visual overexposure). Therefore, a color truncation operation (Clamp operation) is introduced for boundary protection. Specifically, the upper and lower threshold values ​​of the color space set by the current target image encoding format are read. For standard 8-bit deep digital images, the preset maximum value of the color space boundary is set to 255, and the minimum value is 0. Each fused channel value in the initial fused pixel values ​​is iterated one by one to determine whether it exceeds the upper limit of the color space; if a channel value is detected to be greater than the maximum value of the boundary, the truncation protection mechanism is triggered, and the channel value is forcibly assigned the maximum value of the boundary (i.e., 255); if the value is less than the minimum value of the boundary, it is assigned 0; if the value is within the legal range of the upper and lower limits of the boundary, the original calculated value remains unchanged. After the above channel-level boundary security truncation process, a new pixel vector that conforms to the physical storage specifications and has no color overflow distortion is output. This new pixel vector is defined as the mapped texture pixel, thereby completing the standardized numerical fusion of the original physical material features and the light and shadow features.

[0060] Preferably, the following steps are used to fill pixels with a bilinear interpolation algorithm and perform rasterization rendering to output a power grid inspection image from multiple viewpoints: On the two-dimensional virtual viewpoint plane, due to projection transformation, the coordinates of the grid vertices of the two-dimensional background grid are typically represented as sub-pixel level coordinates carrying decimals. To overcome the jagged texture mapping and resampling errors caused by coordinate discretization, for each sub-pixel level two-dimensional projection coordinate, the mapped texture pixel values ​​of the four adjacent integer-level pixels are extracted, and the horizontal and vertical offset weights of the two-dimensional projection coordinates relative to these four adjacent pixels are calculated. Using the horizontal and vertical offset weights, the bilinear interpolation algorithm formula is substituted to perform a weighted summation in the two-dimensional dimension to calculate a smoothly transitioned interpolated mapped pixel, which is then accurately filled into the grid vertex positions in the two-dimensional background grid that coincide with the two-dimensional projection coordinates. After all valid grid vertices of the two-dimensional background grid have been assigned mapped texture pixel values, rasterization rendering is performed on the two-dimensional background grid after pixel filling. Specifically, a scanline algorithm or a barycentric coordinate interpolation shading algorithm is employed. Each valid triangular facet in the 2D background mesh is used as the basic rendering unit. Based on the interpolated mapping pixels of the three vertices constituting each triangular facet, pixel-by-pixel interpolation shading and region filling are performed on all blank discrete pixel regions within the continuous space of that triangular facet. By traversing and shading all valid triangular facets in the 2D background mesh, the spatial data structure composed of discrete meshes and vector topology is transformed into a rasterized data set consisting of continuously and tightly arranged rows and columns, thus obtaining the 2D pixel array. Finally, the 2D pixel array, encompassing the reconstructed texture of the target object and the surrounding physical background, is encoded in a standard image format to output the cross-view power grid inspection image.

[0061] In this embodiment, the following technical effects are achieved through the aforementioned illumination multiplication fusion, color truncation protection, and bilinear interpolation and rasterization rendering mechanisms targeting texture pixels. Existing image geometric transformation techniques directly transfer original pixels, failing to reflect illumination changes caused by perspective shifts, and easily producing abrupt seams and false light and shadow misalignments at illumination conflict points. This embodiment, through rigorous channel-level multiplication and truncation operations, accurately imbues the original texture with the illumination attenuation and specular characteristics of the real physical world, while effectively preventing white spot distortion caused by specular overflow at the data level. Simultaneously, the bilinear interpolation and rasterization rendering algorithms effectively eliminate sub-pixel jaggedness and mesh surface holes caused by cross-dimensional projection and discretization, ensuring that the final output cross-view power grid inspection image not only strictly follows the three-dimensional perspective projection rules in terms of geometric shape but also achieves a high degree of physical realism in visual optical characteristics. This provides high-fidelity sample data for intelligent defect recognition models, improving the accuracy of subsequent model training and defect detection.

[0062] likeFigure 2 As shown, another embodiment of the present invention also provides a device for generating power grid inspection images from multiple perspectives, including: a data acquisition module, a three-dimensional reconstruction module, a mesh construction module, a feature calculation module, and an image generation module; The data acquisition module is used to acquire a sequence of power grid inspection images containing target objects taken by an image acquisition device mounted on a drone, and simultaneously acquire attitude parameters and positioning data corresponding to the power grid inspection image sequence collected by an attitude sensor and a satellite positioning receiver. The 3D reconstruction module is used to calculate the camera relative pose matrix of adjacent frames in the power grid inspection image sequence based on the pose parameters and the positioning data, and to perform 3D reconstruction of the power grid inspection image sequence in combination with the camera relative pose matrix to generate a 3D point cloud containing the target object. The mesh construction module is used to obtain the set virtual camera pose, and project the three-dimensional point cloud onto the two-dimensional virtual view plane according to the virtual camera pose to construct a two-dimensional background mesh corresponding to the target object. The feature calculation module is used to extract the three-dimensional normal vector of the surface of the target object from the three-dimensional point cloud, and calculate the physical brightness and darkness variation features of the target object under the two-dimensional virtual view plane based on the three-dimensional normal vector, the obtained physical environment lighting direction vector and the observation direction vector determined by the virtual camera pose. The image generation module is used to extract the original texture pixels of the target object from the power grid inspection image sequence, fuse the original texture pixels with the physical brightness and darkness variation features to generate mapped texture pixels, and fill the mapped texture pixels into the two-dimensional background grid to generate a power grid inspection image from a cross-view perspective.

[0063] It is understood that the above-described device embodiments correspond to the method embodiments of the present invention, and can realize the method for generating power grid inspection images from a cross-view perspective provided by any of the above-described method embodiments of the present invention.

[0064] It should be noted that the device embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can specifically be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0065] Based on the above embodiment of the method for generating power grid inspection images from multiple perspectives, another embodiment of the present invention provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements any one of the methods for generating power grid inspection images from multiple perspectives according to the present invention.

[0066] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the terminal device.

[0067] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0068] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.

[0069] Based on the above-described method embodiments, another embodiment of the present invention provides a computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the method for generating a power grid inspection image from any of the above-described method embodiments of the present invention.

[0070] The modules / units integrated in the device / terminal equipment, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0071] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for generating power grid inspection images from multiple perspectives, characterized in that, include: A sequence of power grid inspection images containing the target object is acquired from an image acquisition device mounted on a drone. Simultaneously, attitude parameters and positioning data corresponding to the power grid inspection image sequence are acquired from an attitude sensor and a satellite positioning receiver. Based on the attitude parameters and positioning data, the relative camera pose matrix of adjacent frames in the power grid inspection image sequence is calculated. The power grid inspection image sequence is then reconstructed in three dimensions using the relative camera pose matrix to generate a three-dimensional point cloud containing the target object. A set virtual camera pose is acquired, and the three-dimensional point cloud is projected onto a two-dimensional virtual view plane based on the virtual camera pose to construct a two-dimensional background mesh corresponding to the target object. The three-dimensional normal vector of the target object surface is extracted from the three-dimensional point cloud, and the physical light and shadow change characteristics of the target object under the two-dimensional virtual view plane are calculated based on the three-dimensional normal vector, the obtained physical environment lighting direction vector and the observation direction vector determined by the virtual camera pose. The original texture pixels of the target object are extracted from the power grid inspection image sequence. The original texture pixels are fused with the physical brightness and darkness variation features to generate mapped texture pixels. The mapped texture pixels are then filled into the two-dimensional background grid to generate a power grid inspection image from a cross-view perspective.

2. The method for generating power grid inspection images from multiple perspectives as described in claim 1, characterized in that, Based on the attitude parameters and the positioning data, the camera relative pose matrix of adjacent frames in the power grid inspection image sequence is calculated, including: converting the attitude parameters into an initial rotation matrix in the world coordinate system and converting the positioning data into an initial translation vector in the world coordinate system; calculating the initial rotation difference matrix and the initial translation difference vector between adjacent frames based on the initial rotation matrix and the initial translation vector, and concatenating them to form an initial relative pose matrix; performing singular value decomposition on the rotation components in the initial relative pose matrix to obtain a left singular matrix, a singular value diagonal matrix, and a right singular matrix; The modified singular value matrix is ​​obtained by forcibly setting all non-zero diagonal elements of the singular value diagonal matrix to 1. The matrix is ​​then reconstructed by combining the left singular matrix and the right singular matrix to obtain the orthogonal rotation matrix. Based on the velocity vector provided by the positioning data and the time difference between the corresponding frames, the inertial prediction translation vector is calculated, and the candidate translation vector derived from the orthogonal rotation matrix is ​​weighted and averaged with the inertial prediction translation vector to obtain the fused translation vector; the camera relative pose matrix is ​​generated by combining the orthogonal rotation matrix and the fused translation vector.

3. The method for generating power grid inspection images from multiple perspectives as described in claim 2, characterized in that, The process of obtaining a fused translation vector by weighted averaging of the candidate translation vector derived from the orthogonal rotation matrix and the inertial prediction translation vector includes: extracting the three-dimensional position accuracy factor from the positioning data; calculating the inertial prediction weight based on the three-dimensional position accuracy factor and a preset normalization coefficient; calculating the difference between a constant 1 and the inertial prediction weight, and using the difference as a candidate translation weight; weighting the candidate translation weight with the candidate translation vector to obtain a first weighted translation vector; weighting the inertial prediction weight with the inertial prediction translation vector to obtain a second weighted translation vector; and combining the first weighted translation vector with the second weighted translation vector to obtain the fused translation vector.

4. The method for generating power grid inspection images from multiple perspectives as described in claim 3, characterized in that, The method involves reconstructing a 3D point cloud containing the target object from the power grid inspection image sequence using the camera relative pose matrix, including: extracting pixel-level dense optical flow correspondences between adjacent frames in the power grid inspection image sequence; performing triangulation calculations using the camera relative pose matrix and the pixel-level dense optical flow correspondences to obtain an initial 3D point cloud; back-projecting the initial 3D point cloud onto the image plane based on the camera relative pose matrix to generate back-projected 2D coordinates, calculating the coordinate residual distance between the back-projected 2D coordinates and the actual 2D coordinates obtained through the pixel-level dense optical flow correspondences, removing outliers with a coordinate residual distance greater than a preset threshold based on the coordinate residual distance, and obtaining a filtered 3D point cloud; acquiring the spatial region of interest (ROI) of the target object in the power grid inspection image sequence using a target detection algorithm, saving the 3D data points of the filtered 3D point cloud located within the ROI, and performing density clustering processing on the saved 3D data points to generate a 3D point cloud containing the target object.

5. The method for generating power grid inspection images from multiple perspectives as described in claim 4, characterized in that, The method of projecting the 3D point cloud onto a 2D virtual view plane based on the virtual camera pose to construct a 2D background mesh corresponding to the target object includes: projecting the 3D point cloud onto the 2D virtual view plane based on the virtual camera pose to obtain a set of 2D projection coordinate points; performing Deloni triangulation on the set of 2D projection coordinate points to generate an initial planar topology composed of multiple triangular facets; reading the 3D spatial direct adjacency relationship of each point in the set of 2D projection coordinate points in the 3D point cloud; traversing each triangular facet in the initial planar topology and checking whether there is a 3D spatial direct adjacency relationship between the two 2D projection coordinate points connected by the three edges of each triangular facet in the 3D point cloud; if there is no direct adjacency relationship between the two endpoints of any edge in 3D space, the corresponding triangular facet is determined to be a degenerate facet, and the triangular facet corresponding to the degenerate facet is removed from the initial planar topology, and all the remaining valid triangular facets constitute the 2D background mesh.

6. The method for generating power grid inspection images from multiple perspectives as described in claim 5, characterized in that, Extracting the three-dimensional normal vector of the target object surface from the three-dimensional point cloud, and calculating the physical brightness and darkness variation characteristics of the target object under the two-dimensional virtual view plane based on the three-dimensional normal vector, the obtained physical environment lighting direction vector, and the observation direction vector determined by the virtual camera pose, including: taking each three-dimensional data point in the three-dimensional point cloud as the center, searching for a set of adjacent three-dimensional data points within a preset search radius; Calculate the decentralized coordinate matrix and covariance matrix of the adjacent 3D data point set, and perform eigenvalue decomposition on the covariance matrix, taking the normalized eigenvector corresponding to the smallest eigenvalue as the 3D normal vector; parse the exchangeable image file format information of the power grid inspection image sequence, extract the shooting timestamp and geographical location data from the exchangeable image file format information, and calculate the solar azimuth angle and solar zenith angle based on the shooting timestamp and geographical location data, and determine the physical environment illumination direction vector based on the solar azimuth angle and solar zenith angle; extract the spatial coordinates of the virtual camera optical center from the virtual camera pose, and construct a direction vector with the virtual camera optical center as the starting point and the corresponding 3D data point as the ending point, taking the direction vector as the observation direction vector; substitute the 3D normal vector, the physical environment illumination direction vector, and the observation direction vector into the half-range vector illumination model to calculate the reflection intensity value of the target object, and take the reflection intensity value as the physical brightness change feature.

7. The method for generating power grid inspection images from multiple perspectives as described in claim 6, characterized in that, The process of fusing the original texture pixels with the physical brightness variation features to generate mapped texture pixels, and filling the mapped texture pixels into the two-dimensional background grid to generate a power grid inspection image from multiple viewpoints, includes: multiplying the color channel values ​​of the original texture pixels by the reflection intensity values ​​corresponding to the physical brightness variation features to obtain multiple fusion channel values, and recombinating the multiple fusion channel values ​​into initial fusion pixel values; performing a color truncation operation on the initial fusion pixel values, forcing values ​​exceeding the upper limit of the color space to the maximum value of the color space boundary, thereby generating the mapped texture pixels; using a bilinear interpolation algorithm to fill the mapped texture pixels into the grid vertices of the two-dimensional background grid that coincide with the two-dimensional projection coordinates, and performing rasterization rendering on the two-dimensional background grid after filling the pixels to obtain a two-dimensional pixel array, and outputting the two-dimensional pixel array as the power grid inspection image from multiple viewpoints.

8. A device for generating power grid inspection images from multiple perspectives, characterized in that, include: The module includes a data acquisition module, a 3D reconstruction module, a mesh construction module, a feature calculation module, and an image generation module. The data acquisition module is used to acquire a sequence of power grid inspection images containing target objects taken by an image acquisition device mounted on a drone, and simultaneously acquire attitude parameters and positioning data corresponding to the power grid inspection image sequence collected by an attitude sensor and a satellite positioning receiver. The 3D reconstruction module is used to calculate the camera relative pose matrix of adjacent frames in the power grid inspection image sequence based on the pose parameters and the positioning data, and to perform 3D reconstruction of the power grid inspection image sequence in combination with the camera relative pose matrix to generate a 3D point cloud containing the target object. The mesh construction module is used to obtain the set virtual camera pose, and project the three-dimensional point cloud onto the two-dimensional virtual view plane according to the virtual camera pose to construct a two-dimensional background mesh corresponding to the target object. The feature calculation module is used to extract the three-dimensional normal vector of the surface of the target object from the three-dimensional point cloud, and calculate the physical brightness and darkness variation features of the target object under the two-dimensional virtual view plane based on the three-dimensional normal vector, the obtained physical environment lighting direction vector and the observation direction vector determined by the virtual camera pose. The image generation module is used to extract the original texture pixels of the target object from the power grid inspection image sequence, fuse the original texture pixels with the physical brightness and darkness variation features to generate mapped texture pixels, and fill the mapped texture pixels into the two-dimensional background grid to generate a power grid inspection image from a cross-view perspective.

9. A terminal device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements the method for generating cross-view power grid inspection images as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, include: A stored computer program, wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform the method for generating cross-view power grid inspection images as described in any one of claims 1-7.

Images