Unmanned aerial vehicle tilt photography overhead pipeline image data efficient processing system

Abstract

The application discloses an unmanned aerial vehicle oblique photography overhead pipeline image data efficient processing system and belongs to the field of photographic data processing. The system comprises a high-precision synchronous control unit, which is used for microsecond-level synchronous triggering collection of a visible light camera, an infrared thermal imaging sensor and an ultraviolet discharge detection sensor carried on the unmanned aerial vehicle; a positioning and clock synchronization module, which is used for acquiring three-dimensional space position information of the unmanned aerial vehicle and marking time stamps on the collected data of each sensor; a pixel-level space-time alignment module, which is used for aligning multi-source image data to the same time and space reference; a cross-modal Transformer feature extractor, which is used for extracting features and cross-modal fusion of the multi-modal image data after space-time alignment and outputting a fusion feature map; and a multi-task independent decoder, which is used for performing three tasks of defect detection, defect classification and defect positioning in parallel according to the fusion feature map. The application realizes comprehensive, accurate and automatic detection of various defect types of overhead pipelines.

Description

Technical Field

[0001] This invention belongs to the technical field of photographic data processing, specifically relating to a high-efficiency processing system for aerial pipeline image data from oblique photography by unmanned aerial vehicles. Background Technology

[0002] With the continuous advancement of smart grid construction, the inspection of overhead transmission lines is gradually transforming towards intelligence and automation. Unmanned aerial vehicle (UAV) oblique photography technology, as a mainstream method in the current power inspection field, enables long-distance observation and data acquisition of transmission line equipment by carrying various image sensors, effectively reducing the operational risks and labor intensity of manual inspection. Against this technological backdrop, visible light imaging, infrared thermal imaging, and ultraviolet discharge detection technologies are widely used in defect identification scenarios for key equipment such as overhead conductors, insulators, and towers.

[0003] However, existing technologies have significant limitations in sensor configuration and data fusion. Specifically, most current mainstream drone inspection solutions use a single sensor for data acquisition, such as using only a visible light camera for video monitoring or only a laser sensor to measure tower displacement. Even those involving multi-sensor concepts fail to address the effective fusion of heterogeneous data. This single or simplistic sensor configuration results in the system only acquiring single physical characteristic information of the target object, leading to insufficient detection capabilities: visible light sensors can clearly reveal structural defects on the equipment surface, but cannot effectively identify latent defects caused by material degradation or abnormal current; infrared thermal imaging sensors can detect abnormal heating of equipment, but struggle to accurately determine the specific type and severity of defects; ultraviolet discharge detection sensors can capture partial discharge phenomena, but cannot independently perform spatial location and morphological analysis of defects.

[0004] The limitations of single-sensor detection make it difficult for existing technologies to achieve comprehensive and accurate detection of multi-dimensional defects in overhead pipelines. Some solutions have attempted to mount multiple sensors on the same UAV platform to obtain richer equipment status information. However, when simultaneously acquiring multi-dimensional equipment status information, the data collected by different sensors exhibit significant differences in temporal and spatial coordinates. This temporal asynchrony and spatial misalignment of heterogeneous data makes it difficult to effectively fuse multi-source data within a unified framework, resulting in poor fusion performance. Consequently, this severely restricts the practical application of multimodal information fusion technology in defect identification.

[0005] In summary, the lack of a technical solution in the current technology to efficiently align and deeply fuse multi-source sensor data in time and space, and on this basis to realize the synchronous detection of multi-dimensional defects in overhead pipelines, has become the core bottleneck restricting the further development of UAV inspection technology. Summary of the Invention

[0006] This application provides a high-efficiency processing system for overhead pipeline image data obtained by oblique photography of unmanned aerial vehicles, which aims to overcome the problem that the existing technology cannot achieve comprehensive detection of multi-dimensional defects in overhead pipelines using a single sensor, or at least partially solve the above problems.

[0007] This application provides a high-efficiency processing system for overhead pipeline image data obtained from UAV oblique photography, comprising: The high-precision synchronization control unit is used to perform microsecond-level synchronous triggering and acquisition of visible light cameras, infrared thermal imaging sensors, and ultraviolet discharge detection sensors mounted on UAVs. The system includes a GPS and BeiDou dual-mode positioning module and a high-precision clock synchronization module. The GPS and BeiDou dual-mode positioning module is used to acquire the three-dimensional spatial position information of the UAV, and the high-precision clock synchronization module is used to timestamp the data collected by each sensor. A pixel-level spatiotemporal alignment module is used to unify the visible light image data, infrared image data and ultraviolet image data collected by each sensor to the same time and space reference based on the three-dimensional spatial position information and the timestamp mark. A cross-modal Transformer feature extractor is used to extract features and perform cross-modal fusion on the spatiotemporally aligned visible light image data, infrared image data, and ultraviolet image data, and output a fused feature map. A multi-task independent decoder is used to perform three tasks in parallel: defect detection, defect classification, and defect localization, based on the fused feature map.

[0008] In summary, this application includes at least one of the following beneficial technical effects: 1. This invention achieves microsecond-level synchronous triggering and acquisition of visible light, infrared, and ultraviolet sensors through a high-precision synchronous control unit. Combined with dual-mode positioning and a high-precision clock synchronization module, each frame of data is precisely spatiotemporally marked. Then, a pixel-level spatiotemporal alignment module unifies the multi-source images to the same time and space reference, effectively solving the problems of time asynchrony and spatial misalignment of heterogeneous sensor data. This provides a high-quality aligned data foundation for subsequent multimodal fusion and significantly improves the accuracy and reliability of multi-source information fusion.

[0009] 2. This invention captures long-distance dependencies between different modal features through learnable modal identifier vectors and multi-head self-attention mechanisms. Combined with parallel mechanisms of channel attention and spatial attention, it achieves adaptive weighted fusion of multimodal features, fully exploiting the complementary advantages between appearance information of visible light images, thermal anomaly information of infrared images, and discharge information of ultraviolet images, thereby significantly enhancing the comprehensive identification capability of various latent and overt defects of overhead pipelines.

[0010] 3. This invention adopts a deep learning multi-task architecture that combines a shared feature encoder with multi-task independent decoders. It executes three tasks—defect detection, defect classification, and defect localization—in parallel. Most parameters are shared among the tasks, reducing the computational load and parameter count of the model. At the same time, the three decoders output bounding boxes and confidence scores, multi-label classification probabilities, and pixel-level segmentation masks, respectively, enabling multi-dimensional synchronous output of defect type, location, and severity. This significantly improves the operational efficiency and automation and intelligence level of overhead pipeline inspection. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the overall technical architecture of the UAV oblique photography aerial pipeline image data high-efficiency processing system; Figure 2 This is a schematic diagram of the core principle framework of a multimodal feature fusion network based on an attention mechanism; Figure 3 It is a logical flowchart of multi-source sensor data synchronous acquisition and pixel-level spatiotemporal alignment; Figure 4 This is a schematic diagram of the overall framework of a deep learning multi-task learning network architecture; Figure 5 This is a schematic diagram of the system composition of a multi-source sensor integrated system and a distributed mounting structure for UAVs. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this invention clearer, the following description is provided in conjunction with the appendix. Figure 1 To be continued Figure 5 The present invention will be further described in detail below with reference to specific embodiments.

[0013] The proposed system for efficient processing of overhead pipeline image data by UAV oblique photography includes: an integrated module of multi-source sensors, a UAV-mounted multispectral sensor array scheme, a multimodal data synchronous acquisition system, a multimodal feature fusion network based on attention mechanism, a deep learning multi-task learning network architecture, a hard example sample mining strategy, and an adaptive defect classification model.

[0014] In this embodiment, the UAV oblique photography overhead pipeline image data high-efficiency processing system achieves multi-dimensional defect detection of overhead pipelines through a multi-source sensor integrated module. This module uses a high-precision synchronous control unit to achieve unified clock synchronization control of the visible light camera, infrared thermal imaging sensor, and ultraviolet discharge detection sensor. The parallelism of the optical axes of each sensor is better than 0.5 degrees, and the sampling frequency achieves microsecond-level synchronous triggering. This solves the problem that existing technologies using a single sensor cannot simultaneously detect multi-dimensional defects such as cracks, corrosion, and abnormal heating, achieving comprehensive detection coverage of defects in overhead conductors, insulators, towers, and other equipment.

[0015] The UAV-mounted multispectral sensor array solution is used to design a dedicated UAV carrier platform. It adopts a distributed mounting structure, and a single flight can carry three or more types of sensors, including visible light cameras, infrared thermal imaging sensors, and ultraviolet discharge detection sensors. The endurance meets the inspection range requirement of 200 kilometers, and the total weight of the sensors is controlled within 5 kg. This solves the problems of low efficiency and high risk of traditional manual inspection, realizes large-scale automated inspection, improves inspection efficiency, and reduces labor costs and operational risks.

[0016] The multimodal data synchronous acquisition system is used to achieve pixel-level spatiotemporal alignment of visible light, infrared, and ultraviolet data based on GPS and Beidou dual-mode positioning modules and high-precision clock synchronization modules. The timestamp error is less than 1ms and the spatial position error is less than 2cm, thereby solving the problem of the inability to effectively fuse data from different sensors in existing technologies and providing a high-quality data foundation for subsequent multimodal feature fusion.

[0017] A multimodal feature fusion network based on an attention mechanism is used to construct a cross-modal Transformer feature extractor. It achieves adaptive weighted fusion of visible light, infrared, and ultraviolet feature maps through a learnable weight matrix. The fusion layer adopts a parallel mechanism of channel attention module and spatial attention module, thereby solving the problems of inconsistent feature expression dimensions and poor fusion effect of multi-source heterogeneous data, and improving the accuracy and robustness of defect identification.

[0018] The deep learning multi-task learning network architecture is used to design a two-stage architecture with a shared feature encoder and multiple task independent decoders, which simultaneously realizes three tasks: defect detection, defect classification, and defect localization. The shared parameters among tasks account for no less than 60%, thereby solving the problem that a single task model cannot output multiple diagnostic results at the same time, and realizing adaptive identification and classification of defects in overhead equipment.

[0019] The difficult sample mining strategy employs a dynamic training mechanism that combines online difficult sample mining with focus loss. It automatically adjusts sample weights based on sample loss values ​​to achieve targeted enhancement training for difficult samples such as micro-defects and occlusion defects. This addresses the problem of low detection rate and high false detection rate of micro-defects in existing technologies, and improves the detection capability of latent defects such as micro-cracks and early corrosion.

[0020] The adaptive defect classification model is used for end-to-end classification models based on deep neural networks. The output layer adopts a multi-label classification mechanism, which supports automatic identification of more than 20 defect types such as conductor breakage, insulator damage, tower corrosion, abnormal heating, and ultraviolet discharge. This solves the problem of traditional methods relying on manual experience and having low efficiency, and achieves the goal of improving the detection accuracy to over 95%.

[0021] In summary, the UAV oblique photography aerial pipeline image data high-efficiency processing system proposed in this application constructs a complete technical chain from data acquisition, data alignment, feature fusion to defect identification.

[0022] At the data acquisition end, the system integrates a multi-source sensor module, simultaneously mounting a visible light camera, an infrared thermal imaging sensor, and an ultraviolet discharge detection sensor on the same UAV platform. This enables the synchronous capture of multi-dimensional information on the appearance, heating anomalies, and discharge phenomena of overhead pipelines. Each sensor employs a distributed mounting structure, allowing for the simultaneous carrying of three or more sensors per flight, with a total weight controlled within 5 kg and a flight time exceeding 90 minutes, supporting an inspection range of 200 km. A high-precision synchronization control unit achieves microsecond-level synchronous triggering for data acquisition, ensuring precise time synchronization of sampling actions by each sensor.

[0023] In the data preprocessing stage, the system utilizes a GPS and BeiDou dual-mode positioning module to acquire 3D spatial positioning information with an accuracy better than 2cm. Combined with a high-precision clock synchronization module, it achieves timestamp marking accuracy better than 10ns, keeping the timestamp error within 1ms. Based on this, the system performs spatial coordinate transformation and time registration, unifying visible light, infrared, and ultraviolet image data collected by sensors with different physical mechanisms onto the same time and spatial reference. This provides a pixel-level aligned data foundation for subsequent multimodal fusion. Spatial coordinate transformation is achieved by mapping the pixel coordinates of each sensor image to a geographic coordinate system with the UAV's current location as the origin. Time registration uses cubic spline interpolation to unify the data from each sensor to the same time reference point.

[0024] In the feature fusion stage, the system constructs a multimodal feature fusion network based on an attention mechanism. The core of this network is a cross-modal Transformer feature extractor, which distinguishes features from different sensor sources by assigning learnable modal identifier vectors to each modality of data and captures long-distance dependencies between features from different modalities through a multi-head self-attention mechanism. Building upon this, the system further employs a parallel fusion mechanism of channel attention and spatial attention modules. The channel attention module recalibrates features along the channel dimension to enhance the representation of key channels, while the spatial attention module focuses on key regions along the spatial dimension to suppress background interference. The two modules work together to achieve adaptive weighted fusion of multimodal features.

[0025] In the defect identification stage, the system adopts a deep learning multi-task learning network architecture. Through a hard-shared structure of a shared feature encoder and three independent decoder heads, it simultaneously executes three tasks: defect detection, defect classification, and defect localization. The shared parameters among tasks account for no less than 60%. The shared feature encoder uses a hybrid architecture of convolutional neural networks and Transformers. It extracts low-level texture features using shallow convolutional modules, models long-distance dependencies through multiple residual attention blocks, and finally outputs multi-scale feature maps via a feature pyramid module for use by the decoders of each task. The three parallel decoders respectively output defect bounding boxes and confidence scores, multi-label classification probability distributions, and pixel-level segmentation masks.

[0026] During the network training phase, the system introduces a dynamic training mechanism combining online hard example mining and focus loss. For classification tasks, a focus loss function is used to reduce the loss contribution of easily classified samples by focusing parameters, thus enabling model training to focus on hard samples. For detection tasks, an IoU loss function is used to directly optimize the overlap between predicted and ground truth bounding boxes. During training, the system calculates the loss value of each sample in real time, sorts them in descending order of loss value, and prioritizes hard samples with the highest loss ranking to form training batches. The sampling proportion of hard samples is no less than 70%, and the rest are randomly sampled to ensure training diversity.

[0027] During the inference and deployment phase, the system employs an independent end-to-end adaptive defect classification model for final discrimination. The model's backbone network uses a lightweight structure, ensuring feature extraction capabilities while keeping computational load low, making it suitable for deployment on UAVs. After fusing multi-scale features, the model outputs multi-label classification results through a classification output layer, supporting automatic discrimination of 25 defect categories, including conductor breakage, conductor wear, conductor corrosion, insulator damage, insulator cracks, tower corrosion, tower deformation, abnormal heating, and ultraviolet corona discharge.

[0028] In addition, the system integrates a data storage and transmission module and a defect annotation and management platform. The data storage and transmission module stores the collected data and detection results in a solid-state storage array and transmits them back to the ground control station via a wireless communication module. The defect annotation and management platform provides data support for continuous model iteration, supports multiple annotation modes such as bounding boxes, pixel-level segmentation, and key points, employs a double-blind review mechanism to ensure annotation quality, and can automatically generate training annotation files that conform to standard formats.

[0029] In summary, this application presents a complete technical solution, encompassing high-precision synchronous acquisition and spatiotemporal alignment of multi-source data, deep feature fusion based on attention mechanisms, joint inference via multi-task learning networks, and targeted hard example mining training strategies. This solution organically addresses a series of technical challenges in the acquisition, fusion, and analysis of multi-source heterogeneous data, ultimately achieving comprehensive, accurate, and automated detection of various defect types in overhead pipelines, providing a reliable technical path for intelligent inspection of overhead pipelines.

[0030] In the high-efficiency processing system for aerial pipeline image data from UAV oblique photography, the high-precision synchronization control unit is the core component for achieving microsecond-level synchronous acquisition from multiple sensors. This unit specifically comprises three sub-modules: a clock signal generator, a trigger delay compensation module, and multiple synchronous trigger output channels.

[0031] The clock signal generator employs a high-precision crystal oscillator or atomic clock reference source to generate a high-frequency stable clock signal with a frequency of not less than 100MHz. This clock signal serves as the timing reference for the entire system, providing a unified clock reference for the subsequent synchronous sampling of various sensors.

[0032] The trigger delay compensation module is used to eliminate the inherent trigger delay caused by the different physical response characteristics of each sensor. During the system initialization phase, this module needs to pre-measure and store the trigger delay of each sensor. The specific measurement method is as follows: a standard trigger signal is sent to each sensor, and a high-precision time interval measuring instrument is used to record the time interval between the issuance of the trigger signal and the actual start of sampling by the sensor; the measurement is repeated more than 1000 times for each sensor, and the average value of the measurement results is taken as the inherent trigger delay of that sensor and stored in the module.

[0033] During actual data acquisition, when a synchronization trigger signal is issued, the trigger delay compensation module automatically adds a corresponding delay compensation value to each trigger signal based on the pre-stored delay values ​​of each sensor. This ensures that the deviation between the actual sampling start time of each sensor and the system's predetermined sampling time is less than a preset threshold. In this embodiment, the preset threshold is set to 10 μs. Through this mechanism, even if the response characteristics of each sensor differ, near-synchronous acquisition can be achieved after calibration.

[0034] The number of multi-channel synchronous trigger output channels is set to no less than three, corresponding to the visible light camera, infrared thermal imaging sensor, and ultraviolet discharge detection sensor, respectively. The output signal of each channel is independent, and its time accuracy is better than 100ns. The channel independence is achieved by using independent drive circuits and signal amplifiers for each channel to ensure that there is no crosstalk between the signals, and that an abnormality in the signal of any channel will not affect the normal trigger output of other channels.

[0035] The three sub-modules work together to enable synchronous triggering and acquisition by three sensors with different physical mechanisms within a microsecond time accuracy. This precision ensures that even when the UAV flies at a high speed of 15 m / s, the images acquired by each sensor at the same moment maintain a high degree of spatial consistency. Measured by a displacement of 15 mm corresponding to 1 μs of flight, this spatial offset is within an acceptable range for overhead pipeline inspection scenarios and will not have a substantial impact on subsequent image fusion and defect identification. Therefore, this implementation effectively solves the problem of data fusion difficulties caused by asynchronous acquisition times of different sensors in existing technologies.

[0036] In the efficient processing system for aerial pipeline image data from UAV oblique photography, the distributed mounting structure is the mechanical foundation for achieving integrated multi-sensor operation. The distributed mounting structure specifically comprises three components: a central load-bearing plate, sensor mounting brackets, and vibration damping devices.

[0037] The central load-bearing plate is the core load-bearing component of the entire mounting structure. In this embodiment, the central load-bearing plate is made of aerospace-grade aluminum alloy and is fixed to the belly of the UAV. The central load-bearing plate connects to the main power supply interface and high-speed data bus from the UAV flight platform via standard aerospace connectors. The input voltage range of the main power supply interface is 22.8V to 26.4V, and the high-speed data bus uses a gigabit Ethernet interface with a data transmission rate of no less than 1Gbps. The central load-bearing plate is also responsible for distributing the power supplied by the main power supply interface to each sensor and providing a high-speed data bus interface for each sensor, thereby achieving unified data aggregation and power management for sensor data.

[0038] The sensor mounting bracket is a key component for achieving precise assembly and optical axis calibration of each sensor. In this embodiment, the sensor mounting bracket employs a three-dimensional angle adjustment mechanism, and each sensor is fixed to the sensor mounting bracket via a precision adjustment mechanism. Each sensor's precision adjustment mechanism includes a pitch adjustment mechanism, a yaw adjustment mechanism, and a roll adjustment mechanism. The pitch adjustment mechanism has an adjustment range of ±15° and an adjustment accuracy of 0.1°; the yaw adjustment mechanism has an adjustment range of ±15° and an adjustment accuracy of 0.1°; and the roll adjustment mechanism has an adjustment range of ±10° and an adjustment accuracy of 0.1°.

[0039] By adjusting the three-axis angles of the pitch, yaw, and roll adjustment mechanisms described above, the parallelism of the optical axes of each sensor can be calibrated to better than 0.5 degrees. The specific calibration method is as follows: Before takeoff, align the entire system with a standard calibration target. The operator adjusts the angles of the pitch, yaw, and roll adjustment mechanisms of each sensor until the center point of the images captured by all three sensors coincides with the center point of the standard calibration target. This completes the optical axis parallelism calibration. This calibration operation ensures that each sensor has a consistent observation direction during subsequent flight data acquisition.

[0040] A vibration damping device is positioned between the sensor and its mounting bracket. Its function is to suppress the adverse effects of high-frequency vibrations during UAV flight on the sensor's imaging quality. In this embodiment, the vibration damping device can be implemented using either damping rubber material or a spring assembly. When damping rubber material is used, its damping coefficient is not less than 0.8; when a spring assembly is used, its stiffness coefficient is 50 N / mm.

[0041] The vibration damping device has an effective frequency reduction range of 20Hz to 500Hz, covering the main high-frequency vibration bands during UAV flight. With the above design, the vibration damping device can achieve a vibration suppression rate of over 85%, reducing the interference of vibration on the clarity and stability of sensor imaging.

[0042] Through the coordinated operation of the three components—the central load-bearing plate, the sensor mounting bracket, and the vibration damping device—this embodiment achieves stable and precise integration of multiple sensors on the UAV platform. Based on the precise design of the aforementioned distributed mounting structure, a single UAV can simultaneously carry three or more sensors, including a visible light camera, an infrared thermal imaging sensor, and an ultraviolet discharge detection sensor.

[0043] The visible light camera weighs approximately 800g, the infrared thermal imaging sensor approximately 1200g, and the ultraviolet discharge detection sensor approximately 900g, with the total sensor weight controlled within 5kg. Combined with the weight of the drone itself and the battery, the overall takeoff weight is controlled within 7.5kg. With this configuration, the system's endurance can reach over 90 minutes, meeting the operational requirements for a single sortie of 200km inspection distance. Therefore, this implementation method effectively solves the problems of low efficiency and high risk associated with traditional manual inspections, providing hardware platform support for large-scale automated inspections.

[0044] This application provides a high-efficiency processing system for aerial pipeline image data from UAV oblique photography. A GPS and BeiDou dual-mode positioning module is the core component for achieving accurate spatial position measurement. Specifically, the GPS and BeiDou dual-mode positioning module comprises four parts: a main receiving antenna, a backup receiving antenna, an RF front-end circuit, and a baseband processing chip. The outputs of the main and backup receiving antennas are connected to the inputs of the RF front-end circuit, and the output of the RF front-end circuit is connected to the input of the baseband processing chip. The baseband processing chip processes the down-converted signal from the RF front-end circuit.

[0045] The primary and backup receiving antennas are arranged in a spatial diversity configuration. In this embodiment, the antenna spacing is set to be no less than the maximum geometric dimension of the sensor array. The maximum geometric dimension of the sensor array is 400 mm, therefore the antenna spacing is set to 450 mm. This antenna spacing setting enhances the decorrelation effect of multipath signals and reduces the impact of multipath effects on positioning accuracy.

[0046] The RF front-end circuit has the capability for simultaneous reception of multiple frequencies, supporting parallel processing of GPS L1, GPS L5, and BeiDou B1, BeiDou B2, and BeiDou B3 frequencies. Specifically, the GPS L1 frequency is 1575.42MHz, the GPS L5 frequency is 1176.45MHz, the BeiDou B1 frequency is 1561.098MHz, the BeiDou B2 frequency is 1207.14MHz, and the BeiDou B3 frequency is 1268.52MHz. Through this multi-frequency parallel reception, the RF front-end circuit eliminates ionospheric delay errors and improves positioning accuracy.

[0047] The baseband processing chip incorporates a real-time kinematic positioning algorithm, also known as carrier phase differential technology, abbreviated as RTK. Under open field-of-view conditions, the baseband processing chip achieves a 3D positioning accuracy better than 2cm based on carrier phase differential technology.

[0048] The real-time kinematic positioning algorithm is implemented as follows: The baseband processing chip receives differential correction data broadcast from a reference station in real time via its internal wireless receiving unit. The reference station is a fixed GNSS receiver positioned at known precise coordinates. This fixed GNSS receiver continuously observes satellites and generates differential correction data, which is then broadcast via radio link or cellular network. Simultaneously, the baseband processing chip acquires its own observed carrier phase data. The differential correction data and carrier phase data are then input into the Kalman filter algorithm built into the baseband processing chip for fusion calculation, thereby obtaining accurate position information. The position update frequency of the baseband processing chip is set to 10Hz.

[0049] Through the coordinated operation of the main receiving antenna, backup receiving antenna, RF front-end circuit, and baseband processing chip, this embodiment provides high-precision three-dimensional spatial position information for the entire system. The positioning accuracy of better than 2cm provides a precise position reference for subsequent spatial coordinate transformation and pixel-level spatiotemporal alignment of multi-source sensor data.

[0050] In one embodiment, the high-precision clock synchronization module is the core component for achieving microsecond-level time synchronization in the UAV oblique photography aerial pipeline image data high-efficiency processing system provided in this application. The high-precision clock synchronization module specifically includes three parts: an atomic clock reference source, a clock distribution network, and a timestamp marking unit. The output of the atomic clock reference source is connected to the input of the clock distribution network, and the output of the clock distribution network is connected to the timestamp marking unit and the clock inputs of each sensor and data acquisition unit.

[0051] An atomic clock reference source is used to provide a highly stable time and frequency reference signal. In this embodiment, the atomic clock reference source can be either a miniature rubidium atomic clock or a temperature-controlled crystal oscillator, which can be selected based on cost and size requirements.

[0052] When a miniature rubidium atomic clock is selected as the atomic clock reference source, the short-term frequency stability of the miniature rubidium atomic clock is better than 1×10⁻ 9 The sampling time interval τ for measuring the short-term frequency stability is 1 second; the long-term frequency stability of the miniature rubidium atomic clock is better than 1×10⁻ 8 The sampling time interval τ for measuring the long-term frequency stability is 100 s. When a cryogenic crystal oscillator is selected as the atomic clock reference source, the short-term frequency stability of the cryogenic crystal oscillator is better than 5 × 10⁻¹. 0 The sampling time interval τ for measuring the short-term frequency stability is 1 s; the long-term frequency stability of the temperature-controlled crystal oscillator is better than 1 × 10⁻ 7 The sampling time interval τ for measuring the long-term frequency stability is 100s.

[0053] The clock distribution network is used to distribute the highly stable clock signal generated by the atomic clock reference source to each sensor and data acquisition unit. In this embodiment, the clock distribution network uses a low-jitter clock buffer chip, which has a phase jitter of less than 1 ps RMS and a root mean square phase jitter of less than 1 picosecond.

[0054] After clock distribution via the clock distribution network, the clock phase deviation of each output channel is less than 50 ps, ​​meaning the clock phase difference between each output channel is less than 50 picoseconds. This distribution method enables all sensors to acquire data under a unified time reference.

[0055] The timestamp marking unit is used to accurately mark the actual trigger time of each frame of acquired data. In this embodiment, the timestamp marking unit uses the clock signal output by the atomic clock reference source as the time reference. The timestamp resolution of the timestamp marking unit is better than 10 ns.

[0056] The specific implementation of the timestamp marking is as follows: when the synchronization trigger signal arrives at the sensor, the synchronization trigger signal is simultaneously transmitted to the timestamp marking unit; the timestamp marking unit uses the clock signal output by the atomic clock reference source as a reference to record the precise arrival time of the synchronization trigger signal; then the precise arrival time is encoded into a 64-bit timestamp, and the 64-bit timestamp is appended to the header of the acquisition data frame corresponding to the synchronization trigger signal.

[0057] In the 64-bit timestamp, the high 32 bits store the number of seconds that have elapsed since the preset start time, and the low 32 bits store the nanosecond count value within the current second. A one-to-one correspondence is established between the arrival time of the synchronization trigger signal recorded by the timestamp unit and the data frame acquired by the sensor in response to the synchronization trigger signal, using the unique identifier of the synchronization trigger signal.

[0058] Through the coordinated operation of the atomic clock reference source, clock distribution network, and timestamp marking unit, this embodiment achieves timestamp marking accuracy better than 10 ns and controls the timestamp error within 1 ms. This high-precision time synchronization provides an accurate time reference for subsequent pixel-level spatiotemporal alignment of multi-source sensor data.

[0059] In one implementation, pixel-level spatiotemporal alignment is a key preprocessing step for achieving multi-source heterogeneous data fusion. Pixel-level spatiotemporal alignment includes two steps: spatial coordinate transformation and temporal registration, which are performed by the spatial coordinate transformation module and the temporal registration module, respectively.

[0060] The spatial coordinate transformation module is used to map the image data collected by each sensor to a geographic coordinate system with the current position of the UAV as the origin, thus completing the spatial domain alignment.

[0061] The spatial coordinate transformation module first acquires the relative attitude parameters of the sensor array. In this embodiment, the relative attitude parameters are acquired by scanning and measuring the sensor array already installed on the UAV using a lidar or structured light scanning device to obtain the installation yaw angle, installation pitch angle, and installation roll angle of each sensor relative to the UAV body, as well as the position offset of each sensor relative to the UAV's center of gravity. These relative attitude parameters are obtained through precise calibration after system assembly, with a calibration accuracy better than 0.05°.

[0062] The spatial coordinate transformation module also acquires precise location information provided by the positioning and orientation system. In this embodiment, the precise location information comes from the aforementioned GPS and BeiDou dual-mode positioning module and high-precision clock synchronization module.

[0063] The spatial coordinate transformation module, combining the aforementioned relative attitude parameters and precise position information, uses the pinhole camera model and spatial coordinate transformation matrix to uniformly map the image data collected by each sensor to a geographic coordinate system with the UAV's current location as the origin. The coordinate transformation formula is as follows:

[0064] In the above coordinate transformation formula, Image pixel coordinates, Three-dimensional coordinates in a geographic coordinate system. This is the transformation matrix from sensor to machine body. This is the transformation matrix from the organism to the geographic coordinate system. Wherein, It is constructed from the installation yaw angle, installation pitch angle, installation roll angle and position offset in the aforementioned relative attitude parameters; The coordinates are calculated from the real-time position and attitude information of the positioning and orientation system. The constant 1 in the homogeneous coordinate representation is used to maintain the translational consistency of the coordinate transformation, ensuring that the affine property of the transformation matrix holds in three-dimensional space. Through the above coordinate transformation, the image data collected by each sensor is unified into a geographic coordinate system with the UAV's current position as the origin, completing the spatial domain alignment.

[0065] The time registration module is used to unify image data acquired by different sensors to the same time reference point. In this embodiment, the time registration module achieves the unification of the time reference based on the timestamps of each sensor through interpolation or resampling methods.

[0066] The time registration module employs the following strategy: For timestamp errors less than 1ms, cubic spline interpolation is used for time registration. The basic principle of cubic spline interpolation is as follows: using the actual sampling time of each sensor as the interpolation node and the image data acquired at that time as the function value, a cubic spline interpolation function is constructed. Then, the interpolation function is evaluated at the target time to obtain an estimated value of the image data at that target time. The target time is the same time reference point to which all sensors are uniformly aligned. For timestamp errors greater than or equal to 1ms, a sub-pixel registration algorithm based on optical flow is used for fine synchronization. The basic principle of the sub-pixel registration algorithm based on optical flow is as follows: calculating the optical flow field between two adjacent frames, obtaining the motion vector of each pixel in the image between adjacent frames, and then inferring the precise position of the target at any time based on this motion vector, thereby achieving sub-pixel-level spatiotemporal synchronization.

[0067] Through the coordinated work of the above two steps of spatial coordinate transformation and time registration, this embodiment unifies visible light images, infrared images and ultraviolet images collected by sensors with different physical mechanisms onto the same time and space reference, providing a pixel-level aligned data foundation for subsequent multimodal feature fusion.

[0068] In one implementation, the cross-modal Transformer feature extractor is a core component of a multimodal feature fusion network based on an attention mechanism. Specifically, the cross-modal Transformer feature extractor comprises three parts: a visual Transformer encoder backbone network, a modality embedding layer, and a positional encoding module.

[0069] The visual Transformer encoder backbone network adopts a hierarchical Transformer architecture. In this embodiment, the visual Transformer encoder backbone network includes 12 attention layers and 12 feedforward network layers, with each layer containing 12 attention heads and a feature dimension of 768. This architecture can extract global contextual information from the image and capture long-distance dependencies between features of different modalities. The feedforward network layer adopts a two-layer fully connected structure. The first fully connected layer maps the feature dimension from 768 dimensions to 3072 dimensions, and the second fully connected layer maps the 3072 dimensions back to 768 dimensions. The ReLU activation function is used between the two layers.

[0070] The modality embedding layer is used to assign a learnable modality identifier vector to each type of sensor data. In this embodiment, the sensor data includes three modalities: visible light image data, infrared image data, and ultraviolet image data. The modality embedding layer assigns learnable modality identifier vectors to each of these three modalities. The dimension of the modality identifier vector is the same as the feature dimension, both being 768-dimensional. Before the feature map enters the visual Transformer encoder backbone network, the modality embedding layer adds the modality identifier vector to the original feature map along the channel dimension, achieving implicit injection of modality information. The modality identifier vector is broadcast and copied along the spatial dimension of the feature map, making the spatial size of the modality identifier vector consistent with the spatial size of the feature map, and then element-wise addition along the channel dimension is performed.

[0071] The location encoding module provides location information representation for each location in the feature map. In this embodiment, the location encoding module employs a two-dimensional learnable location encoding, with the dimension of the location encoding matching the feature dimension, both being 768-dimensional. The two-dimensional learnable location encoding is implemented through a learnable parameter matrix, the size of which is... ,in For feature map height, The width of the feature map. The feature dimension is 768. During training, the above parameter matrix is ​​optimized along with the network parameters, thereby automatically adapting to the spatial feature distribution of different scenarios.

[0072] The complete workflow of the cross-modal Transformer feature extractor is as follows: The first step involves extracting preliminary feature maps for each mode using the feature extraction networks corresponding to each sensor. In this embodiment, ResNet50 is used as the feature extraction network for visible light image data, while ResNet34 is used for both infrared and ultraviolet image data. The resolution of the preliminary feature maps output by each feature extraction network is reduced to 1 / 16 of the original image resolution.

[0073] The second step is to flatten the preliminary feature map into a sequence. The flattening operation involves flattening the spatial dimension of the feature map into a one-dimensional sequence, resulting in a sequence of length [length missing]. The sequence, where , and These represent the height and width of the feature map, respectively.

[0074] The third step involves adding a corresponding modality identifier vector to the beginning of the sequence, and simultaneously adding a two-dimensional learnable positional code to each position in the sequence. Sequences of different modalities have their own corresponding modality identifier vectors added, while sequences of all modalities have a unified two-dimensional learnable positional code added.

[0075] The fourth step involves inputting the processed sequence into a 12-layer Transformer encoder. Each Transformer encoder layer contains a multi-head self-attention mechanism and a feedforward network. The multi-head self-attention mechanism has 12 attention heads, and the calculation formula for the self-attention mechanism is as follows:

[0076] In the above formula, The query matrix is ​​formed by linear transformation of the input sequence. Mapped to obtain; The key matrix is ​​formed by the linear transformation matrix of the input sequence. Mapped to obtain; The value matrix is ​​formed by the linear transformation matrix of the input sequence. Mapped to obtain; Key matrix The transpose operation; The dimension of the key vector is 64. This is a scaling factor used to control the numerical range of the attention score and prevent gradient vanishing. The attention score is converted into a probability distribution by using a normalized exponential function; The attention score matrix is ​​obtained by multiplying the query matrix by the transpose of the key matrix. The weighted aggregation of attention outputs represents the features of each position in the sequence aggregated according to attention weights. Using a multi-head mechanism, the model calculates the outputs of 12 attention heads separately, concatenates the outputs of the 12 attention heads along the feature dimension, and finally obtains the final output of this layer through linear projection.

[0077] The fifth step involves processing the sequence representation by the 12 layers of Transformer encoders, which then fuses multimodal information. This sequence representation is then restored to a feature map form, i.e., a two-dimensional spatial structure, through a reshaping operation, which serves as the input for the subsequent multi-task learning network.

[0078] Through the processing of the aforementioned cross-modal Transformer feature extractor, image data from the visible light, infrared, and ultraviolet modalities achieve deep interaction and fusion at the feature level. The output fused feature map simultaneously contains complementary information from the three modalities, providing a unified feature input for subsequent defect detection, classification, and localization tasks.

[0079] In one implementation, the channel attention module is the core component for achieving adaptive calibration of feature channels. The channel attention module employs the classic Squeeze-and-Excitation channel attention mechanism, abbreviated as SE channel attention mechanism. Specifically, the channel attention module consists of three parts: a global average pooling layer, a fully connected layer, and a sigmoid activation function.

[0080] Global average pooling layers are used to compress the input feature map along the spatial dimension, aggregating spatial information into global statistical features along the channel dimension.

[0081] In this embodiment, the input feature map The dimension is ,in Batch size, which is the number of samples input into the network each time; The number of channels refers to the number of channels in the input feature map. The height of the feature map, in pixels; The width of the feature map, in pixels. The global average pooling layer extends along the spatial dimension. Perform global average pooling to obtain dimension . Global statistical characteristics Global average pooling compresses spatial information into a global description along the channel dimension. The calculation formula is as follows:

[0082] In the above formula, For the first The global statistical characteristic value of each channel is a scalar; Input feature map In the individual channels, spatial locations Eigenvalues ​​at; The height of the feature map; The width of the feature map; This is the spatial location index along the height direction of the feature map, with a value range of [value range missing]. to ; This is the spatial location index along the width direction of the feature map, with a value range of [value range missing]. to Based on the above calculations, each channel Obtain a global statistical feature value All channels Together they form global statistical features .

[0083] The fully connected layer is used to perform a non-linear transformation on the global statistical features, generating attention weights for each channel. In this embodiment, the global statistical features... It passes through two fully connected layers in sequence. The first fully connected layer converts the channel dimension... Mapped to The intermediate layer dimension is introduced using the ReLU activation function; the second fully connected layer then converts the intermediate layer dimension... Map back to the original number of channels The channel attention weight vector is obtained. .

[0084] The Sigmoid activation function is used to normalize the channel attention weight vector. In this embodiment, the Sigmoid activation function normalizes the channel attention weight vector. Each element in the map is mapped to The interval is used to generate a channel attention weight map. This channel attention weight map is compared with the input feature map. Channel-by-channel multiplication is performed to recalibrate features along the channel dimension. The complete computation process of the channel attention module is as follows:

[0085] In the above formula, The output feature map of the channel attention module has the same dimensions as the input feature map. Consistent, all are ; The Sigmoid activation function is used to map values ​​to... interval; This is the weight matrix of the first-level fully connected layer, with size . ; This is the weight matrix of the second-level fully connected layer, with size [size missing]. ; For global statistical features, the dimension is ReLU is an activation function used to introduce nonlinearity. This refers to matrix multiplication. This is a channel-wise multiplication operation, meaning each channel of the channel attention weight map is multiplied by the input feature map. The corresponding channels are multiplied element by element.

[0086] Through the aforementioned processing steps of global average pooling, two-stage fully connected layers, and sigmoid normalization, the channel attention module automatically learns the importance of different channels, enhances the feature representation of key channels, and suppresses the feature responses of redundant channels. After processing by the channel attention module, the output feature map is completed. While maintaining the spatial resolution, the feature representation of each channel is adaptively calibrated, providing channel-dimension optimized feature input for subsequent spatial attention modules or feature fusion operations.

[0087] In one implementation, the spatial attention module employs the classic spatial attention mechanism, namely the spatial attention part of CBAM. Specifically, the spatial attention module comprises four parts: an average pooling layer along the channel dimension, a max pooling layer along the channel dimension, a convolutional layer with a 7×7 kernel size, and a sigmoid activation function.

[0088] The average pooling layer performs average pooling on the input feature map along the channel dimension, extracting the mean statistics of the feature map along the channel dimension. The max pooling layer performs max pooling on the input feature map along the channel dimension, extracting the peak statistics of the feature map along the channel dimension. In this embodiment, the input feature map... The dimension is ,in Batch size, which is the number of samples input into the network each time; The number of channels refers to the number of channels in the input feature map. The height of the feature map, in pixels; The width of the feature map, in pixels.

[0089] The average pooling layer and the max pooling layer are respectively along the channel dimension. Perform pooling operation to obtain two dimensions. The feature descriptors for the spatial dimension. The feature descriptors obtained by average pooling contain the average response intensity of each spatial location across all channels, while the feature descriptors obtained by max pooling contain the maximum response intensity of each spatial location across all channels. Both describe the importance of spatial location from different perspectives.

[0090] Concatenate the two feature descriptors above along the channel dimension to obtain the dimension as follows: The fusion feature contains both mean and peak information for the channel dimension.

[0091] The aforementioned fusion features are fused using a convolutional layer with a kernel size of 7×7. In this embodiment, the convolutional layer has 2 input channels and 1 output channel, a kernel size of 7×7, a stride of 1, and a padding method set to ensure that the spatial dimensions of the output feature map are consistent with those of the input feature map, i.e., zero padding with a padding size of 3. After the above convolutional operation, the number of channels is compressed from 2 to 1, resulting in a single-channel spatial attention feature map.

[0092] The Sigmoid activation function is used to normalize the output of the convolutional layer. In this embodiment, the Sigmoid activation function maps each element of the spatial attention feature map of the convolutional layer output to... The interval is used to generate a spatial attention weight map. This spatial attention weight map is compared with the input feature map. Element-wise multiplication achieves feature focusing in the spatial dimension. The complete computation process of the spatial attention module is as follows:

[0093] In the above formula, represents the output feature map of the spatial attention module, and its dimension is the same as the input feature map. Figure 1 To, all; For average pooling along the channel dimension, the output dimension is ; for max pooling along the channel dimension, the output dimension is ; This is a convolution operation with a kernel size of 7×7. The input is the concatenation of two pooling results along the channel dimension, and the output dimension is... ; is the Sigmoid activation function, used to map numerical values ​​to a range; This is an element-wise multiplication operation, meaning the value at each spatial location of the spatial attention weight map is multiplied by the input feature map. The values ​​of all channels at the corresponding spatial locations are multiplied together.

[0094] Through the aforementioned processing steps of average pooling, max pooling, convolutional fusion, and sigmoid normalization, the spatial attention module automatically learns the importance of different spatial locations. Specifically, the model adaptively generates a spatial attention weight map based on the content of the input feature map, assigning higher weights to key spatial locations to enhance their feature responses and lower weights to background regions to suppress their feature responses. After processing by the spatial attention module, the output feature map is... While keeping the number of channels constant, the feature representations at each spatial location are adaptively calibrated, providing spatially optimized feature inputs for subsequent defect detection and identification tasks.

[0095] In one implementation, the shared feature encoder employs a hybrid architecture of convolutional neural networks and Transformers, specifically comprising three parts: a shallow convolutional module, multiple residual attention blocks, and a feature pyramid module. These three parts are connected sequentially, and the input image data is processed sequentially through the shallow convolutional module, multiple residual attention blocks, and the feature pyramid module, ultimately outputting a multi-scale feature map.

[0096] The shallow convolutional module is used to extract the low-level texture and edge features of the input image. In this embodiment, the shallow convolutional module contains three convolutional layers, each with a 3×3 kernel size and channel numbers of 32, 64, and 128 respectively. Each convolutional layer uses the ReLU activation function to introduce non-linearity. After processing by the shallow convolutional module, the resolution of the output feature map is reduced to 1 / 4 of the original image resolution.

[0097] Multiple residual attention blocks are used to model long-range dependencies and extract high-level semantic features. In this embodiment, a total of 6 residual attention blocks are set, and each residual attention block is connected sequentially, that is, the output of the previous residual attention block is used as the input of the next residual attention block. Each residual attention block adopts a residual connection structure, which adds the input feature map of the current block to the output feature map of the current block element-wise along the channel dimension, so that the gradient can be effectively propagated in the deep network.

[0098] Each residual attention block contains two sub-modules: a multi-head self-attention mechanism and a feedforward network. The multi-head self-attention mechanism contains eight attention heads used to capture long-distance dependencies between different locations in the feature map. The feedforward network consists of two fully connected layers with a hidden layer dimension of 2048. The first fully connected layer maps the input features from the current dimension to the 2048 dimension and applies the ReLU activation function. The second fully connected layer maps the 2048 dimension back to the original feature dimension. The two layers are connected by residual connections and layer normalization. After layer-by-layer processing through six residual attention blocks, the model can extract high-level semantic features with global context awareness.

[0099] The feature pyramid module is used for multi-scale feature fusion, outputting feature maps at different resolutions. In this embodiment, the feature pyramid module adopts a feature pyramid network structure, abbreviated as FPN structure. The feature pyramid module receives feature maps output from different levels in multiple residual attention blocks, and fuses the feature maps of different resolutions through top-down paths and lateral connections, finally outputting four feature maps of different resolutions. The resolutions of the above four feature maps are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image resolution, respectively, with corresponding channel numbers of 256, 512, 1024, and 2048.

[0100] Feature maps of different resolutions correspond to defect detection needs at different scales. High-resolution feature maps contain rich detail information and are suitable for detecting small-scale defects, while low-resolution feature maps contain sufficient semantic information and are suitable for detecting large-scale defects.

[0101] Through the collaborative work of the aforementioned shallow convolutional module, multiple residual attention blocks, and feature pyramid module, the shared feature encoder transforms the input multimodal fusion feature map into a multi-scale feature map, providing feature inputs of different scales for the three tasks of defect detection, defect classification, and defect localization in the subsequent multi-task independent decoder.

[0102] In one implementation, the multi-task independent decoder includes three parallel branches: a detection decoder, a classification decoder, and a positioning decoder.

[0103] The detection decoding head outputs the bounding box coordinates and confidence scores of defects, achieving the target detection function. In this embodiment, the detection decoding head adopts an anchorless detection head, specifically an FCOS architecture. The detection decoding head performs detection head operations using the multi-scale feature map output by the feature pyramid module to achieve multi-scale defect detection. The detection decoding head contains a total of 4 convolutional layers and 1 classification convolutional layer. The kernel size of the 4 convolutional layers is 3×3, and the number of channels is 256, 256, 256 and 4+1 respectively, where 4+1 represents the 4 parameters of the bounding box coordinates plus 1 confidence score. The kernel size of the classification convolutional layer is 3×3, the number of channels is 256, and the number of output nodes is equal to the number of defect categories, used to predict the defect category for each spatial location. The detection decoding head outputs the bounding box coordinates x_min, y_min, x_max, y_max of the defect and the confidence score.

[0104] The classification decoding head outputs the probability distribution of each defect category, supporting multi-label classification. In this embodiment, the classification decoding head adopts a multilayer perceptron structure, containing three fully connected layers. The hidden layer dimensions of the three fully connected layers are 1024 and 512, respectively, and the number of output nodes equals the number of defect categories. The classification decoding head receives the highest-level feature map output by the feature pyramid module, which has a resolution of 1 / 32 of the original image resolution. The classification decoding head compresses the spatial dimension of the highest-level feature map using global average pooling before inputting it into the fully connected layers. The classification decoding head uses the Sigmoid activation function to support multi-label classification and outputs the probability distribution of each defect category.

[0105] The localization decoding head is used to output a pixel-level segmentation mask of defects in the image, achieving precise localization. In this embodiment, the localization decoding head adopts a fully convolutional segmentation head, containing four convolutional layers and one upsampling layer. The kernel size of the four convolutional layers is 3×3, and the number of channels is 256, 128, 64, and 1 respectively. The upsampling layer is placed after the last convolutional layer and is used to restore the spatial resolution of the feature map to the same level as the input feature map. Figure 1 To. Locate the pixel-level segmentation mask of the final output defect of the decoding head, with resolution matching the input features. Figure 1 To.

[0106] A hard-sharing mechanism is used between the shared feature encoder and the multi-task independent decoder. Specifically, this means that all parameters of the shared feature encoder are fully shared across the three branches of the multi-task independent decoder, while parameters are not shared between the individual task-specific decoder parts. In this embodiment, the shared parameters between tasks account for no less than 60%, primarily concentrated in the shallow convolutional modules and residual attention blocks. Parameters are not shared between the individual task-specific decoder parts. This architecture design reduces the computational load and number of parameters while maintaining the accuracy of each task.

[0107] Through the collaborative work of the three parallel branches of the detection decoding head, classification decoding head, and positioning decoding head, the multi-task independent decoder simultaneously outputs three results: the bounding box coordinates and confidence score of the defect, the multi-label classification probability distribution, and the pixel-level segmentation mask, providing multi-dimensional output for the comprehensive detection of overhead pipeline defects.

[0108] In one implementation, the online hard example mining mechanism specifically includes three parts: a sample loss calculation module, a loss ranking module, and a batch sampling module.

[0109] The sample loss calculation module is used to calculate the classification loss or detection loss for each sample in the training batch. In this embodiment, the classification loss is the focal loss, and the detection loss is the IoU loss (Intersection over Union Loss).

[0110] For classification tasks, the formula for calculating focus loss is as follows:

[0111] In the above formula for focus loss, This is the model's predicted probability of the true class, that is, the probability value of the correct class label in the model output, with a value ranging from 0 to 1; This is a focusing parameter used to adjust the proportion of loss contribution between easily separated and difficult-to-separate samples, and its value is 2. The balancing parameter is used to adjust the loss weights for positive and negative samples, with a value of 0.25 for positive samples and 0.75 for negative samples. Positive samples are those containing defects in the image, while negative samples are those not containing defects.

[0112] For detection tasks, the formula for calculating IoU loss is as follows:

[0113] In the above IoU loss formula, The bounding box predicted by the model contains the coordinates of the four vertices of the bounding box; These are the actual bounding boxes annotated manually, i.e., the actual locations of defects annotated in the training data; To predict the intersection area between the bounding box and the true bounding box; To predict the area of ​​the union of the bounding box and the true bounding box; The IoU loss value ranges from 0 to 1. The loss value is 0 when the predicted bounding box completely overlaps with the true bounding box, and 1 when they do not overlap at all.

[0114] The loss ranking module is used to sort the samples in descending order based on their loss values. In this embodiment, after the sample loss calculation module calculates the loss value for each sample, the loss ranking module arranges all samples in descending order of their loss values, placing samples with larger loss values ​​at the beginning of the sequence. Samples with larger loss values ​​are considered hard samples; these samples are difficult for the model to classify or locate correctly during training and require more attention during training.

[0115] The batch sampling module prioritizes samples with the highest loss values ​​to form training batches. In this embodiment, the batch sampling module selects samples with the highest loss values ​​from the descending sequence output by the loss ranking module. The sampling ratio is adjusted via hyperparameters; the default hard sample sampling ratio is set to 70%, meaning that 70% of the samples in each training batch are hard samples with high loss values, and 30% are randomly sampled samples. Random sampling ensures the diversity of training data and prevents the model from overfitting to hard samples while ignoring the overall data distribution.

[0116] Through the collaborative work of the above three parts—sample loss calculation module, loss ranking module, and batch sampling module—the online hard example mining mechanism dynamically adjusts the sampling distribution of training samples during model training, enabling the model to obtain more learning opportunities from hard samples during training, thereby improving the model's ability to identify samples that are difficult to detect, such as minor defects and occlusion defects.

[0117] In one implementation, the adaptive defect classification model employs an end-to-end deep neural network architecture, specifically comprising three parts: a backbone feature extraction network, a feature fusion module, and a classification output layer. These three parts are connected sequentially, and the input image data is processed sequentially through the backbone feature extraction network, the feature fusion module, and the classification output layer, ultimately outputting the probability of existence for each defect category.

[0118] The backbone feature extraction network is used to extract multi-scale feature maps from the input image, reducing computational cost while maintaining feature extraction capabilities to adapt to the computing power limitations of the UAV. In this embodiment, the backbone feature extraction network adopts the lightweight backbone network EfficientNet-B0. This network uses a composite scaling method to jointly scale the network depth, width, and input resolution, keeping the computational cost below 0.39 G FLOPs and the number of model parameters below 5.3 M while maintaining feature extraction capabilities. The feature fusion module is used to upsample and concatenate the multi-scale features output by the backbone network to form a fused feature map. In this embodiment, the feature fusion module receives multi-scale features output by EfficientNet-B0, specifically from layers 3, 5, and 7 of EfficientNet-B0, with corresponding feature map resolutions of 1 / 8, 1 / 16, and 1 / 32 of the original image resolution, respectively. The feature fusion module upsamples the feature maps at the three scales using bilinear interpolation, adjusting the spatial resolution of the three feature maps to 1 / 8 of the original image resolution. Then, the maps are concatenated along the channel dimension to form a fused feature map. The fused feature map has 320 channels.

[0119] The classification output layer outputs the probability distribution of each defect category, supporting multi-label classification. In this embodiment, the classification output layer adopts a structure of global average pooling followed by a fully connected layer. Global average pooling compresses the spatial dimension of the fused feature map to 1×1, that is, it takes the average of all spatial locations of each channel, aggregating the spatial information of each channel into a scalar value. The number of input nodes of the fully connected layer is 320, which is the number of channels of the fused feature map; the number of output nodes is set to 25, corresponding to the 25 defect categories. The classification output layer uses the Sigmoid activation function to support multi-label classification. The Sigmoid activation function maps the output of the fully connected layer to the interval between 0 and 1, outputting 25 probability values, each probability value representing the probability of the existence of the corresponding defect category.

[0120] In this embodiment, the adaptive defect classification model supports 25 defect categories, including conductor breakage, conductor wear, conductor corrosion, conductor rust, abnormal conductor heating, insulator damage, insulator cracks, insulator contamination, insulator zero value, tower rust, tower deformation, tower tilting, grounding device damage, clamp overheating, lead wire overheating, abnormal discharge of overhead pipeline, ultraviolet corona discharge, ultraviolet spark discharge, and other visible defects.

[0121] Through the collaborative work of the aforementioned backbone feature extraction network, feature fusion module, and classification output layer, the adaptive defect classification model achieves automatic multi-label discrimination of 25 defect categories in the input image, providing the final classification result for overhead pipeline defect detection.

[0122] In one implementation, the data storage and transmission module uniformly stores the raw data collected by multi-source sensors, the preprocessed aligned data, the intermediate output of the feature fusion network, and the final defect detection results according to a preset data format. The aforementioned raw data includes visible light images, infrared images, and ultraviolet images.

[0123] Data storage utilizes a high-reliability solid-state storage array, specifically a RAID 0 array with an NVMe interface. Sequential write speeds are no less than 3000MB / s, and sequential read speeds are no less than 2500MB / s, supporting high-speed writing and real-time reading. The RAID 0 array combines multiple solid-state drives into a single logical storage unit, with data stored in stripes across the drives, achieving parallel performance improvements for both read and write operations. The wireless communication module supports 4G cellular networks, 5G cellular networks, or dedicated wireless links, with an upload bandwidth of no less than 20Mbps. Under 5G network coverage, the wireless communication module enables real-time video streaming; under 4G network or dedicated wireless link conditions, it enables timed transmission of image data.

[0124] Through the aforementioned data storage and transmission modules, the raw data collected by multi-source sensors, the pre-processed aligned data, the intermediate output of the feature fusion network, and the final defect detection results can be reliably stored and transmitted back to the ground control station in real time, providing data support for ground-based monitoring and subsequent analysis.

[0125] In one implementation, the defect labeling and management platform provides manual labeling tools for defect samples, a labeling quality review process, and version management functions for labeling data.

[0126] The manual annotation tool supports three annotation modes: bounding box annotation, pixel-level segmentation annotation, and keypoint annotation. Bounding box annotation is used for object detection tasks, where annotators draw rectangles to mark the image regions where defects are located; pixel-level segmentation annotation is used for semantic segmentation tasks, where annotators draw the precise outline of defects pixel by pixel; and keypoint annotation is used for keypoint localization tasks, where annotators mark the key feature points of defects to achieve precise localization.

[0127] The annotation quality review process adopts a double-blind review mechanism, that is, two annotators independently annotate the same image. When the annotation results are consistent, the review is approved. When the annotation results are inconsistent, the senior annotator will arbitrate.

[0128] The version management function for labeled data adopts the Git version control mechanism. Each label change automatically generates a new version number, supporting version rollback and version comparison. The defect labeling and management platform automatically generates label file formats that meet the requirements of deep learning training, including COCO, YOLO, and Cityscapes formats. The COCO format is used for object detection and instance segmentation tasks, the YOLO format is used for object detection tasks, and the Cityscapes format is used for semantic segmentation tasks.

[0129] Through the aforementioned defect annotation and management platform, the system can perform standardized annotation and quality control on defect samples, providing high-quality labeled data support for the training and continuous iteration of deep learning models.

[0130] In summary, the efficient processing system for UAV oblique photography of overhead pipeline imagery proposed in this application revolves around four stages: data acquisition, alignment, fusion, and recognition. At the data acquisition end, a high-precision synchronous control unit enables microsecond-level synchronous triggering of visible light, infrared, and ultraviolet sensors, while a distributed mounting structure ensures stable integration of multiple sensors. For positioning and timing, a dual-mode GPS and BeiDou positioning module and a high-precision clock synchronization module are configured to achieve positioning accuracy better than 2cm and timestamp marking accuracy better than 10ns, providing a benchmark for subsequent spatiotemporal alignment. Based on this, the system performs pixel-level spatiotemporal alignment, unifying multi-source images into the same time and space coordinate system. In the feature fusion stage, a cross-modal Transformer feature extractor is constructed, coupled with a parallel mechanism of channel attention and spatial attention, to achieve adaptive weighted fusion of multimodal features. In the defect recognition stage, a multi-task learning network architecture is used to simultaneously execute detection, classification, and localization tasks, combined with online hard example mining and focus loss to enhance hard sample learning. In the inference and deployment stage, a lightweight classification model supports multi-label discrimination of 25 defect types. In addition, the system integrates a data storage and transmission module and a defect labeling and management platform. Through the above complete technical chain, this application effectively solves the problem of multi-source heterogeneous data fusion and realizes comprehensive automated detection of multi-dimensional defects in overhead pipelines.

[0131] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects.

[0132] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A high-efficiency processing system for aerial pipeline image data obtained by oblique photography from unmanned aerial vehicles (UAVs), characterized in that, include: The high-precision synchronization control unit is used to perform microsecond-level synchronous triggering and acquisition of visible light cameras, infrared thermal imaging sensors, and ultraviolet discharge detection sensors mounted on UAVs. The system includes a GPS and BeiDou dual-mode positioning module and a high-precision clock synchronization module. The GPS and BeiDou dual-mode positioning module is used to acquire the three-dimensional spatial position information of the UAV, and the high-precision clock synchronization module is used to timestamp the data collected by each sensor. A pixel-level spatiotemporal alignment module is used to unify the visible light image data, infrared image data and ultraviolet image data collected by each sensor to the same time and space reference based on the three-dimensional spatial position information and the timestamp mark. A cross-modal Transformer feature extractor is used to extract features and perform cross-modal fusion on the spatiotemporally aligned visible light image data, infrared image data, and ultraviolet image data, and output a fused feature map. A multi-task independent decoder is used to perform three tasks in parallel: defect detection, defect classification, and defect localization, based on the fused feature map.

2. The high-efficiency processing system for UAV oblique photography aerial pipeline image data according to claim 1, characterized in that, The high-precision synchronization control unit includes a clock signal generator, a trigger delay compensation module, and multiple synchronous trigger output channels; The clock signal generator is used to generate a clock signal with a frequency of not less than 100MHz; The trigger delay compensation module is used to pre-determine and store the inherent trigger delay of each sensor, and automatically add the corresponding delay compensation value to each trigger signal according to the pre-stored delay of each sensor when the synchronous trigger signal is issued, so that the deviation between the actual sampling time of each sensor and the predetermined sampling time of the system is less than 10μs. The number of the multi-channel synchronous trigger output channels is 3, corresponding to the visible light camera, the infrared thermal imaging sensor and the ultraviolet discharge detection sensor respectively. Each output channel in the multi-channel synchronous trigger output channel is independent of each other and has a time accuracy better than 100ns.

3. The high-efficiency processing system for UAV oblique photography aerial pipeline image data according to claim 1, characterized in that, The GPS and BeiDou dual-mode positioning module includes a main receiving antenna, a backup receiving antenna, an RF front-end circuit, and a baseband processing chip. The main receiving antenna and the backup receiving antenna are arranged in a spatial diversity configuration, and the antenna spacing is not less than the maximum geometric dimension of the sensor array. The radio frequency front-end circuit supports parallel reception of GPS L1 frequency, GPS L5 frequency, and BeiDou B1 frequency, BeiDou B2 frequency, and BeiDou B3 frequency. The baseband processing chip has a built-in carrier phase differential positioning algorithm, which achieves a three-dimensional positioning accuracy better than 2cm and a position update frequency of 10Hz. The high-precision clock synchronization module includes an atomic clock reference source, a clock distribution network, and a timestamp marking unit. The atomic clock reference source is used to provide a time and frequency reference signal; The clock distribution network is used to distribute the clock signal generated by the atomic clock reference source to each sensor and data acquisition unit, and the clock phase deviation of each output channel is less than 50ps; The timestamp marking unit is used to mark the actual trigger time of each frame of acquired data, with the clock signal output by the atomic clock reference source as the time reference. The timestamp resolution is better than 10ns.

4. The high-efficiency processing system for UAV oblique photography aerial pipeline image data according to claim 3, characterized in that, The atomic clock reference source is a miniature rubidium atomic clock or a temperature-controlled crystal oscillator; When the atomic clock reference source is a miniature rubidium atomic clock, the short-term frequency stability of the miniature rubidium atomic clock at a sampling time interval of 1 second is better than 1×10⁻. 9 The long-term frequency stability at a sampling time interval of 100s is better than 1×10⁻ 8 ; When the atomic clock reference source is a temperature-controlled crystal oscillator, the short-term frequency stability of the temperature-controlled crystal oscillator at a sampling time interval of 1 second is better than 5×10⁻¹. 0 The long-term frequency stability at a sampling time interval of 100s is better than 1×10⁻ 7 .

5. The high-efficiency processing system for UAV oblique photography aerial pipeline image data according to claim 1, characterized in that, It also includes a defect labeling and management platform; The defect annotation and management platform provides manual annotation tools for defect samples, annotation quality review processes, and version management functions for annotation data; The manual annotation tool supports three annotation modes: rectangular box annotation, pixel-level segmentation annotation, and key point annotation. The annotation quality review process adopts a double-blind review mechanism, in which two annotators independently annotate the same image. When the annotation results are consistent, the review is approved; when the annotation results are inconsistent, a senior annotator will arbitrate. The version management function of the labeled data adopts the Git version control mechanism. Each time the label is changed, a new version number is automatically generated, and version rollback and version comparison are supported. The defect annotation and management platform automatically generates annotation file formats that meet the requirements of deep learning training, including COCO format, YOLO format, and Cityscapes format.

6. The high-efficiency processing system for UAV oblique photography aerial pipeline image data according to claim 1, characterized in that, The pixel-level spatiotemporal alignment module includes a spatial coordinate transformation module and a temporal registration module; The spatial coordinate transformation module is used to obtain the installation yaw angle, installation pitch angle, and installation roll angle of each sensor relative to the UAV body, as well as the position offset of each sensor relative to the UAV's center of gravity. The calibration accuracy is better than 0.05°. The spatial coordinate transformation module, combined with the position information provided by the GPS and BeiDou dual-mode positioning module and the high-precision clock synchronization module, uses a pinhole camera model and a spatial coordinate transformation matrix to uniformly map the image data collected by each sensor to a geographic coordinate system with the UAV's current position as the origin. The coordinate transformation formula is: ； in, Image pixel coordinates, Three-dimensional coordinates in a geographic coordinate system. This is the transformation matrix from sensor to machine body. This is the transformation matrix from the organism to the geographic coordinate system; The time registration module is used to perform time registration using cubic spline interpolation when the timestamp error is less than 1ms, and to perform fine synchronization processing using an optical flow-based subpixel registration algorithm when the timestamp error is greater than or equal to 1ms.

7. The high-efficiency processing system for UAV oblique photography aerial pipeline image data according to claim 1, characterized in that, The cross-modal Transformer feature extractor includes a visual Transformer encoder backbone network, a modality embedding layer, and a position encoding module; The visual Transformer encoder backbone network consists of 12 attention layers and 12 feedforward network layers, with 12 attention heads per layer and 768 feature dimensions. The modality embedding layer is used to assign learnable modality identifier vectors to the visible light image data, the infrared image data, and the ultraviolet image data, respectively. The modality identifier vectors have a dimension of 768. Before the feature map enters the visual Transformer encoder backbone network, the modality embedding layer adds the modality identifier vectors to the original feature map in the channel dimension. The location encoding module employs two-dimensional learnable location encoding, which uses a size of... The learnable parameter matrix is ​​implemented, where and These represent the height and width of the feature map, respectively.

8. The high-efficiency processing system for UAV oblique photography aerial pipeline image data according to claim 1, characterized in that, It also includes a shared feature encoder, which adopts a hybrid architecture of convolutional neural network and Transformer, including a shallow convolutional module, 6 residual attention blocks and a feature pyramid module; The shallow convolutional module contains three convolutional layers, each with a kernel size of 3×3 and channel numbers of 32, 64, and 128 respectively. Each of the residual attention blocks is connected sequentially. Each residual attention block adopts a residual connection structure and contains a multi-head self-attention mechanism and a feedforward network. The multi-head self-attention mechanism contains 8 attention heads, and the feedforward network contains two fully connected layers with a hidden layer dimension of 2048. The feature pyramid module adopts a feature pyramid network structure, receives feature maps output from different levels in multiple residual attention blocks, and fuses feature maps of different resolutions through top-down paths and lateral connections to output four feature maps of different resolutions. The resolutions of the four feature maps are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original image resolution, respectively.

9. The high-efficiency processing system for UAV oblique photography aerial pipeline image data according to claim 8, characterized in that, The multi-task independent decoder includes three parallel branches: a detection decoder, a classification decoder, and a localization decoder. The detection decoding head is an anchorless detection head, which includes 4 convolutional layers and 1 classification convolutional layer. The detection head performs calculations using the multi-scale feature map output by the feature pyramid module, and outputs the bounding box coordinates and confidence scores of the defects. The classification decoding head adopts a multilayer perceptron structure, which includes three fully connected layers. The hidden layer dimensions of the three fully connected layers are 1024 and 512, respectively. The number of output nodes is equal to the number of defect categories. The classification decoding head receives the highest-level feature map output by the feature pyramid module, inputs it into the fully connected layer after global average pooling, and uses the Sigmoid activation function to output the probability distribution of each defect category. The positioning decoding head adopts a fully convolutional segmentation head, which includes 4 convolutional layers and 1 upsampling layer. The kernel size of the 4 convolutional layers is 3×3, and the number of channels is 256, 128, 64 and 1 respectively. The upsampling layer is set after the last convolutional layer and is used to restore the spatial resolution of the feature map to be consistent with the input feature map and output a pixel-level segmentation mask of the defect. All parameters of the shared feature encoder are fully shared among the three branches of the multi-task independent decoder, with the percentage of shared parameters between tasks being no less than 60%.

10. The high-efficiency processing system for UAV oblique photography aerial pipeline image data according to claim 1, characterized in that, It also includes an online hard case mining mechanism and an adaptive defect classification model; The online hard example mining mechanism includes a sample loss calculation module, a loss ranking module, and a batch sampling module; The sample loss calculation module is used to calculate the loss value for each sample in the training batch. The classification loss uses focus loss and the detection loss uses IoU loss. The loss sorting module is used to sort all samples in descending order according to the loss value; The batch sampling module is used to select samples with the highest loss values ​​from the descending sequence to form a training batch, wherein the sampling ratio of difficult samples is 70% and the remaining 30% are random samples. The adaptive defect classification model adopts an end-to-end deep neural network architecture, including a backbone feature extraction network, a feature fusion module, and a classification output layer; The backbone feature extraction network adopts the lightweight backbone network EfficientNet-B0; The feature fusion module receives the feature maps of the 3rd, 5th and 7th layers output by the backbone feature extraction network, which correspond to 1 / 8, 1 / 16 and 1 / 32 of the original image resolution, respectively. After performing bilinear interpolation upsampling on the feature maps of the three scales, the feature fusion module uniformly adjusts the spatial resolution to 1 / 8 of the original image resolution and splices them in the channel dimension to form a fused feature map. The number of channels in the fused feature map is 320. The classification output layer adopts a structure of global average pooling followed by a fully connected layer. The number of input nodes in the fully connected layer is 320, and the number of output nodes is 25. The Sigmoid activation function is used to output the existence probability of 25 defect categories.

Images