Detection method based on mooring platform unmanned aerial vehicle and electronic device

Abstract

The invention relates to the technical field of unmanned aerial vehicles, and discloses a detection method and an electronic device based on a mooring platform unmanned aerial vehicle, and the method comprises the steps: obtaining visible light, infrared, laser radar, positioning and inertial measurement data; constructing a local geographic coordinate system and realizing space projection and time sequence synchronization of multi-modal data; performing dynamic weighted fusion on each modal feature through a multi-modal cross attention fusion network to generate a high-dimensional semantic feature map; and performing target detection and thermal anomaly discrimination based on the feature map. The electronic device comprises a multi-modal sensing unit, a space-time alignment module, a time synchronization module, a multi-modal fusion module and a target discrimination module. According to the invention, through a unified space-time alignment and adaptive fusion mechanism, high-precision and high-robustness comprehensive detection can still be realized under severe conditions such as dark light, haze or shielding, and long-time stable operation is guaranteed by relying on a mooring platform.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of unmanned aerial vehicles, and particularly relates to a detection method and an electronic device based on a tethered platform unmanned aerial vehicle. BACKGROUND

[0002] With the wide application of unmanned aerial vehicles in power inspection, infrastructure monitoring and emergency rescue fields, intelligent detection systems based on aerial platforms have become a key technical means to improve operation efficiency and safety. Traditional unmanned aerial vehicle detection schemes mostly rely on single sensor modalities (such as visible light cameras or infrared thermal imagers) for target recognition and state evaluation, and their sensing ability is limited by the physical characteristics and environmental adaptability of the sensors. In complex field or urban environments, factors such as light changes, weather disturbances (such as fog, rain and snow), and target occlusion can easily cause single-source data distortion or information loss, resulting in misjudgment or missed detection, which is difficult to meet the needs of high-reliability detection tasks.

[0003] Multi-modal sensor fusion technology is considered an important direction to break through the above limitations. This technology processes data from multiple sensors such as visible light, infrared, laser radar, etc. in a coordinated manner, aiming to build a more comprehensive and robust environmental perception capability. However, existing fusion methods mostly use back-end software alignment or rough timestamp matching for data integration, lacking high-precision spatio-temporal synchronization mechanisms for multi-source signals on the edge side. Especially in the tethered platform scenario with high-speed flight or dramatic attitude changes, the data collected by different sensors have significant deviations in time phase and spatial coordinates, leading to semantic misplacement or feature conflicts in the fusion results. For example, dark areas formed by shadows in visible light images may be misjudged as equipment failure, but if combined with the synchronous acquisition of infrared temperature field data, such false positive judgments can be quickly ruled out.

[0004] The real-time, accurate synchronization and efficient fusion of multi-modal sensor information have not been effectively solved on the tethered unmanned aerial vehicle platform; the on-board computing resources are limited, making it difficult to support complex time calibration algorithms and large-scale data caching; on the other hand, there is a lack of lightweight fusion architecture for specific detection tasks, making the multi-source data collected unable to form a synergistic gain at the edge. SUMMARY

[0005] The application provides a detection method and electronic device based on a tethered platform unmanned aerial vehicle, aiming to solve the information island and misjudgment problem caused by independent analysis of multi-source sensor data in the prior art, and the technical defects of poor adaptability and insufficient robustness of single modal perception in complex environmental conditions. By constructing a multi-modal spatio-temporal alignment fusion architecture for the tethered platform unmanned aerial vehicle, the application realizes the deep fusion of visible light, infrared thermal imaging, laser radar point cloud and high-precision positioning information in a unified spatio-temporal coordinate system, thereby maintaining high confidence target recognition and state discrimination ability in dark light, haze, shielding and other harsh environments.

[0006] The application provides a detection method based on a tethered platform unmanned aerial vehicle, the method comprising the following steps:

[0007] Obtaining first image data output by a visible light imaging unit carried by the tethered platform unmanned aerial vehicle;

[0008] Obtaining second image data output by an infrared thermal imaging unit carried by the tethered platform unmanned aerial vehicle;

[0009] Obtaining three-dimensional point cloud data output by a laser radar unit carried by the tethered platform unmanned aerial vehicle;

[0010] Obtaining real-time three-dimensional position coordinates output by a global positioning system module carried by the tethered platform unmanned aerial vehicle;

[0011] Obtaining attitude angle, angular velocity and acceleration data output by an inertial measurement unit module carried by the tethered platform unmanned aerial vehicle;

[0012] Based on the real-time three-dimensional position coordinates and attitude angle data, a local geographic coordinate system with the unmanned aerial vehicle body as the origin is constructed;

[0013] The first image data, the second image data and the three-dimensional point cloud data are projected into the local geographic coordinate system respectively to generate a multi-modal perception data set with a unified spatial reference base;

[0014] Timestamp synchronization processing is performed on the multi-modal perception data set, so that the modal data is aligned to the same sampling time in the time dimension;

[0015] Based on the synchronized multi-modal perception data set, a multi-channel feature tensor is constructed, wherein each channel corresponds to a spatial feature map of one modal;

[0016] The multi-channel feature tensor is input into a pre-trained multi-modal cross-attention fusion network, the network dynamically adjusts the contribution degree of each modal under the current environmental condition through a channel weight distribution mechanism, and outputs a fused high-dimensional semantic feature map;

[0017] Based on the high-dimensional semantic feature map, target detection and state discrimination operations are performed to generate comprehensive detection results containing target categories, spatial positions, geometric contours and thermodynamic abnormality indicators.

[0018] As an embodiment of the present application, the timestamp synchronization process specifically includes: taking the data stream output by the inertial measurement unit as the main clock source, interpolating and resampling the raw data streams of the visible light imaging unit, the infrared thermal imaging unit and the laser radar unit, and unifying the time resolution of all modal data to 100 Hz.

[0019] As an embodiment of the present application, the projection of the first image data, the second image data and the three-dimensional point cloud data into the local geographic coordinate system specifically includes: using a pinhole camera model to back-project the pixel points in the visible light image and the infrared image into rays, and combining the depth information provided by the laser radar point cloud to calculate the three-dimensional coordinates of each pixel point in the local geographic coordinate system; for the pixel region without corresponding depth information, a neighborhood depth propagation algorithm is used for filling.

[0020] As an embodiment of the present application, the multi-modal cross-attention fusion network includes a modal embedding layer, a cross-modal interaction layer and a feature reconstruction layer connected in turn; the modal embedding layer performs channel normalization and position coding on the spatial feature maps of each modality; the cross-modal interaction layer uses a multi-head self-attention mechanism to calculate the correlation weight between any two modal feature channels, and aggregates the cross-modal information according to the weight; the feature reconstruction layer performs nonlinear transformation and dimension compression on the aggregated features to output a single-channel fusion semantic feature map.

[0021] As an embodiment of the present application, the target detection and state discrimination operation specifically includes: sliding a detection window of a preset size on the high-dimensional semantic feature map, extracting a local feature vector for each window region; inputting the local feature vector into a classification subnetwork and a regression subnetwork; the classification subnetwork outputs a probability value of whether there is a target to be detected in the window, and the regression subnetwork outputs the offset of the target bounding box and the thermal anomaly score; when the probability value is greater than a preset threshold and the thermal anomaly score exceeds the normal equipment operating temperature range, the target is determined to be a faulty equipment.

[0022] As an embodiment of the present application, the calculation method of the thermal anomaly score is: extracting the average radiation temperature value of the target region from the second image data output by the infrared thermal imaging unit, subtracting the reference temperature value of the same type of equipment under the same working condition, and then dividing by the maximum temperature rise threshold allowed by the type of equipment, and the obtained ratio is the thermal anomaly score.

[0023] The present application provides a kind of detection electronic device based on tethered platform unmanned aerial vehicle, the device includes:

[0024] a visible light imaging unit for acquiring visible light image data of the object under inspection;

[0025] an infrared thermal imaging unit for acquiring infrared radiation image data of the object under inspection;

[0026] a laser radar unit for acquiring three-dimensional point cloud data of the environment around the object under inspection;

[0027] a global positioning system module for outputting real-time three-dimensional position coordinates of the UAV;

[0028] an inertial measurement unit module for outputting attitude angle, angular velocity and acceleration data of the UAV;

[0029] a space-time alignment module for constructing a local geographic coordinate system based on the three-dimensional position coordinates and attitude angle data, and projecting the visible light image data, infrared image data and three-dimensional point cloud data into the coordinate system to form spatially aligned multi-modal data;

[0030] a time synchronization module for resampling each sensor data stream with the inertial measurement unit data as the master clock to achieve accurate alignment in the time dimension;

[0031] a multi-modal fusion module for constructing a multi-channel feature tensor from the time-synchronized multi-modal data, and generating a fused semantic feature map through a multi-modal cross-attention fusion network;

[0032] a target discrimination module for performing target detection and state discrimination based on the fused semantic feature map, and outputting a comprehensive detection result containing target class, location, contour and thermal anomaly indicators.

[0033] As an embodiment of the present application, the visible light imaging unit adopts a global shutter complementary metal oxide semiconductor image sensor with a resolution of not less than 4000x3000 pixels and a frame rate of not less than 30 Hz; the infrared thermal imaging unit adopts a non-cooled microbolometer focal plane array with a thermal sensitivity of not more than 50 mK and a spatial resolution of 640x480 pixels; and the laser radar unit is a mechanical rotating single-line or multi-line laser radar with a ranging range of 0.5 meters to 200 meters and a ranging accuracy of ±2 centimeters.

[0034] As an embodiment of the present application, the space-time alignment module internally integrates a coordinate transformation engine, which performs data projection according to a rigid body transformation formula determined by the position coordinates and Euler angle parameters of the UAV, and the rotation matrix of which is composed of the roll angle, pitch angle and yaw angle in a specific order.

[0035] As an embodiment of the present application, the multi-modal cross attention fusion network is deployed in an airborne embedded graphics processor, which has a floating point operation capability of not less than 20 trillion times per second, a display memory capacity of not less than 8 gigabytes, and supports a real-time inference delay of less than 50 milliseconds.

[0036] As an embodiment of the present application, the target discrimination module is built-in with a device type knowledge base, which stores standard appearance templates, typical heat distribution patterns and allowable temperature rise thresholds of various inspected devices; when performing state discrimination, the target discrimination module matches the detected target features with corresponding entries in the knowledge base to determine whether an abnormality is constituted.

[0037] Compared with the prior art, the present application has the beneficial effects that:

[0038] 1. The present application fundamentally solves the misjudgment problem caused by isolated processing of sensor data in traditional detection methods by constructing a multi-modal perception fusion system based on a tethered platform unmanned aerial vehicle as a carrier.

[0039] 2. The shadow area presenting as a suspected defect in the visible light image, if its corresponding infrared temperature data is within the normal range, the fusion network will automatically suppress the response weight of the visible light channel to avoid false alarms; otherwise, in the fog or low light environment, the visible light imaging quality is reduced, and the system increases the weight of the infrared and laser radar channels to ensure the detection continuity.

[0040] 3. By accurately aligning and dynamically weighting the multi-source data in a unified space-time coordinate system, the present application significantly improves the detection robustness and accuracy under complex weather and lighting conditions. At the same time, the continuous power supply provided by the tethered platform ensures the long-term stable operation of high-power sensors and computing units, overcoming the bottleneck of limited endurance of ordinary battery-powered unmanned aerial vehicles, and is suitable for key infrastructure scenarios such as substations, transmission lines, chemical plants and other scenes that require long-term fixed-point inspection.

[0041] 4. The multi-modal cross attention mechanism used in the present application can dynamically adjust the credibility of each sensor according to the environment, without the need for manual setting of fixed fusion rules, realizing truly adaptive intelligent detection. DETAILED DESCRIPTION

[0042] Fig. 1 is the overall technical scheme architecture schematic diagram of the detection method and electronic device based on a tethered platform unmanned aerial vehicle proposed by the present application;

[0043] Fig. 2 is the core principle framework schematic diagram of the multi-modal cross attention fusion network in the present application;

[0044] Fig. 3is a logical flow framework diagram of the spatio-temporal alignment processing of multi-source sensor data in the application;

[0045] Fig. 4 is a logical flow framework diagram of target detection and state discrimination operation in the application;

[0046] Fig. 5 is a multi-level interaction relationship and data flow diagram of the tethered platform UAV and the ground control terminal in the application;

[0047] Fig. 6 is a principle diagram of the projection and fusion of multi-modal perception data in the local geographic coordinate system in the application. DETAILED DESCRIPTION

[0048] Please refer to the accompanying Figs. 1 to 6 The application provides a detection method and electronic device based on a tethered platform UAV, which is characterized by accurate alignment and dynamic fusion of multi-modal sensors in a unified spatio-temporal coordinate system, realizing high-confidence recognition and state discrimination of the detected object. The steps of the method will be described layer by layer, and the structure of the electronic device supporting the operation of the method will be simultaneously disclosed.

[0049] The method starts from step S1: acquiring first image data output by a visible light imaging unit carried by the tethered platform UAV. The visible light imaging unit adopts a global shutter complementary metal oxide semiconductor image sensor with a resolution not less than 4000*3000 pixels, and a frame rate not less than 30 Hz. The image sensor captures optical reflection information of the detected area at a fixed sampling period, generating a two-dimensional pixel array containing red, green and blue channels. Each pixel records the radiation intensity value of the corresponding spatial position in a specific wavelength range. The image data is output in raw Bayer format or RGB format after on-chip processing, and is attached with a hardware timestamp generated by an internal crystal oscillator. The timestamp has a precision of microseconds and is used for subsequent time alignment with other sensor data. The image data stream is transmitted to the onboard main control unit through a high-speed serial interface and stored in a double-buffer memory area, ensuring that the current frame is read and preprocessed before the next frame arrives.

[0050] S2: Obtain second image data output by an infrared thermal imaging unit carried by the tethered platform UAV. The infrared thermal imaging unit uses a non-cooled microbolometer focal plane array, which has a thermal sensitivity of no more than 50 mK and a spatial resolution of 640*480 pixels. The unit converts the medium-long wave infrared radiation energy emitted by the detected object into an electrical signal, and outputs the absolute temperature value corresponding to each pixel point after non-uniformity correction and temperature calibration. The second image data also has an independent time stamp, and its generation mechanism is decoupled from the visible light imaging unit, so there is an inherent time sequence offset. The infrared image data is transmitted to the main control unit in the form of a single-channel floating-point matrix, with a value unit of degrees Celsius, and a dynamic range covering -40 degrees Celsius to 500 degrees Celsius, which is sufficient to cover the full working condition spectrum of industrial equipment from low temperature shutdown to high temperature failure.

[0051] S3: Obtain three-dimensional point cloud data output by a laser radar unit carried by the tethered platform UAV. The laser radar unit is a mechanical rotating multi-line laser radar, with a ranging range of 0.5 meters to 200 meters and a ranging accuracy of ±2 centimeters. The unit emits laser pulses at a rotation frequency of 10 Hz per second and receives target reflection echoes, and calculates the distance by time of flight method. Combined with the horizontal and vertical angle information provided by the built-in encoder, each valid echo is converted into a three-dimensional space point, whose coordinates are represented as (x, y, z), and is attached with a reflection intensity value. The point cloud data is continuously output in the form of an unordered point set, and each point carries a time stamp generated by the internal clock of the laser radar. Since the laser radar scanning mechanism is continuous rotation, the point cloud data is non-uniformly distributed in time, and needs to be resampled by an interpolation algorithm to match the sampling time of other sensors.

[0052] S4: Obtain real-time three-dimensional position coordinates output by a global positioning system module carried by the tethered platform UAV. The global positioning system module is a high-precision receiver that supports multiple frequencies and multiple constellations, and its output includes differential corrected geodetic coordinates (longitude, latitude, altitude) and the corresponding covariance matrix. The module updates the position solution result at a frequency of 20 Hz, and adds an accurate coordinated universal time time label to each output. The position coordinate data is transmitted to the main control unit through a serial communication interface as the basis for building a global reference frame.

[0053] S5: Obtain the attitude angle, angular velocity and acceleration data output by the inertial measurement unit module carried by the tethered platform UAV. The inertial measurement unit module integrates a three-axis gyroscope and a three-axis accelerometer, and its output frequency is 200 Hz. The attitude angle is expressed in Euler angles, including roll angle, pitch angle and yaw angle, which are calculated in real time by fusing gyroscope and accelerometer data with an internal extended Kalman filter. Angular velocity and acceleration data are raw sensor readings without filtering, preserving high-frequency dynamic information. All inertial data are attached with high-precision timestamps, and their data stream is used as the master clock source of the entire system because of its highest sampling frequency and lowest transmission delay.

[0054] S6: Based on the real-time three-dimensional position coordinates and attitude angle data, construct a local geographic coordinate system with the UAV body as the origin. The coordinate system is defined as follows: the origin is located at the center of mass of the UAV; the Z-axis is perpendicular upward and parallel to the local gravity direction; the X-axis points to the UAV nose direction in the horizontal plane; and the Y-axis forms a right-handed coordinate system. The construction process first converts the geodetic coordinates output by the global positioning system module into Cartesian coordinates in the geocentric coordinate system, and then converts them to the local east-north-up coordinate system through the 7-parameter Bursa model. Subsequently, the roll angle, pitch angle and yaw angle output by the inertial measurement unit are used to construct a rotation matrix R in the order of yaw-pitch-roll. This rotation matrix relates the body coordinate system to the local east-north-up coordinate system, thereby establishing the spatial orientation of the local geographic coordinate system. This coordinate system is updated in real time with the UAV motion, with an update frequency of 100 Hz driven by the timer interrupt of the master control unit.

[0055] S7: Project the first image data, second image data and three-dimensional point cloud data into the local geographic coordinate system to generate a multi-modal perception data set with a unified spatial reference. Different strategies are used for different modalities in the projection process. For visible light and infrared images, the pinhole camera model is used for back projection. First, according to the camera intrinsic matrix K and extrinsic matrix [R|t], where t is the position of the camera optical center in the local geographic coordinate system, the image pixel coordinates (u, v) are converted to the ray direction vector in the normalized camera coordinate system . The ray can be represented as , where is the optical center position, is the depth scale factor.

[0056] Depth information is provided by the LiDAR point cloud: each point in the point cloud is mapped to the camera coordinate system by a rigid transformation, and a depth map is established. For each pixel in the image, if it has a valid value in the depth map, then λ is the depth value, and the three-dimensional coordinates of the pixel in the local geographic coordinate system are calculated. For the pixel area without valid depth, the neighborhood depth propagation algorithm is used to fill in: the weighted average value of the valid depth points in the 8-neighborhood is taken as the initial estimate, and the hole region is filled by an iterative diffusion process, and the weight is determined by the color similarity and spatial distance between pixels. For the LiDAR point cloud, the original coordinates are already in the body coordinate system, and only a rigid transformation by the rotation matrix R and the translation vector t is needed to map it directly to the local geographic coordinate system. All projected data are stored in the form of three-dimensional point cloud or pixel grid with coordinates, forming a spatially aligned multi-modal perception dataset.

[0057] S8 step is: performing timestamp synchronization processing on the multi-modal perception dataset, so that each modal data is aligned to the same sampling time in the time dimension. The data stream of the inertial measurement unit is used as the master clock, and the target sampling frequency is set to 100 Hz. The master control unit maintains a master time queue, taking the arrival time of each data of the inertial measurement unit as the reference node. For visible light, infrared and LiDAR data, the system traverses the timestamp sequence to find the nearest two original data frames before and after each master time node. Then, a cubic spline interpolation algorithm is used to resample the data between the two frames. For image data, interpolation is performed at the pixel level to generate a virtual image frame at the target time; for point cloud data, interpolation is performed on the position and reflectivity of the points to generate a virtual point cloud at the target time. The interpolation process ensures that each modal data is strictly aligned in time, eliminating motion blur and spatial misalignment caused by asynchronous acquisition of sensors.

[0058] S9 step is: based on the synchronized multi-modal perception dataset, a multi-channel feature tensor is constructed, where each channel corresponds to a spatial feature map of a modal. The construction process first rasterizes each modal data. Taking the ground projection plane in the local geographic coordinate system as the reference, a fixed-size detection region is defined, for example, 50m x 50m. The region is divided into a regular grid, and the grid resolution is 10 cm. For visible light and infrared images, the projected three-dimensional points are mapped back to the two-dimensional grid plane, and each grid cell stores the average color value or average temperature value of all corresponding pixels in the region. For LiDAR point cloud, each grid cell stores the maximum value, minimum value and point density of the point cloud height. In this way, three two-dimensional feature maps are generated: visible light feature map, infrared feature map and LiDAR feature map. The three feature maps have the same size and spatial resolution, and are stacked along the channel dimension to form a three-dimensional tensor, whose size is where and H and W of the feature map, is 3. This tensor is the multi-channel feature tensor, which is the input of the subsequent fusion network.

[0059] S10: input the multi-channel feature tensor into the pre-trained multi-modal cross-attention fusion network, which dynamically adjusts the contribution of each modality under the current environmental condition through the inter-channel weight distribution mechanism, and outputs the fused high-dimensional semantic feature map. The fusion network is deployed in an airborne embedded graphics processing unit, which has a floating point operation capability of not less than 2 trillion times per second and a video memory capacity of not less than 8 gigabytes. The network structure includes a modality embedding layer, a cross-modal interaction layer and a feature reconstruction layer. The modality embedding layer first performs batch normalization on each channel of the input tensor to eliminate the dimensional difference between modalities, and then adds a learnable position encoding vector to each spatial position to preserve the geometric structure information. The cross-modal interaction layer is the core component, which adopts a multi-head self-attention mechanism. Let the input feature tensor be , which is flattened into a sequence , where N = H x W. For the th modality channel and the th modality channel, calculate its attention weight:

[0060] ;

[0061] where , , and are learnable projection matrices, is the dimension of the key vector. This weight reflects the degree of information dependence of modality on modality at the current spatial position. Then, all modality channels are weighted and aggregated:

[0062] ;

[0063] where W_V is the value projection matrix. The aggregated feature is reshaped back to the tensor form of H x W x C. The feature reconstruction layer consists of two convolutional layers, the first layer expands the channel number to 64 and introduces a nonlinear activation function, and the second layer compresses the channel number to 1, outputting a single-channel high-dimensional semantic feature map. Each pixel value of the feature map represents the comprehensive confidence after multi-modal collaborative verification at the corresponding spatial position.

[0064] S11 step is: based on the high-dimensional semantic feature map, performing target detection and state discrimination operation, generating comprehensive detection results containing target category, spatial position, geometric contour and thermodynamic abnormality index. The target discrimination module has a built-in device type knowledge base, which stores the standard appearance templates, typical thermal distribution patterns and allowed temperature rise thresholds of various devices under test. The detection process adopts a sliding window strategy, moving a fixed-size window on the high-dimensional semantic feature map with a preset step size. For each window region, extract the local feature vector inside it, which is composed of the mean, variance and gradient amplitude of the pixel values in the window. The feature vector is input into the classification subnetwork and the regression subnetwork. The classification subnetwork is a three-layer fully connected network, which outputs the probability value of whether there is a target to be detected in the window. The regression subnetwork is also a three-layer fully connected network, which outputs four boundary box offset values (center point coordinate offset, width and height scaling factor) and a thermal anomaly score. The calculation method of the thermal anomaly score is: extract the average radiation temperature value of the target region from the second image data output by the infrared thermal imaging unit , subtract the baseline temperature value of the same type of device under the same working condition , and divide by the maximum temperature rise threshold of the type of device , the resulting ratio is the thermal anomaly score :

[0065] ;

[0066] When the probability value is greater than the preset threshold 0.7 and the thermal anomaly score exceeds 0.5, the target is determined to be a faulty device. Finally, all targets determined to be faulty are restored to real-world coordinates in the local geographic coordinate system with the bounding box obtained by regression, and are packaged with the semantic label of the category, the geometric contour polygon and the thermal anomaly score to generate a comprehensive detection result, which is transmitted in real time to the ground control station through the optical fiber link in the tethered cable.

[0067] The complete execution of the above method relies on a set of highly integrated electronic devices. The device includes a visible light imaging unit, an infrared thermal imaging unit, a laser radar unit, a global positioning system module, an inertial measurement unit module, a space-time alignment module, a time synchronization module, a multi-modal fusion module, and a target discrimination module. All sensor units are rigidly fixed to the unmanned aerial vehicle body through a physical support, and their relative position and attitude are precisely calibrated before leaving the factory. The calibration parameters are stored in a non-volatile memory. The space-time alignment module internally integrates a coordinate transformation engine, which performs rigid body transformation determined by position coordinates and Euler angle parameters in real time. The time synchronization module is implemented by a field programmable gate array, and the hardware logic ensures that the resampling process based on the inertial measurement unit clock has deterministic low delay. The multi-modal fusion module and the target discrimination module share the same embedded graphics processor resources, and the pipeline parallel processing is realized through a task scheduler. The entire device is powered by a ground power supply system through a tether cable to provide continuous power, ensuring the long-term stable operation of high-power computing units, and is suitable for key infrastructure scenarios such as substations, transmission lines, chemical plants, and other scenarios that require 24-hour uninterrupted inspection.

Claims

1. A detection method for unmanned aerial vehicles based on a tethered platform, characterized in that, include: Generate a multimodal sensing dataset with a unified spatial reference benchmark; The multimodal sensing dataset is subjected to timestamp synchronization processing so that the data of each modality are aligned to the same sampling time in the time dimension; Based on the synchronized multimodal sensing dataset, a multi-channel feature tensor is constructed, where each channel corresponds to a spatial feature map of a modality. The multi-channel feature tensor is input into a pre-trained multimodal cross-attention fusion network. The network dynamically adjusts the contribution of each modality under the current environmental conditions through a channel weight allocation mechanism, and outputs a fused high-dimensional semantic feature map. Based on the high-dimensional semantic feature map, target detection and state discrimination operations are performed to generate a comprehensive detection result that includes target category, spatial location, geometric contour and thermodynamic anomaly indicators.

2. The detection method for tethered platform UAVs according to claim 1, characterized in that, The method for generating the multimodal sensing dataset is as follows: Acquire the first image data output by the visible light imaging unit carried by the tethered platform UAV; Acquire the second image data output by the infrared thermal imaging unit mounted on the tethered platform UAV; Acquire the three-dimensional point cloud data output by the lidar unit mounted on the tethered platform UAV; Obtain the real-time three-dimensional position coordinates output by the GPS module carried by the tethered platform UAV; Acquire the attitude angle, angular velocity, and acceleration data output by the inertial measurement unit module mounted on the tethered platform UAV; Based on the real-time three-dimensional position coordinates and attitude angle data, a local geographic coordinate system is constructed with the UAV body as the origin. The first image data, the second image data, and the three-dimensional point cloud data are projected onto the local geographic coordinate system to generate a multimodal perception dataset with a unified spatial reference benchmark.

3. The detection method for tethered platform UAVs according to claim 2, characterized in that, Perform timestamp synchronization processing on the multimodal sensing dataset to align the data of each modality to the same sampling time in the time dimension, including: Using the data stream output by the inertial measurement unit as the main clock source, the original data streams of the visible light imaging unit, infrared thermal imaging unit, and lidar unit are interpolated and resampled to unify the time resolution of all modal data to 100 Hz.

4. The detection method for tethered platform UAVs according to claim 3, characterized in that, The first image data, the second image data, and the 3D point cloud data are projected onto the local geographic coordinate system to generate a multimodal perception dataset with a unified spatial reference datum, including: Using a pinhole camera model, pixels in visible light and infrared images are back-projected into rays, and combined with depth information provided by lidar point clouds, the three-dimensional coordinates of each pixel in the local geographic coordinate system are calculated. For pixel regions without corresponding depth information, a neighborhood depth propagation algorithm is used to fill them in.

5. The detection method for tethered platform UAVs according to claim 4, characterized in that, The multi-channel feature tensor is input into a pre-trained multimodal cross-attention fusion network. The network dynamically adjusts the contribution of each modality under the current environmental conditions through an inter-channel weight allocation mechanism, and outputs a fused high-dimensional semantic feature map, including: The spatial feature maps of each modality are normalized by channel and encoded by position through a modality embedding layer; By employing a multi-head self-attention mechanism in the cross-modal interaction layer, the correlation weight between any two modal feature channels is calculated, and cross-modal information is aggregated accordingly using this weighted approach. The aggregated features are nonlinearly transformed and compressed in a feature reconstruction layer to output a fused semantic feature map in a single channel.

6. The detection method for unmanned aerial vehicles based on tethered platforms according to claim 5, characterized in that, The spatial feature maps of each modality are normalized by channel and encoded by a modality embedding layer, including: Batch normalization is performed on each channel of the input multichannel feature tensor to eliminate intermodal dimensional differences; Add a learnable location encoding vector to each spatial location to preserve geometric information.

7. The detection method for tethered platform UAVs according to claim 6, characterized in that, A multi-head self-attention mechanism is employed in the cross-modal interaction layer to calculate the correlation weights between any two modal feature channels, and cross-modal information is aggregated accordingly using this weighted approach, including: After flattening the input feature tensor into a spatial position sequence, attention weights are calculated for the i-th modal channel and the j-th modal channel; Based on the attention weights, all modal channels are weighted and aggregated to generate a feature sequence with enhanced cross-modal interaction.

8. The detection method for tethered platform UAVs according to claim 7, characterized in that, Based on the high-dimensional semantic feature map, target detection and state discrimination operations are performed to generate a comprehensive detection result that includes target category, spatial location, geometric contour, and thermodynamic anomaly indicators, including: A detection window of a preset size is slid across the high-dimensional semantic feature map, and local feature vectors are extracted for each window region. The local feature vectors are input into the classification subnetwork and the regression subnetwork; The classification subnetwork outputs the probability value of whether the target to be detected exists within the window, and the regression subnetwork outputs the offset of the target bounding box and the thermal anomaly score; When the probability value is greater than a preset threshold and the thermal anomaly score exceeds the normal operating temperature range of the equipment, the target is determined to be a faulty device.

9. The detection method for tethered platform UAVs according to claim 8, characterized in that, The thermal anomaly score is calculated as follows: The average radiation temperature value of the target area is extracted from the second image data output by the infrared thermal imaging unit, subtracted from the reference temperature value of the same type of equipment under the same operating conditions, and then divided by the maximum allowable temperature rise threshold of the same type of equipment. The resulting ratio is the thermal anomaly score.

10. A detection electronic device based on a tethered platform unmanned aerial vehicle (UAV), characterized in that, include: The visible light imaging unit is used to acquire visible light image data of the object under inspection; Infrared thermal imaging unit, used to acquire infrared radiation image data of the object being inspected; The lidar unit is used to collect three-dimensional point cloud data of the environment surrounding the object being inspected. The Global Positioning System (GPS) module is used to output the real-time three-dimensional position coordinates of the UAV. The inertial measurement unit module is used to output the attitude angle, angular velocity, and acceleration data of the UAV. The spatiotemporal alignment module is used to construct a local geographic coordinate system based on the three-dimensional position coordinates and attitude angle data, and project visible light image data, infrared image data and three-dimensional point cloud data into the coordinate system to form spatially aligned multimodal data; The time synchronization module is used to resample the data streams of each sensor using the data from the inertial measurement unit as the main clock, so as to achieve precise alignment in the time dimension. The multimodal fusion module is used to construct a multi-channel feature tensor from time-synchronized multimodal data and generate a fused semantic feature map through a multimodal cross-attention fusion network. The target discrimination module is used to perform target detection and state discrimination based on the fused semantic feature map, and output a comprehensive detection result including target category, location, contour and thermal anomaly index.

Images