A method and system for fusion of visible light and lidar features for oilfield UAV inspection

Abstract

A method and system for target feature fusion in oilfield UAV inspection using visible light and lidar in collaboration is presented. This invention relates to the fields of industrial IoT, UAV inspection, and multimodal target detection. Addressing the difficulties in spatiotemporal alignment of visible light images and lidar point clouds, and insufficient fusion of two-dimensional texture features and three-dimensional geometric features in oilfield UAV inspection, the following solution is proposed: Visible light images, lidar point clouds, and IMU / GPS data are acquired; motion compensation and image-point cloud alignment are performed on the lidar point cloud; image features and point cloud geometric features are extracted separately; the point cloud geometric features are mapped and expanded and then fused with the image features; and the detection result is output based on the fused features. This invention is applicable to intelligent UAV inspection, equipment fault detection, leak identification, and safety hazard investigation in scenarios such as oilfield stations, oil pipelines, and oil well platforms.

Description

Technical Field

[0001] Belonging to the fields of industrial IoT and drone inspection technology, it specifically involves computer vision image processing, lidar point cloud processing, multimodal feature fusion, and deep learning target detection technology. It is specifically applied to equipment fault detection, leak identification, and safety hazard investigation in scenarios such as oilfield stations, oil pipelines, and oil well platforms. Background Technology

[0002] As oilfield production management moves towards intelligence, unmanned operation, and precision, the use of drones for inspections of key areas such as oilfield stations, pipelines, well sites, storage tank areas, and valve pipelines has become an important technical means for oilfield safety production and equipment maintenance. Compared to traditional manual inspection methods, drone inspections can complete large-scale, multi-angle data collection in a shorter time, making them particularly suitable for application scenarios where oilfields are widely distributed, have a large number of equipment, some areas are difficult to access manually, and require high inspection frequency. In existing oilfield drone inspection solutions, a common approach is to acquire images of the equipment's appearance using visible light cameras and then use image recognition or target detection models to identify targets such as pipeline damage, equipment abnormalities, oil leaks, and abnormal valve conditions. Some solutions further incorporate lidar sensors to use point cloud data to obtain the spatial location, contour structure, and three-dimensional size information of targets, supplementing the limitations of visible light images in terms of depth information and spatial scale representation.

[0003] Existing visible light inspection technologies typically rely on convolutional neural networks, object detection networks, or semantic segmentation networks to analyze inspection images. They extract surface texture, color variations, edge contours, and local anomalies to identify the condition of oilfield equipment. While these technologies perform well under conditions of sufficient lighting, unobstructed targets, and simple backgrounds, real-world oilfield scenarios present challenges. The diverse equipment types and complex pipeline networks make inspection images susceptible to variations in lighting, shadows, oil reflections, dust, fog, and changes in shooting angle. Relying solely on visible light images often fails to reliably distinguish between genuine equipment anomalies and environmental interference. For example, for small targets such as valves, joints, pipeline bends, and pumping unit components, extracting texture and color features from only two-dimensional images can easily lead to missed or false detections. Furthermore, for anomalies requiring spatial structural analysis, such as leaks, deformation, tilting, and positional shifts, relying solely on two-dimensional images is insufficient to accurately determine the target's three-dimensional scale and spatial relationships.

[0004] To improve the reliability of inspection results, existing technologies are increasingly adopting multi-sensor inspection schemes that utilize both visible light cameras and lidar for collaborative acquisition. LiDAR can acquire 3D point cloud data of the target and its surrounding environment, providing geometric information such as distance, shape, contour, and spatial location, which can, to some extent, compensate for the lack of depth information in visible light images. A common approach is to first preprocess the visible light image and lidar point cloud separately, then extract image features and point cloud features separately, and finally perform simple fusion at the detection result level or feature level. For example, some schemes use image detection models to output 2D target bounding boxes, while simultaneously using point cloud clustering or 3D target detection methods to output 3D target positions, and then obtain a comprehensive detection result through coordinate transformation or result matching; other schemes concatenate or weight point cloud features with image features to utilize both image texture information and point cloud geometric information simultaneously.

[0005] However, in practical applications of UAV inspection in oilfields, multi-sensor collaborative detection cannot be stably improved simply by adding more sensors. During flight, UAVs experience speed changes, attitude variations, and body vibrations. The corresponding UAV attitudes are not entirely consistent when the lidar scans different points, easily leading to motion distortion in the point cloud. Simultaneously, the data sampling frequencies, timestamps, and coordinate systems of visible light cameras, lidar, and IMU / GPS sensors are all different. If motion compensation for the point cloud is not performed, and the visible light image and lidar point cloud are not accurately aligned in time and space, there will be discrepancies between the pixel positions of the subsequently fused image and the spatial positions of the point cloud. This discrepancy is particularly noticeable in small target detection in oilfield scenarios. For example, targets such as valves, joints, leak points, and localized rust areas are small in size. Once the image features and point cloud features are misaligned, the fused features not only fail to enhance recognition capabilities but may also introduce incorrect geometric information, reducing detection accuracy.

[0006] Furthermore, existing visible light and lidar fusion methods still fall short in terms of feature fusion depth. Many solutions merely perform post-processing fusion of the detection results from the two types of sensors, or coarsely stitch and weighted superposition of image features and point cloud features, lacking fine-grained modeling of the correspondence between two-dimensional image coordinates and three-dimensional point cloud features. For oilfield inspection, equipment failures, leaks, and component anomalies often manifest as both two-dimensional texture changes and three-dimensional structural changes. Only by fully associating the texture details in the visible light image with the spatial geometric information in the lidar point cloud at corresponding locations can the target category, location, and size be more accurately identified. If fusion is only performed at the result level, the image branch and the point cloud branch are independent of each other during the initial feature extraction process, making it impossible to achieve deep interaction between texture and geometric information; if only overall feature stitching is performed, it is also difficult to ensure an effective correspondence between the point cloud geometric information and the image pixel region, resulting in insufficient expressive power of the fused features.

[0007] Meanwhile, existing inspection and detection models face limitations in training and deployment within oilfield scenarios. Oilfield stations, well sites, and pipelines exhibit strong industry-specific characteristics. Equipment layouts, pipeline structures, ground environments, lighting conditions, and anomaly types vary significantly across different oilfield areas. Furthermore, the collection and annotation of real inspection data are typically constrained by safety production requirements, operational costs, flight conditions, and the difficulty of manual annotation, making it challenging to obtain datasets that are large enough, fully annotated, and cover multiple anomaly states. If the detection model relies solely on a small number of real samples for training, it is prone to overfitting to specific scenarios. The detection performance will significantly decline after changing the inspection area, equipment type, or environmental conditions. Therefore, balancing real and synthetic data, expanding the training sample coverage, and improving the model's generalization ability across different oilfield inspection scenarios are problems that existing technologies need to address in engineering implementation.

[0008] Furthermore, most existing multimodal inspection algorithms remain at the level of individual data processing or model validation. In practical applications, they often require specialized personnel to perform operations such as data loading, parameter configuration, model training, inference detection, and result analysis, making them difficult for oilfield maintenance personnel to use directly. For oilfield inspection operations, the detection results not only need to be output by the algorithm but also need to intuitively display information such as visible light images, lidar point clouds, fused detection results, target category, target location, confidence level, 3D dimensions, and inspection logs, facilitating on-site personnel to make judgments, verifications, and traceability. Without an integrated visual interactive system, even if the algorithm itself has certain detection capabilities, it is difficult to effectively integrate into the oilfield on-site inspection process, affecting technical deployment and practical application value.

[0009] In summary, existing technologies suffer from several drawbacks during oilfield drone inspections, including difficulty in achieving high-precision spatiotemporal alignment between visible light images and lidar point clouds, insufficient fusion of two-dimensional texture features and three-dimensional geometric features, lack of end-to-end deep interaction in the detection network, limited model generalization ability due to insufficient real-world labeled data from oilfields, and difficulty in providing integrated visualization of inspection results. Summary of the Invention

[0010] To address the shortcomings of existing technologies in oilfield UAV inspection processes, such as difficulty in achieving high-precision spatiotemporal alignment between visible light images and lidar point clouds, insufficient fusion of 2D texture features and 3D geometric features, lack of end-to-end deep interaction in the detection network, limited model generalization ability due to insufficient real-world oilfield labeled data, and difficulty in providing integrated visualization of inspection results, the technical solution provided by this invention is as follows: A method for fusing target features in oilfield UAV inspection using a combination of visible light and lidar includes: The steps to acquire visible light images, lidar point clouds, and IMU / GPS data during the oilfield drone inspection process, and perform time correlation to obtain multi-source inspection data; The steps are as follows: motion compensation is performed on the lidar point cloud based on the IMU / GPS data, and the motion-compensated lidar point cloud is synchronized with the visible light image to obtain an image point cloud data pair; The step of projecting the lidar point cloud in the image point cloud data pair onto the image coordinate system of the visible light image based on the camera intrinsic and extrinsic parameters to obtain the image point cloud association data; The steps include extracting features from the visible light image and lidar point cloud in the image point cloud association data to obtain image feature maps and point cloud geometric features; The step of mapping and extending the geometric features of the point cloud to fit the image feature map to obtain the point cloud projection feature map; The step of fusing the image feature map with the point cloud projection feature map to obtain a fused feature map; The steps for performing target detection based on the fused feature map to obtain the detection results of oilfield UAV inspection targets.

[0011] Furthermore, in a preferred embodiment, the data sources of the multi-source inspection data include real-time data collected by UAVs, historical real data, and synthetic data. The multi-source inspection data includes visible light images, lidar point clouds, IMU / GPS data, and target annotation data.

[0012] Furthermore, in a preferred embodiment, motion compensation corrects the coordinates of laser points in the lidar point cloud based on attitude, angular velocity, acceleration, and position information in the IMU / GPS data.

[0013] Furthermore, in a preferred embodiment, the image point cloud association data is obtained through timestamp synchronization, camera intrinsic parameter conversion, camera extrinsic parameter conversion, and image coordinate projection, while retaining the pixel correspondence between the lidar point cloud and the visible light image.

[0014] Furthermore, in a preferred embodiment, after performing voxel downsampling, normal estimation, and outlier removal on the lidar point cloud in the image point cloud association data, point cloud feature extraction is then performed.

[0015] Furthermore, in a preferred embodiment, the point cloud projection feature map is obtained by linearly mapping the geometric features of the point cloud and broadcasting it according to the spatial dimensions of the image feature map.

[0016] A target feature fusion system for oilfield UAV inspection, combining visible light and lidar, includes: This module acquires visible light images, lidar point clouds, and IMU / GPS data during the oilfield drone inspection process, and performs time correlation to obtain multi-source inspection data. The module performs motion compensation on the lidar point cloud based on the IMU / GPS data, and synchronizes the motion-compensated lidar point cloud with the visible light image to obtain an image point cloud data pair. Based on camera intrinsic and extrinsic parameters, the module projects the lidar point cloud in the image point cloud data pair onto the image coordinate system of the visible light image to obtain the image point cloud association data. A module that extracts features from the visible light image and lidar point cloud in the image point cloud association data to obtain image feature maps and point cloud geometric features; A module that maps and extends the geometric features of the point cloud to fit the image feature map to obtain a point cloud projection feature map; The module that fuses the image feature map with the point cloud projection feature map to obtain the fused feature map; A module that performs target detection based on the fused feature map to obtain the detection results of oilfield UAV inspection targets.

[0017] A computer storage medium for storing a computer program, which, when read by the computer, is executed by the computer using the method described.

[0018] A computer, including a processor and a storage medium, executes the method when the processor reads a computer program stored in the storage medium.

[0019] A computer program product, which, as a computer program, implements the method when the computer program is executed.

[0020] Compared with the prior art, the advantages of the technical solution provided by the present invention are as follows: By simultaneously acquiring data using visible light cameras, lidar, and IMU / GPS sensors mounted on UAVs, inspection data is no longer limited to single two-dimensional images. Instead, it simultaneously obtains information on target appearance texture, spatial contours, distance scale, and flight attitude. Visible light images can reflect visual features such as oil stains, rust, leak traces, and equipment surface conditions. LiDAR point clouds can reflect the three-dimensional geometric structure of targets such as pipelines, valves, storage tanks, and pumping units. IMU / GPS data can provide attitude and position information for subsequent motion compensation and coordinate transformation. This addresses the problem at the data source level that single visible light inspection is easily affected by lighting, occlusion, and angle changes, and lacks spatial scale information.

[0021] Using the lidar timestamp as a reference, linear interpolation is used to obtain the corresponding IMU attitude data and visible light image frames, enabling sensor data with different sampling frequencies and trigger times to be unified under the same time reference. Compared to the method of coarse matching based on nearest neighbor frames, this processing can reduce the image and point cloud correspondence deviation caused by time misalignment during UAV flight, and maintain a more stable temporal correspondence between the same inspection target in the visible light image and lidar point cloud, providing a reliable data foundation for subsequent pixel-level fusion.

[0022] The position of each laser point is corrected in real time based on the angular velocity and acceleration measured by the IMU, compensating for point cloud distortion caused by UAV attitude changes, body jitter, and flight displacement during lidar scanning. Since oilfield inspections typically require the identification of small targets such as valves, joints, pipeline connections, and leak points, point cloud distortion can directly lead to target contour stretching, offset, or local ghosting. This feature can restore the authenticity of the point cloud spatial structure before feature extraction, improving the usability of point cloud geometric features and target positioning accuracy.

[0023] By utilizing camera intrinsic and extrinsic parameters, the compensated 3D point cloud is projected onto a 2D image plane, enabling the spatial positions in the LiDAR point cloud to correspond to pixel regions in the visible light image. Traditional result-layer fusion methods typically only match at the detection box or target category level, making it difficult to ensure that 3D geometric information accurately falls into the corresponding image target area. This case, through coordinate system transformation and projection alignment, enables the point cloud information of targets such as pipelines, valves, tank edges, and leak areas to correspond to image coordinates, thereby reducing the problem of misalignment between 2D texture features and 3D geometric features.

[0024] Adaptive voxel downsampling is applied to the motion-compensated lidar point cloud to reduce redundant point cloud data while preserving the key geometric contours of the target. This allows subsequent point cloud feature extraction to avoid processing a large number of repetitive or densely distributed invalid points. In oilfield inspection scenarios, structures such as pipelines, equipment supports, and tank walls generate large-scale point clouds. Directly inputting these into the network would increase the computational burden and affect real-time performance. Voxel downsampling improves point cloud processing efficiency by retaining representative points and compressing the data volume, while avoiding the destruction of key edges and small target structures caused by simple random sampling.

[0025] By searching the neighborhood points of each point using KD-trees and calculating the normal vector through principal component analysis, the point cloud not only contains discrete spatial coordinates but also represents local surface orientation, edge variations, and structural undulations. For targets in oilfield equipment such as pipe bends, valve protrusions, tank edges, and support connections, relying solely on point coordinates is insufficient to reliably distinguish different structural morphologies. Normal estimation provides local geometric relationships for point cloud feature extraction, which helps enhance the model's ability to identify small targets, irregular structures, and local abnormal deformations.

[0026] By statistically analyzing the neighborhood distance of each point and removing outliers whose mean distance exceeds a threshold, noise, flying points, and environmental interference points in the LiDAR point cloud are eliminated before feature extraction. Oilfield UAV inspections are easily affected by airborne particles, reflective metal surfaces, long-distance sparse scanning, and complex backgrounds, potentially resulting in anomalies in the point cloud unrelated to the actual equipment structure. Outlier removal improves point cloud data quality, reduces the participation of erroneous geometric information in fusion detection, and thus lowers the probability of false detections.

[0027] Noise reduction, normalization, and standardization to a preset input size are performed on visible light images to ensure stable network input for inspection images acquired under different lighting conditions, shooting distances, and image resolutions. Oilfield environments often present challenges such as strong light, shadows, reflections, dust, and cluttered backgrounds; unprocessed images can lead to unstable network feature distributions. Image preprocessing reduces some imaging interference, allowing subsequent convolutional features to more effectively reflect information such as equipment appearance, oil stains, leak areas, and component edges.

[0028] This approach leverages lightweight convolutional neural networks to extract multi-scale texture features from visible light images, enabling the simultaneous capture of large equipment outlines and small anomalies in inspection images across different receptive fields. Oilfield scenarios contain both large-scale targets such as storage tanks, pumping units, and pipeline routes, as well as small-scale targets like valves, joints, bolts, and localized leaks. Single-scale features tend to favor obvious targets while ignoring subtle anomalies; multi-scale convolutional features enhance adaptability to targets of varying sizes, while the lightweight backbone network helps meet the high single-frame processing speed requirements of UAV inspections.

[0029] Mapping multi-scale image features to a unified dimension allows texture features from different convolutional layers to participate in subsequent fusion in a consistent data format. Since image features at different levels differ in channel count, semantic depth, and spatial resolution, directly concatenating them with point cloud features can easily lead to inconsistent feature scales and low fusion efficiency. A unified dimension mapping can establish a common representation space for image features and point cloud features, reducing information loss during the fusion process.

[0030] A point cloud feature extractor is built based on the PointNet++ architecture. Through sampling, grouping, multilayer perceptron processing, and global pooling, irregular point cloud data is processed, enabling the effective extraction of local structural and global geometric features from LiDAR point clouds. Compared to simply rasterizing point clouds or performing only clustering detection, this feature extractor can directly adapt to the disordered, sparse, and unevenly dense characteristics of point clouds, forming a more stable geometric representation of structures such as valves, pipelines, and tank edges in oilfield scenarios.

[0031] The sampling and grouping strategies of PointNet++ are optimized to make the point cloud feature extraction process more adaptable to the distribution characteristics of small targets and sparse point clouds in oilfield scenarios. When UAVs equipped with LiDAR scan at long distances or at an angle, the auxiliary structures near valves, joints, and leaks may only correspond to a small number of point clouds. If the sampling and grouping are too coarse, they are easily overwhelmed by the large background structures during feature extraction. Optimizing the sampling and grouping can increase the probability of small target areas being preserved and represented, and enhance the sensitivity of point cloud geometric features to local anomalies.

[0032] By mapping the global features of point clouds to the same dimension as image features through linear transformation, the three-dimensional geometric information can be kept consistent with the two-dimensional texture information in the feature dimension. In traditional fusion, directly stitching features of different dimensions and semantic levels can easily produce redundant channels or invalid information; linear transformation can adjust the geometric representation of point clouds to a form suitable for image feature fusion, so that subsequent stitching and convolution transformation can be completed within a unified feature space.

[0033] By broadcasting, the mapped point cloud features are expanded to the same spatial size as the image features, enabling point cloud geometric information to participate in the feature representation of each pixel location in the image. Unlike methods that only introduce point cloud information at the bounding box or final result level, this process allows geometric features to be embedded in the spatial representation of the two-dimensional feature map, providing a foundation for subsequent pixel-by-pixel fusion, thereby enhancing the correlation between local texture regions and global spatial geometric information in the image.

[0034] Image features and expanded point cloud features are concatenated along the channel dimension and then transformed nonlinearly using 1×1 convolution. This allows visible light texture information and LiDAR geometric information to interact at the pixel-level feature level. Simple weighting or result voting is insufficient to express the complex complementary relationship between the two modalities. 1×1 convolution, however, can recombine texture and geometric features from different channels at each spatial location, forming a fused feature that combines appearance and spatial information. This improves the ability to identify leaks, equipment malfunctions, small target components, and targets in complex backgrounds.

[0035] By using fused features as detection input and simultaneously predicting target bounding boxes, class probabilities, confidence levels, and 3D size information through a detection head, the detection results no longer limit themselves to two-dimensional image bounding boxes but can simultaneously reflect the spatial scale of the target and the reliability of the detection. Oilfield maintenance personnel not only need to know if anomalies exist but also need to determine the location, size, and severity of abnormal targets; this detection output can provide more complete information for equipment fault identification, leak investigation, and on-site verification.

[0036] An end-to-end fusion detection network enables integrated processing from raw multi-sensor data to detection results, allowing features from the visible light branch and point cloud branch to be jointly optimized around the same detection target during training. Existing architectures that process data separately first and then fuse the results are prone to information isolation in the early feature extraction stage, leading to remedial matching results in subsequent fusion. The end-to-end network allows the fused features to be directly constrained by the target detection loss, reducing information loss in intermediate stages and improving detection accuracy and robustness.

[0037] The model training is constrained by both classification loss and bounding box regression loss, enabling the detection model to simultaneously focus on target category judgment and target location fitting. False alarms and false negatives in oilfield inspections often stem not only from category recognition errors but also from detection box offsets or incomplete coverage. Cross-entropy loss helps improve the ability to distinguish between categories such as faults, leaks, and normal equipment, while smoothing L1 loss helps improve the stability of target location regression, thus optimizing the fused detection results in both category and location aspects.

[0038] Non-maximum suppression is applied to the detection results to remove duplicate detection boxes and retain results with high confidence, ensuring that the same valve, pipeline joint, leak point, or abnormal equipment area is not output multiple times. Oilfield equipment has complex edges and dense structures, and the fusion detection model may generate multiple overlapping candidate boxes in adjacent locations; non-maximum suppression can reduce duplicate alarms and result confusion, improving the readability and field usability of the final detection results.

[0039] It supports three data sources: real-time acquired data, historical real data, and synthetic data, so that model training and inference no longer rely entirely on limited real labeled samples. The acquisition of oilfield inspection data is limited by safety, cost, and the scarcity of abnormal samples. Using only real data can easily lead to model overfitting. Synthetic data can supplement the sample coverage under different equipment, different faults, and different environmental conditions. Historical real data can maintain the authenticity of the scene. Real-time acquired data can adapt to the needs of on-site inspection. The three types of data work together to improve the model's generalization ability and scene adaptability.

[0040] The loaded data is divided into training and test sets. After training, the detection accuracy, precision, and recall are verified using the test set, providing quantifiable performance feedback for the model optimization process. Compared to solutions that only train the model without scenario validation, this approach can promptly reflect the actual recognition capability of the fusion detection network on oilfield inspection targets, facilitating parameter adjustment, preserving effective weights, and reducing the uncertainty caused by direct model deployment.

[0041] By automatically identifying and adapting to the CPU / GPU environment, the system can automatically select computing resources based on the conditions of on-site equipment, reducing reliance on manual configuration during deployment. Oilfield on-site operation and maintenance environments may have different hardware configurations; requiring manual modification of operating parameters would increase the barrier to entry for system use. Hardware self-adaptation improves the system's deployability on different inspection terminals or workstations and ensures the stability of model training and inference processes.

[0042] This system, built on PyQt5, integrates a unified visual interactive system encompassing data loading, parameter setting, model training, inference detection, log display, and result presentation. This allows oilfield maintenance personnel to complete the entire inspection and detection process from a single interface. Compared to solutions that only provide algorithm models or command-line scripts, this graphical system lowers the barrier to data processing and model invocation, enabling non-algorithm personnel to load multi-source data, initiate detection, view results, and manage inspection records. This facilitates the practical application of this technical solution in oilfield settings.

[0043] The asynchronous thread training mechanism avoids interface lag, ensuring the visualization system remains responsive during model training or inference. In oilfield applications, maintenance personnel need to continuously monitor logs, detect status changes, and track results. If the training task occupies the main interface thread, it can cause system inactivity or even lead to the mistaken belief of a program malfunction. Asynchronous processing improves system stability and user experience, guaranteeing continuous human-computer interaction during long-term inspection tasks.

[0044] Visible light images, LiDAR point clouds, and fused detection results are displayed in a labeled manner, with detected faulty targets highlighted. This allows maintenance personnel to simultaneously verify the 2D appearance, 3D spatial structure, and model detection conclusions. Displaying detection boxes individually makes it difficult to interpret the basis of model judgments, while simultaneous display of multimodal results helps on-site personnel determine the rationality of alarms, quickly locate abnormal targets, and improve the interpretability and review efficiency of inspection results.

[0045] The system automatically saves the entire process operation log, detection results, and alarm information, and generates inspection reports, creating a traceable record of the oilfield drone inspection process. Traditional algorithm detection results, without log and report support, are difficult to use for subsequent maintenance decisions, accountability, and historical status comparisons. This case, by saving detection process and result information, creates a continuous data chain between equipment anomaly detection, review, maintenance scheduling, and inspection file management.

[0046] It is suitable for intelligent inspection, equipment fault detection, leak identification and safety hazard investigation of oilfield stations, oil pipelines, oil well platforms and other scenarios. Attached Figure Description

[0047] Figure 1System overall architecture diagram; Figure 2 Core processing flowchart; Figure 3 : 2D-3D projection fusion module structure diagram. Detailed Implementation

[0048] To make the advantages and benefits of the technical solution provided by the present invention clearer, the technical solution provided by the present invention will now be described in further detail with reference to the accompanying drawings, specifically: Implementation Method 1: This implementation method provides a target feature fusion method for oilfield UAV inspection using a combination of visible light and lidar, including: The steps to acquire visible light images, lidar point clouds, and IMU / GPS data during the oilfield drone inspection process, and perform time correlation to obtain multi-source inspection data; The steps are as follows: motion compensation is performed on the lidar point cloud based on the IMU / GPS data, and the motion-compensated lidar point cloud is synchronized with the visible light image to obtain an image point cloud data pair; The step of projecting the lidar point cloud in the image point cloud data pair onto the image coordinate system of the visible light image based on the camera intrinsic and extrinsic parameters to obtain the image point cloud association data; The steps include extracting features from the visible light image and lidar point cloud in the image point cloud association data to obtain image feature maps and point cloud geometric features; The step of mapping and extending the geometric features of the point cloud to fit the image feature map to obtain the point cloud projection feature map; The step of fusing the image feature map with the point cloud projection feature map to obtain a fused feature map; The steps for performing target detection based on the fused feature map to obtain the detection results of oilfield UAV inspection targets.

[0049] The sources of multi-source inspection data include real-time data collected by UAVs, historical real data, and synthetic data. The multi-source inspection data includes visible light images, lidar point clouds, IMU / GPS data, and target annotation data.

[0050] Motion compensation corrects the coordinates of laser points in the lidar point cloud based on attitude, angular velocity, acceleration, and position information from the IMU / GPS data.

[0051] The image point cloud association data is obtained through timestamp synchronization, camera intrinsic parameter conversion, camera extrinsic parameter conversion, and image coordinate projection, and retains the pixel correspondence between the lidar point cloud and the visible light image.

[0052] After performing voxel downsampling, normal estimation, and outlier removal on the lidar point cloud in the image point cloud association data, point cloud feature extraction is then performed.

[0053] The point cloud projection feature map is obtained by linearly mapping the geometric features of the point cloud and broadcasting it according to the spatial dimensions of the image feature map.

[0054] A target feature fusion system for oilfield UAV inspection, combining visible light and lidar, includes: This module acquires visible light images, lidar point clouds, and IMU / GPS data during the oilfield drone inspection process, and performs time correlation to obtain multi-source inspection data. The module performs motion compensation on the lidar point cloud based on the IMU / GPS data, and synchronizes the motion-compensated lidar point cloud with the visible light image to obtain an image point cloud data pair. Based on camera intrinsic and extrinsic parameters, the module projects the lidar point cloud in the image point cloud data pair onto the image coordinate system of the visible light image to obtain the image point cloud association data. A module that extracts features from the visible light image and lidar point cloud in the image point cloud association data to obtain image feature maps and point cloud geometric features; A module that maps and extends the geometric features of the point cloud to fit the image feature map to obtain a point cloud projection feature map; The module that fuses the image feature map with the point cloud projection feature map to obtain the fused feature map; A module that performs target detection based on the fused feature map to obtain the detection results of oilfield UAV inspection targets.

[0055] A computer storage medium for storing a computer program, which, when read by the computer, is executed by the computer using the method described.

[0056] A computer, including a processor and a storage medium, executes the method when the processor reads a computer program stored in the storage medium.

[0057] A computer program product, which, as a computer program, implements the method when the computer program is executed.

[0058] Implementation Method Two: This implementation method is a further detailed description of the technical solution provided in Implementation Method One, specifically: This embodiment provides a method for fusing the features of oilfield UAV inspection targets using a combination of visible light and lidar. It is used to continuously process visible light images, lidar point clouds, and IMU / GPS attitude positioning data collected during UAV inspections. After loading, compensation, alignment, feature extraction, projection fusion, model training, inference detection, and result display, the multi-source data outputs the category, location, confidence level, and three-dimensional size information of the oilfield inspection targets.

[0059] The inspection target feature fusion system is launched and its operating parameters are configured to establish a unified operating environment for subsequent multi-source data processing, model training, and inference detection. Specifically, the visual interactive system is first launched. The system identifies the hardware environment of the currently running device, determining whether it has GPU computing resources. If GPU computing resources are available, subsequent model training and inference tasks are loaded onto the GPU for execution; otherwise, the CPU is used for processing. Subsequently, the point cloud voxel size, image input size, model training batch size, learning rate, training epochs, detection confidence threshold, non-maximum suppression threshold, camera intrinsic parameters, camera extrinsic parameters, and sensor mounting pose parameters are configured in the visual interface. The visible light image feature extraction network, point cloud feature extraction network, projection fusion module, and target detection head are also initialized. The inputs to this step are the user-configured operating parameters, device hardware information, and preset model structure. The output is the initialized operating environment and each processing module, which serves as the basis for subsequent multi-source data loading and processing.

[0060] The system loads multi-source data from oilfield drone inspections, providing raw input for motion compensation, spatiotemporal alignment, and subsequent feature fusion. In practice, the system selects data sources via a visualization interface, including real-time acquired data, historical real data, and synthetic data. When real-time acquired data is selected, the system activates the drone's onboard visible light camera, lidar, and IMU / GPS sensors to acquire visible light image frames, lidar point cloud frames, and the drone's attitude, angular velocity, acceleration, position, and timestamp information, respectively. When historical real data is selected, the system reads previously acquired visible light images, point cloud files, and IMU / GPS log files from a specified path. When synthetic data is selected, the system generates corresponding visible light images, lidar point clouds, and target annotation information based on scene elements such as oilfield stations, pipelines, well platforms, valves, storage tanks, and pumping units. The system organizes the loaded data according to timestamps, data types, and scene numbers, forming a sample set containing image data, point cloud data, attitude positioning data, and target annotation data. The input to this step is real-time sensor data, historical file data, or synthetic sample data, and the output is a multi-source inspection data set organized by sample. This output is then processed for motion compensation and spatiotemporal alignment.

[0061] Motion distortion compensation is applied to the lidar point cloud to prevent attitude changes and body jitter during UAV flight from directly affecting the spatial structure of the point cloud. Specifically, each lidar point in the lidar point cloud is processed. The acquisition time corresponding to that point is read, and the attitude, angular velocity, acceleration, and position information of the UAV at that time are obtained or interpolated from IMU / GPS data based on that time. The system performs rotation and translation corrections on the original coordinates of each lidar point according to the attitude changes of the UAV during scanning, converting them into compensated point coordinates at a unified reference time. For points formed at different scanning times within a single frame of the point cloud, corrections are made separately according to their respective attitude changes, avoiding treating the entire frame of the point cloud as data from the same moment. The inputs to this step are the original lidar point cloud and IMU / GPS attitude positioning data, and the output is the motion-compensated lidar point cloud, which serves as the basis for subsequent point cloud preprocessing and spatial alignment.

[0062] Time synchronization is performed between the visible light image and the motion-compensated lidar point cloud to ensure that both types of data correspond to the same inspection time. Specifically, using the timestamp of the lidar point cloud frame as a reference, image frames adjacent to that timestamp are searched in the visible light image sequence. When a perfectly matching image frame exists, the correspondence between the point cloud frame and the image frame is directly established. When no perfectly matching image frame exists, image frames with timestamps before or after the reference time are selected, and linear interpolation or nearest-neighbor matching is performed based on the time interval to obtain the visible light image data corresponding to the lidar point cloud frame. For IMU / GPS data, corresponding attitude information is also obtained according to the same time reference, ensuring that the image, point cloud, and attitude positioning data are in a unified temporal sequence. The inputs to this step are the compensated point cloud frame, the visible light image frame, and the corresponding timestamp; the output is the time-synchronized image-point cloud data pair, which is then processed for spatial coordinate alignment.

[0063] Spatially aligning the time-synchronized point cloud with the visible light image enables the mapping of 3D point cloud coordinates to the 2D image plane. Specifically, the system reads pre-calibrated camera intrinsic and extrinsic parameters, transforms the motion-compensated LiDAR point cloud from the LiDAR coordinate system to the camera coordinate system, and then projects the 3D points onto the 2D pixel coordinates of the visible light image according to the camera imaging relationship. Points that fall outside the image boundary after projection are discarded; points projected into the effective image area retain their 2D pixel coordinates, 3D spatial coordinates, depth values, and corresponding point cloud attributes. After this processing, each effective point cloud point can correspond to a pixel position or local pixel region in the visible light image. The inputs to this step are time-synchronized image point cloud data pairs, camera intrinsic and extrinsic parameters, and the output is spatially aligned image point cloud associated data, which is then fed into image preprocessing, point cloud preprocessing, and feature extraction processes.

[0064] The spatially aligned visible light image is preprocessed to ensure it meets the input requirements of the image feature extraction network. Specifically, the system performs noise suppression, scaling, and pixel normalization on the visible light image, adjusting images with different resolutions, brightness ranges, and shooting conditions to a preset input size and value range. For edge regions, highly reflective areas, or localized noise areas in the image, filtering and normalization can reduce the impact of abnormal pixels on the network input. The processed image maintains its spatial correspondence with the aforementioned point cloud projection results, and the scaling relationship of the point cloud projection coordinates is updated synchronously when the image size changes. The input to this step is the spatially aligned visible light image and its point cloud projection coordinates; the output is the normalized visible light image and the updated point cloud projection correspondence, which serves as the input for image feature extraction.

[0065] The motion-compensated LiDAR point cloud is preprocessed to reduce redundancy and noise while preserving key geometric structures. Specifically, the system first divides the point cloud space into multiple voxel grids based on a preset voxel size. Representative points are selected from multiple points falling within the same voxel grid to reduce duplicate and dense points. Next, a KD-tree search is performed to find the neighborhood points of each representative point. The local surface orientation of each point is calculated based on the spatial distribution of its neighbors, yielding the corresponding normal information. Then, the distance distribution between each point and its neighbors is statistically analyzed, and points whose neighborhood distances significantly deviate from the overall statistical range are removed as outliers. After this processing, the point cloud data is transformed from the original dense point cloud into a preprocessed point cloud containing representative points, normal information, and effective spatial structure. The input to this step is the motion-compensated LiDAR point cloud, and the output is the point cloud data after voxel downsampling, normal estimation, and outlier removal. This output serves as the input for point cloud feature extraction.

[0066] Texture and semantic features of visible light images are extracted to provide a two-dimensional feature representation for subsequent fusion with point cloud geometric features. Specifically, the standardized visible light image is input into an image feature extraction network. This network uses multi-layer convolutional, activation, and pooling structures to process the image layer by layer. Shallow layers extract edge, texture, color variations, and local contour information, while deeper layers extract semantically relevant features such as equipment components, pipeline areas, tank edges, leak traces, and fault areas. For targets of different scales, the network captures the contours of large targets and the details of small targets through convolutional features with different receptive fields, mapping image features from different levels to a unified dimension to form an image feature map that can be used for fusion. The input to this step is the standardized visible light image, and the output is the image feature map, which is then fed into the 2D-3D projection fusion process.

[0067] Geometric features of LiDAR point clouds are extracted to provide a 3D spatial representation for subsequent fusion with image texture features. Specifically, the preprocessed point cloud is input into a point cloud feature extraction network. This network samples and groups the point cloud, organizing spatially adjacent points into local point sets. A multilayer perceptron extracts the geometric relationships between these local point sets, and pooling is used to aggregate local structural information, forming point cloud features that represent the overall spatial contour and local geometric changes of the point cloud. For small targets or local structures such as valves, joints, pipeline bends, and tank edges, point sets reflecting local contour changes are prioritized during sampling and grouping to prevent small target point clouds from being overwhelmed by large background point clouds. The input to this step is the preprocessed point cloud data, and the output is the global geometric features and local geometric representations of the point cloud. This output is then fed into point cloud feature mapping and projection fusion processing.

[0068] The system maps point cloud geometric features to an expression space adapted to image features, enabling 3D point cloud features to participate in 2D image feature fusion. Specifically, the system inputs the point cloud geometric features output from the point cloud feature extraction network into a linear transformation layer. The linear transformation adjusts the number of channels and feature dimensions to match the channel representation requirements of the image feature map. Then, based on the spatial dimensions of the image feature map, the mapped point cloud features are expanded to the same height and width, ensuring that the point cloud geometric features correspond to pixel positions in the 2D feature plane. During the expansion process, the correlation between the point cloud features and the aforementioned projection coordinates is preserved, allowing the spatial geometric information provided by the point cloud to be embedded into the corresponding image feature representation. The inputs to this step are the point cloud geometric features, the image feature map dimensions, and the point cloud projection correspondence; the output is a point cloud projection feature map with the same spatial dimensions as the image feature map, which serves as the input for pixel-level fusion.

[0069] The system performs pixel-level fusion of image feature maps and point cloud projection feature maps to form a fused feature map that combines two-dimensional texture information and three-dimensional geometric information. Specifically, the system concatenates the image feature map and the point cloud projection feature map along the channel dimension, ensuring that each spatial location simultaneously contains texture, edge, color, and semantic information from the visible light image, as well as distance, contour, and spatial structure information from the LiDAR point cloud. The concatenated feature map is then input to a one-to-one convolutional layer for nonlinear transformation. This convolutional layer recombines and filters information from different channels, enhancing effective features relevant to the inspected target and reducing background interference and redundant features. After this processing, the output fused feature map retains both the image's ability to express details and appearance anomalies and the point cloud's ability to express spatial structure and three-dimensional scale. The input to this step is the image feature map and the point cloud projection feature map, and the output is the fused feature map, which is then used for training or inference detection processing of the fusion detection model.

[0070] In training mode, the fusion detection model is iteratively trained using fused feature maps and target annotation information, enabling the model to learn the category, location, and spatial size representation of oilfield inspection targets. Specifically, the system reads the fused feature maps and their corresponding annotation information from the training set. The annotation information includes the target category, 2D bounding box position, confidence label, and 3D size information. The fused feature maps are input into the target detection head, which outputs the bounding box coordinates, category probability, confidence score, and 3D size prediction results of candidate targets through convolution processing. The system compares the prediction results with the annotation information. For classification, the classification loss is calculated based on the difference between the predicted and true categories; for bounding boxes, the regression loss is calculated based on the positional difference between the predicted and true boxes; and for confidence and 3D size, constraints are applied based on the deviation between the predicted and labeled values. Then, the trainable parameters in the image feature extraction network, point cloud feature extraction network, projection fusion module, and detection head are updated in reverse based on the total loss. The input to this step is the fused feature map and target annotation information from the training set, and the output is the weights of the trained fusion detection model, which serves as the basis for subsequent testing, verification, and inference detection.

[0071] The system monitors training parameters and status during model training to ensure optimization and weight saving under preset conditions. Specifically, the system inputs training samples into the model in batches according to the user-configured batch size and performs iterative updates based on the preset learning rate and training epochs. During training, the system records real-time changes in loss, detection accuracy, precision, and recall for each training epoch and displays the training status in the log area of ​​the visualization interface. When the preset number of training epochs is reached or the model performance meets the saving conditions, the system saves the current model weights. If an abnormal interruption, data loss, or insufficient hardware resources occur during training, the system records the abnormal information and stops the corresponding task. The inputs to this step are the model output, loss calculation results, and performance verification results during training. The output is the optimized model file and training logs, which are used for inference detection and inspection record traceability.

[0072] In test or verification mode, the performance of the trained fusion detection model is verified using a test set to confirm whether the model meets the requirements of oilfield inspection applications. Specifically, visible light images, lidar point clouds, and IMU / GPS data from the test set are sequentially processed according to the aforementioned steps: motion compensation, spatiotemporal alignment, image preprocessing, point cloud preprocessing, image feature extraction, point cloud feature extraction, and 2D-3D projection fusion, resulting in a fusion feature map of the test samples. This fusion feature map is then input into the trained detection model to obtain target detection results. The detection results are compared with the labeled test set data, and the detection accuracy, precision, recall, average precision, and single-frame inference time are statistically analyzed. If the test metrics meet the preset requirements, the model weights are used as the final inference model; if the test metrics do not meet the requirements, the training parameters, data samples, or model weights are adjusted, and the model is retrained. The input to this step is the multi-source data of the test set and the trained model weights; the output is the model performance verification results and the usable inference model, which is used for subsequent on-site inspection inference.

[0073] In inference detection mode, the oilfield UAV inspection data to be detected is input into the trained fusion detection model, and the inspection target detection results are output. Specifically, the user selects the data to be detected in the visual interface, and the system processes the data according to the same process as in the training phase, sequentially completing data loading, point cloud motion compensation, time synchronization, spatial projection alignment, image preprocessing, point cloud preprocessing, single-modal feature extraction, and 2D-3D projection fusion. The resulting fusion feature map is input into the detection model with saved weights, and the detection head outputs the bounding boxes, class probabilities, confidence scores, and 3D dimensions of the candidate targets. The input for this step is the visible light image to be detected, LiDAR point cloud, IMU / GPS data, and the trained model weights; the output is the candidate target detection results, which are then processed for duplicate box removal and result filtering.

[0074] The system filters and deduplicates candidate target detection results to make the final output more stable and clear. Specifically, the system first filters out candidate targets with low confidence based on a preset confidence threshold, and then performs non-maximum suppression on the remaining candidate boxes according to their overlap. For multiple highly overlapping candidate boxes pointing to the same target, the system retains the detection box with the highest confidence and deletes other duplicate detection boxes. For detection results of different categories but close locations, the system comprehensively judges their retention relationship based on category probability, confidence, and target spatial size. After filtering, the final retained results include target category, 2D image location, confidence, spatial correspondence, and 3D size information. The input to this step is the candidate target detection results, and the output is the deduplicated final detection result, which is then processed for visualization, logging, and report generation.

[0075] The final detection results are displayed in conjunction with visible light images and LiDAR point clouds, allowing inspection personnel to intuitively view target locations and anomalies. Specifically, the system includes tabs for visible light images, LiDAR point clouds, and fused detection results in the visualization interface. In the visible light image tab, the final detection bounding box is overlaid on the original inspection image, with target category and confidence level labeled. The LiDAR point cloud tab displays the corresponding 3D point cloud data and marks the spatial region corresponding to the target based on the detection results. The fused detection results tab simultaneously displays the image detection bounding box, point cloud spatial location, and 3D dimension information. Targets identified as faults, leaks, or safety hazards are highlighted for quick location by inspection personnel. The inputs to this step are the final detection results, the original image, and the corresponding point cloud data; the output is a visualized inspection result, used for manual review and inspection decision-making.

[0076] The system saves the inspection process, detection results, and alarm information to form a traceable inspection log. Specifically, during each data loading, model training, inference detection, and result output process, the system records the operation time, data source, parameter configuration, model version, target category, target location, confidence level, 3D dimensions, and alarm status. When the detection results show faulty targets, suspected leak targets, or targets with safety hazards, the system saves the corresponding image area, point cloud area, and detection information to the alarm log. After the inspection task is completed, the system generates an inspection report based on the saved detection results and log information. The report includes the inspection data source, target list, abnormal target location, confidence level, 3D dimensions, visualization screenshots, and processing logs. The inputs to this step are the visualized inspection results and system operation logs, and the outputs are the inspection record, alarm information, and inspection report, which are used for subsequent maintenance review, hazard investigation, and historical data management.

[0077] In continuous inspection or batch processing scenarios, the system executes the above processing flow cyclically according to the sample order, enabling multiple sets of inspection data to automatically complete fusion detection. Specifically, for batch-loaded historical real data or synthetic data, the system sequentially reads each set of visible light images, point clouds, and attitude positioning data, and completes motion compensation, spatiotemporal alignment, feature extraction, projection fusion, detection output, and result saving for each set. For real-time acquired data, the system continuously generates image-point cloud data pairs according to sensor data frames, and updates the results to the visualization interface after processing each frame. After the detection results, logs, and alarm information of the current set of data are saved, the system uses the next set of data as new input to continue processing, thus forming a continuous detection process for oilfield UAV inspection tasks.

[0078] Implementation Method 3: This implementation method is described in detail with reference to the accompanying drawings. Specific embodiments are provided to further illustrate the technical solutions offered above. Specifically: The technical solution of this implementation is based on a closed-loop process of multi-sensor data acquisition → motion compensation and spatiotemporal alignment → single-modal feature extraction → 2D-3D projection fusion → end-to-end target detection → visualization interaction, covering seven core functional modules. All modules have been improved to address the shortcomings of existing technologies. The core formulas and judgment rules are derived from the core logic of motion compensation calculation, point cloud projection transformation, feature fusion calculation, and target detection confidence calculation, realizing deep fusion of visible light and lidar data and accurate target detection.

[0079] System Overall Architecture The system architecture of this implementation is based on 2D-3D pixel-level feature fusion, supported by motion-compensated spatiotemporal alignment, single-modal feature extraction, projection fusion, and end-to-end detection. It utilizes full-process visualization as the operational platform and constructs a 6-layer closed-loop architecture. Data exchange and result feedback are achieved between each layer. The specific architecture layers are as follows (from top to bottom): Visual interaction layer: Integrates the entire process of data loading / generation, model training, inference detection, and result display into a visual operation, enabling one-click operation and real-time result display.

[0080] Detection result output layer: Generates target detection results based on fused features, and outputs target category, location, confidence level and three-dimensional size information.

[0081] 3.2D-3D Feature Fusion Layer: Projects 3D point cloud features onto a 2D image coordinate system to achieve pixel-level feature fusion and outputs fused features that combine texture and geometric information.

[0082] Single-modal feature extraction layer: It is divided into visible light image feature extraction branch and lidar point cloud feature extraction branch, which extract the texture features of the image and the geometric features of the point cloud, respectively.

[0083] Data preprocessing layer: Implements motion compensation, voxel downsampling, and normal estimation for lidar point clouds, as well as denoising and normalization of visible light images.

[0084] Data Acquisition Layer: Supports the synchronous acquisition of data from visible light cameras, LiDAR, and IMU / GPS sensors mounted on drones, as well as the loading of synthetic data and real historical data.

[0085] Core module design The seven core modules of this implementation are: multi-sensor data acquisition module, motion compensation and spatiotemporal alignment module, lidar point cloud preprocessing module, visible light image feature extraction module, point cloud feature extraction module, 2D-3D projection fusion module, end-to-end fusion detection module, and integrated visualization interaction module. The design ideas, core improvements, and core formulas / judgment rules of each module are as follows: Motion compensation and spatiotemporal alignment module Design concept: Based on the IMU / GPS data carried by the UAV, motion distortion compensation is performed on the lidar point cloud. At the same time, through timestamp synchronization and coordinate system transformation, high-precision spatiotemporal alignment between the visible light image and the lidar point cloud is achieved.

[0086] Core improvements: Solving the problem of LiDAR point cloud distortion caused by drone motion, achieving microsecond-level time synchronization and millimeter-level spatial alignment of multi-sensor data, with a spatiotemporal alignment accuracy improvement of ≥30%.

[0087] Core Formulas and Design: Point cloud motion compensation formula: Based on the angular velocity and acceleration measured by the IMU, the position of each laser point is corrected in real time. The formula is as follows:

[0088] in, z represents the original laser point coordinates, R(t) is the rotation matrix at time t, and T(t) is the translation vector at time t. These are the coordinates of the laser point after compensation.

[0089] Spatiotemporal alignment mechanism: Based on the timestamp of the lidar, the IMU attitude data and visible light image frames at the corresponding time are obtained through linear interpolation. Then, the compensated 3D point cloud is projected onto the 2D image plane through camera intrinsic and extrinsic parameters to achieve spatial alignment.

[0090] LiDAR point cloud preprocessing module Design concept: Voxel downsampling, normal estimation, and outlier removal are performed on the motion-compensated LiDAR point cloud to reduce the amount of point cloud data and improve the efficiency and accuracy of subsequent feature extraction.

[0091] Core improvement: Adopting an adaptive voxel downsampling method, the point cloud data volume is compressed to 10%-20% of the original while preserving the key geometric features of the target, improving processing efficiency by ≥60%.

[0092] Key Design: Voxel downsampling: The point cloud space is divided into a uniform voxel grid, and only one representative point is retained in each voxel to reduce data redundancy.

[0093] Normal estimation: Based on the KD tree, search the neighborhood points of each point, and calculate the normal vector of the point through principal component analysis to provide geometric information for subsequent feature extraction.

[0094] Outlier removal: By calculating the neighborhood distance of each point, outliers with a mean distance exceeding a threshold are removed, thereby improving the quality of the point cloud.

[0095] Visible light image feature extraction module Design concept: A visible light image feature extraction architecture is constructed based on convolutional neural networks. Multi-scale texture features and semantic information of the image are extracted through multi-layer convolution and pooling operations, providing high-quality 2D features for subsequent fusion.

[0096] Core improvement: A lightweight backbone network is adopted, which significantly improves inference speed while ensuring feature extraction accuracy. The feature extraction time for a single frame image is ≤5ms.

[0097] Key Design: Input preprocessing: Denoise and normalize the visible light image, and unify the image size to a preset size.

[0098] Multi-scale feature extraction: Extracting multi-scale features of images by using convolution kernels of different sizes to capture texture information of targets of different sizes.

[0099] Feature mapping: The extracted multi-scale features are mapped to a unified dimension to prepare for subsequent fusion with point cloud features.

[0100] Point cloud feature extraction module Design concept: A point cloud feature extractor is built based on the PointNet++ architecture. The global geometric features and local structural features of the point cloud are extracted through multilayer perceptron and max pooling operations, so as to achieve effective processing of irregular point cloud data.

[0101] Core improvements: Optimize the sampling and grouping strategy of PointNet++ to enhance the feature extraction capability of small targets and sparse point clouds in oilfield scenarios, and improve the point cloud feature representation capability by ≥40%.

[0102] Core Formulas and Design: The core computational logic for point cloud feature extraction is as follows:

[0103] in, The data is preprocessed point cloud data; MLP stands for Multilayer Perceptron. This is a global max pooling operation. This refers to the extracted global features of the point cloud.

[0104] 2D-3D projection fusion module Design concept: Project 3D point cloud features onto a 2D image coordinate system, and perform pixel-by-pixel stitching and nonlinear transformation with visible light image features to achieve pixel-level feature fusion and output fused features that combine texture and geometric information.

[0105] Core improvement: For the first time, 2D-3D pixel-level feature fusion in oilfield drone inspection scenarios has been achieved, fully exploring the complementary value of the two modalities of data, and improving the expressive power of the fused features by ≥50%.

[0106] Core Formulas and Design: Point cloud feature projection: The extracted global features of the point cloud are mapped to the same dimension as the image features through a linear transformation, and then expanded to the same spatial size as the image features through a broadcast operation. The formula is as follows:

[0107] Where Linear represents the linear transformation layer, and Expand represents the broadcast operation. Visible light image features, The features of the projected point cloud are shown.

[0108] Pixel-level fusion: Image features and projected point cloud features are concatenated along the channel dimension, and then a 1×1 convolution is applied for non-linear transformation to output fused features. The formula is as follows:

[0109] Where Cat represents the channel-level concatenation operation. It is a 1×1 convolutional layer. This is the final fusion feature.

[0110] End-to-end fusion detection module Design concept: The target detection head is built based on fused features. It predicts the bounding box, category, confidence and 3D size information of the target through convolution operation, realizing end-to-end processing from raw data input to detection result output.

[0111] Core improvement: The construction of an integrated end-to-end fusion detection network avoids the information loss of the traditional "process separately first, then fuse the results" architecture, improving the detection accuracy by ≥25%.

[0112] Core Formulas and Design: Detection head output: Input the fused features into the detection head, and output the bounding box coordinates, class probability, and confidence score of the target. The formula is:

[0113] The Output dimension is (B, 6, H, W), which corresponds to 4 bounding box coordinates, 1 class probability, and 1 confidence level, respectively.

[0114] Non-maximum suppression: Non-maximum suppression is applied to the detection results to remove duplicate detection boxes and retain the detection results with the highest confidence.

[0115] Integrated visual interactive module Design concept: Based on PyQt5, we developed an integrated visual interactive system that integrates the entire process of data loading, model training, inference detection, and result display. It adopts an asynchronous thread training mechanism to avoid interface lag and can complete the entire process of oilfield drone inspection without the need for professional code operation.

[0116] Core improvements: Lowering the technical operation threshold, adapting to the usage needs of oilfield field operation and maintenance personnel, and improving the implementation and practicality of the technology.

[0117] Key Design: Data loading area: Supports batch loading and management of synthetic and real data.

[0118] Parameter settings area: Allows you to configure hyperparameters such as learning rate, batch size, and number of training epochs for model training.

[0119] Control button area: Provides function buttons such as start training, start inference, and stop operation.

[0120] Log area: Displays system running status and log information in real time.

[0121] Visualization area: Displays visible light images, LiDAR point clouds, and fusion detection results under different labels.

[0122] Core Implementation Steps The visible light and lidar-coordinated oilfield UAV inspection target feature fusion method of this embodiment is specifically implemented in steps S1-S8, with each step executed sequentially and data exchanged to form a complete closed loop of data acquisition-preprocessing-feature extraction-fusion detection-result display: S1: System Initialization and Parameter Configuration Start the visual interactive system to complete hardware environment adaptation (automatic CPU / GPU identification) and initialization of each module.

[0123] Configure core parameters in the visualization interface: point cloud voxel size, image input size, model training hyperparameters (batch size, learning rate, training epochs), camera intrinsic and extrinsic parameters.

[0124] Initialize the visible light image feature extraction network, point cloud feature extraction network, projection fusion module, and detection head to prepare for subsequent processes.

[0125] S2: Preparation and Loading of Multi-Source Data Select the data source in the visualization interface: real-time collected data, historical real data, or synthetic data.

[0126] If you choose to collect data in real time: activate the visible light camera, lidar, and IMU / GPS sensor on the drone to collect data from multiple sensors simultaneously.

[0127] If you choose historical real data: Select the data file path, and the system will automatically load visible light images, lidar point clouds, and IMU / GPS data.

[0128] If synthetic data is selected: the system automatically generates visible light images and lidar point cloud data that match the oilfield inspection scenario in batches, with complete target annotation information.

[0129] The loaded data is divided into training and testing sets to provide data support for model training and performance verification.

[0130] S3: Motion Compensation and Spatiotemporal Alignment Based on IMU / GPS data, motion distortion compensation is performed on the lidar point cloud to eliminate point cloud deformation caused by drone movement.

[0131] By synchronizing with timestamps, each frame of LiDAR point cloud is matched with the corresponding visible light image.

[0132] By utilizing camera intrinsic and extrinsic parameters, the compensated 3D point cloud is projected onto a 2D image plane to achieve spatial alignment.

[0133] S4: Single-modal feature extraction The aligned visible light image is input into the image feature extraction module, and multi-scale texture features are extracted through convolution and pooling operations, and the image feature map is output.

[0134] The aligned LiDAR point cloud is input into the point cloud feature extraction module, and global geometric features of the point cloud are extracted through voxel downsampling, normal estimation, multilayer perceptron and global pooling operations.

[0135] S5: 2D-3D Projection Blending The global features of the point cloud are mapped to the same dimension as the image features through a linear transformation.

[0136] The mapped point cloud features are expanded to the same spatial size as the image features through a broadcast operation.

[0137] Image features and expanded point cloud features are concatenated along the channel dimension.

[0138] The concatenated features are non-linearly transformed using 1×1 convolution to output a fused feature map.

[0139] S6: Training and Optimization of Fusion Detection Model Set the optimizer and loss function for model training, using cross-entropy loss as the classification loss and smooth L1 loss as the bounding box regression loss.

[0140] The model is iteratively trained according to the preset training rounds and batch size, using fused features as input and target annotations as supervision.

[0141] After training, the model performance is validated using a test set to verify the model's detection accuracy, precision, and recall, and the model optimization and weight saving are completed.

[0142] S7: Inference Detection and Result Output Start the inference detection function in the visual interface and select the multi-sensor data to be detected.

[0143] The data to be detected is sequentially input into the motion compensation and spatiotemporal alignment module, the single-modal feature extraction module, the projection fusion module, and the trained and optimized detection model.

[0144] Non-maximum suppression is applied to the detection results to remove duplicate detection frames.

[0145] Output the target's category, location, confidence level, and 3D size information.

[0146] S8: Results Visualization and Log Recording Visible light images, lidar point clouds, and fused detection results are displayed in a visualization interface with labels, and detected faulty targets are highlighted in red.

[0147] Automatically saves the entire process of operation logs, test results and alarm information, forming a traceable inspection record.

[0148] Generate inspection reports to provide data support for oilfield operation and maintenance decisions.

[0149] The core innovations of this implementation method are reflected in five aspects. All innovations address the core shortcomings of existing visible light and lidar-based collaborative oilfield UAV inspection target feature fusion technologies, possessing significant technical originality and practical application value. The specific innovations are as follows: Motion compensation and spatiotemporal alignment mechanism based on IMU / GPS: For the first time, a high-precision spatiotemporal alignment mechanism for multiple sensors suitable for oilfield drone inspection scenarios is constructed. The mechanism corrects the motion distortion of the lidar point cloud in real time by using IMU angular velocity and acceleration data. Combined with timestamp synchronization and coordinate system transformation, it achieves microsecond-level time synchronization and millimeter-level spatial alignment between visible light images and lidar point clouds, solving the industry pain point of difficult multi-sensor data alignment.

[0150] 2.2D-3D Pixel-Level Feature Projection Fusion Method: A pixel-level fusion architecture is designed to project 3D point cloud features onto a 2D image coordinate system. Through linear transformation and broadcasting operations, the global features of the point cloud are expanded to the same spatial size as the image features. Then, pixel-by-pixel stitching and nonlinear transformation are performed to fully explore the complementary value of visible light image texture features and LiDAR point cloud geometric features, which significantly improves the accuracy and robustness of target detection.

[0151] End-to-end fusion detection network architecture: An integrated end-to-end network is constructed from the input of raw multi-sensor data to the output of target detection results. It realizes deep interaction and information sharing of visible light and lidar features, avoids the information loss of the traditional "process separately first, then fuse the results" architecture, and improves the detection accuracy by ≥25%.

[0152] Dual data source processing mechanism for oilfield inspection scenarios: It supports input from three data sources: real-time collected data, historical real data, and synthetic data. Synthetic data can simulate inspection data and fault types under different oilfield scenarios in batches, effectively solving the problems of scarce and difficult annotation of real labeled data in oilfield scenarios, and improving the generalization ability and scenario adaptability of the model.

[0153] Integrated Visual Interaction and Asynchronous Training Design: An integrated visual interaction system based on PyQt5 was developed, integrating the entire process of data loading, model training, inference detection, and result display. An asynchronous thread training mechanism is adopted to avoid interface lag. The entire process of oilfield UAV inspection can be completed without professional code operation, which greatly reduces the technical operation threshold and improves the implementation and practicality of the technology.

[0154] To verify the effectiveness and technical advantages of the proposed implementation method, two sets of comparative experiments were designed for the core scenario of oilfield UAV inspection. These experiments verified the performance advantages of the motion compensation and spatiotemporal alignment mechanism and the 2D-3D projection fusion method of the proposed implementation method. The experimental setup and results are as follows: Experimental Environment and Dataset Setup Experimental hardware environment: CPU is Intel Xeon W-2245, GPU is NVIDIA RTX 3090Ti (24G VRAM), memory is 64G, software environment is Python 3.8, PyTorch 1.12, CUDA 11.6, Open3D 0.15, OpenCV 4.5.

[0155] Experimental dataset: The dataset used is the UAV inspection dataset from a certain oil production plant in Daqing Oilfield from 2024 to 2025. It includes visible light images and lidar point cloud data of core equipment such as pumping units, oil pipelines, storage tanks, and valves, totaling 860 sets of samples, covering various states such as normal operation, equipment failure, and leakage. 80% of the dataset is used as the training set and 20% is used as the test set.

[0156] Evaluation metrics: Detection accuracy, precision, recall, mean average precision (mAP), and frame inference time (FPS) were selected as the core evaluation metrics to comprehensively verify the model's detection accuracy and real-time performance.

[0157] Comparative Experiment of Motion Compensation and Spatiotemporal Alignment Mechanism This experiment compares the IMU / GPS-based motion compensation and spatiotemporal alignment mechanism proposed in this implementation with traditional methods without motion compensation. Training and testing were conducted under the same dataset and hardware environment. The core performance comparison is as follows:

[0158] Experimental results show that the motion compensation and spatiotemporal alignment mechanism of this implementation can effectively eliminate motion distortion of lidar point clouds, significantly improve the alignment accuracy of multi-sensor data, and thus greatly improve the accuracy of target detection, especially for small targets (such as valves and joints) in oilfield scenarios.

[0159] Comparative Experiment of 2D-3D Projection Fusion Methods This experiment compares the proposed 2D-3D pixel-level projection fusion method with traditional feature stitching fusion and weighted fusion methods. Training and testing were conducted under the same dataset and hardware environment. The core performance comparison is as follows:

[0160] Experimental results show that the 2D-3D pixel-level projection fusion method of this embodiment can more fully explore the complementary value of visible light and lidar data compared with traditional fusion methods, significantly improve the detection accuracy of equipment faults and leaks in oilfield inspection scenarios, and maintain good real-time performance, which can meet the real-time requirements of oilfield UAV inspection.

[0161] The above description of several specific embodiments further details the technical solution provided by the present invention in order to highlight the advantages and benefits of the technical solution provided by the present invention. However, the above-described specific embodiments are not intended to limit the present invention. Any reasonable modifications and improvements to the present invention, combinations of embodiments, and equivalent substitutions based on the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for fusing target features in oilfield UAV inspection using a combination of visible light and lidar, characterized in that, include: The steps to acquire visible light images, lidar point clouds, and IMU / GPS data during the oilfield drone inspection process, and perform time correlation to obtain multi-source inspection data; The steps are as follows: motion compensation is performed on the lidar point cloud based on the IMU / GPS data, and the motion-compensated lidar point cloud is synchronized with the visible light image to obtain an image point cloud data pair; The step of projecting the lidar point cloud in the image point cloud data pair onto the image coordinate system of the visible light image based on the camera intrinsic and extrinsic parameters to obtain the image point cloud association data; The steps include extracting features from the visible light image and lidar point cloud in the image point cloud association data to obtain image feature maps and point cloud geometric features; The step of mapping and extending the point cloud geometric features to fit the image feature map to obtain a point cloud projection feature map; The step of fusing the image feature map with the point cloud projection feature map to obtain a fused feature map; The steps for performing target detection based on the fused feature map to obtain the detection results of oilfield UAV inspection targets.

2. The method for fusing target features of oilfield UAV inspection in collaboration with visible light and lidar as described in claim 1, characterized in that, The sources of multi-source inspection data include real-time data collected by UAVs, historical real data, and synthetic data. The multi-source inspection data includes visible light images, lidar point clouds, IMU / GPS data, and target annotation data.

3. The method for fusing target features in oilfield UAV inspection using visible light and lidar in coordination, as described in claim 1, is characterized in that... Motion compensation corrects the coordinates of laser points in the lidar point cloud based on attitude, angular velocity, acceleration, and position information from the IMU / GPS data.

4. The method for fusing target features of oilfield UAV inspection in collaboration with visible light and lidar as described in claim 1, characterized in that, The image point cloud association data is obtained through timestamp synchronization, camera intrinsic parameter conversion, camera extrinsic parameter conversion, and image coordinate projection, and retains the pixel correspondence between the lidar point cloud and the visible light image.

5. The method for fusing target features of oilfield UAV inspection in collaboration with visible light and lidar as described in claim 1, characterized in that, After performing voxel downsampling, normal estimation, and outlier removal on the lidar point cloud in the image point cloud association data, point cloud feature extraction is then performed.

6. The method for fusing target features of oilfield UAV inspection in collaboration with visible light and lidar as described in claim 1, characterized in that, The point cloud projection feature map is obtained by linearly mapping the geometric features of the point cloud and broadcasting it according to the spatial dimensions of the image feature map.

7. A target feature fusion system for oilfield UAV inspection using visible light and lidar in coordination, characterized in that, include: This module acquires visible light images, lidar point clouds, and IMU / GPS data during the oilfield drone inspection process, and performs time correlation to obtain multi-source inspection data. The module performs motion compensation on the lidar point cloud based on the IMU / GPS data, and synchronizes the motion-compensated lidar point cloud with the visible light image to obtain an image point cloud data pair. Based on camera intrinsic and extrinsic parameters, the module projects the lidar point cloud in the image point cloud data pair onto the image coordinate system of the visible light image to obtain the image point cloud association data. A module that extracts features from the visible light image and lidar point cloud in the image point cloud association data to obtain image feature maps and point cloud geometric features; A module that maps and extends the geometric features of the point cloud to fit the image feature map to obtain a point cloud projection feature map; The module that fuses the image feature map with the point cloud projection feature map to obtain the fused feature map; A module that performs target detection based on the fused feature map to obtain the detection results of oilfield UAV inspection targets.

8. A computer storage medium for storing computer programs, characterized in that, When the computer program is read by the computer, the computer executes the method of claim 1.

9. A computer, comprising a processor and a storage medium, characterized in that, When the processor reads the computer program stored in the storage medium, the computer executes the method of claim 1.

10. A computer program product, as a computer program, is characterized by: When the computer program is executed, it implements the method of claim 1.

Images