Two-stage oil and gas pipeline occupation pressure monitoring method and system based on unmanned aerial vehicle space perception and deep learning fusion

Abstract

The invention belongs to the technical field of oil and gas pipeline safety monitoring and unmanned aerial vehicle image analysis, and discloses a two-stage oil and gas pipeline occupation pressure monitoring method and system based on unmanned aerial vehicle space perception and deep learning fusion, and the method comprises the steps: obtaining image data and space data of a to-be-monitored region, combining the image data and the space data with a DEM, and obtaining a DEM image; realizing projection of geographic coordinates of the pipeline to image pixel coordinates to obtain a plane image of the pipeline; constructing and deploying a lightweight target detection network DRF-YOLOv8n, performing reasoning on a plane image of the pipeline based on the lightweight target detection network DRF-YOLOv8n, outputting a candidate bounding box, and performing screening and key frame judgment; a Promp-guide SAM pixel level segmentation module is used to realize pixel level segmentation of the key frame, and a candidate pixel mask is generated; and judging whether occupation occurs or not through the intersection of the pipeline projection line and the candidate pixel mask, and outputting alarm information with geographic coordinates.

Description

Technical Field

[0001] This invention relates to the field of oil and gas pipeline safety monitoring and UAV image analysis technology, specifically to a two-stage oil and gas pipeline occupancy monitoring method and system based on the fusion of UAV spatial perception and deep learning. Background Technology

[0002] Oil and gas pipelines are critical infrastructure for energy transportation, and their safe operation is directly related to national energy supply and the safety of the ecological environment along the pipeline route. In complex terrain environments such as mountainous areas, pipelines are vulnerable to serious threats from third-party encroachment or damage (such as ponds, buildings, etc.), and such incidents must be detected promptly. Current inspections mainly rely on manual foot patrols, which are not only extremely inefficient and costly, but also easily affected by complex terrain and environment, resulting in high rates of missed and false detections. In addition, existing UAV inspection methods often focus only on target detection or segmentation, lacking deep integration and utilization of pipeline spatial location information; traditional image processing and target detection algorithms are not robust enough to multi-scale, medium and large-sized targets; and semantic segmentation technology based on deep learning requires a large number of pixel-level annotations, which are costly. Although SegmentAnythingModel (SAM) can segment without pre-annotated pixel-level labels, its computational load is enormous, making it difficult to meet real-time segmentation requirements on edge devices. Current technologies have not effectively combined UAV dynamic attitude information with deep learning models, resulting in the inaccurate mapping of pipeline geographical location onto aerial images, thus limiting the practical value of intelligent monitoring.

[0003] Scholars have proposed risk assessment systems based on remote sensing imagery for detecting occupancy of oil and gas pipelines. For example, the Shandong Provincial Oil and Gas Pipeline Protection Service Center proposed "A Method, System, and Medium for Risk Assessment of Oil and Gas Pipelines Based on Remote Sensing Images." Its main technical features include: acquiring pipeline network vector data and high-resolution remote sensing images of the area to be identified; constructing a hazard buffer zone of 5m to 200m for the pipeline and slicing the image within the buffer zone (including overlapping areas); using CSPDarknet53 as the backbone for feature extraction (converting the original 608×608×3 image into a 304×304×32 feature map); combining feature fusion structures such as SPP and FPN+PAN with multi-scale prediction to output target location, category, and rotation angle information; and using loss functions such as GIoU-Loss during training to improve regression accuracy. This type of solution is suitable for offline batch investigation based on high-resolution remote sensing images. It has high detection stability and overall coverage in the detection of building encroachment and construction hazards. However, when facing engineering applications that require frame-by-frame geolocation or pixel-level boundary determination, it suffers from high computational and memory overhead due to slicing and multi-scale fusion, and it is difficult to directly use the UAV pose and DEM for pixel-geographic coordinate coupling. In addition, China University of Petroleum (East China) and others proposed a "Method and System for Detecting Excavator Occupation in Oil and Gas Pipelines Based on YOLOv5". Its main technical features include: acquiring synthetic or on-site captured images of excavator occupancy and performing preprocessing such as histogram equalization and edge detection using Sobel / Laplacian / Canny; labeling targets to construct a dataset; using YOLOv5 as the detector and combining it with DeepSORT for multi-target tracking and early warning; adopting adaptive anchor boxes (k-means clustering + genetic algorithm mutation), adaptive image scaling, and various data augmentations (scaling, color space adjustment, mosaic) during the training phase to improve detection robustness; and reducing false detections by programming the ROI (recognition region) according to the camera installation angle during prediction. This type of solution performs excellently in real-time detection and continuous tracking based on fixed cameras or surveillance video, enabling rapid and automated early warning of dangerous behaviors such as those of excavators. However, its limitations include: reliance on large-scale, representative datasets and annotation work; insufficient robustness for scenarios requiring geographic coordinate positioning, pixel-level pipeline masks, or complex airborne perspectives; and limited versatility of camera-based ROI settings in heterogeneous aerial photography or mobile platforms. In summary, the problems of inefficiency in traditional pipeline inspection methods, insufficient spatial perception in existing image analysis methods, high annotation costs, and poor real-time performance urgently need to be addressed. Summary of the Invention

[0004] To address the technical problems of low efficiency of manual inspection, insufficient utilization of three-dimensional spatial information, the need for extensive annotation in semantic segmentation, and poor real-time performance in existing technologies, this invention provides a two-stage oil and gas pipeline occupancy monitoring method and system based on the fusion of UAV spatial perception and deep learning. This method utilizes a UAV platform to acquire high-definition aerial images along the pipeline and combines deep learning target detection and large model segmentation technology to achieve real-time monitoring and accurate judgment of pipeline occupancy threats such as ponds and buildings.

[0005] To achieve the above objectives, the present invention provides the following solution: A two-stage method for monitoring oil and gas pipeline occupancy based on the fusion of UAV spatial perception and deep learning, the method comprising: Acquire image and spatial data of the area to be monitored, combine the image and spatial data with the DEM, and project the pipeline's geographic coordinates onto the image pixel coordinates to obtain a planar image of the pipeline. A lightweight object detection network, DRF-YOLOv8n, is constructed and deployed. Based on the lightweight object detection network DRF-YOLOv8n, inference is performed on the planar image of the pipeline, candidate bounding boxes are output, and then filtered and keyframes are determined. The Prompt-guided SAM pixel-level segmentation module is used to perform pixel-level segmentation on keyframes and generate candidate pixel masks. The intersection of the pipeline projection line and the candidate pixel mask is used to determine whether an overload has occurred, and an alarm message with geographic coordinates is output.

[0006] Preferably, the method for combining image data, spatial data, and DEM to project pipeline geographic coordinates onto image pixel coordinates includes: Use the PyProj library to query the surface elevation in the DEM based on the center latitude and longitude of each image frame; Calculate the drone's altitude relative to the ground based on the ground elevation; Calculate the horizontal / vertical field of view based on the camera's intrinsic parameters; The image coverage size is calculated based on the drone's relative height to the ground and its horizontal / vertical field of view. Calculate the pixel ground resolution (GSD) based on the image coverage size; Based on pixel-level ground resolution (GSD), the pipeline's latitude and longitude are converted into local projection coordinates of the UAV, realizing the projection of the pipeline's geographical coordinates onto the image's pixel coordinates.

[0007] Preferably, the construction method of the lightweight object detection network DRF-YOLOv8n includes: embedding DRF modules at two C2f positions in the YOLOv8n backbone; The DRF module consists of two branches: one is a 3×3 depthwise separable convolutional branch with a 1×1 point convolution, and the other is a 5×5 dilated convolutional branch with a dilation rate of 3, followed by a 1×1 point convolution to adjust the number of channels. Each branch extracts features from different receptive fields in parallel, then concatenates them along the channel dimension and performs channel compression and feature fusion through a 1×1 convolution. The fused features are refined through channel attention and spatial attention. Channel attention is generated by global average pooling followed by a 1×1 convolution and a sigmoid function to generate channel weights. Spatial attention is generated by concatenating max pooling and average pooling along the channel dimension followed by a 7×7 convolution and a sigmoid function to generate spatial weights. The module is then fused with the backbone features using 1×1 compression and residual connections.

[0008] Preferably, the method for filtering candidate bounding boxes and determining keyframes includes: Geometric relationship determination is performed between the candidate bounding box and the pipeline pixel polyline superimposed on the corresponding frame: first calculate whether the bounding box intersects with each segment of the pipeline polyline, and mark it as "suspected occupancy" when the pipeline intersects with the bounding box; A simple time consistency strategy within a frame or on a sequence is used as a weighted decision, and the corresponding frame is marked as a key frame when a preset condition is met.

[0009] Preferably, the method for determining whether occupancy has occurred by the intersection of the pipeline projection line and the candidate pixel mask includes: Each pipeline segment and each mask is first coarsely screened using AABB; if the AABBs do not overlap, they are skipped. For candidates that pass the coarse screening, a line segment-polygon edge intersection detection is performed. If an intersection exists, the occupancy is determined to be valid. If there are no intersecting edges, use the ray method to determine whether the endpoints of the line segment fall within the mask. If any endpoint is contained within the mask, the occupancy is determined to be valid. For cases with blurred boundaries or near-touch, a confidence threshold is set: the final occupancy confidence is obtained by fusing detection confidence, mask confidence, and projection error estimation; results below the threshold are reported as items requiring manual verification. If the occupancy determination is valid, the pixel coordinates of the mask are back-projected into a geographic polygon through pixel-geographic mapping, and the geographic attributes of the occupant object are calculated.

[0010] The present invention also provides a two-stage oil and gas pipeline occupancy monitoring system based on the fusion of UAV spatial perception and deep learning. The system is used to implement the aforementioned method and includes: a projection module, an inference module, a segmentation module, and a judgment module. The projection module is used to acquire image data and spatial data of the area to be monitored, and combine the image data, spatial data and DEM to realize the projection of the pipeline's geographical coordinates to the image pixel coordinates to obtain a planar image of the pipeline. The inference module is used to build and deploy a lightweight object detection network DRF-YOLOv8n, perform inference on the planar image of the pipeline based on the lightweight object detection network DRF-YOLOv8n, output candidate bounding boxes and perform filtering and keyframe determination; The segmentation module is used to perform pixel-level segmentation of keyframes using the Prompt-guided SAM pixel-level segmentation module to generate candidate pixel masks. The judgment module is used to determine whether occupancy has occurred by intersecting the pipeline projection line with the candidate pixel mask, and outputs alarm information with geographic coordinates.

[0011] Preferably, the projection module includes: a query unit, a first calculation unit, a second calculation unit, a third calculation unit, a fourth calculation unit, and a projection unit; The query unit is used to query the surface elevation in the DEM based on the center latitude and longitude of each image frame using the PyProj library. The first calculation unit is used to calculate the height of the UAV relative to the ground based on the ground elevation; The second calculation unit is used to calculate the horizontal / vertical field of view based on the camera intrinsic parameters; The third calculation unit is used to calculate the image coverage size based on the UAV's relative ground altitude and horizontal / vertical field of view. The fourth calculation unit is used to calculate the pixel ground resolution (GSD) based on the image coverage size; The projection unit is used to convert the pipeline's latitude and longitude into local projection coordinates of the UAV based on the pixel ground resolution (GSD), thereby realizing the projection of the pipeline's geographical coordinates onto the image pixel coordinates.

[0012] Preferably, the construction process of the lightweight object detection network DRF-YOLOv8n includes: embedding DRF modules at two C2f positions in the YOLOv8n backbone; The DRF module consists of two branches: one is a 3×3 depthwise separable convolutional branch with a 1×1 point convolution, and the other is a 5×5 dilated convolutional branch with a dilation rate of 3, followed by a 1×1 point convolution to adjust the number of channels. Each branch extracts features from different receptive fields in parallel, then concatenates them along the channel dimension and performs channel compression and feature fusion through a 1×1 convolution. The fused features are refined through channel attention and spatial attention. Channel attention is generated by global average pooling followed by a 1×1 convolution and a sigmoid function to generate channel weights. Spatial attention is generated by concatenating max pooling and average pooling along the channel dimension followed by a 7×7 convolution and a sigmoid function to generate spatial weights. The module is then fused with the backbone features using 1×1 compression and residual connections.

[0013] Preferably, the process of filtering candidate bounding boxes and determining keyframes includes: Geometric relationship determination is performed between the candidate bounding box and the pipeline pixel polyline superimposed on the corresponding frame: first calculate whether the bounding box intersects with each segment of the pipeline polyline, and mark it as "suspected occupancy" when the pipeline intersects with the bounding box; A simple time consistency strategy within a frame or on a sequence is used as a weighted decision, and the corresponding frame is marked as a key frame when a preset condition is met.

[0014] Preferably, the judgment module includes: a coarse screening unit, a cross detection unit, an occupancy judgment unit, a threshold judgment unit, and an attribute calculation unit; The coarse screening unit is used to perform AABB coarse screening on each pipeline segment and each mask. If the AABB does not overlap, it is skipped. The cross detection unit is used to perform line segment-polygon edge cross detection on the candidates that have passed the coarse screening. If a cross exists, the occupancy is determined to be valid. The occupancy determination unit is used to determine whether the endpoints of the line segment fall into the mask if there are no intersecting edges, and to determine that occupancy is established if any endpoint is included. The threshold determination unit is used to set a confidence threshold for cases with blurred boundaries or near-touch: the final occupancy confidence is obtained by fusing detection confidence, mask confidence and projection error estimation; results below the threshold are reported as items requiring manual verification. The attribute calculation unit is used to back-project the pixel coordinates of the mask into a geographic polygon through pixel-geographic mapping if the occupancy determination is valid, and to calculate the geographic attributes of the occupancy object.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) High detection accuracy: The framework combining DRF-enhanced YOLOv8 model and fine segmentation significantly improves the detection and segmentation accuracy of pipeline occupancy targets. In the experiment, this method can accurately identify medium and large occupancy targets and obtain accurate segmentation regions. (2) Reduced computation and improved real-time performance: The two-stage processing only uses complex SAM segmentation on a small number of high-risk frames, which greatly reduces the overall computation. Compared with the traditional full-image instance segmentation method, this scheme significantly improves processing speed and real-time performance while maintaining high accuracy. (3) Elimination of manual annotation costs: The second-stage segmentation uses Prompt-Guided SAM, which eliminates the need for manual pixel-level annotation and greatly reduces data preparation costs. By generating prompts using the target detection results, high-quality segmentation masks can be obtained without constructing segmentation labels for each image, avoiding tedious annotation work. (4) Strong spatial awareness and adaptability to complex terrain: The multi-source spatiotemporal fusion strategy ensures the accurate positioning of pipelines in aerial images and effectively adapts to mountain undulations and UAV altitude changes. By combining DEM to compensate for terrain height, this invention can accurately overlay pipeline paths under different terrain conditions, providing a reliable geographical context for the correlation analysis of pipeline location and detection results. (5) Improve monitoring efficiency and reliability: The dual-stage intelligent recognition framework constructed in this invention realizes the deep integration of UAV spatiotemporal data and deep learning technology, which greatly improves the efficiency of pipeline occupancy event recognition and reduces the system's computing resource consumption. It has a high degree of automation and fast response speed, effectively supporting pipeline safety monitoring in complex scenarios. Attached Figure Description

[0016] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the two-stage oil and gas pipeline occupancy monitoring method based on the fusion of UAV spatial perception and deep learning in an embodiment of the present invention; Figure 2 The following is a schematic diagram of the DRF module structure according to an embodiment of the present invention, wherein (a) is a schematic diagram of the DRF module, (b) is a schematic diagram of two parallel branch structures, (c) is a schematic diagram of the channel attention structure, and (d) is a schematic diagram of the spatial attention structure; Figure 3 This is a schematic diagram of the DRF-YOLOv8n embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] Example 1 like Figure 1 As shown, this invention provides a two-stage oil and gas pipeline occupancy monitoring method based on the fusion of UAV spatial perception and deep learning. By combining the UAV's dynamic pose, camera parameters, and DEM, it achieves accurate projection of pipeline geographic coordinates to image pixel coordinates. While ensuring the detection speed of the first stage, it improves detection recall and localization accuracy for medium and large-scale targets. The second stage uses detection-driven cue segmentation to obtain pixel-level masks, reducing pixel-level annotation costs and the number of segmentation calculations. It provides a deployable occupancy determination process and outputs alarm information with geographic coordinates to meet engineering application requirements.

[0021] The method includes: Acquire image and spatial data of the area to be monitored, combine the image and spatial data with the DEM, and project the pipeline's geographic coordinates onto the image pixel coordinates to obtain a planar image of the pipeline. A lightweight object detection network, DRF-YOLOv8n, is constructed and deployed. Based on the lightweight object detection network DRF-YOLOv8n, inference is performed on the planar image of the pipeline, candidate bounding boxes are output, and then filtered and keyframes are determined. The Prompt-guided SAM pixel-level segmentation module is used to perform pixel-level segmentation on keyframes and generate candidate pixel masks. The intersection of the pipeline projection line and the candidate pixel mask is used to determine whether an overload has occurred, and an alarm message with geographic coordinates is output.

[0022] The specific implementation process includes: S1: Data Acquisition and Synchronization (UAV Spatial Data and Imagery) S11: Plan and execute drone inspection routes to acquire image and spatial data of the area to be monitored. Each image frame must simultaneously record a timestamp, latitude and longitude (lon, lat), and absolute altitude (h). UInformation such as yaw, pitch, roll, and camera parameters (focal length f, sensor sizes Sw, Sh, resolution W, H) is extracted (from spatiotemporal fusion frames). This ensures a one-to-one correspondence between the image and attitude information and saves it as a temporal frame stream.

[0023] S12: Preprocess the raw image data: remove distortion (if camera distortion parameters are available), normalize brightness / contrast as needed, resample at a fixed resolution and record the original resolution information; mark invalid / blurred frames and optionally remove them.

[0024] S13: Prepare pipeline vector data (latitude and longitude coordinates) and flight range DEM data. The pipeline vector data comes from the existing pipeline geographic information system, and the digital elevation model data (DEM data) comes from existing surveying results.

[0025] S2: Relative altitude and pixel-geographic projection calculation (spatial data preprocessing) is used to combine the UAV's spatiotemporal data with the terrain to establish a mapping between pixels and geographic coordinates, preparing for subsequent judgment.

[0026] S21: Use the PyProj library to look up the surface elevation in the DEM for the latitude and longitude of the image center for each frame. Let E be the surface elevation obtained from the DEM for the image center latitude and longitude. ground (rice).

[0027] S22: Calculate the drone's altitude relative to the ground.

[0028] Among them, h rel h is the drone's altitude relative to the ground. U This refers to the absolute altitude of the drone. S23: Calculate the horizontal / vertical field of view based on camera intrinsic parameters.

[0029] In the pinhole model, let the camera focal length be f (meters) and the sensor width / height be S. w S h (meters), then the horizontal and vertical field of view angles are: Wherein, FOVh is the horizontal field of view and FOVv is the vertical field of view; S24: Combining relative ground height h rel Calculate the ground cover size and pixel ground resolution GSD: Ground coverage width: Ground coverage height: Horizontal and vertical pixel ground resolution: Among them G w G h These represent the width and height of the image's ground coverage, respectively; W and H represent the image resolution; and GSD... x GSD y This represents the actual geographic distance corresponding to the pixel. S25: Convert the pipeline's latitude and longitude to the UAV's local projected coordinates, and let the pipeline's local coordinates be (x... p ,y p ), UAV local coordinates (x u ,y u First, calculate the coordinate offset: Then combine the yaw angle to convert it into radians. The heading deviation is corrected using a rotation matrix, as shown in the formula: in , This refers to the coordinate offset after heading correction; The corrected coordinate offset is mapped to image pixel coordinates to generate a pipeline pixel polyline, using the following formula: in Here are the pixel coordinates of the center point of the image, and u and v are the total pixel coordinates of the pipeline re-image; Supports pixel-to-geographic coordinate inverse projection correction: for any pixel ( , First, convert it to local meter coordinates relative to the UAV: Then, the original coordinate offset is recovered by reversing the matrix: Final restored absolute geographic coordinates of the pipeline: in , For pixels ( , The corresponding pipeline geographic coordinates are then converted into latitude and longitude for alarms and GIS display.

[0030] S3: Phase 1: Lightweight and Fast Detection (DRF-YOLOv8n) DRF-YOLOv8n: Addressing the challenges of drastic target size changes, severe attitude disturbances, and complex backgrounds in UAV inspection, a Dilated Receptive Field YOLOv8n (DRF-YOLOv8n) is proposed. By expanding the receptive field and fusing spatial attitude perception features, it significantly improves the recall capability for large targets and distant occupants while maintaining the lightweight advantages of YOLOv8n. DRF-YOLOv8n consists of three parts: a backbone feature extraction network, an expanded receptive field feature enhancement module (DRF Block), and a multi-scale spatial perception detection head (Pose-Aware Head), as shown below. Figure 3 As shown.

[0031] S31: Build and deploy the lightweight object detection network DRF-YOLOv8n: Embed DRF modules at two C2f positions in the YOLOv8n backbone. For example... Figure 2 As shown in (a), the DRF module takes the feature map output by the backbone network as input. The module includes, in sequence, a parallel multi-receptive field feature extraction unit, a feature concatenation and 1×1 compression fusion unit, a channel attention unit, a spatial attention unit, and a residual fusion output unit. This structure is used to perform multi-scale and attention enhancement on the backbone features without significantly increasing the network's computational cost, and to feed the enhanced features back to the backbone network through residuals.

[0032] like Figure 2 As shown in (b), the input feature map simultaneously enters two parallel branches, thereby acquiring a wider range of spatial context information. During runtime, the input feature map... Where C is the number of channels, H is the feature map height (in pixels), and W is the feature map width (in pixels). It is simultaneously fed into a dual-branch system. The first branch is for local detail extraction, employing a 3×3 depthwise separable convolution and a 1×1 pointwise convolution. The first branch outputs the feature map... , This represents the number of channels after channel transformation. It is calculated as follows: In the second branch, a 5×5 dilated convolution is performed on the input feature map X to expand the feature receptive field, with a dilation rate of 3. Then, a 1×1 point convolution is performed to adjust the number of channels, resulting in the output feature of the second branch. , The number of channels output by the second branch. Its calculation method is as follows: With the above settings, the first branch focuses on fine-grained local information representation, while the second branch focuses on medium- and large-scale spatial context information modeling.

[0033] Output features of the first branch With the output features of the second branch By splicing along the channel dimension, the splicing features are obtained: For the splicing features Applying a 1×1 convolution to perform channel compression and feature fusion yields fused features. , This represents the number of output channels after 1×1 compression / fusion. The calculation method is as follows: in, Here, b represents the weights of a 1×1 convolution kernel used for channel compression / mapping of the concatenated features, and b is the bias vector associated with the convolution / linear transformation. This method of concatenating and then compressing and fusing avoids premature fusion during the branching stage, which could lead to the loss of multi-scale information.

[0034] like Figure 2 As shown in (c), the spliced ​​and compressed fused features First, a global average pooling operation is performed to obtain channel description vectors that reflect the overall response intensity of each channel. Then, the fused features are... Perform global average pooling to obtain channel-level description vectors. ,in: Among them, z c Let z be the c-th component of the channel description vector, where c is the channel index, i is the row index, and j is the column index. The channel description vector z is input into a 1×1 convolution (or an equivalent linear mapping) and then processed by a Sigmoid activation function to generate the channel attention weight vector. : in, It is the Sigmoid activation function. The linear mapping weights in the channel attention module, This represents the bias vector in the channel attention mapping. The channel attention weights s are used to fuse the features. Channel-by-channel weighting is performed to obtain the channel attention enhancement feature F. ca : in, This is the scaling factor for channel attention.

[0035] like Figure 2 As shown in (d), after completing the channel attention weighting, the channel attention enhancement feature F...ca Perform max pooling and average pooling operations along the channel dimension respectively to obtain the space mapping: The max pooling map and the average pooling map are concatenated along the channel dimension, and a spatial attention weight map is generated by a 7×7 convolution and a sigmoid activation function. : Using the spatial attention weight map M s For feature F ca Pixel-wise weighting is performed to obtain the spatial attention-enhanced feature F. sa .

[0036] When the spatial attention enhancement feature F sa When the number of channels is inconsistent with the input feature map X, a 1×1 convolution is performed on it to align the channels, resulting in feature F. out .

[0037] The feature F out The residual is added to the original input feature map X to output the final enhanced features of the DRF module: Y represents the fused output feature map.

[0038] S32: Prepare near-field / mid-field / far-field samples covering the drone's perspective and label them hierarchically according to GSD. During training, perform multi-scale training within the input scale range of 320–1024 (main scale 640), and combine data augmentation such as Mosaic, MixUp, random rotation (±15°), random cropping, HSV color gamut perturbation, and small target enhancement. Use CIoU (DIoU / GIoU can be used as alternatives) loss for bounding box regression to improve localization, and use BCE+Focal loss for class confidence to alleviate class imbalance.

[0039] S33: Perform fast detection on each frame during the inference phase, output candidate bounding boxes (supporting rotated or axis-aligned boxes), categories, and confidence scores; apply NMS (example IoU threshold 0.45) and confidence threshold to the output to obtain a candidate detection set.

[0040] S4: Candidate Selection and Keyframe Determination S41: Perform geometric relationship determination between the candidate detection boxes from the first stage and the pipeline pixel polyline superimposed on the frame: First, calculate whether each segment of the detection box and the pipeline polyline intersects. Specifically, first, perform coordinate alignment processing on the detection box and the pipeline polyline so that they are in the same two-dimensional plane coordinate system. The detection box can be represented as an axis-aligned rectangle or a rotated rectangle, and its vertex set is denoted as... The pipeline curve is composed of several continuous line segments, each segment being denoted as . , where A k With B k These are the start and end points of the line segment, respectively. To improve computational efficiency, axis-aligned bounding boxes are preferentially used for rapid exclusion. When the axis-aligned bounding box of the detection frame is aligned with line segment S... k If the axis-aligned bounding boxes do not overlap in the horizontal or vertical directions, the line segment is determined not to intersect with the detection box; only when the bounding boxes overlap will a precise intersection determination be performed. When the detection box is an axis-aligned rectangle, the intersection determination of the line segment and the detection box includes any of the following situations: first, any endpoint of the line segment is located inside the detection box; second, the line segment intersects with any boundary line segment of the detection box. The line segment intersection relationship is determined by the direction of the vector cross product. If two line segments satisfy the conditions of opposite direction signs or collinear overlap in the plane, they are determined to intersect.

[0041] When the detection box is a rotated rectangle, the endpoints of the line segments are projected onto the local coordinate system of the detection box, and the range of line segment parameters is calculated along the two orthogonal axes of the detection box. Let the parameterized representation of the line segment be: By cropping the projection intervals in each axis direction, the parameter intervals corresponding to the line segment within the detection box are obtained; when the parameter intervals intersect with [0,1], it is determined that the line segment intersects with the detection box.

[0042] Based on the intersection determination, the effective length of the pipeline segment within the detection frame is further calculated. Let the parameter interval corresponding to the segment within the detection frame be [t1, t2], then the length of the segment within the detection frame is: The effective lengths of all intersecting line segments within the same detection frame are summed to obtain the pipeline occupancy length corresponding to that detection frame. When a pipeline intersects with a detection frame, it is marked as "suspected occupancy".

[0043] Through the above methods, this invention achieves precise segment-by-segment intersection determination and quantitative calculation of occupancy length for pipeline polygons at the detection frame level. This step can accurately distinguish the degree of contact between the pipeline and the target, enabling refined identification of occupancy behavior and providing a reliable basis for subsequent risk classification and control decisions.

[0044] S42: Keyframe determination: In order to reduce false alarms in a single frame and trigger the second-stage segmentation only when the target appears stably in the time series, this embodiment introduces a time consistency weighted determination strategy based on a sliding time window on the first-stage detection results.

[0045] Let the j-th candidate detected in the t-th frame be... ,in For bounding box, ∈[0,1] represents the interval confidence level; the projection of the pipeline onto the t-th frame is P. t Define the geometric prior function: Within a time window of length k, if the candidate in frame t... With the candidate of frame ti satisfy: Then it is considered that the two match in time, where, The threshold for determining the intersection-union ratio (IU). Let k be the geometric / projective distance threshold, and k be the sliding window length. Let the matching indicator be... Otherwise, it is 0.

[0046] Using exponentially decaying weights: in is the time decay weight corresponding to the i-th historical frame, and Wr is the time consistency weight coefficient.

[0047] The historical matching results are weighted and accumulated to obtain the time consistency score: in, Indicates that among all candidates in frame ti, with The candidate index that best matches. , Historical detection boxes that match the current candidate and their confidence levels.

[0048] The current frame confidence score is fused with the temporal consistency score, and combined with the geometric prior, to obtain the final judgment score: in, The fusion basis score of candidate j at time t, To incorporate the final decision score after integrating geometric priors, This provides the geometric prior indication / confidence of candidate j in the current frame t. This indicates that in historical frame ti, compared with the current candidate The geometric confidence of the matched historical candidates, where λ,β∈[0,1] are the fusion coefficients.

[0049] When the following conditions are met: Then the t-th frame (or its corresponding candidate) is marked as a keyframe and sent to the second stage of segmentation processing, where The threshold is set, and m is the minimum number of frames set; otherwise, segmentation is not triggered, and temporal consistency evidence continues to accumulate only in subsequent frames.

[0050] This strategy effectively suppresses false detections in a single frame by combining constraints of "continuous occurrence of multiple frames + stable spatial location + pipeline geometric prior", while ensuring that the real occupancy target can stably trigger fine segmentation after satisfying the time consistency condition.

[0051] S5: Second Stage: Prompt-guided SAM Pixel-level Segmentation S51: For keyframes, the bounding box output from the first stage detection is used as the bounding-box prompt input to SAM (Segment Anything Model), and SAM is called to generate one or more candidate pixel masks.

[0052] S53: To save computation, SAM is only run on candidate boxes that are suspected of being occupied.

[0053] S6: Post-mask processing and merging S61: Perform small-area noise filtering on multiple masks output by SAM; S62: Perform union / topological merging strategies (merging by category or confidence) on overlapping masks, and perform morphological closing operations as necessary to repair small holes and remove thin artifacts; S63: Calculates geometric properties for each final mask: pixel area, boundary polygon, AABB (axis-aligned bounding box), and centroid pixel coordinates.

[0054] S7: Precise Occupancy Determination (Segment-by-Segment Pipeline - Mask / Frame Association) S71: Perform AABB coarse screening on each pipeline segment and each mask / detection box. If the AABBs do not overlap, skip them quickly. S72: For candidates that have passed the coarse screening, perform line segment-polygon edge intersection detection (line segment intersection algorithm based on vector cross product). If an intersection exists, the occupancy is determined to be valid. S73: If there are no intersecting edges, use the ray method to determine whether the endpoints of the line segment fall within the mask (point-in-polygon). If any endpoint is contained within the mask, the occupancy is determined to be valid. S74: For cases with blurred boundaries or near-touch, a confidence threshold can be set: the final occupancy confidence is obtained by fusing detection confidence, mask confidence (if any) and projection error estimation; results below the threshold are reported as items requiring manual verification.

[0055] S75: If the occupancy determination is valid, the pixel coordinates of the mask are back-projected into a geographic polygon (latitude and longitude coordinate sequence) through the pixel-geographic mapping in S2, and the geographic attributes of the occupancy object (area m², approximate occupancy length, coordinates of the nearest point to the pipeline, etc.) are calculated.

[0056] S8: Result Recording, Reporting, and Visualization Database In summary, this invention achieves (i) an information fusion method based on UAV attitude, camera parameters, and digital elevation model, including: acquiring the latitude and longitude, attitude, and camera intrinsic and extrinsic parameters of the UAV at various times, converting image pixel coordinates into geographic coordinates through collinear projection, and performing height correction in conjunction with the DEM to achieve accurate mapping of pipeline geographic location to the image plane. (ii) The structural design and deployment strategy of the DRF attention module, namely, introducing deformable convolution and dynamic convolution units in parallel in the YOLOv8 network, supplemented by multi-scale spatial attention, to dynamically adjust the receptive field of the feature map and highlight the target area. (iii) A two-stage collaborative detection and segmentation architecture in the field of oil and gas pipeline occupancy detection, including an improved YOLOv8 fast detection module and a Prompt-Guided SAM segmentation module; specifically, outputting the target bounding box in the first stage, and using the bounding box as input prompts for SAM in the second stage to achieve pixel-level segmentation of the target. (iv) Segmentation decision logic based on bounding box cues, including: using the YOLO detection box coordinates as the prediction input of the SAM model, and determining whether occupancy occurs by the intersection of the pipeline projection line and the segmentation mask (or bounding box).

[0057] Example 2 The present invention also provides a two-stage oil and gas pipeline occupancy monitoring system based on the fusion of UAV spatial perception and deep learning. The system is used to implement the method described in Embodiment 1. The system includes: a projection module, an inference module, a segmentation module, and a judgment module. The projection module is used to acquire image data and spatial data of the area to be monitored, and combine the image data, spatial data and DEM to project the pipeline's geographical coordinates to the image pixel coordinates to obtain a planar image of the pipeline. The inference module is used to build and deploy the lightweight object detection network DRF-YOLOv8n. Based on the lightweight object detection network DRF-YOLOv8n, it performs inference on the planar image of the pipeline, outputs candidate bounding boxes, and performs filtering and keyframe determination. The segmentation module is used to perform pixel-level segmentation of keyframes using the Prompt-guided SAM pixel-level segmentation module and generate candidate pixel masks. The judgment module is used to determine whether occupancy has occurred by intersecting the pipeline projection lines with the candidate pixel mask, and outputs alarm information with geographic coordinates.

[0058] In this embodiment, the projection module includes: a query unit, a first calculation unit, a second calculation unit, a third calculation unit, a fourth calculation unit, and a projection unit; The query unit is used to query the surface elevation in the DEM based on the center latitude and longitude of each image frame using the PyProj library. The first calculation unit is used to calculate the drone's altitude relative to the ground based on the ground elevation; The second calculation unit is used to calculate the horizontal / vertical field of view based on the camera's intrinsic parameters; The third computing unit is used to calculate the image coverage size based on the UAV's relative height to the ground and the horizontal / vertical field of view. The fourth calculation unit is used to calculate the pixel ground resolution GSD based on the image coverage size; The projection unit is used to convert the pipeline's latitude and longitude into local projected coordinates of the UAV based on the pixel ground resolution (GSD), thereby realizing the projection of the pipeline's geographical coordinates onto the image pixel coordinates.

[0059] In this embodiment, the construction process of the lightweight object detection network DRF-YOLOv8n includes: embedding DRF modules at two C2f positions in the YOLOv8n backbone; The DRF module consists of two branches: one is a 3×3 depthwise separable convolutional branch with a 1×1 point convolution, and the other is a 5×5 dilated convolutional branch with a dilation rate of 3, followed by a 1×1 point convolution to adjust the number of channels. Each branch extracts features from different receptive fields in parallel, then concatenates them along the channel dimension and performs channel compression and feature fusion through a 1×1 convolution. The fused features are refined through channel attention and spatial attention. Channel attention is generated by global average pooling followed by a 1×1 convolution and a sigmoid function to generate channel weights. Spatial attention is generated by concatenating max pooling and average pooling along the channel dimension followed by a 7×7 convolution and a sigmoid function to generate spatial weights. The module is then fused with the backbone features using 1×1 compression and residual connections.

[0060] In this embodiment, the process of filtering candidate bounding boxes and determining keyframes includes: Geometric relationship determination is performed between the candidate bounding box and the pipeline pixel polyline superimposed on the corresponding frame: first calculate whether the bounding box intersects with each segment of the pipeline polyline, and mark it as "suspected occupancy" when the pipeline intersects with the bounding box; A simple time consistency strategy within a frame or on a sequence is used as a weighted decision, and the corresponding frame is marked as a key frame when a preset condition is met.

[0061] In this embodiment, the judgment module includes: a coarse screening unit, a cross detection unit, an occupancy judgment unit, a threshold judgment unit, and an attribute calculation unit; The coarse screening unit is used to perform AABB coarse screening on each pipeline segment and each mask. If the AABBs do not overlap, they are skipped. The cross detection unit is used to perform line segment-polygon edge cross detection on the candidates that have passed the coarse screening. If a cross exists, the occupancy is determined to be valid. The occupancy determination unit is used to determine whether the endpoints of a line segment fall within the mask if there are no intersecting edges. If any endpoint is included, the occupancy is determined to be valid. The threshold determination unit is used to set a confidence threshold for cases with blurred boundaries or near-touch: the final occupancy confidence is obtained by fusing detection confidence, mask confidence and projection error estimation; results below the threshold are reported as items requiring manual verification. The attribute calculation unit is used to backproject the pixel coordinates of the mask into a geographic polygon through pixel-geographic mapping if the occupancy determination is valid, and to calculate the geographic attributes of the occupancy object.

[0062] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A two-stage oil and gas pipeline occupancy monitoring method based on the fusion of UAV spatial perception and deep learning, characterized in that, The method includes: Acquire image and spatial data of the area to be monitored, combine the image and spatial data with the DEM, and project the pipeline's geographic coordinates onto the image pixel coordinates to obtain a planar image of the pipeline. A lightweight object detection network, DRF-YOLOv8n, is constructed and deployed. Based on the lightweight object detection network DRF-YOLOv8n, inference is performed on the planar image of the pipeline, candidate bounding boxes are output, and then filtered and keyframes are determined. The Prompt-guided SAM pixel-level segmentation module is used to perform pixel-level segmentation on keyframes and generate candidate pixel masks. The intersection of the pipeline projection line and the candidate pixel mask is used to determine whether an overload has occurred, and an alarm message with geographic coordinates is output.

2. The method according to claim 1, characterized in that, Methods for combining image data, spatial data, and DEM to project pipeline geographic coordinates onto image pixel coordinates include: Use the PyProj library to query the surface elevation in the DEM based on the center latitude and longitude of each image frame; Calculate the drone's altitude relative to the ground based on the ground elevation; Calculate the horizontal / vertical field of view based on the camera's intrinsic parameters; The image coverage size is calculated based on the drone's relative height to the ground and its horizontal / vertical field of view. Calculate the pixel ground resolution (GSD) based on the image coverage size; Based on pixel-level ground resolution (GSD), the pipeline's latitude and longitude are converted into local projection coordinates of the UAV, realizing the projection of the pipeline's geographical coordinates onto the image's pixel coordinates.

3. The method according to claim 1, characterized in that, The construction method of the lightweight object detection network DRF-YOLOv8n includes embedding DRF modules at two C2f positions in the YOLOv8n backbone; The DRF module consists of two branches: one is a 3×3 depthwise separable convolutional branch with a 1×1 point convolution, and the other is a 5×5 dilated convolutional branch with a dilation rate of 3, followed by a 1×1 point convolution to adjust the number of channels. Each branch extracts features from different receptive fields in parallel, then concatenates them along the channel dimension and performs channel compression and feature fusion through a 1×1 convolution. The fused features are refined through channel attention and spatial attention. Channel attention is generated by global average pooling followed by a 1×1 convolution and a sigmoid function to generate channel weights. Spatial attention is generated by concatenating max pooling and average pooling along the channel dimension followed by a 7×7 convolution and a sigmoid function to generate spatial weights. The module is then fused with the backbone features using 1×1 compression and residual connections.

4. The method according to claim 1, characterized in that, Methods for filtering candidate bounding boxes and determining keyframes include: Geometric relationship determination is performed between the candidate bounding box and the pipeline pixel polyline superimposed on the corresponding frame: first calculate whether the bounding box intersects with each segment of the pipeline polyline, and mark it as "suspected occupancy" when the pipeline intersects with the bounding box; A simple time consistency strategy within a frame or on a sequence is used as a weighted decision, and the corresponding frame is marked as a key frame when a preset condition is met.

5. The method according to claim 1, characterized in that, Methods for determining whether occupancy has occurred by analyzing the intersection of the pipeline projection lines and the candidate pixel mask include: Each pipeline segment and each mask is first coarsely screened using AABB; if the AABBs do not overlap, they are skipped. For candidates that pass the coarse screening, a line segment-polygon edge intersection detection is performed. If an intersection exists, the occupancy is determined to be valid. If there are no intersecting edges, use the ray method to determine whether the endpoints of the line segment fall within the mask. If any endpoint is contained within the mask, the occupancy is determined to be valid. For cases with blurred boundaries or near-touch, a confidence threshold is set: the final occupancy confidence is obtained by fusing detection confidence, mask confidence, and projection error estimation; results below the threshold are reported as items requiring manual verification. If the occupancy determination is valid, the pixel coordinates of the mask are back-projected into a geographic polygon through pixel-geographic mapping, and the geographic attributes of the occupant object are calculated.

6. A two-stage oil and gas pipeline occupancy monitoring system based on the fusion of UAV spatial perception and deep learning, the system being used to implement the method described in any one of claims 1-5, characterized in that, The system includes: a projection module, an inference module, a segmentation module, and a judgment module; The projection module is used to acquire image data and spatial data of the area to be monitored, and combine the image data, spatial data and DEM to realize the projection of the pipeline's geographical coordinates to the image pixel coordinates to obtain a planar image of the pipeline. The inference module is used to build and deploy a lightweight object detection network DRF-YOLOv8n, perform inference on the planar image of the pipeline based on the lightweight object detection network DRF-YOLOv8n, output candidate bounding boxes and perform filtering and keyframe determination; The segmentation module is used to perform pixel-level segmentation of keyframes using the Prompt-guided SAM pixel-level segmentation module to generate candidate pixel masks. The judgment module is used to determine whether occupancy has occurred by intersecting the pipeline projection line with the candidate pixel mask, and outputs alarm information with geographic coordinates.

7. The system according to claim 6, characterized in that, The projection module includes: a query unit, a first calculation unit, a second calculation unit, a third calculation unit, a fourth calculation unit, and a projection unit; The query unit is used to query the surface elevation in the DEM based on the center latitude and longitude of each image frame using the PyProj library. The first calculation unit is used to calculate the height of the UAV relative to the ground based on the ground elevation; The second calculation unit is used to calculate the horizontal / vertical field of view based on the camera intrinsic parameters; The third calculation unit is used to calculate the image coverage size based on the UAV's relative ground altitude and horizontal / vertical field of view. The fourth calculation unit is used to calculate the pixel ground resolution (GSD) based on the image coverage size; The projection unit is used to convert the pipeline's latitude and longitude into local projection coordinates of the UAV based on the pixel ground resolution (GSD), thereby realizing the projection of the pipeline's geographical coordinates onto the image pixel coordinates.

8. The system according to claim 6, characterized in that, The construction process of the lightweight object detection network DRF-YOLOv8n includes: embedding DRF modules at two C2f positions in the YOLOv8n backbone; The DRF module consists of two branches: one is a 3×3 depthwise separable convolutional branch with a 1×1 point convolution, and the other is a 5×5 dilated convolutional branch with a dilation rate of 3, followed by a 1×1 point convolution to adjust the number of channels. Each branch extracts features from different receptive fields in parallel, then concatenates them along the channel dimension and performs channel compression and feature fusion through a 1×1 convolution. The fused features are refined through channel attention and spatial attention. Channel attention is generated by global average pooling followed by a 1×1 convolution and a sigmoid function to generate channel weights. Spatial attention is generated by concatenating max pooling and average pooling along the channel dimension followed by a 7×7 convolution and a sigmoid function to generate spatial weights. The module is then fused with the backbone features using 1×1 compression and residual connections.

9. The system according to claim 6, characterized in that, The process of filtering candidate bounding boxes and determining keyframes includes: Geometric relationship determination is performed between the candidate bounding box and the pipeline pixel polyline superimposed on the corresponding frame: first calculate whether the bounding box intersects with each segment of the pipeline polyline, and mark it as "suspected occupancy" when the pipeline intersects with the bounding box; A simple time consistency strategy within a frame or on a sequence is used as a weighted decision, and the corresponding frame is marked as a key frame when a preset condition is met.

10. The system according to claim 6, characterized in that, The judgment module includes: a coarse screening unit, a cross detection unit, an occupancy judgment unit, a threshold judgment unit, and an attribute calculation unit; The coarse screening unit is used to perform AABB coarse screening on each pipeline segment and each mask. If the AABB does not overlap, it is skipped. The cross detection unit is used to perform line segment-polygon edge cross detection on the candidates that have passed the coarse screening. If a cross exists, the occupancy is determined to be valid. The occupancy determination unit is used to determine whether the endpoints of the line segment fall into the mask if there are no intersecting edges, and to determine that occupancy is established if any endpoint is included. The threshold determination unit is used to set a confidence threshold for cases with blurred boundaries or near-touch: the final occupancy confidence is obtained by fusing detection confidence, mask confidence and projection error estimation; results below the threshold are reported as items requiring manual verification. The attribute calculation unit is used to back-project the pixel coordinates of the mask into a geographic polygon through pixel-geographic mapping if the occupancy determination is valid, and to calculate the geographic attributes of the occupancy object.

Images