A pipeline threat behavior identification method and device and a storage medium

Abstract

The application discloses a pipeline threat behavior identification method and device and a storage medium, and relates to the technical field of machine vision. The method comprises the following steps: acquiring a to-be-detected video photographed by unmanned aerial vehicle inspection; performing target detection on image frames of the to-be-detected video, determining target detection boxes of detection objects in the image frames and target categories of the detection objects; performing trajectory tracking on the target detection boxes to obtain target trajectories of the detection objects; extracting behavior features of the detection objects, wherein the behavior features comprise morphological change features of the detection boxes and trajectory features of the target trajectories of the detection objects; and identifying pipeline threat behaviors of the detection objects based on the target categories, the behavior features of the detection objects and a pre-trained classification model. The method can effectively detect small targets such as personnel and vehicles, effectively determine threat behaviors such as personnel loitering, personnel digging and mechanical digging, and the algorithm has the light-weight feature.

Description

A method, apparatus, and storage medium for identifying pipeline threat behaviors. Technical Field

[0001] This application relates to the field of machine vision technology, and in particular to a method, apparatus and storage medium for identifying pipeline threat behaviors. Background Technology

[0002] As the "main artery" connecting oil and gas production and consumption, long-distance oil and gas pipelines have large diameters, high pressures, and long distances, involving various complex environments and numerous potential risk factors. Once a leak or explosion occurs, it will seriously threaten the lives and property of the people. Planting deep-rooted plants, engaging in construction activities, etc. are prohibited within a five-meter radius on both sides of the pipeline's centerline.

[0003] However, these behaviors that endanger pipeline safety persist despite repeated prohibitions, especially excavation, which can damage accompanying fiber optic cables and even the pipeline itself, causing significant harm and irreversible losses. Therefore, it is essential to strengthen pipeline patrols and promptly detect and stop any threatening activities. Unmanned aerial vehicle (UAV) pipeline inspection, due to its high degree of automation and efficiency, has been widely used in the inspection of long-distance oil and gas pipelines.

[0004] Drones equipped with high-definition cameras can perform imaging monitoring of pipeline threats such as excavation. Target recognition technology based on video / images has achieved great success in some fields after years of development. Using online monitoring cameras and drone image recognition to detect construction machinery and encroachment has also become a hot research topic, with some positive results achieved.

[0005] However, the identification of construction machinery, personnel, etc. currently has the following problems: (1) It is easy to misreport normally driving construction machinery, normally walking personnel, etc. as pipeline threat targets, increasing the workload of alarm review; (2) It cannot effectively warn of the severity of the threat, such as the different threat levels of excavators in operation and those not in operation to the pipeline, and general target detection methods cannot achieve this kind of discrimination; (3) Data is sent back to the server for target identification, which is greatly affected by the link, and the data transmission pressure is high. During long-distance inspection, it is easy to miss the detection due to unstable link signals, and the computing power of the airborne edge computing equipment is limited.

[0006] Therefore, there is an urgent need for a method to identify pipeline threat behaviors with high accuracy and lightweight deployment. Summary of the Invention

[0007] The purpose of this application is to provide a method, device and storage medium for identifying pipeline threat behaviors, aiming to solve the technical problems of low identification accuracy and inability to be deployed in a lightweight manner in existing behavior identification methods.

[0008] To achieve the above objectives, this application adopts the following technical solution:

[0009] In a first aspect, this application provides a method for identifying pipeline threat behaviors, the method comprising:

[0010] Acquire the video of the object to be inspected captured by the drone during inspection;

[0011] Target detection is performed on the image frames of the video to be detected to determine the target detection box of the detected object in each image frame and the target category of each detected object;

[0012] Trajectory tracking is performed on the target detection box to obtain the target trajectory of each of the detected objects;

[0013] Extract the behavioral features of each of the detected objects, the behavioral features including: the shape change features of the detection box of the detected object and the trajectory features of the target trajectory of the detected object;

[0014] For each of the detected objects, based on the target category, the behavioral characteristics of the detected object, and the pre-trained classification model, the pipeline threat behavior of the detected object is identified.

[0015] As can be seen, the pipeline threat behavior identification method provided in this application embodiment is applicable to the application scenario of UAV pipeline inspection. It can effectively detect small targets such as personnel and vehicles, and effectively determine threat behaviors such as personnel loitering, personnel digging, and mechanical digging. Moreover, it is not based on a large model with high computational complexity requirements. The algorithm has lightweight characteristics and can be deployed on airborne edge computing devices, effectively reducing false alarms, improving the accuracy of pipeline threat target identification, effectively supporting the construction of "intelligent pipeline lines," and ensuring the safe and stable operation of long-distance oil and gas pipelines.

[0016] In some embodiments, the trajectory features include:

[0017] The trajectory features include at least one of the following: trajectory curvature, trajectory smoothness, and trajectory direction change rate; the morphological change features include: aspect ratio change parameter.

[0018] In some embodiments, the step of tracking the trajectory of the target detection box to obtain the target trajectory of each detected object includes:

[0019] Based on the attribute information of the target detection box, the association relationship between the target detection boxes in adjacent image frames is established;

[0020] Based on the target detection boxes that are related and located in different image frames, the target trajectory of the detected object corresponding to the target detection box is constructed.

[0021] In some embodiments, the attribute information includes at least one of the following: the center coordinates of the detection box, the size of the detection box, the aspect ratio, the rate of change of the coordinates of the detection box, and the rate of change of the size of the detection box.

[0022] In some embodiments, acquiring the video to be inspected captured by the drone inspection includes:

[0023] Acquire aerial video taken by the drone while it is flying along the flight path;

[0024] Based on the transition time point, the candidate video is segmented to obtain the video to be detected; wherein, the transition time point represents the time point when the UAV turns while flying along the flight path.

[0025] In some embodiments, the detection targets include: personnel and / or construction machinery, and the pipeline threat behaviors include at least one of: personnel stationary, personnel digging, personnel loitering, construction machinery stationary, and construction machinery digging;

[0026] The method also includes: graded early warning based on the identified pipeline threat behaviors.

[0027] In some embodiments, the target detection for the image frames of the video to be detected includes: performing target detection on the image frames based on a detection model;

[0028] The head layer of the detection model is provided with a first grid structure. The first grid structure is used to generate a query feature vector based on the query key vector and the query value vector. The query key vector is used to represent the coarse predicted position of the target in the coarse feature map, and the query value vector is used to represent the feature information of the fine feature map. The query feature vector is used to realize category prediction and / or bounding box regression.

[0029] Secondly, this application provides a device for identifying pipeline threat behaviors, the device comprising:

[0030] The acquisition module is used to acquire the video of the object to be inspected captured by the drone during inspection;

[0031] The detection module is used to perform target detection on the image frames of the video to be detected, and to determine the target detection box of the detected object in each image frame and the target category of each detected object;

[0032] The tracking module is used to track the trajectory of the target detection box to obtain the target trajectory of each of the detected objects;

[0033] An extraction module is used to extract behavioral features of each of the detected objects, the behavioral features including: the shape change features of the detection box of the detected object and the trajectory features of the target trajectory of the detected object;

[0034] The identification module is used to identify pipeline threat behaviors of each of the detected objects based on the target category, the behavioral characteristics of the detected objects, and a pre-trained classification model.

[0035] As can be seen, the pipeline threat behavior identification method provided in this application embodiment is applicable to the application scenario of UAV pipeline inspection. It can effectively detect small targets such as personnel and vehicles, and effectively determine threat behaviors such as personnel loitering, personnel digging, and mechanical digging. Moreover, it is not based on a large model with high computational complexity requirements. The algorithm has lightweight characteristics and can be deployed on airborne edge computing devices, effectively reducing false alarms, improving the accuracy of pipeline threat target identification, effectively supporting the construction of "intelligent pipeline lines," and ensuring the safe and stable operation of long-distance oil and gas pipelines.

[0036] In some embodiments, the trajectory features include:

[0037] The trajectory features include at least one of the following: trajectory curvature, trajectory smoothness, and trajectory direction change rate; the morphological change features include: aspect ratio change parameter.

[0038] In some embodiments, the tracking module is specifically used for:

[0039] Based on the attribute information of the target detection box, the association relationship between the target detection boxes in adjacent image frames is established;

[0040] Based on the target detection boxes that are related and located in different image frames, the target trajectory of the detected object corresponding to the target detection box is constructed.

[0041] In some embodiments, the attribute information includes at least one of the following: the center coordinates of the detection box, the size of the detection box, the aspect ratio, the rate of change of the coordinates of the detection box, and the rate of change of the size of the detection box.

[0042] In some embodiments, the acquisition module is specifically used for:

[0043] Acquire aerial video taken by the drone while it is flying along the flight path;

[0044] Based on the transition time point, the candidate video is segmented to obtain the video to be detected; wherein, the transition time point represents the time point when the UAV turns while flying along the flight path.

[0045] In some embodiments, the detection targets include: personnel and / or construction machinery, and the pipeline threat behaviors include at least one of: personnel stationary, personnel digging, personnel loitering, construction machinery stationary, and construction machinery digging;

[0046] The device further includes:

[0047] The early warning module is used to provide tiered early warnings based on the identified pipeline threat behaviors.

[0048] Thirdly, this application provides a pipeline threat behavior identification device, the device comprising: a processor and a memory; the processor and the memory being coupled; the memory being used to store one or more programs, the one or more programs including computer-executable instructions, wherein when the pipeline threat behavior identification device is running, the processor executes the computer-executable instructions stored in the memory to implement the method as described in the first aspect and any possible implementation thereof.

[0049] Fourthly, this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods described in the first aspect and any possible implementation thereof.

[0050] Fifthly, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run computer programs or instructions to implement the methods described in the first aspect and any possible implementation thereof.

[0051] Sixthly, this application provides a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods described in the first aspect and any possible implementation thereof.

[0052] The technical problems that can be solved and the technical effects that can be achieved by the pipeline threat behavior identification device, computer equipment, computer storage medium, chip or computer program product can be referred to the technical problems and technical effects solved in the first aspect above, and will not be repeated here. Attached Figure Description

[0053] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 is a flowchart illustrating a pipeline threat behavior identification method provided in an embodiment of this application;

[0055] Figure 2 is a schematic diagram of a target detection network provided in an embodiment of this application;

[0056] Figure 3 is a schematic diagram of a trajectory tracking process provided in an embodiment of this application;

[0057] Figure 4 is a schematic diagram of a pipeline threat behavior identification device provided in an embodiment of this application;

[0058] Figure 5 is a schematic diagram of another pipeline threat behavior identification device provided in an embodiment of this application;

[0059] Figure 6 is a conceptual partial view of a computer program product provided in an embodiment of this application. Detailed Implementation

[0060] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0061] In the description of this application, it should be understood that the terms "upper," "lower," "left," "right," "front," "rear," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or relative positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and for simplification, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Unless otherwise specified, the above-mentioned orientational descriptions can be flexibly set in practical applications, provided that the relative positional relationships shown in the accompanying drawings are satisfied.

[0062] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0063] In embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, article, or apparatus that includes that element.

[0064] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0065] Currently, solutions for identifying pipeline threats by construction machinery and personnel are difficult to determine and have low accuracy because the targets (such as personnel and construction machinery) carrying out pipeline threats in video images are small in scale and have indistinct image features.

[0066] Furthermore, the high complexity of typical behavior recognition models poses significant challenges for airborne deployment. Specifically, if deep learning networks, such as the SlowFast network, are used for behavior recognition, their complex structure and high computational complexity make them difficult to implement on airborne systems.

[0067] To address the aforementioned technical problems, this application provides a method for identifying pipeline threat behaviors. In this method, target detection is performed on image frames in a video captured by a drone during inspection to determine the target detection box and the target category of the detected object. Then, trajectory tracking is performed on the target detection box to obtain the target trajectory of each detected object. The trajectory features of each detected object's target trajectory are further extracted. Using a pre-trained classification model, the target category and trajectory features are processed to identify pipeline threat behaviors of the detected object.

[0068] Therefore, instead of directly analyzing the image features of the detected object, it identifies pipeline threat behaviors based on the trajectory features of the detected object. This overcomes the problems of small target scale and insignificant image features, and improves the accuracy of behavior recognition. It also solves the technical problem that current intelligent recognition systems easily misreport normally moving construction machinery and normally walking personnel as pipeline threat targets.

[0069] Furthermore, compared to deep learning models with high computational complexity, classification models significantly reduce the computational and deployment requirements of airborne terminals, enabling deployment on-board without transmitting video data to server nodes. Compared to solutions that send video data back to servers for recognition, this reduces data transmission pressure and avoids missed detections due to unstable link signals during long-distance inspections.

[0070] The application scenarios of the embodiments of this application are described below by way of example.

[0071] In this embodiment, to promptly detect and prevent pipeline threats, unmanned aerial vehicles (UAVs) are used for pipeline inspection. When inspecting the pipeline, the UAV can pre-set a flight path, flying at a constant speed diagonally above the pipeline without hovering, changing the pod's attitude or focal length, etc., resulting in a straight line or a zigzag path.

[0072] As a possible implementation provided in the embodiments of this application, the drone for pipeline inspection can be used to execute the pipeline threat behavior identification method provided in the embodiments of this application. That is, the drone does not need to send the video captured during the inspection process to the server node, but can process it locally on the drone.

[0073] This approach eliminates the need to transmit video data to server nodes, reducing data transmission pressure and preventing missed detections due to unstable link signals during long-distance inspections.

[0074] As another possible implementation provided in this application embodiment, the drone can send the video captured during the inspection process to a server node, which then executes the pipeline threat behavior identification method provided in this application embodiment. This reduces the computational power requirements of the drone.

[0075] Referring to Figure 1, which is a flowchart illustrating a pipeline threat behavior identification method provided in an embodiment of this application, the method includes the following steps:

[0076] S101: Acquire the video of the device to be inspected captured by the drone during inspection.

[0077] In this embodiment, the drone flies along a pre-defined inspection route, which can be determined based on the distribution of pipelines. During flight, the drone captures video footage from a bird's-eye view.

[0078] S102: Perform target detection on the image frames of the video to be detected, and determine the target detection box of the detected object in each image frame and the target category of each detected object.

[0079] In this embodiment, personnel and construction machinery in the video to be detected can be used as detection targets, and target detection can be performed on the image frames in the video to be detected.

[0080] The task of object detection is to find objects of interest in an image and determine their category and location.

[0081] In this embodiment, object detection algorithms can be used to process image frames to obtain object detection results. These algorithms include traditional object detection algorithms and deep learning-based regression algorithms (such as YOLO).

[0082] YOLO is an end-to-end object detection algorithm that predicts based on global image information. Its core idea is to divide the image into a grid and predict the bounding box and class of objects in each grid cell. This design makes YOLO well-suited for real-time object detection applications, enabling it to complete object detection tasks in a relatively short time.

[0083] As a state-of-the-art (SOTA) model, YOLOv8 builds upon the foundation of the YOLO series, incorporating the experience of previous versions while introducing innovative features and improvements to further enhance its performance and flexibility.

[0084] Yolov8 employs a novel backbone network architecture that enhances feature extraction and processing capabilities, resulting in more accurate object detection. It introduces an Anchor-Free detection head, eliminating reliance on anchor boxes and providing greater flexibility to better adapt to various target shapes and sizes.

[0085] As a possible implementation of this application, Yolov8 can be used to perform target detection on image frames in the video to be detected, so as to identify targets such as personnel and construction machinery that may pose a potential threat to the pipeline.

[0086] Specifically, considering the small size of targets in aerial videos captured by drones, this application embodiment improves the Yolov8 network to detect small targets in order to adapt to the detection requirements of small targets in this application scenario. See Figure 2, which is a schematic diagram of the target detection network provided in this application embodiment. In Figure 2, conv represents the convolution operation. c2f represents the coarse-to-fine module, the core idea of ​​which is to gradually transition from coarse feature extraction to fine feature extraction through multi-stage processing. sppf represents an improved version of the spp (spatial pyramid pooling) module, which divides the input image into sub-regions of different sizes by constructing multiple pyramid structures of different scales and performs pooling operations on each sub-region. concat represents the operation of concatenating two or more tensors along a certain dimension. upsample represents the upsampling operation. detect is the last layer of the network model, which can be understood as the target detection layer, responsible for predicting bounding boxes, categories, and confidence scores.

[0087] As a possible implementation of this application, the head layer of the classification model is provided with a first grid structure. The first grid structure is used to generate a query feature vector based on the query key vector and the query value vector. The query key vector is used to represent the coarse predicted position of the target in the coarse feature map, and the query value vector is used to represent the feature information of the fine feature map. The query feature vector is used to realize category prediction and / or bounding box regression.

[0088] Specifically, as shown in Figure 2, this application modifies the head layer of the Yolov8 network by incorporating low-level target location prediction information: first, the approximate location of the small target is predicted on the coarse feature map, and then the detailed location of the small target is densely calculated on the fine feature map. The approximate location of the small target is represented as query keys. The high-resolution features used to detect the small target are represented as query values, and the two generate a query value feature. Based on this query feature vector, class prediction (cls) and bounding box regression are implemented. The query keys also serve as a branch for detection. Bounding box regression represents the prediction of the bounding box location.

[0089] Specifically, the query takes the feature map as input and outputs a heatmap, which represents the probability that grid (i,j) contains a small target. During inference, the positions with predicted scores greater than the threshold σ are mapped to their four nearest neighbors on the feature as key positions, thus constructing a sparse tensor of the feature to obtain the query value feature.

[0090] By adopting the above network structure, the accuracy of small target detection can be enhanced by focusing on the position of small targets through cascading. This requires minimal additional computation.

[0091] S103: Track the target detection box to obtain the target trajectory of each detected object.

[0092] Trajectory tracking algorithms for detected targets belong to computer vision technology and are used to detect and track moving targets, such as people and objects, in video or image sequences. These algorithms utilize image processing and machine learning techniques to identify and track the target's position, trajectory, and other attributes by analyzing changes between consecutive frames. Common target tracking algorithms include Kalman filters, particle filters, correlation filters, and deep learning-based algorithms, such as those based on convolutional neural networks and recurrent neural networks.

[0093] As a possible implementation of this application embodiment, after processing in S102, a detection object is detected in the image frame, and the position of the detection object in the image and the category of the detection object are marked in the form of a detection box. Subsequently, a data association algorithm is used to associate the detected object with the detection objects in the tracking list.

[0094] Referring to Figure 3, which is a schematic flowchart of trajectory tracking provided in an embodiment of this application, as shown in Figure 3, trajectory tracking is performed on the target detection box to obtain the target trajectory of each detected object, which may specifically include:

[0095] S301: Based on the attribute information of the target detection box, establish the association relationship between target detection boxes between adjacent image frames.

[0096] S302: Based on the target detection boxes that are related and located in different image frames, construct the target trajectory of the detected object corresponding to the target detection box.

[0097] Specifically, if two object detection boxes belonging to two different image frames establish a relationship, it means that these two object detection boxes correspond to the same detected object. Therefore, based on the object detection boxes in different image frames that have a relationship, the target trajectory of the detected object can be constructed.

[0098] For example, in order to measure whether the above-mentioned relationship can be established between target detection boxes, the following attribute information of the target detection boxes can be defined: the center coordinates of the detection box, the size of the detection box, the aspect ratio, the rate of change of the coordinates of the detection box, and the rate of change of the size of the detection box.

[0099] Furthermore, relevant algorithms can be used for data association, such as the Hungarian assignment algorithm. The difference metric for determining the cost matrix is ​​the overlap between the predicted position of the original target in the current frame and the target detection box in the current frame. If the overlap exceeds a threshold, the aforementioned association is considered to have been established.

[0100] Furthermore, when the overlap between a detected target and the detection boxes of all existing target predictions is less than a specified threshold, a new target is considered to have appeared. The position information of the new target is initialized using the detection box information, and the velocity is set to 0. The set velocity variance is very large, indicating significant uncertainty. The new target needs to undergo a waiting period to be associated with the detection results to accumulate the confidence of the new target's appearance, preventing the erroneous creation of new tracking targets due to false alarms in target detection.

[0101] S104: Extract the behavioral features of each detected object. The behavioral features include: the shape change features of the detection box of the detected object and the trajectory features of the target trajectory of the detected object.

[0102] Specifically, potential threats to pipelines can include: mechanical digging, manual digging, and loitering (personnel loitering may be a precursor to drilling for oil theft or construction excavation). These behaviors are generally represented in images as targets exhibiting certain morphological changes and trajectories that differ from those of normally moving vehicles and personnel.

[0103] The features of a target captured by a drone in an image are related to the target's own characteristics, the drone's state (drone altitude, speed, attitude, and heading), the pod's attitude (pitch, roll, and yaw angles), and the camera's focal length. It is necessary to convert the target's features in the image into the target's actual features.

[0104] In this embodiment, the drone for pipeline inspection has a long flight distance, generally uses an automated airport, and has a pre-set flight path. The drone flies at a constant speed diagonally above the pipeline without hovering, and generally does not change the pod attitude or focal length, resulting in a zigzag flight path. For stationary targets or targets moving along roads, the target's trajectory has strong homogeneity. However, for mechanical excavation operations and personnel moving around, the target's trajectory exhibits abrupt changes and irregularities. Furthermore, the aspect ratio of the target changes due to the swaying of the excavator and the squatting and standing movements of the personnel.

[0105] Based on the above considerations, the trajectory of the target of potential pipeline threat behavior is abrupt and irregular, and the aspect ratio of the target will change due to squatting and digging actions. In this embodiment, the corresponding behavioral features of the detected target are extracted and classified based on the behavioral features.

[0106] Specifically, behavioral features include: the shape change features of the detection box of the detected object and the trajectory features of the target trajectory of the detected object.

[0107] Among them, the morphological change features can be represented by the aspect ratio change parameter of the detection box, and the trajectory features of the target trajectory can include at least one of the following: trajectory curvature, trajectory smoothness, and trajectory direction change rate.

[0108] As a possible implementation of this application, the trajectory curvature can be defined based on the following formula:

[0109] ;

[0110]

[0111] in, The sum of the distances between points on the trajectory represents the sum of the distances between points on the trajectory, and L represents the distance between the start and end points of the trajectory. It represents the curvature of the trajectory. Represents the sum of the trajectory points. Represents the i-th trajectory point. This represents the x-coordinate of the i-th trajectory point. This represents the ordinate of the i-th trajectory point.

[0112] As a possible implementation of this application, trajectory smoothness can be defined based on the following formula:

[0113]

[0114]

[0115] Trajectory smoothness through trajectory points The offset distance Di from the line connecting the start and end points of the trajectory is used to represent the smoothness of the trajectory. The smaller the offset distance, the better the smoothness of the trajectory, and vice versa.

[0116] in, These are the starting coordinates of the trajectory. These are the coordinates of the endpoint of the trajectory.

[0117] As a possible implementation of this application, the trajectory direction change rate can be defined based on the following formula:

[0118]

[0119]

[0120] Rate of change of trajectory direction It represents the ratio of the sum of the distances between two segments of a set of three adjacent coordinate points to the distance between the starting point of the three points.

[0121] As a possible implementation of this application, the aspect ratio change parameter of the detection box can be defined based on the following formula:

[0122] ;

[0123]

[0124] Where w represents the width of the target bounding box and h represents the height of the target bounding box. This represents the aspect ratio parameter, which is associated with the frame number of the image frame. This represents the aspect ratio parameter corresponding to image frame number j. This represents the aspect ratio change parameter corresponding to image frame j.

[0125] As a possible implementation of this application, the aspect ratio variation parameter and the characteristic values ​​such as trajectory curvature, smoothness, and direction change rate can be represented by the minimum value, maximum value, mean, and variance.

[0126] S105: For each detected object, based on the target category, the behavioral characteristics of the detected object, and the pre-trained classification model, identify the pipeline threat behavior of the detected object.

[0127] In this embodiment of the application, a classification model can be pre-trained. The classification model can process the target category and behavioral characteristics of the detected object to identify whether the detected object performs pipeline threat behavior.

[0128] To facilitate understanding, the training process of the classification model will be introduced.

[0129] In this embodiment of the application, several video clips of people loitering, people digging, and mechanical digging behaviors taken by drones can be collected as an initial dataset, and the location information of the target in the video screenshots can be labeled to form a target detection labeled dataset.

[0130] Further, continuous frame annotation is performed on the video target to establish the target trajectory. The minimum, maximum, mean, and variance of the target aspect ratio change, trajectory curvature, trajectory smoothness, and trajectory direction change rate are calculated as feature quantities, and whether they belong to threatening behavior are marked to form a behavior discrimination dataset.

[0131] The dataset is divided into a training set and a test set in a 2:1 ratio. The training set data for the discriminant dataset is denoted as D, and the training feature set is denoted as A. D = {A, C}, where C represents the category, which can specifically include stationary personnel, personnel digging, personnel loitering, stationary construction machinery, and construction machinery digging, etc.

[0132] As a possible implementation of this application, the Adaboost algorithm is used to identify threatening behaviors based on the above-mentioned features.

[0133] Adaboost is an ensemble learning algorithm that combines multiple weak classifiers (such as decision trees) to build a strong classifier. Its core idea is to progressively adjust sample weights, allowing the classifier to focus on samples that are difficult to classify, thereby improving the overall model's accuracy.

[0134] The main steps of Adaboost are as follows:

[0135] (1) Initialization: Each training sample is assigned the same weight.

[0136] (2) Iterative training of weak learners:

[0137] a. Train a weak learner using the current sample weights.

[0138] b. Calculate the error rate of the weak learner on the training set.

[0139] c. Calculate the weight of the weak learner based on the error rate: the lower the error rate, the greater the weight of the learner.

[0140] d. Update the weights of the training samples: increase the weight of misclassified samples and decrease the weight of correctly classified samples. The weight update is based on the weights of the weak learner, which ensures that misclassified samples receive more attention in the next round of training.

[0141] (3) Combining weak learners: Combine all trained weak learners according to their weights to form the final strong learner.

[0142] (4) Classification decision: For a new input sample, each weak learner will give a classification result. These results are weighted and voted on according to their respective weights to obtain the final classification result.

[0143] In this embodiment of the application, a weak learner is trained based on the above dataset, the weights of the training samples are updated, and the weak learner is combined. Finally, if the classification decision and the real classification behavior contained in the dataset meet the convergence condition, the classification model training is confirmed to be complete.

[0144] For each detection object, the target category and the behavioral characteristics of the detection object are input into the pre-trained classification model to obtain the classification result output by the classification model. The classification result represents the identification result of the pipeline threat behavior of the detection object.

[0145] As can be seen, the pipeline threat behavior identification method provided in this application embodiment is applicable to the application scenario of UAV pipeline inspection. It can effectively detect small targets such as personnel and vehicles, and effectively determine threat behaviors such as personnel loitering, personnel digging, and mechanical digging. Moreover, it is not based on a large model with high computational complexity requirements. The algorithm has lightweight characteristics and can be deployed on airborne edge computing devices, effectively reducing false alarms, improving the accuracy of pipeline threat target identification, effectively supporting the construction of "intelligent pipeline lines," and ensuring the safe and stable operation of long-distance oil and gas pipelines.

[0146] As a possible implementation of this application embodiment, acquiring the video to be inspected captured by the drone inspection may specifically include:

[0147] The process involves acquiring bird's-eye view videos taken by the drone while it is flying along the flight path; segmenting the candidate videos based on the transition time points to obtain the videos to be detected; where the transition time point represents the time point when the drone makes a turn while flying along the flight path.

[0148] Specifically, in the embodiments of this application, the drones for pipeline inspection have a long flight distance, generally use automatic airports, and pre-set flight paths. The drones fly at a constant speed diagonally above the pipeline without hovering, generally without changing the attitude of the pod or the focal length, and the flight path is a zigzag line.

[0149] As a drone travels along its flight path, it may need to turn. However, the position of the same detected object in the image frame changes abruptly before and after a turn, leading to significant calculation errors. Therefore, the drone's turning point is recorded, and based on this turning point, the captured bird's-eye view video is split into two parts, which are then processed and calculated separately.

[0150] In this embodiment of the application, when a pipeline threat is identified, a tiered early warning system can be implemented based on the identified pipeline threat. The warning level for each pipeline threat can be preset. For example, the warning level for construction machinery excavation is higher, while the warning level for personnel loitering is lower.

[0151] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. It is understood that the pipeline threat behavior identification device, in order to achieve the above functions, includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that the pipeline threat behavior identification method steps described in conjunction with the embodiments disclosed in this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0152] This application embodiment can divide the pipeline threat behavior identification device into functional modules or functional units according to the above method example. For example, each function can be divided into a separate functional module or functional unit, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or in software functional modules or functional units. The module or unit division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0153] This application provides a pipeline threat behavior identification device. Referring to Figure 4, Figure 4 is a structural schematic diagram of the pipeline threat behavior identification device provided in this application embodiment. The pipeline threat behavior identification device may include: an acquisition module 401, a detection module 402, a tracking module 403, an extraction module 404, and an identification module 405.

[0154] The acquisition module 401 is used to acquire the video to be inspected captured by the drone during inspection;

[0155] Detection module 402 is used to perform target detection on image frames of the video to be detected, and to determine the target detection box of the detected object in each image frame and the target category of each detected object;

[0156] Tracking module 403 is used to track the trajectory of the target detection box to obtain the target trajectory of each of the detected objects;

[0157] The extraction module 404 is used to extract the behavioral features of each of the detected objects, the behavioral features including: the shape change features of the detection box of the detected object and the trajectory features of the target trajectory of the detected object;

[0158] The identification module 405 is used to identify pipeline threat behaviors of each of the detected objects based on the target category, the behavioral characteristics of the detected objects, and a pre-trained classification model.

[0159] Figure 5 is a schematic diagram of another pipeline threat behavior identification device provided in an embodiment of this application. The pipeline threat behavior identification device may include a processor 502, which is used to execute application code to implement the pipeline threat behavior identification method in this application.

[0160] Processor 502 may be a central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application.

[0161] As shown in Figure 5, the pipeline threat behavior identification device may further include a memory 503. The memory 503 stores the application code that executes the scheme of this application, and its execution is controlled by the processor 502.

[0162] Memory 503 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory 503 may exist independently and be connected to processor 502 via bus 504. Memory 503 may also be integrated with processor 502.

[0163] As shown in Figure 5, the pipeline threat behavior identification device may further include a communication interface 501, wherein the communication interface 501, processor 502, and memory 503 may be coupled to each other, for example, through a bus 504. The communication interface 501 is used for information exchange with other devices, such as supporting information exchange between the pipeline threat behavior identification device and other devices.

[0164] It should be noted that the device structure shown in Figure 5 does not constitute a limitation on the identification device for the pipeline threat behavior. In addition to the components shown in Figure 5, the identification device for the pipeline threat behavior may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0165] This application also provides a computer-readable storage medium storing instructions that, when executed by a processor of a computer device, enable the computer to perform the pipeline threat behavior identification method provided in the embodiments described above. For example, the computer-readable storage medium may be a memory 503 including instructions, which may be executed by a processor 502 of a computer device to perform the above method.

[0166] Computer-readable storage media can be non-transitory computer-readable storage media, such as ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices.

[0167] Figure 6 is a conceptual partial view of a computer program product provided in an embodiment of this application. The computer program product includes a computer program for executing computer processes on a computing device.

[0168] In one embodiment, the computer program product is provided using a signal carrying medium 600. The signal carrying medium 600 may include one or more program instructions that, when executed by one or more processors, can provide the functions or parts thereof described above with reference to FIG. 1. Therefore, for example, referring to the embodiment shown in FIG. 1, one or more features of S101-S105 may be borne by one or more instructions associated with the signal carrying medium 600. Furthermore, example instructions are also described in FIG. 6.

[0169] In some examples, the signal carrying medium 600 may include a computer-readable medium 601, such as, but not limited to, a hard disk drive, a compact disc (CD), a digital video optical disc (DVD), a digital magnetic tape, a memory, a read-only memory (ROM), or a random access memory (RAM), and so on.

[0170] In some implementations, the signal carrying medium 600 may include a computer recordable medium 602, such as, but not limited to, a memory, a read / write (R / W) CD, an R / W DVD, and the like.

[0171] In some implementations, the signal carrying medium 600 may include a communication medium 603, such as, but not limited to, digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, etc.).

[0172] The signal-bearing medium 600 can be transmitted by a wireless communication medium 603. One or more program instructions may be, for example, computer-executable instructions or logical implementation instructions.

[0173] In some examples, the pipeline threat behavior identification device can be configured to provide various operations, functions, or actions in response to one or more program instructions in a computer-readable medium 601, a computer-recordable medium 602, and / or a communication medium 603.

[0174] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0175] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0176] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the constituent units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0177] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0178] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, essentially, or the part that contributes to the prior art, or the entirety or part of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute the entirety or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0179] In the description of this specification, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0180] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for identifying pipeline threat behaviors, characterized in that, Applied to unmanned aerial vehicles (UAVs), the method includes: acquiring a video to be inspected captured by the UAV during inspection; the video to be inspected is a video captured by the UAV flying along a preset route above a pipeline and looking towards the ground; performing target detection on the image frames of the video to be inspected, determining the target detection bounding boxes and target categories of the detected objects in each image frame; performing trajectory tracking on the target detection bounding boxes to obtain the target trajectories of each detected object; including: establishing a correlation relationship between the target detection bounding boxes in adjacent image frames based on the attribute information of the target detection bounding boxes; the attribute information includes the center coordinates of the detection bounding box, the size of the detection bounding box, the aspect ratio, and the rate of change of the coordinates of the detection bounding box. The method involves selecting at least one of the following: rate of change of detection box size; constructing the target trajectory of the detected object corresponding to the target detection box based on the target detection boxes in different image frames that are related; extracting behavioral features of each detected object, the behavioral features including: morphological change features of the detection box of the detected object and trajectory features of the target trajectory of the detected object; the trajectory features including at least one of: trajectory curvature, trajectory smoothness, and trajectory direction change rate; the morphological change features including: aspect ratio change parameter; the aspect ratio change parameter characterizes the ratio between the aspect ratios of the target detection boxes in adjacent frames; wherein, the trajectory curvature satisfies the following formula: ； in, The sum of the distances between points on the trajectory represents the sum of the distances between points on the trajectory, and L represents the distance between the start and end points of the trajectory. Indicates the curvature of the trajectory; Represents the sum of the trajectory points. Represents the i-th trajectory point. This represents the x-coordinate of the i-th trajectory point. This represents the ordinate of the i-th trajectory point; the smoothness of the trajectory is achieved through the trajectory point p. i The offset distance Di from the line connecting the start and end points of the trajectory is represented by the following formula: in, These are the starting coordinates of the trajectory. These are the coordinates of the endpoint of the trajectory; the rate of change of the trajectory direction satisfies the following formula: Rate of change of trajectory direction This represents the ratio of the sum of the distances between two segments of a set of three adjacent coordinate points to the distance between the starting points of the three points; the aspect ratio parameter satisfies the following formula: ； Where w represents the width of the target box and h represents the height of the target box; This represents the aspect ratio parameter, which is associated with the frame number of the image frame; This represents the aspect ratio parameter corresponding to image frame number j. The aspect ratio change parameter corresponding to image frame j is used to represent the detection object. For each of the detected objects, based on the target category, the behavioral characteristics of the detected object, and a pre-trained classification model, the pipeline threat behavior of the detected object is identified. The detected objects include personnel and / or construction machinery, and the pipeline threat behavior includes at least one of: personnel digging, personnel loitering, construction machinery remaining stationary, and construction machinery digging. Acquiring the video to be detected captured by the UAV inspection includes: acquiring a bird's-eye view video captured by the UAV while it is flying along the flight path; segmenting the bird's-eye view video based on a conversion time point to obtain the video to be detected; wherein, the conversion time point represents the time point at which the UAV turns while flying along the flight path.

2. The method according to claim 1, characterized in that, The method also includes: graded early warning based on the identified pipeline threat behaviors.

3. The method according to claim 1, characterized in that, The target detection for the image frames of the video to be detected includes: performing target detection on the image frames based on a detection model; the head layer of the detection model is provided with a first grid structure, the first grid structure is used to generate a query feature vector based on a query key vector and a query value vector, the query key vector is used to represent the coarse predicted position of the target in the coarse feature map, the query value vector is used to represent the feature information of the fine feature map, and the query feature vector is used to realize category prediction and / or bounding box regression.

4. A device for identifying pipeline threat behaviors, characterized in that, The device, applied to unmanned aerial vehicles (UAVs), includes: an acquisition module for acquiring a video to be inspected captured by the UAV during inspection; the video to be inspected is a video captured by the UAV flying along a preset route above a pipeline and looking towards the ground; a detection module for performing target detection on image frames of the video to be inspected, determining the target detection box and the target category of each detected object in each image frame; and a tracking module for tracking the trajectory of the target detection boxes to obtain the target trajectory of each detected object; specifically, the tracking module is used to: establish the association relationship between the target detection boxes in adjacent image frames based on the attribute information of the target detection boxes; the attribute information includes the center coordinates of the detection box, the size of the detection box, and its length and width. The method includes at least one of the following: the ratio of the detection box's coordinate change rate to the detection box's size change rate; constructing the target trajectory of the detected object corresponding to the detection box based on the target detection boxes in different image frames that have a correlation; and an extraction module for extracting behavioral features of each detected object, the behavioral features including: the morphological change features of the detection box of the detected object and the trajectory features of the target trajectory of the detected object; the trajectory features include at least one of the following: trajectory curvature, trajectory smoothness, and trajectory direction change rate; the morphological change features include: aspect ratio change parameter; the aspect ratio change parameter characterizes the ratio between the aspect ratios of the target detection boxes in adjacent frames; wherein, the trajectory curvature satisfies the following formula: ； in, The sum of the distances between points on the trajectory represents the sum of the distances between points on the trajectory, and L represents the distance between the start and end points of the trajectory. Indicates the curvature of the trajectory; Represents the sum of the trajectory points. Represents the i-th trajectory point. This represents the x-coordinate of the i-th trajectory point. This represents the ordinate of the i-th trajectory point; the smoothness of the trajectory is achieved through the trajectory point p. i The offset distance Di from the line connecting the start and end points of the trajectory is represented by the following formula: in, These are the starting coordinates of the trajectory. These are the coordinates of the endpoint of the trajectory; the rate of change of the trajectory direction satisfies the following formula: Rate of change of trajectory direction This represents the ratio of the sum of the distances between two segments of a set of three adjacent coordinate points to the distance between the starting points of the three points; the aspect ratio parameter satisfies the following formula: ； Where w represents the width of the target box and h represents the height of the target box; This represents the aspect ratio parameter, which is associated with the frame number of the image frame; This represents the aspect ratio parameter corresponding to image frame number j. The aspect ratio change parameter corresponding to image frame j is represented by the following: The identification module is used to identify pipeline threat behaviors of each detected object based on the target category, the behavioral characteristics of the detected object, and a pre-trained classification model; the detected objects include personnel and / or construction machinery, and the pipeline threat behaviors include at least one of: personnel digging, personnel loitering, construction machinery stationary, and construction machinery digging; the acquisition module is specifically used to: acquire bird's-eye view video taken by the UAV while it is flying along the flight path; and segment the bird's-eye view video based on the conversion time point to obtain the video to be detected; wherein, the conversion time point represents the time point when the UAV turns while flying along the flight path.

5. A device for identifying pipeline threat behaviors, characterized in that, include: Processor and memory; The processor and the memory are coupled; The memory is used to store one or more programs, the one or more programs including computer-executable instructions, and when the pipeline threat behavior identification device is running, the processor executes the computer-executable instructions stored in the memory to cause the pipeline threat behavior identification device to perform the method as described in any one of claims 1-3.

6. A computer-readable storage medium storing instructions, characterized in that, When the computer executes the instruction, the computer performs the method as described in any one of claims 1-3.

Images