A multi-modal fusion oil and gas pipeline perimeter security method and system and storage medium

Abstract

The present invention provides a multimodal fusion oil and gas pipeline perimeter security method and system, as well as a storage medium. These methods relate to the field of oil and gas pipeline security and address the existing lack of a radar and image / video multimodal fusion oil and gas pipeline perimeter security method. The method comprises: step S100, collecting point cloud data of the oil and gas pipeline perimeter environment via radar to identify suspicious targets, enabling a camera to track the targets and collect image data; step S200, performing feature extraction and fusion on the point cloud data to obtain multimodal fusion data; step S300, constructing a YOLO-based target detection model, introducing a MaSA module to introduce explicit spatial priors into the visual backbone network to enable the model to focus on target features; an RCS-OSA module for multi-scale feature extraction and aggregation; and an MLIA module to enable the module to have second-order information interaction capabilities and calculate the local importance of pixels; and step S400, detecting intruders in the multimodal fusion data using the YOLO-based target detection model.

Description

Technical Field

[0001] The present invention relates to the technical field of oil and gas pipeline security, and in particular to a multi-modal fusion oil and gas pipeline perimeter security method and system and a storage medium. Background Art

[0002] Oil and gas pipeline perimeters often span complex terrains such as deserts and mountains. With the continuous advancement of perimeter security technology, perimeter security systems are gradually enhancing their capabilities, enabling them to quickly identify and respond to various intrusions. However, most equipment is susceptible to interference from environmental factors, resulting in high false alarm rates and often relying on a single detection method to identify intrusions.

[0003] At present, some devices can only rely on radar detection. Although it can provide long-distance detection and the ability to penetrate obstacles, it has limitations in target recognition and classification. Some devices only rely on cameras for visual recognition. Although it can provide rich color and shape information, its performance will be affected in severe weather and lighting conditions.

[0004] In recent years, there has been considerable research on oil and gas pipelines. For example, the prior art document CN 113206978 A discloses an intelligent monitoring and early warning system and method for oil and gas pipeline station security. By incorporating a video intelligent recognition module, a perimeter fiber optic security module, a personnel location and health monitoring module, and a pipeline monitoring module, the system automatically monitors the entire pipeline in real time from multiple perspectives. However, the prior art lacks research on oil and gas pipeline perimeter security using multimodal fusion of radar and images / video for high accuracy and low false alarm rates. Summary of the Invention

[0005] The technical problems to be solved by the present invention are:

[0006] The existing technology lacks a high-accuracy oil and gas pipeline perimeter security method based on multimodal fusion of radar and image / video.

[0007] The present invention is to solve the above technical problems using the following technical solutions:

[0008] The present invention provides a multi-modal fusion oil and gas pipeline perimeter security method, comprising the following steps:

[0009] Step S100: collecting point cloud data of the oil and gas pipeline perimeter environment through radar, determining a suspicious target based on the radar point cloud data, and causing a camera to track the target and collect image data of the target;

[0010] Step S200: performing dimensionality reduction processing on the point cloud data, performing feature extraction on the point cloud data and image data after dimensionality reduction processing, and then performing feature fusion to obtain multimodal fusion data;

[0011] Step S300: Construct a target detection model based on YOLO. The head network of the model introduces the MaSA attention module and the RCS-OSA module. The MaSA attention module is used to introduce an explicit spatial prior in the visual backbone network by constructing a two-dimensional bidirectional spatial attenuation matrix, so that the model focuses on the target features; the RCS-OSA module is used to perform multi-scale feature extraction and feature aggregation based on the dynamic structure reparameterization capability and the cross-channel interaction mechanism of ShuffleNet; the MLIA attention module is also introduced into the backbone network of the model. The MLIA attention module is used to integrate local importance learning and channel gating mechanism to enable the module to have second-order information interaction capability to calculate the local importance of pixels;

[0012] The YOLO-based object detection model uses a loss function based on DynamicFocaler-IoU to help the model better learn to extract features from moderately difficult samples;

[0013] Step S400: Detect intrusion targets in multimodal fusion data using a YOLO-based target detection model.

[0014] Furthermore, in step S100, the camera is made to track the target and collect image data of the target. Specifically, dual radars are used to collect point cloud data of the oil and gas pipeline perimeter environment, with each radar connected to a camera. The distance of the target from radar 1 and radar 2 is first calculated, and the angle of the target relative to the camera is further calculated:

[0015]

[0016] θ=δ-α

[0017] μ=ε-β

[0018] Where a is the distance between the target and radar 1, b is the distance between the target and radar 2, c is the distance between the two radars, α and β are the angles between camera 1 and camera 2 and the target, δ and ε are the initial angles of camera 1 and camera 2, and θ and μ are the rotation angles of camera 1 and camera 2.

[0019] Furthermore, the point cloud data is subjected to dimensionality reduction processing as described in S200, specifically, a mapping transformation neural network is used to perform dimensionality reduction processing on the point cloud data. The mapping transformation neural network is based on a BP neural network, and a convolution layer and a residual module are added. The convolution layer is used to extract features of radar data, and the residual module is used to combine deep features with shallow features, thereby reducing the increase in the number of parameters and the degradation in performance as the network depth increases.

[0020] Furthermore, the mapping transformation neural network uses mean square error as the loss function:

[0021]

[0022] Where E is the average error of the data, n is the number of data, y i is the correct value of the i-th data in the data, is the predicted value given by the neural network.

[0023] Furthermore, the function implementation process of the MaSA module is as follows:

[0024] MaSA(X)=(Softmax(QK T )⊙D 2d )V

[0025]

[0026] Where Q, K and V represent query, key and value matrices respectively, and D 2d is the spatial attenuation matrix, is the two-dimensional spatial attenuation matrix, γ is the attenuation coefficient, x n 、y n 、x m 、y m is the two-dimensional coordinate of the token in the image, T is the transpose, Softmax is the normalization process, and ⊙ is the element-by-element multiplication.

[0027] Furthermore, the function implementation process of the RCS-OSA module is as follows:

[0028] RepVGG uses a multi-branch structure for inference during training and merges it into a single 3x3 convolution through an equivalent transformation:

[0029] X train =X·Identity+Conv1×1(X)+Conv3×3(X)

[0030] Among them, Identity is the identity mapping, Convlx1 and Conv3x3 are convolution operations;

[0031] By reparameterizing the structure, the multi-branch is converted to a single 3x3RepConv:

[0032] X inference =RepConv(X)

[0033] Rearrange the order of channels to facilitate information exchange between different channel groups:

[0034] ChannelShuffle(X)=X·shuffle(W)

[0035] Where W is the channel rearrangement weight matrix, which enables cross-channel information flow through grouped convolution;

[0036] Stack multiple RCS modules and aggregate features in the last layer:

[0037] OSA(X)=Concat(OSA1(X),OSA2(X),...,OSA n (X)

[0038] Among them, OSA i represents the output of the i-th OSA submodule, and Concat represents channel concatenation.

[0039] Furthermore, the functional implementation process of the MLIA module is as follows:

[0040] The functional implementation process of the local importance of the module is:

[0041]

[0042] where Γ(X)| x is the local importance value of pixel x, R represents the neighborhood centered on x, and w k are learnable weights for refining the importance of the measurement;

[0043] The implementation mechanism of the module's gating mechanism is:

[0044] A(X)=σ(X [0] )⊙ψ(σ(Γ(X)))⊙X

[0045] Among them, σ(·) and ψ(·) are sigmoid activation and bilinear interpolation operations respectively; X [0] is the first channel map of the input features.

[0046] Furthermore, the loss function based on DynamicFocaler-IoU is:

[0047]

[0048] Among them, IoU is the intersection-over-union ratio, IoU focaler is the reconstructed Focaler-IoU value, 1 is the lower threshold, u is the upper threshold; ω is the nonlinear function introduced to differentiate the weighted prediction error, eproch is the total number of trainings, and eps is the actual number of trainings.

[0049] The present invention provides a multimodal fusion oil and gas pipeline perimeter security system, which has a program module corresponding to the steps of the method described in any of the above technical solutions, and executes the steps in the above multimodal fusion oil and gas pipeline perimeter security method during operation.

[0050] The present invention also provides a computer-readable storage medium, which stores a computer program. The computer program is configured to implement the steps of the multimodal fusion oil and gas pipeline perimeter security method described in any one of the above technical solutions when called by a processor.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] A multimodal data fusion method for oil and gas pipelines will be adopted. Suspicious targets will be identified through radar with strong penetration and unaffected by light and weather. Combined with the high-resolution images provided by the camera, data fusion and identification of visual features and point cloud features will be used to greatly reduce the device's false alarm rate. High target detection accuracy can still be maintained in complex scenarios such as low light and occlusion, effectively identifying all intrusion events and responding promptly. The YOLO-based target detection model of the present invention introduces explicit spatial priors into the visual backbone network by introducing the MaSA module, introduces the RCS-OSA module to achieve cascading between features at different levels to enhance information flow, and introduces the MLIA module to integrate local importance learning and channel gating mechanisms, giving the module a synergistic effect of second-order information interaction capabilities, comprehensively improving the model's target detection accuracy and providing a more reliable technical solution for oil and gas pipeline perimeter security. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a structural diagram of the oil and gas pipeline perimeter security monitoring system with multimodal information fusion according to an embodiment of the present invention;

[0054] Figure 2 Schematic diagram of the radar-guided camera rotation in an embodiment of the present invention;

[0055] Figure 3 Schematic diagram of accurate camera pixel tracking in an embodiment of the present invention;

[0056] Figure 4 Schematic diagram of the mapping transformation neural network structure in an embodiment of the present invention;

[0057] Figure 5 is a spatial attenuation matrix diagram in the MaSA module in an embodiment of the present invention;

[0058] Figure 6 Schematic diagram of the structure of the RCS-OSA module in the example of the present invention;

[0059] Figure 7 Graph showing the loss curves of the model in the embodiment of the present invention after adding multiple loss functions;

[0060] Figure 8 Schematic diagram of data fusion of a multimodal information fusion oil and gas pipeline perimeter security monitoring system in an embodiment of the present invention;

[0061] Figure 9 This is a technical roadmap for the oil and gas pipeline perimeter security monitoring system with multimodal information fusion in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to enable those skilled in the art to better understand the present invention, exemplary embodiments or examples of the present invention will be described below with reference to the accompanying drawings. Obviously, the described embodiments or examples are only some of the embodiments or examples of the present invention, and not all of them. Based on the embodiments or examples of the present invention, all other embodiments or examples obtained by those skilled in the art without creative work should fall within the scope of protection of the present invention.

[0063] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0064] Specific implementation scheme 1: The present invention provides a multi-modal fusion oil and gas pipeline perimeter security method, such as Figure 8 and 9 As shown, the following steps are included:

[0065] Step S100: collecting point cloud data of the oil and gas pipeline perimeter using radar, identifying a suspicious target based on the radar point cloud data, calculating the angle of the target relative to the camera, and enabling the camera to track the target and collect image data of the target;

[0066] Step S200: performing dimensionality reduction processing on the point cloud data, performing feature extraction on the point cloud data and image data after dimensionality reduction processing, and then performing feature fusion to obtain multimodal fusion data;

[0067] Step S300, constructing a target detection model based on YOLO, introducing the MaSA attention module and the RCS-OSA module into the Head network of the model, wherein the MaSA attention module is used to construct a two-dimensional bidirectional spatial attenuation matrix based on the Manhattan distance, and introduce an explicit spatial prior into the visual backbone network so that the model focuses on target features; the RCS-OSA module is used to reduce computational complexity based on the dynamic structure reparameterization capability and the cross-channel interaction mechanism of ShuffleNet, and perform multi-scale feature extraction and feature aggregation; the MLIA attention module is also introduced into the backbone network of the model, and the MLIA attention module is used to integrate local importance learning and channel gating mechanism to enable the module to have second-order information interaction capability to calculate the local importance of pixels;

[0068] The model uses a loss function based on DynamicFocaler-IoU to help the model better learn to extract features from moderately difficult samples;

[0069] Step S400: Detect intrusion targets in multimodal fusion data using a YOLO-based target detection model.

[0070] Specific implementation plan 2: Figure 2 As shown, the calculation of the angle of the target relative to the camera in step S100 is specifically as follows: radar 1 and radar 2 collect the point cloud data of the oil and gas pipeline perimeter, calculate the distance of the target from the two different radars, and further calculate the angle of the target relative to the two cameras:

[0071]

[0072] θ=δ-α

[0073] Where a is the distance between the target and radar 1, b is the distance between the target and radar 2, c is the distance between the two radars, α and β are the angles between camera 1 and camera 2 and the target, δ and ε are the initial angles of camera 1 and camera 2, and θ and μ are the angles that camera 1 and camera 2 should rotate to.

[0074] like Figure 3 As shown, after the radar locates the target angle, when the distance is less than 200 pixels, the camera uses a slower tracking speed, making target tracking more stable and eliminating overshoot. The tracking time is dynamically adjusted based on the required movement distance. This embodiment is otherwise identical to the first embodiment.

[0075] Specific implementation scheme three: performing dimensionality reduction processing on the point cloud data as described in S200, specifically using a mapping transformation neural network to perform dimensionality reduction processing on the point cloud data, wherein the mapping transformation neural network is based on a BP neural network and adds a convolution layer and a residual module;

[0076] like Figure 4 As shown in Figure 1, the input layer of the model is set to 7, corresponding to the seven state information of the object detected by the millimeter wave radar, including: longitudinal distance, lateral distance, longitudinal speed, lateral speed, category, length and width.

[0077] The convolution layer adds a one-dimensional convolution operation to extract the features of the radar data.

[0078] The residual module is used to reduce the increase in the number of parameters and the degradation in performance as the network depth increases by combining deep features with shallow features.

[0079] The output layer is set to 4, representing the pixel coordinates of the top left and bottom right corners of the bounding box.

[0080] By combining the convolution operation and the residual module, the model can effectively extract features and perform information fusion. The rest of this embodiment is the same as the specific embodiment 2.

[0081] Specific implementation scheme 4: The mapping transformation neural network uses mean square error (MSE) as a loss function to measure the difference between the network output and the label:

[0082]

[0083] Where E is the average error of this batch of data, n is the number of this batch of data, y i is the correct value of the i-th data in this batch of data, is the predicted value given by the neural network. The rest of this embodiment is the same as the specific embodiment three.

[0084] Specific implementation plan 5: The functional implementation process of the Manhattan Self-Attention (MaSA) module is as follows:

[0085] MaSA(X)=(Softmax(QK Γ )⊙D 2d )V

[0086]

[0087] Where Q, K, and V represent the query, key, and value matrices, respectively, which are used to calculate the attention weights. is a two-dimensional spatial attenuation matrix, which calculates the attenuation factor of the attention weight based on the Manhattan distance. γ is the attenuation coefficient, which is used to control the attenuation speed. n 、y n 、x m 、y mis the two-dimensional coordinate of the token in the image, T is the transpose, Softmax is the normalization process, and ⊙ is the element-by-element multiplication.

[0088] like Figure 5 As shown in Figure 2, the MaSA module extends the temporal decay mechanism in RetNet to the spatial domain, constructing a two-dimensional bidirectional spatial decay matrix based on the Manhattan distance, thereby introducing an explicit spatial prior into the visual backbone network. The rest of this implementation is the same as Specific Implementation Plan 4.

[0089] Specific implementation plan six: The function implementation process of the RCS-OSA module is as follows:

[0090] RepVGG uses a multi-branch structure for inference during training and merges it into a single 3x3 convolution through an equivalent transformation. The formula is as follows:

[0091] X train =X·Identity+Conv1×1(X)+Conv3×3(X)

[0092] Among them, Identity is the identity mapping, Convlx1 and Conv3x3 are convolution operations.

[0093] By reparameterizing the structure, the multi-branch is converted to a single 3x3RepConv:

[0094] X inference =RepConv(X)

[0095] Channel shuffling rearranges the order of channels to facilitate information exchange between different channel groups:

[0096] ChannelShuffle(X)=X·shuffle(W)

[0097] Where W is the channel rearrangement weight matrix, which enables cross-channel information flow through grouped convolution;

[0098] Stack multiple RCS modules and aggregate features in the last layer:

[0099] OSA(X)=Concat(OSA1(X),OSA2(X),...,OSA n (X)

[0100] Among them, OSA i represents the output of the i-th OSA submodule, and Concat represents channel concatenation.

[0101] This implementation uses the RCS module to implement a multi-branch topology during the training phase and a simplified single-branch structure during the inference phase, improving feature information richness and inference speed. The OSA module aggregates multi-scale features, reducing network fragmentation and computational complexity. Multiple RCS modules are stacked within each OSA branch, enhancing feature extraction capabilities through repeated stacking.

[0102] like Figure 6 As shown, the RCS-OSA module (RepVGG / RepConv ShuffleNet One-Shot Aggregation) is a lightweight feature enhancement unit designed for target detection tasks. Its technical architecture integrates the dynamic structural reparameterization capability of RepVGG / RepConv and the cross-channel interaction mechanism of ShuffleNet. The RCS-OSA module can effectively cascade between features at different levels to enhance information flow. By adopting a scalable multi-branch structure to capture diverse features during training, and merging them into a single computing path through structural reparameterization during the inference stage, the computing efficiency is optimized. The rest of this implementation plan is the same as the specific implementation plan five.

[0103] Specific implementation plan seven: The MLIA (Mix Local Importance-based Attention) module integrates local importance learning and channel gating mechanism, and has second-order information interaction capabilities. The functional implementation process of the MLIA module is as follows:

[0104] The functional implementation process of the local importance of the module is:

[0105]

[0106] where Γ(X)| x is the local importance value of pixel x, R represents the neighborhood centered on x, and w k are learnable weights for refining the importance of the measurement;

[0107] In order to recalibrate local importance and avoid artifacts introduced by strided convolution and bilinear interpolation, the gating mechanism of the module is implemented as follows:

[0108] A(X)=σ(X [0] )⊙ψ(σ(r(X)))⊙X

[0109] Among them, σ(·) and ψ(·) are sigmoid activation and bilinear interpolation operations respectively; X [0] It is the first channel map of the input feature, which is used to simplify the gate control unit. The rest of this embodiment is the same as the specific embodiment six.

[0110] Specific implementation plan eight: The loss function based on DynamicFocaler-IoU adopts the linear interval mapping method to construct the IoU loss:

[0111]

[0112] Among them, IoU is the intersection-over-union ratio, IoU focaler is the reconstructed Focaler-IoU value, 1 is the lower threshold, and u is the upper threshold. By adjusting the values ​​of 1 and u, the Focaler-IoU can be controlled to focus on regression samples in different ranges. ω is the nonlinear function introduced to differentiate the weighted prediction error, eproch is the total number of training times, and eps is the actual number of training times.

[0113] Based on the loss function of DynamicFocaler-IoU, the loss is adjusted according to the value of the intersection-over-union ratio. When loU is less than a lower threshold of 1, the loss is 0; when loU is greater than an upper threshold u, the loss is 1; and when loU is between 1 and u, the loss is not a function that increases linearly according to the loU value, but a nonlinear function transformed according to ω. The method of the present invention allows the loss function to be sensitive to the IoU value within a certain range, so that the model can focus more on samples with a medium degree of overlap between the predicted bounding box and the true bounding box, which helps the model better learn to extract features from samples of medium difficulty, rather than just focusing on the easiest or most difficult samples. The rest of this implementation plan is the same as the specific implementation plan seven.

[0114] In order to verify the effectiveness of the Dynamic Focaler-IoU loss function of the present invention, training was carried out under the same conditions, and the effects were compared with the common loss functions CIoU, EIoU, DIoU, GIoU, SIoU, WIoU, MPDIoU, ShapeIoU, and FocalerIoU. The results are as follows: Figure 7 As shown in Figure 3, the results show that as the number of training times changes, the model's focus also changes, allowing it to focus more on samples where the predicted bounding boxes have a moderate overlap with the true bounding boxes.

[0115] The multimodal fusion oil and gas pipeline perimeter security method (algorithm) proposed in the present invention is the underlying technical core of the present invention, and various products can be derived based on the algorithm.

[0116] Based on the method proposed in the present invention, a multi-modal fusion oil and gas pipeline perimeter security system is developed using a programming language. The system has a program module corresponding to the steps of the above technical solution, and executes the steps of the above multi-modal fusion oil and gas pipeline perimeter security method when running. Figure 1As shown in the figure, the system is based on the "smart pipeline" architecture and follows the principles of intelligence, panoramic view, and integration for its system framework design. The system consists of three main layers: the perception layer is used to collect environmental information around the oil and gas pipelines; the transmission layer is used to ensure that the collected data is transmitted to the data processing center in real time and accurately; and the application layer is used to store and preliminarily process the transmitted data, enabling intelligent analysis and comprehensive application of data to provide decision support. The system also includes an emergency alarm processing module and a storage module for collecting relevant information data, ensuring a rapid response in emergency situations and saving key data for subsequent analysis.

[0117] The developed system (software) computer program is stored on a computer-readable storage medium. The computer program is configured to implement the steps of the aforementioned multimodal fusion oil and gas pipeline perimeter security method when invoked by a processor. This materializes the present invention on a carrier, becoming a computer program product.

[0118] Various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuitry, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0119] The computer programs (also referred to as programs, software, software applications, or code) of the present invention include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., a magnetic disk, an optical disk, a memory, a programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0120] The beneficial effects of the present invention will be described below with reference to specific embodiments.

[0121] Example 1

[0122] The proposed network model was compared with the existing YOLOv5-YOLOv12. A dataset of 5,000 images was constructed, encompassing five key security elements: personnel, vehicles, drones, helmets, and safety clothing. This comprehensive dataset contains complex environmental features and security semantics. The training, test, and validation sets were arranged in a 7:2:1 ratio. The experimental configuration was as follows: Windows 10 operating system, a 12th Gen Intel(R) Core(TM) i9-12900H 2.50GHz processor, an NVIDIA GeForce GTX 3060 GPU, CUDA version 12.1, PyTorch 2.5.1 deep learning framework, and Python 3.10.16 scripting language. Throughout the training phase, a batch size of 16 images was used, with each image having an input dimension of 640×640, and training spanned 100 epochs. The experiment was conducted while keeping the configuration environment and initial training settings unchanged. The results are shown in Table 1. It can be seen that the target detection accuracy and model performance of the network model of the present invention are significantly improved.

[0123] Table 1

[0124] algorithm Precision Recall F1 mAP0.5 mAP (0.5: 0.95) FPS GFLOPS YOLOv5 87.52％ 84.34％ 0.85 88.89％ 63.44％ 839.17 7.1 YOLOv6 85.79％ 81.47％ 0.83 86.37％ 60.78％ 840.28 11.8 YOLOv7 87.71％ 84.34％ 0.80 87.89％ 63.21％ 841.17 8.3 YOLOv8 89.08％ 85.57％ 0.86 90.45％ 66.14％ 813.54 8.1 YOLOv9 87.82％ 85.79％ 0.86 89.97％ 65.70％ 152.55 102.3 YOLOv10 83.20％ 80.69％ 0.81 86.53％ 63.07％ 935.79 8.2 YOLOv11 88.41％ 84.91％ 0.86 89.58％ 64.95％ 765.45 6.3 YOLOv12 85.02％ 78.57％ 0.81 84.58％ 57.88％ 524.59 6.3 MR-YOLO 91.75％ 88.92％ 0.90 93.14％ 71.21％ 340.66 33.5

[0125] Although the present invention is disclosed as above, the scope of protection disclosed by the present invention is not limited thereto. Those skilled in the art of the present invention may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A multi-modal fusion oil and gas pipeline perimeter security method, characterized by: The steps include: Step S100: collecting point cloud data of the oil and gas pipeline perimeter environment through radar, determining a suspicious target based on the radar point cloud data, and causing a camera to track the target and collect image data of the target; Step S200: performing dimensionality reduction processing on the point cloud data, performing feature extraction on the point cloud data and image data after dimensionality reduction processing, and then performing feature fusion to obtain multimodal fusion data; Step S300: Build a YOLO-based target detection model. The head network of the model introduces the MaSA attention module and the RCS-OSA module. The MaSA attention module is used to introduce an explicit spatial prior into the visual backbone network by constructing a two-dimensional bidirectional spatial attenuation matrix, so that the model focuses on target features. The RCS-OSA module is used to perform multi-scale feature extraction and feature aggregation based on the dynamic structure reparameterization capability and the cross-channel interaction mechanism of ShuffleNet. The MLIA attention module is also introduced into the backbone network of the model. The MLIA attention module is used to integrate local importance learning and channel gating mechanisms to enable the module to have second-order information interaction capabilities to calculate the local importance of pixels. The YOLO-based object detection model uses a loss function based on DynamicFocaler-IoU to help the model better learn to extract features from moderately difficult samples; Step S400: Detect intrusion targets in multimodal fusion data using a YOLO-based target detection model. The functional implementation process of the MLIA attention module is as follows: The functional implementation process of the local importance of the module is: where Γ(X)| x is the local importance value of pixel x, R represents the neighborhood centered on x, and w k are learnable weights for refining the importance of the measurement; The implementation mechanism of the module's gating mechanism is: A(X)=σ(X [0] )⊙ψ(σ(Γ(X)))⊙X Among them, σ(·) and ψ(·) are sigmoid activation and bilinear interpolation operations respectively; X [0] is the first channel map of the input features.

2. The multi-modal fusion oil and gas pipeline perimeter security method according to claim 1 is characterized in that: Step S100 involves making the camera track the target and collect image data of the target. Specifically, dual radars are used to collect point cloud data of the oil and gas pipeline perimeter environment, with each radar connected to a camera. The distance of the target from radar 1 and radar 2 is first calculated, and the angle of the target relative to the camera is further calculated. θ=δ-α μ=ε-β Where a is the distance between the target and radar 1, b is the distance between the target and radar 2, c is the distance between the two radars, α and β are the angles between camera 1 and camera 2 and the target, δ and ε are the initial angles of camera 1 and camera 2, and θ and μ are the rotation angles of camera 1 and camera 2.

3. The multi-modal fusion oil and gas pipeline perimeter security method according to claim 2 is characterized in that: The point cloud data is subjected to dimensionality reduction processing as described in S200, specifically using a mapping transformation neural network to perform dimensionality reduction processing on the point cloud data. The mapping transformation neural network is based on a BP neural network and adds a convolution layer and a residual module. The convolution layer is used to extract features of radar data, and the residual module is used to combine deep features with shallow features, thereby reducing the increase in the number of parameters and the degradation in performance caused by the increase in network depth.

4. The multi-modal fusion oil and gas pipeline perimeter security method according to claim 3 is characterized in that: The mapping transformation neural network uses mean square error as the loss function: Where E is the average error of the data, n is the number of data, y i is the correct value of the i-th data in the data, is the predicted value given by the neural network.

5. The multi-modal fusion oil and gas pipeline perimeter security method according to claim 4 is characterized in that: The functional implementation process of the MaSA attention module is as follows: MaSA(X)=(Soft max(QK T )⊙D 2d )V Where Q, K and V represent query, key and value matrices respectively, and D 2d is the spatial attenuation matrix, is the two-dimensional spatial attenuation matrix, γ is the attenuation coefficient, x n 、y n 、x m 、y m is the two-dimensional coordinate of the token in the image, T is the transpose, Softmax is the normalization process, and ⊙ is the element-by-element multiplication.

6. The multi-modal fusion oil and gas pipeline perimeter security method according to claim 5 is characterized in that: The function implementation process of the RCS-OSA module is as follows: RepVGG uses a multi-branch structure for inference during training and merges it into a single 3x3 convolution through an equivalent transformation; Through structural reparameterization, the multi-branch is converted into a single 3x3RepConv; Rearrange the order of channels to promote information interaction between different channel groups; Multiple RCS modules are stacked and features are aggregated in the last layer.

7. The multi-modal fusion oil and gas pipeline perimeter security method according to claim 6 is characterized in that: The loss function based on DynamicFocaler-IoU is: Among them, IoU is the intersection-over-union ratio, IoU focaler is the reconstructed Focaler-IoU value, 1 is the lower threshold, u is the upper threshold; ω is the nonlinear function introduced to differentiate the weighted prediction error, eproch is the total number of trainings, and eps is the actual number of trainings.

8. A multi-modal fusion oil and gas pipeline perimeter security system, characterized by: The system has a program module corresponding to the steps of the method described in any one of claims 1 to 7, and executes the steps of the multi-modal fusion oil and gas pipeline perimeter security method during operation.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the multimodal fusion oil and gas pipeline perimeter security method according to any one of claims 1 to 7 when called by a processor.

Images