Pipeline safety early warning method, system and equipment based on multispectral image

Abstract

The invention discloses a pipeline safety early warning method, system and equipment based on a multispectral image, and mainly solves the problems that when a traditional image semantic segmentation method based on monovision (RGB) is applied to a pipeline, the robustness and the accuracy are low when the pipeline faces complex illumination conditions, shielding and diversified environment changes. A hierarchical knowledge distillation-based multi-spectral (RGB-T) image semantic segmentation method is designed for pipeline safety early warning, the model performs knowledge distillation on different network hierarchies, multi-level and multi-scale feature representation ability in a teacher model is fully excavated and transmitted, a feature extraction and fusion mechanism of a student model is optimized, and pipeline safety early warning is realized. Therefore, high efficiency and light weight of the model are realized while high segmentation precision is maintained, and complementarity and collaboration of multispectral information are further improved and adaptive capacity of the model in a complex environment is enhanced in combination with a cross-modal feature alignment and fusion strategy.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a pipeline safety early warning method, system and equipment based on multispectral images. Background Art

[0002] In recent years, pipelines, especially pressure pipelines, have become increasingly complex, characterized by high parameters and harsh operating conditions. As a result, an increasing number of structurally complex pressure pipelines have been put into operation. Defects arising from manufacturing, installation, and operation have reduced the load-bearing capacity of pressure pipelines. Coupled with these harsh operating conditions, these factors have led to an increase in pressure pipeline rupture and damage accidents. Research on the safety assessment of pressure pipelines containing defects has attracted widespread attention.

[0003] Invention patent application number: 202311599107.8, name: Industrial plant production safety early warning method and system based on machine vision, using machine vision to collect infrared and visible light video images, separating motion areas through Gaussian mixture model, and then using image difference technology to extract suspected gas leakage areas that only exist under infrared, and finally combining edge shape index, grayscale center of gravity change and gas diffusion index to determine the real leakage area and issue an early warning. However, the Gaussian mixture model is extremely sensitive to environmental changes. Light fluctuations, noise interference and background complexity will cause unstable motion area detection, resulting in large errors in the extraction of suspected leakage areas; Invention patent application number: 202410359574.1, name: A natural gas pipeline safety monitoring method, system and storage medium, using a fixed camera to collect natural gas pipeline monitoring video, and using a mixed Gaussian model to construct and update the reference background of the monitoring frame at each moment in real time, and then use the background Differentiation and inter-frame difference respectively extract the moving target information, and the moving target image is obtained after binarization and image fusion; at the same time, the monitoring frame is grayed and edge detected to extract the natural gas pipeline contour, and the judgment is made based on the pipeline image integrity and the shortest image distance between the moving target and the pipeline, so as to generate a safety warning when the indicator is lower than the preset threshold; however, background difference and inter-frame difference both rely on fixed thresholds for binarization and lack an adaptive adjustment mechanism. In the real environment, it is very easy to cause false detection or missed detection due to environmental complexity or occlusion, which seriously affects the image integrity and the accuracy of distance calculation; Invention patent application number: 202410365270.6, Name: Pipeline safety warning method, system, equipment and medium based on image features, which utilizes image acquisition and preprocessing, image feature extraction and fusion algorithms to perform multi-level feature analysis on the pipeline surface state, thereby realizing abnormality (such as cracks, corrosion, leakage, etc.) detection and safety warning. This method effectively compensates for the artificial blind spots and delays in traditional inspections by performing deep feature extraction on the details of videos or static images and using pattern recognition and classification algorithms. However, since the system mainly relies on visual information, it is inevitably affected by factors such as insufficient lighting, occlusion interference, and environmental noise under complex working conditions, resulting in significant limitations in the extraction of key feature information. In addition, single image data cannot fully reflect the operating status of the pipeline and ignores the complementary effects of multi-dimensional data such as temperature and pressure, resulting in fundamental shortcomings in the robustness and generalization capabilities of early warning decisions, which are prone to false alarms or missed alarms. At the same time, in large-scale real-time monitoring applications, its computational load and response delay problems are more prominent.

[0004] In view of this, due to the complexity of real-world applications, how to effectively utilize multispectral information, that is, to fuse visible light images and thermal infrared image information, to achieve a pipeline safety early warning method that takes into account both accuracy and efficiency is an urgent problem that needs to be solved. Summary of the Invention

[0005] In order to overcome the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a pipeline safety early warning method, system and equipment based on multispectral images. The main problem is that the traditional single-vision (RGB)-based image semantic segmentation method often exhibits low robustness and accuracy when applied to pipelines in the face of complex lighting conditions, occlusions and diverse environmental changes. A multispectral (RGB-T) image semantic segmentation method based on hierarchical knowledge distillation is designed for pipeline safety early warning. The model performs knowledge distillation at different network levels, fully explores and transfers the multi-level and multi-scale feature representation capabilities of the teacher model, optimizes the feature extraction and fusion mechanism of the student model, thereby maintaining high segmentation accuracy while achieving model efficiency and lightweight. In addition, combined with cross-modal feature alignment and fusion strategies, the complementarity and synergy of multispectral information are further improved, thereby enhancing the model's adaptability in complex environments.

[0006] A pipeline safety early warning method based on multispectral images, the steps are as follows:

[0007] Step 1: Obtain multispectral raw data (including visible light images and thermal infrared images), generate and align the collected data, use data augmentation technology to expand the original dataset, use professional annotation tools (such as Labelme) to accurately annotate the aligned data pixel by pixel, generate annotation files suitable for semantic segmentation tasks, and construct a multispectral (RGB-T) semantic segmentation dataset;

[0008] Step 2: After data preprocessing is completed, the semantic segmentation model adopts a two-stream structure to process RGB visible light images (I RGB ) and thermal infrared images (I THR ) as input;

[0009] Step 3: For the feature extraction process of each modality, a hierarchical structure is designed from shallow to deep and from local to global. The original input resolution is maintained at the L0 layer to extract preliminary low-level features, while the spatial resolution is gradually reduced in the L1 to L4 layers. Through continuous convolutional layers, batch normalization and nonlinear activation operations, higher-level abstract features are further extracted to achieve feature extraction and spatial downsampling.

[0010] Step 4: Based on feature extraction, in order to enhance the expressive power of each layer feature, for each layer L iA channel attention mechanism is introduced for the visible light (RGB) and thermal infrared (THR) features extracted from each layer. First, global average pooling (GAP) is used to perform global statistics on the visible light (RGB) and thermal infrared (THR) features to extract information in the channel dimension to achieve feature aggregation. Then, a multi-layer perceptron (MLP) is used in combination with the ReLU activation function δ and the sigmoid activation function σ, and learnable weight matrices W1 and W2 are introduced to convert the aggregated statistical information into channel attention weights. Finally, the obtained attention weights are broadcast multiplied with the original features channel by channel to enhance the channel features.

[0011] Step 5: To highlight the highly recognizable spatial information in the feature map, a spatial attention module is designed. In this process, the features of the two modalities are first cross-modally connected and feature fusion is achieved through a concatenation operation. Subsequently, a single-channel spatial attention map is generated using a convolution operation to capture salient spatial regions. Finally, the generated spatial attention map is combined with the original features to enhance the spatial features of visible light (RGB) and thermal infrared (THR) features respectively.

[0012] Step 6: Further introduce an adaptive weighted fusion mechanism. Through learnable scalar parameters α and β, the contributions of channel attention and spatial attention features are dynamically balanced. During the fusion process, the features of the two modalities are weighted separately, thereby taking into account the complementarity between the modalities and the prominence of key information in the fused feature representation.

[0013] Step 7: To improve the precision of feature selection, a self-gating mechanism is used. First, a convolution operation is used to generate an adaptive gating map. This process generates gating signals for visible light (RGB) and thermal infrared (THR) features. The generated gating signals are then used to selectively enhance the original features, so that important features are highlighted.

[0014] Step 8: After the gating process is completed, the features of the two modalities are cross-modally integrated by concatenating the gated visible light (RGB) and thermal infrared (THR) features, fusing the concatenated features using a convolution operation, and then passing the fused features to the corresponding decoder layer using a skip connection strategy;

[0015] Step 9: The decoder path gradually reconstructs the feature resolution from D4 to D0. In this process, the features of the previous decoding layer are first upsampled and transferred in combination with the skip connection from the encoder layer. Finally, at the end of the decoder, feature mapping is achieved through 1×1 convolution. The feature map restored to the input resolution is mapped to the target category space, thereby generating the final semantic segmentation prediction result.

[0016] Step 10: Based on the above encoder module (i.e., the two-stream structure in step 2 and the hierarchical feature extraction network in step 3, including the feature extraction hierarchical structure from L0 to L4), the attention module (specifically including the channel attention mechanism implemented for each layer Li feature in step 4, the spatial attention module designed in step 5, the adaptive weighted fusion mechanism introduced in step 6, and the self-gating mechanism adopted in step 7), the fusion module (i.e., the cross-modal integration process of visible light RGB and thermal infrared THR features after concatenation and convolution operation in step 8), and the decoder module (i.e., the process of gradually restoring the feature resolution from D4 to D0 layers through upsampling and jump connections in step 9, and finally mapping the features to the target category space through 1×1 convolution to generate semantic segmentation results), a teacher model T is constructed. The teacher model T uses the same network as in steps 2 to 9 above. Structure and parameter configuration; after construction is completed, the teacher model T outputs normalized or standardized visible light RGB feature maps and thermal infrared THR feature maps in each layer l (l = 1, ..., L), where L represents the total number of layers of the network; the normalization or standardization processing includes but is not limited to batch normalization, layer normalization or instance normalization, which is used to adjust the feature distribution and eliminate the scale and distribution differences of features at different levels; the output feature map of the teacher model T contains multi-scale semantic information, the low-level feature map (corresponding to the L0 and L1 layers in the encoder) retains more spatial details and texture information, and the high-level feature map (corresponding to the L3 and L4 layers in the encoder) contains richer semantic and contextual information; the teacher model T is trained in an end-to-end manner, using the cross-entropy loss function combined with the pixel weight balancing strategy to optimize the network parameters, and is ultimately used to guide the knowledge transfer process of the student model;

[0017] Step 11: Construct a student model S based on the basic network architecture of the teacher model T. The student model S maintains the same overall structure as the teacher model T, including the feature extraction hierarchy structure from L0 to L4), an attention module (specifically including the channel attention mechanism implemented for each layer of Li features in step 4, the spatial attention module designed in step 5, the adaptive weighted fusion mechanism introduced in step 6, and the self-gating mechanism adopted in step 7), a fusion module (i.e., the cross-modal integration process of visible light RGB and thermal infrared THR features after concatenation and convolution operation in step 8), and a decoder module (i.e., the feature resolution is gradually restored from D4 to D0 layers through upsampling and jump connections in step 9, and finally through 1×1 convolution operation). The student model S also generates feature maps for visible light RGB and thermal infrared THR data in each layer l (l = 1, ..., L). The student model S is consistent with the teacher model T in structure, but the initial parameters are randomly set. The student model S learns feature representation capabilities from the teacher model T through knowledge distillation. The feature maps of each layer of the student model S are trained by matching with the feature maps of the corresponding layers of the teacher model T to acquire the knowledge of the teacher model and realize knowledge transfer from T to S. The feature map generation process includes convolution operations, nonlinear activation and normalization processing to ensure that the feature maps generated by the student model S have similar statistical characteristics and representation capabilities as the teacher model T.

[0018] Step 12: To effectively transfer the knowledge of the teacher model, a hierarchical knowledge distillation strategy is designed, where N l Represents the total number of elements in the lth layer. First, the distillation loss is defined in each layer through the DMI module. Then the distillation loss of each layer is multiplied by the corresponding weight coefficient and accumulated to obtain the total hierarchical knowledge distillation loss L. KD ;

[0019] Step 13: In the student model, a dedicated decoder path is designed to further improve segmentation accuracy. At each decoder layer, the output features from the encoder layer 1 are concatenated with the features from the previous decoder layer to achieve feature fusion. After multiple layers of fusion, the semantic segmentation prediction results of the student model are output at the end of the decoder.

[0020] Step 14: During the forward propagation process, the pre-processed visible light (RGB) and thermal infrared (THR) images are simultaneously input into the teacher model and the student model. Each model extracts the corresponding feature representation at each layer and completes feature reconstruction through the decoder. Finally, the semantic segmentation prediction results of the teacher model and the student model are obtained respectively.

[0021] Step 15: Calculate the semantic segmentation loss L during training seg and the layer-wise knowledge distillation loss LKD , where the semantic segmentation loss is used to evaluate the error between the prediction result and the true label, and the layered distillation loss is used to measure the difference in feature expression between the student model and the teacher model. The sum of the two constitutes the total loss L total ;

[0022] Step 16: During the training phase, L total Backpropagation for the target is performed, only the student model is updated, and the teacher model remains unchanged;

[0023] Step 17: Iterate the loop and repeat forward and backward propagation until the student model converges. After training, use the student model for inference to obtain lightweight and efficient multispectral (RGB-T) semantic segmentation.

[0024] Step 18: Use the temperature information in the thermal infrared image to detect whether there are abnormal temperature areas on the pipeline surface. If the temperature of certain areas in the thermal infrared image exceeds the set threshold, these areas may have problems such as heat leakage and insulation damage. These areas will trigger alarms to complete pipeline safety warnings.

[0025] A pipeline safety early warning system based on multispectral images, including the following modules:

[0026] Inspection data storage module, used to store sensor data, infrared images and analysis results generated during pipeline inspection to support subsequent risk assessment and historical query;

[0027] The inspection data query module uses indexing and classification mechanisms to enable rapid query of specific pipe section inspection records, abnormal data, and historical trends;

[0028] The safety knowledge graph module is used to integrate regulations, test data, and inspection results to build a hazardous chemicals safety knowledge network that integrates multi-source heterogeneous data for risk prediction and decision support.

[0029] The UAV mapping module uses a high-precision camera mounted on a UAV to achieve 3D modeling and precise positioning of pressure pipelines in complex terrain;

[0030] The drone inspection module, based on automated path planning and real-time image transmission technology, can complete visible light and infrared dual-mode inspections of pressure pipelines.

[0031] The pipeline semantic segmentation module uses a semantic segmentation algorithm for multispectral (RGB-T) images based on hierarchical knowledge distillation to analyze drone-captured images and accurately identify pipeline shapes and potential structural anomalies.

[0032] The infrared video analysis module extracts the temporal and spatial characteristics of pipeline temperature changes and combines them with the SE-TCN algorithm to identify and predict high-risk points in the pipeline.

[0033] Preferably, a pipeline safety early warning system based on multispectral images, a detection data storage module, includes the following units:

[0034] The data receiving unit is used to receive raw data from site sensors and detection equipment in real time, including numerical values, text, images, videos, infrared information, electromagnetic ultrasound and eddy current detection signals. These data are initially formatted for subsequent analysis and storage;

[0035] A data processing unit is used to perform necessary cleaning and conversion operations on the received data, remove noise and standardize the data format to improve the quality of the data and the efficiency of subsequent processing;

[0036] Data compression and optimization unit, based on data compression technology and algorithms, reduces the storage space occupied by data while maintaining fast data recovery capabilities;

[0037] The data backup and recovery unit is used to regularly back up critical data, using advanced backup technology to ensure data integrity and consistency, while supporting rapid data recovery functions to cope with system failures.

[0038] Preferably, a pipeline safety early warning system based on multispectral images, a detection data query module, includes the following units:

[0039] Index management unit, used to create and maintain data indexes and optimize data retrieval structures to speed up data access during query processes;

[0040] Query processing unit, used to process and respond to user query requests, perform complex query operations, and ensure query accuracy and efficiency;

[0041] The data presentation unit is used to display query results to users through a graphical user interface, supporting multiple data visualization methods, such as charts and images, to improve the readability and interactivity of data.

[0042] Preferably, a pipeline safety early warning system based on multispectral images, a safety knowledge graph module, includes the following units:

[0043] A data integration unit, which integrates multiple data sources, including historical safety records, operational data, and environmental monitoring data, to build a comprehensive knowledge base;

[0044] The knowledge modeling unit uses modeling technology to create and maintain a dynamically updated security knowledge graph that can reflect complex data relationships and dependencies;

[0045] A graph update and maintenance unit, which is used to regularly update the knowledge graph to include new safety information and regulatory changes, ensuring the currentness and accuracy of the graph content;

[0046] The query and analysis unit is used to provide graph query and data analysis functions, support complex logical reasoning and pattern recognition, and help users make better decisions.

[0047] Preferably, a pipeline safety early warning system based on multispectral images, a drone mapping module, includes the following units:

[0048] Surveying and mapping data acquisition unit, used for drones carrying multiple sensors to collect data in a designated area, including high-resolution photos, videos, and topographic measurement data;

[0049] Data processing and modeling unit, used to process the collected data, including image stitching, terrain reconstruction and 3D model generation to provide detailed terrain information;

[0050] User interaction unit: provides an interactive interface that allows users to access, query and operate surveying and mapping data, and supports customized views and data export.

[0051] Preferably, a pipeline safety early warning system based on multispectral images, a drone cruise module, includes the following units:

[0052] Mission planning unit, used to plan the drone's flight path and inspection points based on predetermined inspection requirements and terrain conditions, ensuring coverage of all key areas;

[0053] A data acquisition unit, used to collect critical equipment and facility status information while the drone performs visual and thermal imaging inspections during flight;

[0054] Real-time data transmission unit, used to send collected data back to the control center in real time via a high-speed communication link, supporting remote monitoring and immediate decision-making;

[0055] The data analysis and reporting unit is used to conduct in-depth analysis of the collected data, identify potential risks and anomalies, and generate detailed inspection reports for further analysis and review.

[0056] Preferably, a pipeline safety early warning system based on multispectral images, a pipeline semantic segmentation module, includes the following units:

[0057] Image processing unit, used to enhance and preprocess the collected pipeline images to improve image quality and provide clear input for semantic segmentation;

[0058] Semantic segmentation unit, which is used to identify pipeline shapes and accurately locate pipelines and possible defects by applying a semantic segmentation algorithm of multispectral (RGB-T) images based on hierarchical knowledge distillation;

[0059] The anomaly detection unit is used to analyze pipeline identification results, automatically detect any abnormalities or signs of damage, and promptly alert maintenance personnel to conduct inspections or repairs.

[0060] Preferably, a pipeline safety early warning system based on multispectral images, an infrared video analysis module, comprises the following units:

[0061] Video processing unit, used to process infrared video in real time, extract key temperature and motion information, and provide basic data for analysis;

[0062] A feature analysis unit, used to analyze temperature changes and other key features, identify abnormal patterns, and use them for early fault prediction and diagnosis;

[0063] The risk assessment unit is used to conduct risk assessment based on the analysis results, automatically generate early warning signals, and mark possible risk points on the system map.

[0064] A computer device includes: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor and the memory communicate via the bus. When the machine-readable instructions are executed by the processor, a pipeline safety early warning method based on multispectral images is performed.

[0065] By adopting the above technology, compared with the prior art, the beneficial effects of the present invention are as follows:

[0066] 1) The network of the present invention uses multispectral feature fusion, integrating visible light and thermal infrared data. Through independent but complementary feature extraction modules, it can capture all aspects of scene details. With the help of deep fusion algorithms, the system can compensate for the lack of texture information that may occur in a single modality, thereby significantly improving overall detection accuracy and robustness.

[0067] 2) The network of the present invention deeply combines channel and spatial attention, which can suppress noise while highlighting important information. By learning adjustable weights, it dynamically balances the contributions of channel and spatial features, adaptively emphasizes the importance of features, enhances the model's ability to express multispectral features, and makes the fusion process more flexible and efficient.

[0068] 3) The hierarchical knowledge distillation introduced in the network of the present invention enables the entire network architecture to establish an efficient knowledge transfer mechanism between the teacher model and the student model, optimizing the feature learning process of the student model layer by layer, thereby achieving accurate capture of complex features; through the in-depth utilization of hierarchical feature alignment and multispectral feature complementarity, the network exhibits extremely high segmentation accuracy and efficiency in multispectral feature fusion scenarios, and is particularly suitable for task scenarios that require lightweight deployment and high performance. This architecture achieves a good balance between optimizing model complexity and improving task accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive work.

[0070] Figure 1 This is a schematic diagram of the self-built data set process provided by the present invention;

[0071] Figure 2 It is a schematic diagram of the structure of the multispectral (RGB-T) semantic segmentation model provided by the present invention;

[0072] Figure 3 It is a schematic diagram of the structure of multi-spectral (RGB-T) feature fusion provided by the present invention;

[0073] Figure 4 Schematic diagram of the structure of the hierarchical knowledge distillation provided by the present invention;

[0074] Figure 5 It is a structural diagram of the security knowledge graph module provided by the present invention;

[0075] Figure 6 It is a structural schematic diagram of the drone inspection provided by the present invention. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0077] Implementation 1:

[0078] The purpose of this embodiment is to provide a pipeline safety early warning method based on multispectral images, including:

[0079] Step 1: Please refer to Figure 1, use a drone equipped with a multispectral (RGB-T) camera to obtain multispectral raw data (including visible light images and thermal infrared images), perform multispectral data generation and alignment processing on the collected data, introduce a generative adversarial network (GAN) to generate pseudo-modal images, use the inter-modal consistency check method to screen out abnormal or invalid modal data, use data augmentation technology to expand the original dataset, use professional annotation tools (such as Labelme) to accurately annotate the aligned data pixel by pixel, and generate an annotation file suitable for semantic segmentation tasks. The annotation content includes the category information of the segmented area and pixel-level label data, which is organized into a standardized format adapted to the deep learning model. Construct a multispectral (RGB-T) semantic segmentation dataset and divide it into training and test sets in proportion;

[0080] Step 2: Please refer to Figure 2 , using a dual-stream structure to process RGB visible light images simultaneously (I RGB ) and thermal infrared images (I THR ) as input, the two complementary modalities are processed through their own independent encoder paths, ensuring that the modality-specific information is fully preserved and expressed;

[0081] Step 3: The feature extraction process for each modality follows a hierarchical structure (L0 to L4) from shallow to deep and from local to global, as follows:

[0082] L0 layer: maintain the original input resolution, extract the initial low-level features, and apply formulas (1) and (2) to process visible light (RGB) and thermal infrared (THR):

[0083]

[0084] in, Represents the RGB visible light features extracted at the L0 layer, that is, the initial low-level visible light features, Represents the THR infrared light features extracted at the L0 layer, that is, the initial low-level infrared light features, It is the RGB visible light image input I at the L0 layer RGB The feature encoding function, It is the input of THR infrared image I at L0 layer THR The feature encoding function of

[0085] L1-L4 layers: gradually reduce the spatial resolution and extract higher-level abstract features. The visible light (RGB) and thermal infrared (THR) are processed by formulas (3) and (4) respectively:

[0086]

[0087]

[0088] Each coding block and Contains convolutional layers, batch normalization, and nonlinear activation functions to achieve feature extraction and spatial downsampling; Indicates that L i Layer extracted RGB visible light features, Indicates that L i Layer-extracted THR infrared light features, It is in L i Layer pair L i-1 Feature encoding function of the layer RGB visible light image feature, It is in L i Layer pair L i-1 Feature encoding function of layer THR infrared image features;

[0089] Step 4: See Figure 3 , for each level L i Visible light (RGB) and thermal infrared (THR) characteristics and The channel attention mechanism operates as follows;

[0090] Step 4-1: Global feature aggregation: Apply global average pooling (GAP) to extract statistical information on the channel dimension. The visible light (RGB) and thermal infrared (THR) are processed using formulas (5) and (6) respectively:

[0091]

[0092] Step 4-2: Channel weight calculation: Convert to channel attention weight through multi-layer perceptron (MLP): visible light (RGB) and thermal infrared (THR) are processed by formula (7) (8) respectively:

[0093]

[0094] in, Represents the channel attention weight of RGB visible light features, represents the channel attention weight of THR infrared light features, σ represents the sigmoid activation function, δ represents the ReLU activation function, W1 and W2 are learnable weight matrices;

[0095] Step 4-3: Channel enhancement features: Apply weights to the original features, and apply formulas (9) and (10) to process visible light (RGB) and thermal infrared (THR) respectively:

[0096]

[0097] in, Represents the RGB visible light features after channel enhancement, Indicates the THR infrared light characteristics after channel enhancement, Represents a broadcast multiplication operation in the channel dimension;

[0098] Step 5: The spatial attention module aims to emphasize the most spatially salient regions in the feature map:

[0099] Step 5-1: Cross-modal feature connection: Concatenate the two modal features and apply formula (11):

[0100]

[0101] Step 5-2: Spatial attention map generation: Generate a spatial attention map through convolution operation and apply formula (12) to process:

[0102] W s =σ(f conv (F concat ))∈R H×W×1 (12)

[0103] Among them, W s Represents the spatial attention weight map, with dimension R H×W×1 , where H and W represent the height and width of the feature map respectively, f conv Represents the convolution operation, outputting a single-channel spatial attention map;

[0104] Step 5-3: Spatial enhancement features: Apply spatial attention to the original features, and process the visible light (RGB) and thermal infrared (THR) using formulas (13) and (14) respectively:

[0105]

[0106] in, Represents the RGB visible light features after spatial enhancement, represents the THR infrared light feature after spatial enhancement, ⊙ represents element-by-element multiplication;

[0107] Step 6: Apply the adaptive weighted fusion mechanism to dynamically balance the contribution of channel and spatial attention features, introduce learnable parameters, weight the channel and spatial attention features, and apply formulas (15) (16) to process visible light (RGB) and thermal infrared (THR) respectively:

[0108]

[0109] in, Represents the adaptive fusion result of RGB visible light features, Represents the adaptive fusion result of THR infrared light features, α and β are trainable scalar parameters;

[0110] Step 7: Introduce self-gating mechanism:

[0111] Step 7-1: Gating signal generation: Generate an adaptive gating map through convolution operation, and apply formulas (17) and (18) to process visible light (RGB) and thermal infrared (THR) respectively:

[0112]

[0113]

[0114] Among them, G RGB Represents the adaptive gating signal of RGB visible light features, G THR Represents the adaptive gating signal of THR infrared light feature, * represents the convolution operation, and is a learnable convolution kernel;

[0115] Step 7-2: Feature selection enhancement: Apply gated signals to enhance feature selectivity. Visible light (RGB) and thermal infrared (THR) are processed using formulas (19) and (20) respectively:

[0116]

[0117] in, represents the gated enhanced RGB visible light features, represents the THR infrared light signature after gated enhancement;

[0118] Step 8: Cross-modal feature integration:

[0119] Step 8-1: Modal fusion: Concatenate the gated features and fuse them through convolution, applying formula (21):

[0120]

[0121] in, W represents the feature obtained by fusing RGB visible light features and THR infrared light features after the i-th layer is gated in series, fusion It represents the gating signal obtained by concatenating the RGB visible light feature and the THR infrared light feature;

[0122] Step 8-2: Skip connection transfer: transfer the fused features to the corresponding decoding layer through the skip connection;

[0123] Step 9: The decoder path gradually reconstructs the feature resolution from D4 to D0. First, the features of the previous decoding layer are upsampled and combined with the features transferred by the skip connection. Finally, the decoding layer D0 outputs a feature map with the same resolution as the input, which is used to generate segmentation predictions and mapped to the category space through 1×1 convolution.

[0124] Step 10: See Figure 4 , define the teacher model based on the network model of steps 1-9, denoted as T, which is a high-performance, high-complexity multimodal network; at each layer l, the teacher model will output feature maps of two modalities, visible light (RGB) and thermal infrared (THR), respectively, using formulas (22)(23)(24)(25):

[0125]

[0126]

[0127] Among them, X RGB and X THR is the normalized or standardized input, and are the visible light (RGB) and thermal infrared (THR) features output by the teacher model at layer l, where l = 1,…,L represents the different layers of the network from shallow to deep;

[0128] After forward propagation layer by layer, the feature representation of the teacher model at layer l is obtained and processed by formula (26):

[0129]

[0130] Step 11: Define the student model. Based on the network model in steps 1-9, it is denoted as S. It is a lightweight network with fewer parameters. At each layer l, the teacher model outputs feature maps of two modalities, visible light (RGB) and thermal infrared (THR), respectively, using formulas (27)(28)(29)(30):

[0131]

[0132] Among them, X RGB and X THR is the normalized or standardized input, and are the visible light (RGB) and thermal infrared (THR) features output by the lth layer of the student model, where l = 1,…,L represents the different layers of the network from shallow to deep;

[0133] After forward propagation layer by layer, the feature representation of the teacher model at layer l is obtained and processed by formula (31):

[0134]

[0135] Step 12: Define Hierarchical Knowledge Distillation:

[0136] Step 12-1: Let N l is the total number of elements in the lth layer, defines the distillation loss of the DMI module, and applies formula (32):

[0137]

[0138] in, The distillation loss of the lth layer is used to measure the difference between the student features and the teacher features;

[0139] Step 12-2: Define the total DMI loss and apply formula (33):

[0140]

[0141] Among them, L KD is the sum of the distillation losses of all layers, α l is the weight coefficient;

[0142] Step 13: Define the decoder. In the decoder layer l, the encoder layer l output feature F l S With the previous layer decoder Output feature concatenation, apply formula (34) to process:

[0143]

[0144] After layer-by-layer fusion, the semantic segmentation prediction results of the student model are output at the end of the decoder;

[0145] Step 14: Forward propagation: transform X RGB and X THR Input the teacher model and student model at the same time, and obtain the teacher feature F at each layer l T and student characteristics F l S , and get the final prediction through the decoder;

[0146] Step 15: Loss calculation, calculate the semantic segmentation loss L seg , calculate the loss of layered distillation Accumulate to get L KD , the sum of the two is the total loss L total ;

[0147] Step 16: Back propagation and update, with L totalBackpropagation for the target is performed, only the student model is updated, and the teacher model remains unchanged;

[0148] Step 17: Iterate the loop and repeat forward and backward propagation until the student model converges. After training, use the student model for inference to obtain lightweight and efficient multispectral (RGB-T) semantic segmentation.

[0149] Step 18: Use the temperature information in the thermal infrared image to detect whether there are abnormal temperature areas on the pipeline surface. If the temperature of certain areas in the thermal infrared image exceeds the set threshold, these areas may have problems such as heat leakage and insulation damage. These areas will trigger alarms to complete pipeline safety warnings.

[0150] Implementation 2:

[0151] The purpose of this embodiment is to provide a pipeline safety early warning system based on multispectral images, including:

[0152] The detection data storage module includes:

[0153] The data receiving unit collects and records a variety of data types, including numerical values, text, images, videos, infrared information, electromagnetic ultrasound, and eddy current detection signals, based on the output of various sensors and detection equipment. Specific recording parameters, such as timestamp, device identification, and data type, are set for each data type in the data receiving unit. Data records are updated in real time. By integrating all daily detection data, the data is stored in a central database and indexed by device ID. For the data collection process, data archiving is set up every morning to ensure the timeliness and integrity of the information and generate detailed monitoring data records.

[0154] The data processing unit performs data format conversion, quality control, and cleaning based on the original data records, and is applicable to data types including text, numerical, image, and video. In the data processing unit, based on the original monitoring records, the daily average and peak statistics of each sensor are calculated. For image and video data, the resolution and integrity are checked. For infrared and electromagnetic data, the signal strength and noise ratio are analyzed. The data is processed in a formula to evaluate its integrity. This is used to evaluate the integrity and accuracy of the data records. By performing quality control on the original data, quality-controlled data records are generated.

[0155] The data compression and optimization unit compresses and optimizes the format of data, including images, videos, and other large files, based on data storage requirements and access frequency. Within the unit, appropriate compression algorithms are selected, such as JPEG compression for images, H.264 for videos, and specific algorithms for compressing infrared and electromagnetic ultrasound data. The unit also optimizes data storage structures, reduces storage space requirements, and increases data access speed. By adjusting data storage solutions, storage costs are effectively reduced, ensuring fast data access and long-term preservation.

[0156] The Data Backup and Recovery Unit implements a multimedia data backup and recovery plan based on risk management strategies, including image, video, infrared, and electromagnetic detection data. Within the Data Backup and Recovery Unit, critical data is regularly backed up in full, with particular attention paid to backing up large volumes of video and image data. Backup data is stored in an off-site data center to prevent data loss from natural disasters or system failures. A strict backup schedule, including immediate backup of important changes and periodic full data backups, as well as disaster recovery drills, is implemented to ensure rapid data recovery and continuous system operation under all circumstances.

[0157] The detection data query module includes:

[0158] The index management unit is used to build and maintain an efficient data indexing system that supports rapid retrieval of various monitoring data, including numerical values, text, images, videos, infrared information, electromagnetic ultrasound, and eddy current detection signals. In the index management unit, a dedicated index structure is created for each data type. For image and video data, the index includes timestamp, device location, and key image features. For numerical and text data, the index includes timestamp, sensor type, and measurement value. These indexes support fast access to data and complex queries, ensuring query efficiency and response speed.

[0159] The query processing unit executes data queries based on user requests, supports complex search conditions such as time range, device ID, and data type, and can handle large-scale data sets. SQL query optimization technology is used within the query processing unit to process user query requests, providing a flexible query interface that allows users to customize queries based on specific needs, such as retrieving temperature anomalies within a specific time period or operating data for a specific device. This optimizes query execution plans, reduces query latency, and improves the efficiency and accuracy of data retrieval.

[0160] The data presentation unit displays query results to users in an intuitive form, supporting a variety of data visualization options such as charts, time series graphs, and heat maps. In the data presentation unit, user-friendly reports and graphs are generated based on the query results, including but not limited to using charts to display historical equipment operating data and trend analysis, and using heat maps to display equipment temperature distribution. In addition, interactive data exploration tools are provided, allowing users to deeply analyze specific data points or conduct further data drilling. Advanced graphics processing and presentation technologies ensure that users can easily understand complex data sets and assist in the decision-making process.

[0161] See also Figure 5 , the security knowledge graph module includes:

[0162] The data integration unit integrates multiple data sources, including historical safety records, operational data, and environmental monitoring data, to build a comprehensive knowledge base. Within the unit, all data is preprocessed to standardize formats and eliminate inconsistencies. Complex pipe network single-line diagrams are categorized into vectorized and rasterized drawings. Natural language processing is then performed on the text within the recognized drawings. This involves classifying the recognized text and identifying the pipelines it labels, thereby matching the text with the pipelines and ultimately generating a node-connection relationship diagram. Finally, ETL tools are used to merge this data into a central data warehouse to ensure data integrity and consistency.

[0163] The knowledge modeling unit is used to create and maintain a dynamically updated security knowledge graph using advanced modeling technology. The graph can reflect complex data relationships and dependencies. In the knowledge modeling unit, the entity, attribute, and relationship models of the graph are defined and implemented, and graph database technology is used to realize the physical storage of the model, supporting the continuous evolution of knowledge and the dynamic updating of relationships, so that the graph is always kept up to date and reflects the current situation of actual operations and security environment. Furthermore, the parametric automatic modeling of complex pipe fittings + structured meshing of complex pipe fittings + pipe system assembly and matching steps are realized. In the automated modeling program architecture, the program consists of a human-computer interaction main interface program and an Abaqus PDE secondary development module. In the main interface, the node editor, connection relationship editor, and pipe attribute editor are used to input the geometric parameters of the pipe fittings. When the main interface generates a work file, it will use multi-threaded mode to call five modules in the Abaqus Python development environment to realize work file interpretation, pipe fitting modeling, pipe fitting meshing, pipe system assembly, load, and boundary loading respectively.

[0164] The graph update and maintenance unit is responsible for regularly updating the knowledge graph to include new safety information and regulatory changes, ensuring the currency and accuracy of the graph content. This unit is responsible for monitoring changes in external and internal data sources, such as new safety regulations, accident data, or operational changes, automatically or manually triggering the update process, and using version control and change management techniques to maintain the history and integrity of the graph, ensuring that every update is accurately reflected in the graph.

[0165] The query and analysis unit provides powerful graph query and data analysis capabilities, supporting complex logical reasoning and pattern recognition to help users make better decisions. Within the query and analysis unit, we have developed and implemented a variety of graph-based query interfaces and analysis tools. Users can perform interactive queries through natural language or graphical interfaces. The system can process complex query requests and return real-time analysis results, such as risk assessment reports, operational optimization suggestions, and security warnings.

[0166] Optionally, the text classification network in data integration uses the LAC network (a joint lexical analysis model that can comprehensively complete Chinese word segmentation, part-of-speech tagging, and proper name recognition tasks) to perform part-of-speech classification and tagging. The required key information is selected from the text recognition and saved in the background database. In addition, the Hough transform algorithm can be used to identify straight line segments. The process of Hough line detection is as follows:

[0167] Step 1: In θ oρ Establish the corresponding parameter space and accumulator in , and clear all accumulators to zero;

[0168] Step 2: Perform Hough transform on each pixel with gray value 0 in the image, corresponding to θ oρ There are curves in the space, and 1 is added to the corresponding accumulator;

[0169] Step 3: Count the values of each accumulator. If the value is less than the threshold o, clear the value of the accumulator to zero, find the maximum value in the accumulator, and get the corresponding point (θ p ,ρ p ), represents a straight line in the xoy coordinate system, and (θ p ,ρ p ) into the formula, we can get the equation of the line

[0170] ρ=xcosθ+ysinθ,θ∈[0,π) (35)

[0171] By analyzing the largest inscribed circle near the pixel point, the line type and line width data are obtained;

[0172] The drone mapping module includes:

[0173] The surveying and mapping data acquisition unit is used to collect real-time video from a high-definition camera mounted on a drone. The inspection video stream is encoded and decoded in H.264 (a digital video compression format) and then transmitted to the drone hardware system (remote control or smart drone nest). It is then transmitted via the 5G network RTMP protocol to a streaming media server built on the open source architecture Nginx-rtmp-module. The streaming media server transcodes the video stream. The independently developed drone automatic inspection system is used to pull the video stream from the streaming media server based on the HTTP-FLV live broadcast protocol, enabling front-end playback of multiple remote video feeds.

[0174] The data processing and modeling unit processes the raw data collected by drones, including image stitching, point cloud generation, and 3D model construction. Advanced image processing software and point cloud processing techniques are used in the data processing and modeling unit to convert the collected raw images and scanned data into high-precision 3D models. These models can be used in a variety of applications, such as terrain analysis, building planning, and environmental monitoring. Data is also optimized to reduce file size and improve processing efficiency and usability.

[0175] The surveying and mapping data storage unit is used to securely store and effectively manage processed data, supporting long-term data preservation and rapid retrieval. An efficient data storage system is used to store large amounts of image and 3D data within the surveying and mapping data storage unit. An effective data indexing and classification system is established to ensure that data can be quickly retrieved and easily accessed, providing reliable data support for future analysis and decision-making.

[0176] The user interaction unit is used to provide an intuitive user interface, allowing users to easily access and operate surveying and mapping data, and supports data viewing, analysis, and export. In the user interaction unit, users can view 3D models and maps through simple operations, perform basic data analysis and processing, such as measuring distances and calculating areas, and support exporting data to other formats or systems, thereby enhancing data usability and interoperability.

[0177] See also Figure 6 , the drone inspection module includes:

[0178] The mission planning unit is responsible for generating and optimizing the flight path of the UAV to ensure that all key inspection points are covered. In the mission planning unit, an effective flight path is planned, taking into account risk areas, no-fly zones and environmental factors to ensure that the UAV can complete the inspection task safely and comprehensively. In order to improve the accuracy of the control algorithm data collection, the horizontal safety distance H between the UAV and the distribution network line is first set. a and vertical safety distance D a , apply formula (36) (37) to process:

[0179]

[0180] Among them, H a Indicates the horizontal distance between the center of the vehicle and the drone during autonomous driving, in meters; L indicates the longest cross arm length of the distribution network tower, in meters; n indicates the safe distance of the distribution network line, in meters; D a It represents the vertical safety distance between the UAV and the horizontal ground, in meters; H represents the height of the distribution network structure, in meters; θ represents the range angle between the autonomous UAV and the distribution network, in rad;

[0181] The data acquisition unit uses the drone's onboard sensors and camera systems to collect key data in real time, including videos, photos, temperature, and gas concentrations. The drone is equipped with a variety of sensors, such as optical and infrared cameras, to capture high-quality monitoring data in real time for subsequent analysis and reporting.

[0182] The real-time data transmission unit is responsible for transmitting the collected data back to the control center in real time via a high-speed, secure network connection. In the real-time data transmission unit, 4G / 5G communication technology and RTMP or other suitable protocols are used to ensure that video and other monitoring data can be transmitted quickly and continuously to the ground station, allowing for immediate data analysis and response.

[0183] The data analysis and reporting unit processes and analyzes the received data to generate detailed inspection reports, including anomaly detection and health assessment. Image recognition and data analysis software are used within the unit to automatically identify problem areas, such as equipment loss, leaks, or other potential risks, and generate easy-to-understand inspection reports, providing maintenance teams with accurate maintenance and repair guidance.

[0184] The pipeline semantic segmentation module includes:

[0185] The image acquisition unit is responsible for capturing images of the pipeline and its surrounding area using the drone's high-resolution camera. The drone is equipped with a high-resolution optical camera and an infrared camera, which can capture clear images under various lighting conditions, providing high-quality input data for subsequent image processing and analysis.

[0186] The image preprocessing unit is used to perform necessary preprocessing operations on the collected images, including denoising, contrast enhancement, and color correction. In the image preprocessing unit, various image processing algorithms are applied to optimize image quality and improve the accuracy and reliability of subsequent semantic segmentation.

[0187] The semantic segmentation algorithm unit uses image processing algorithms, such as a semantic segmentation model for multispectral (RGB-T) images based on hierarchical knowledge distillation, to accurately identify pipe shapes. Based on the characteristics of the pipe and the surrounding environment, the most appropriate semantic segmentation technique is selected to automatically extract the pipe shape from the image, providing a precise pipe outline for further analysis.

[0188] The data analysis and reporting unit analyzes semantic segmentation results, identifies possible defects such as cracks and corrosion, and generates detailed inspection reports. Further quantitative analysis of the identified pipeline data, such as measuring crack width or corrosion severity, is performed within the unit. Combined with geographic location information, a maintenance report is automatically generated that includes location, size, severity, and recommended repair measures.

[0189] Infrared video analysis module:

[0190] The infrared video acquisition unit is responsible for capturing thermal images of equipment and facilities using the drone's onboard infrared camera. Equipped with an advanced infrared camera system, the drone is capable of capturing high-resolution thermal images under various environmental conditions. These images can reveal the temperature distribution of the equipment, providing preliminary data for detecting potential problems.

[0191] The video pre-processing unit performs preliminary processing on the captured infrared video, including noise removal, contrast adjustment, and image stabilization. Within the video pre-processing unit, filtering and image enhancement algorithms are applied to improve thermal image quality, making temperature differences more apparent and facilitating subsequent detailed analysis and problem diagnosis.

[0192] The hotspot detection algorithm unit uses image processing technology to identify hotspot areas in the video, which may indicate equipment overheating or energy leakage. Convolutional neural networks are deployed in the hotspot detection algorithm unit to automatically identify and mark abnormal hotspots in the video. These algorithms can identify potential problem areas based on temperature distribution patterns in thermal images.

[0193] The temperature data analysis unit is used to perform temperature analysis on identified hotspots, calculate their temperature range and change trend, and evaluate the health status of related equipment. In the temperature data analysis unit, the temperature data of each hotspot is analyzed, and the temperature change trend is evaluated using time series analysis, which helps to determine the severity and urgency of the problem.

[0194] The steps involved in the apparatus of the above embodiment 2 correspond to those of the method embodiment 1. For the specific implementation method, please refer to the relevant description part of the embodiment 1.

[0195] Implementation three:

[0196] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the method of implementation example 1;

[0197] When it comes to actual deployment, the drone's processor core utilizes highly specialized hardware to meet the rigorous mission requirements. Application-specific integrated circuits (ASICs) deliver highly customized data processing capabilities with low power consumption. Digital signal processors (DSPs) are specifically chosen to accelerate image processing and signal analysis tasks. Digital signal processing devices (DSPDs), programmable logic devices (PLDs), and field-programmable gate arrays (FPGAs) are also well-suited to handling complex drone control logic and image processing algorithms due to their flexibility and high performance.

[0198] In addition, using a microcontroller unit (MCU) or microprocessor as the core processor can ensure the stable operation of the system and basic task management. High-end microprocessors can even support complex artificial intelligence algorithms, enabling drones to have the ability to autonomously learn and adapt to environmental changes.

[0199] This paper mainly solves the problem that traditional single-vision (RGB)-based image semantic segmentation methods often exhibit low robustness and accuracy when applied to pipelines in the face of complex lighting conditions, occlusions and diverse environmental changes. A semantic segmentation method for multispectral (RGB-T) images based on hierarchical knowledge distillation is designed for pipeline safety early warning. The model performs knowledge distillation at different network levels, fully explores and transfers the multi-level and multi-scale feature representation capabilities of the teacher model, optimizes the feature extraction and fusion mechanism of the student model, thereby maintaining high segmentation accuracy while achieving model efficiency and lightweight. In addition, combined with cross-modal feature alignment and fusion strategies, the complementarity and synergy of multispectral information are further improved, thereby enhancing the model's adaptability in complex environments.

Claims

1. A pipeline safety early warning method based on multispectral images, characterized in that: Here are the steps: Step 1: Obtain multispectral raw data, including visible light image I RGB and thermal infrared image I THR The data has two modalities. Multispectral data generation and alignment are performed on the collected data. Data enhancement technology is used to expand the original data set. Annotation tools are used to accurately annotate the aligned data pixel by pixel, generate annotation files suitable for semantic segmentation tasks, and construct a multispectral RGB-T semantic segmentation dataset. Step 2: After data preprocessing is completed, the semantic segmentation model adopts a two-stream structure to process RGB visible light images at the same time RGB and thermal infrared image I THR As input; Step 3: For the feature extraction process of each modality, a hierarchical structure is constructed from shallow to deep and from local to global. The original input resolution is maintained at the L0 layer to extract preliminary low-level features, while the spatial resolution is gradually reduced in the L1 to L4 layers. Through continuous convolutional layers, batch normalization and nonlinear activation operations, higher-level abstract features are further extracted to achieve feature extraction and spatial downsampling. Step 4: For each level L i The visible light RGB and thermal infrared THR features of each layer are extracted using a channel attention mechanism. First, global average pooling (GAP) is used to perform global statistics on the visible light RGB and thermal infrared THR features to extract information in the channel dimension to achieve feature aggregation. Then, a multi-layer perceptron (MLP) is used in combination with the ReLU activation function δ and the sigmoid activation function σ, and learnable weight matrices W1 and W2 are introduced to convert the aggregated statistical information into channel attention weights. Finally, the obtained attention weights are broadcast multiplied with the original features channel by channel to enhance the channel features. Step 5: Design a spatial attention module. First, cross-modally connect the features of the two modalities and achieve feature fusion through concatenation. Then, use convolution to generate a single-channel spatial attention map to capture salient spatial regions. Finally, combine the generated spatial attention map with the original features to enhance the spatial features of visible light RGB and thermal infrared THR features respectively. Step 6: Introduce an adaptive weighted fusion mechanism to dynamically balance the contributions of channel attention and spatial attention features through learnable scalar parameters α and β, and perform weighted processing on the features of the two modalities during the fusion process; Step 7: Using a self-gating mechanism, we first generate an adaptive gating map using a convolution operation to generate gating signals for the visible RGB and thermal infrared THR features. The generated gating signals are then used to selectively enhance the original features. Step 8: After the gating process is completed, the features of the two modalities are cross-modally integrated by concatenating the gated visible light RGB and thermal infrared THR features, fusing the concatenated features using a convolution operation, and then passing the fused features to the corresponding decoder layer using a skip connection strategy; Step 9: The decoder path gradually reconstructs the feature resolution from D4 to D0. In this process, the features of the previous decoding layer are first upsampled and transferred in combination with the skip connection from the encoder layer. Finally, at the end of the decoder, feature mapping is achieved through 1×1 convolution. The feature map restored to the input resolution is mapped to the target category space, thereby generating the final semantic segmentation prediction result. Step 10: Construct a teacher model T. The teacher model T uses the same network structure and parameter configuration as steps 2 to 9 above. After construction, the teacher model T outputs normalized or standardized visible light RGB feature maps and thermal infrared THR feature maps in each layer l (l = 1, ..., L), where L represents the total number of layers in the network. Step 11: Construct a student model S based on the basic network architecture of the teacher model T. The student model S maintains the same overall structure as the teacher model T. The student model S also generates feature maps for visible light RGB and thermal infrared THR data in each layer l (l = 1, ..., L). Step 12: Design a hierarchical knowledge distillation strategy, where N l Represents the total number of elements in the lth layer. First, the distillation loss is defined in each layer through the DMI module. Then the distillation loss of each layer is multiplied by the corresponding weight coefficient and accumulated to obtain the total hierarchical knowledge distillation loss L. KD ; Step 13: In the student model S, a decoder path is designed, where each decoder layer concatenates the output features from the encoder layer l with the features from the previous decoding layer and implements feature fusion. After multiple layers of fusion, the semantic segmentation prediction results of the student model are output at the end of the decoder. Step 14: During the forward propagation process, the pre-processed visible light RGB and thermal infrared THR images are simultaneously input into the teacher model T and the student model S. Each model extracts the corresponding feature representation at each layer and completes feature reconstruction through the decoder. Finally, the semantic segmentation prediction results of the teacher model and the student model are obtained respectively. Step 15: Calculate the semantic segmentation loss L during training seg and the layer-wise knowledge distillation loss L KD , where the semantic segmentation loss is used to evaluate the error between the prediction result and the true label, and the layered distillation loss is used to measure the difference in feature expression between the student model and the teacher model. The sum of the two constitutes the total loss L total ; Step 16: During the training phase, L total Backpropagation is performed for the target, only the student model S is updated, and the teacher model T remains unchanged; Step 17: Iterate the loop and repeat forward and backward propagation until the student model S converges. After training, use the student model S for inference to obtain multispectral RGB-T semantic segmentation. Step 18: Use the temperature information in the thermal infrared image to detect whether there are abnormal temperature areas on the pipeline surface. If the temperature of certain areas in the thermal infrared image exceeds the set threshold, the corresponding area will trigger an alarm to complete the pipeline safety warning.

2. A pipeline safety early warning system based on multispectral images, characterized in that: Contains the following modules: Inspection data storage module, used to store sensor data, infrared images and analysis results generated during pipeline inspection to support subsequent risk assessment and historical query; The inspection data query module uses indexing and classification mechanisms to enable rapid query of specific pipe section inspection records, abnormal data, and historical trends; The safety knowledge graph module is used to integrate regulations, test data, and inspection results to build a hazardous chemicals safety knowledge network that integrates multi-source heterogeneous data for risk prediction and decision support. The UAV mapping module uses a high-precision camera mounted on a UAV to achieve 3D modeling and precise positioning of pressure pipelines in complex terrain; The drone inspection module, based on automated path planning and real-time image transmission technology, can complete visible light and infrared dual-mode inspections of pressure pipelines. The pipeline semantic segmentation module uses a semantic segmentation algorithm for multispectral (RGB-T) images based on hierarchical knowledge distillation to analyze drone-captured images and accurately identify pipeline shapes and potential structural anomalies. The infrared video analysis module extracts the temporal and spatial characteristics of pipeline temperature changes and combines them with the SE-TCN algorithm to identify and predict high-risk points in the pipeline.

3. The pipeline safety early warning system based on multispectral images according to claim 2 is characterized in that: The detection data storage module includes the following units: The data receiving unit is used to receive raw data from site sensors and detection equipment in real time, including numerical values, text, images, videos, infrared information, electromagnetic ultrasound and eddy current detection signals. The raw data undergoes preliminary formatting processing; A data processing unit, used to perform necessary cleaning and conversion operations on the received data, remove noise and standardize the data format; Data compression and optimization unit, based on data compression technology and algorithms, reduces the storage space occupied by data while maintaining fast data recovery capabilities; The data backup and recovery unit is used to regularly back up critical data to ensure data integrity and consistency, while supporting rapid data recovery functions to cope with system failures.

4. The pipeline safety early warning system based on multispectral images according to claim 2 is characterized in that: The detection data query module includes the following units: Index management unit, used to create and maintain data indexes and optimize data retrieval structures to speed up data access during query processes; Query processing unit, used to process and respond to user query requests, perform complex query operations, and ensure query accuracy and efficiency; The data presentation unit is used to display the query results to the user through a graphical user interface, supporting multiple data visualization methods, including charts and images.

5. The pipeline safety early warning system based on multispectral images according to claim 2 is characterized in that: The security knowledge graph module includes the following units: A data integration unit for integrating multiple data sources, including historical safety records, operational data, and environmental monitoring data; The knowledge modeling unit uses modeling technology to create and maintain a dynamically updated security knowledge graph that can reflect complex data relationships and dependencies; A graph update and maintenance unit, which is used to regularly update the knowledge graph to include new safety information and regulatory changes, ensuring the currentness and accuracy of the graph content; The query and analysis unit is used to provide graph query and data analysis functions, support logical reasoning and pattern recognition, and help users make better decisions.

6. The pipeline safety early warning system based on multispectral images according to claim 2 is characterized in that: The UAV mapping module includes the following units: Surveying and mapping data acquisition unit, used for drones carrying multiple sensors to collect data in a designated area, including high-resolution photos, videos, and topographic measurement data; Data processing and modeling unit, used to process the collected data, including image stitching, terrain reconstruction and 3D model generation to provide detailed terrain information; User interaction unit: provides an interactive interface that allows users to access, query and operate surveying and mapping data, and supports customized views and data export.

7. The pipeline safety early warning system based on multispectral images according to claim 2 is characterized in that: The drone cruise module includes the following units: Mission planning unit, used to plan the drone's flight path and checkpoints based on predetermined inspection requirements and terrain conditions to ensure coverage of all areas; A data acquisition unit, used to collect critical equipment and facility status information while the drone performs visual and thermal imaging inspections during flight; Real-time data transmission unit, used to send collected data back to the control center in real time via a high-speed communication link, supporting remote monitoring and immediate decision-making; The data analysis and reporting unit is used to conduct in-depth analysis of the collected data, identify potential risks and anomalies, and generate detailed inspection reports for further analysis and review.

8. The pipeline safety early warning system based on multispectral images according to claim 2 is characterized in that: The pipeline semantic segmentation module includes the following units: Image processing unit, used to enhance and preprocess the collected pipeline images to improve image quality and provide clear input for semantic segmentation; Semantic segmentation unit, which is used to identify pipeline shapes and locate pipelines and possible defects by applying semantic segmentation algorithms of multispectral RGB-T images based on hierarchical knowledge distillation; The anomaly detection unit is used to analyze pipeline identification results, automatically detect any abnormalities or signs of damage, and promptly alert maintenance personnel to conduct inspections or repairs.

9. The pipeline safety early warning system based on multispectral images according to claim 2 is characterized in that: The infrared video analysis module includes the following units: Video processing unit, used to process infrared video in real time, extract temperature and motion information, and provide basic data for analysis; Feature analysis unit, used to analyze temperature changes and other related features, identify abnormal patterns, and use them for early fault prediction and diagnosis; The risk assessment unit is used to conduct risk assessment based on the analysis results, automatically generate early warning signals, and mark possible risk points on the system map.

10. A computer device, characterized in that: include: A processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor and the memory communicate via the bus. When the machine-readable instructions are executed by the processor, a pipeline safety early warning method based on multispectral images is executed.

Images