Pipeline oil leakage detection method and system based on three-mode physical-visual synergy

Abstract

The invention provides a pipeline oil leakage detection method and system based on three-mode physical-visual synergy, and relates to the technical field of pipeline safety detection. The method comprises the following steps: acquiring a visible light image, an infrared image and a laser echo feature sequence of a to-be-detected pipeline area, performing spatial pixel-level alignment and normalization processing, and inputting the processed image into a pipeline oil leakage detection model comprising a shared parameter backbone network, a one-dimensional convolution branch, a cross-modal interactive fusion module and a Transformer decoder; the backbone network extracts visible light and infrared image features, and the one-dimensional convolution branch extracts laser spectrum features; the cross-modal interaction fusion module fuses three-modal features through three-directional attention and constructs a feature pyramid; and the Transform decoder decodes the feature pyramid and outputs an oil leakage detection result. According to the invention, through introducing laser physical characteristics and visual modal deep fusion, material attribute discrimination and visual positioning cooperative detection of an oil leakage target are realized, and the accuracy and robustness of pipeline oil leakage detection in a complex environment are improved.

Description

Technical Field

[0001] This invention relates to a pipeline oil leak detection method and system based on three-modal physical-visual collaboration, belonging to the field of pipeline safety detection technology. Background Technology

[0002] Pipeline leak detection is a key technology for ensuring energy transmission safety and environmental safety. Traditional detection methods mostly rely on manual inspections or pressure sensor monitoring, which suffers from low efficiency, significant safety hazards, and difficulty in accurately locating even minute leaks.

[0003] Visual inspection technologies based on visible light and infrared thermal imaging offer a new approach to addressing the aforementioned problems. However, visible light images are susceptible to environmental influences such as changes in illumination and shadow occlusion, while infrared images are limited by complex surface thermal radiation backgrounds and subtle temperature differences. This results in insufficient detection accuracy for single visual modalities in scenarios involving "different objects with the same spectrum" (e.g., water accumulation or shadows being misidentified as oil stains) and "weak feature recognition" (e.g., early micro-leakage). Furthermore, existing dual-modal fusion methods are mostly shallow or static fusions, making it difficult to achieve deep interaction and adaptive fusion of cross-modal information, and exhibiting poor robustness under complex operating conditions.

[0004] Active laser detection technology can achieve component-level identification by analyzing the characteristic echoes of substances, providing a basis for physical property discrimination in visual inspection. However, when used alone, its detection range is limited and it lacks spatial geometric information. Therefore, how to deeply integrate the physical property identification capability of lasers with the spatial semantic representation capability of images, so that the detection model has both visual perception capability and substance identification capability, has become the key to improving the accuracy and robustness of pipeline oil leak detection.

[0005] In existing technologies, such as the pipeline oil leak detection method based on multimodal and GLCM disclosed in Chinese invention patent application CN119941606A, multispectral images and ordinary RGB images are acquired, and the texture features of the multispectral images are extracted using the Gray-Level Co-occurrence Matrix (GLCM). These are then simply superimposed and fused with the feature maps of the RGB images at the channel level before target detection. While this technical solution utilizes multi-source information to some extent, it relies solely on the visual features of visible light and infrared images, failing to distinguish interfering substances with visual characteristics similar to oil from the perspective of their essential material properties. This can easily lead to false detections under complex operating conditions. Furthermore, its feature fusion method is merely channel stitching, a static linear fusion that cannot achieve nonlinear interaction between physical and visual dimensions, making it difficult to effectively screen suspected areas when visual features are blurred. In addition, this technical solution is prone to modal bias during training, i.e., over-reliance on the strong signal features of a certain modality, failing to adaptively adjust the contribution weights of different modalities according to environmental changes, resulting in insufficient robustness of the detection system.

[0006] In summary, there is an urgent need for a pipeline oil leak detection method that can achieve deep coordination of visible light, infrared and laser three modes, has physical consistency constraints, and can dynamically adjust the mode weights according to the environment, so as to achieve accurate and robust detection of oil leak targets in complex industrial environments. Summary of the Invention

[0007] To address the problems in existing technologies such as shallow cross-modal interaction, imbalanced modal representation, low detection accuracy for weak feature targets, and inability to effectively distinguish foreign objects with the same spectrum (e.g., misjudgment of water accumulation), this invention proposes a pipeline oil leakage detection method and system based on trimodal physical-visual collaboration.

[0008] The technical solution of the present invention is as follows: On the one hand, the present invention provides a pipeline oil leak detection method based on three-modal physical-visual collaboration, the method comprising: Acquire three-modal data of the pipeline region to be inspected, including visible light images, infrared images, and laser echo feature sequences; The three-modal data of the pipeline region to be detected are input into a pre-constructed and trained pipeline oil leak detection model. The pipeline oil leak detection model includes a backbone network with shared parameters, a one-dimensional convolutional branch, a cross-modal interactive fusion module, and a Transformer decoder, wherein: A backbone network with shared parameters is used to extract multi-scale features from visible light and infrared images; a one-dimensional convolutional branch is used to encode the laser echo feature sequence and output the laser spectral feature vector; the cross-modal interactive fusion module fuses the features extracted from the three modal data through a three-way attention mechanism and constructs a cross-modal feature pyramid based on the fused features; The Transformer decoder is used to decode the cross-modal feature pyramid to obtain the oil leak detection results.

[0009] Preferably, the three-modal data are spatially aligned at the pixel level, normalized, and then input into a pre-built and trained pipeline oil leak detection model, including: Scale the visible light and infrared images to a preset size; By projecting the spatial coordinates of the laser detection points corresponding to the laser echo feature sequence onto the image coordinate system using a pre-calibrated extrinsic parameter matrix, the pixel coordinates of each laser detection point on the image are obtained. The RGB channel values ​​of a visible light image are transformed to the [0,1] range using a linear mapping. The original temperature value or radiation intensity value of each pixel in the infrared image is transformed to the [0,1] interval through linear mapping; The laser echo feature sequence is normalized to obtain the normalized laser echo sequence.

[0010] Preferably, the method further includes a three-stage strategy in the training phase of the pipeline oil leak detection model, namely, modal isolation training, material property alignment training, and multimodal fusion training, wherein: In the modal isolation training phase, visible light image datasets, infrared image datasets, and laser echo feature sequence datasets are used respectively to train the backbone network, one-dimensional convolutional branch, and Transformer decoder with shared parameters separately. During training, the weighted sum of classification loss and location regression loss is used as the loss function. In the material property alignment training stage, pairs of visible light images, infrared images, and corresponding laser spectral feature vectors are used as training data. The parameters of the one-dimensional convolution branch are fixed. The visible light high-level features and infrared high-level features extracted by the backbone network with shared parameters are respectively compared with the laser spectral feature vector to calculate the cosine similarity loss. The parameters of the backbone network with shared parameters are optimized by the cosine similarity loss. In the multimodal fusion training phase, paired visible light images, infrared images, and laser echo feature sequences are used to train the complete pipeline oil leak detection model, including the cross-modal interactive fusion module, end-to-end. A composite loss function is used for parameter optimization. The composite loss function includes classification loss, position regression loss, spectral physical consistency loss, and cross-modal distributed alignment loss. The classification loss is the focus loss, the position regression loss is a linear combination of L1 loss and generalized intersection-union loss, the spectral physical consistency loss is the cosine similarity between the image features and the laser spectral features within the detection box, and the cross-modal distributed alignment loss is the KL divergence between the distributions of visible light, infrared, and laser modal features.

[0011] Preferably, the cross-modal interaction fusion module fuses features extracted from the three modal data through a three-way attention mechanism, including: The highest-level visible light feature map and infrared feature map extracted from the multi-scale features of the backbone network are flattened along the spatial dimension and converted into visible light sequence and infrared sequence, respectively; at the same time, the laser spectral feature vector output by the one-dimensional convolution branch is mapped into a laser physical sequence of the same dimension through linear projection. Using the laser physics sequence as the query, the laser physics sequence, visible light sequence, and infrared sequence are transformed through learnable linear transformations to generate query vectors, key vectors, and value vectors, respectively. Specifically: When calculating the visible-infrared attention, the visible light query that incorporates laser physical sequence information is scaled and dot-product attention with the infrared key and infrared value to obtain the visible-infrared attention output; when calculating the infrared-visible attention, the infrared query that incorporates laser physical sequence information is scaled and dot-product attention with the visible light key and visible light value to obtain the infrared-visible attention output. The visible-infrared attention output and the infrared-visible attention output are added element-wise and then added element-wise to the laser physical sequence. The fused multi-scale features are obtained through the reshaping operation.

[0012] Preferably, the construction of a cross-modal feature pyramid based on the fused features is as follows: Starting with the highest-level feature in the fused multi-scale features, a top-down feature fusion is performed, including: The fused multi-scale features are upsampled to obtain upsampled features; the corresponding mid-level visible light feature map and infrared feature map extracted by the backbone network are added element-wise to obtain mid-level visual fusion features; the upsampled features and mid-level visual fusion features are fused using learnable weighted fusion, followed by convolution and batch normalization to obtain the current level weighted fusion features; the upsampling, weighted fusion and convolution normalization operations are repeated, and the features are fused sequentially with the lower-level visible light feature map and infrared feature map extracted by the backbone network until the lowest level is processed to obtain a top-down weighted fusion feature set; The lowest-level weighted fusion features obtained in the top-down path are downsampled to obtain downsampled features. The downsampled features are then fused with the corresponding top-down weighted fusion features using a learnable weighted fusion method. After fusion, the features are processed by convolution and batch normalization to obtain the current layer features of the bottom-up path. The current layer features are then downsampled again and fused with the top-down weighted fusion features of the previous layer using a learnable weighted fusion method. The downsampling and weighted fusion operations are repeated until all feature layers have been processed to obtain the bottom-up cross-modal feature pyramid.

[0013] Preferably, in the learnable weighted fusion, the weight parameters used for fusion are learnable parameters optimized through gradient backpropagation during the training of the pipeline oil leak detection model; during the inference of the pipeline oil leak detection model, the weight parameters and the features of the three-modal data are used to calculate the fusion ratio of each modality feature, and the fusion ratio is dynamically determined based on the average brightness of the visible light image, the gradient amplitude of the infrared image, and the response intensity of the laser spectral feature vector in the three-modal data.

[0014] Preferably, the Transformer decoder consists of multiple stacked decoding layers, which are... The specific process of decoding the cross-modal feature pyramid using a learnable object query vector as input and a Transformer decoder is as follows: For each decoding layer, Each object query vector is input into a multi-head self-attention module, which calculates... Self-attention among query vectors, output the updated... One query vector; The updated Each query vector is used as a query. The feature sequence formed by flattening and splicing the feature maps of all levels in the cross-modal feature pyramid is used as the key and value. It is input into the multi-head cross-attention module to calculate the attention weight between each query and all positional features in the feature sequence. The feature sequence is weighted and summed according to the attention weight to obtain the aggregated feature vector corresponding to each query. The aggregated feature vector is residually joined with the query vector input to the cross-attention module and then normalized to obtain the output of the decoding layer. One query vector; go through After stacking the decoding layers, the first layer will be processed. Layer output Each query vector is input into a linear classification layer and a linear regression layer, respectively. The linear classification layer outputs the target category probability for each query, and the linear regression layer outputs the target bounding box coordinates and the substance determination confidence for each query.

[0015] On the other hand, the present invention also provides a pipeline oil leak detection system based on trimodal physical-visual collaboration, the system being used to execute the pipeline oil leak detection method based on trimodal physical-visual collaboration described in any embodiment of the present invention, the system comprising: The three-modal active sensing hardware module is used to simultaneously acquire visible light images, infrared images, and laser echo feature sequences of the pipeline area to be detected. The multi-source data preprocessing engine, connected to the tri-modal active sensing hardware module, is used to perform spatial pixel-level alignment and normalization processing on the received visible light image, infrared image and laser echo feature sequence, and output the aligned and normalized tri-modal data. The three-dimensional fusion oil leak detection core is connected to the multi-source data preprocessing engine. It contains a pipeline oil leak detection model, which includes a shared parameter backbone network, a one-dimensional convolutional branch, a cross-modal interactive fusion module, a cross-modal feature pyramid construction module, and a Transformer decoder. It is used to output oil leak detection results based on the input aligned and normalized three-modal data. The dynamic feedback and early warning module is connected to the three-dimensional fusion oil leak detection core and is used to trigger an early warning signal and display the fusion image based on the oil leak detection result.

[0016] Preferably, the pipeline oil leak detection model outputs the oil leak detection result based on the input aligned and normalized three-modal data as follows: The visible light and infrared images are extracted in parallel by a backbone network with shared parameters, and multiple levels of visible light feature maps and infrared feature maps are output. The normalized laser echo sequence is encoded by a one-dimensional convolutional branch, and the laser spectral feature vector is output. The cross-modal interactive fusion module, which connects the shared parameter backbone network and the one-dimensional convolutional branch respectively, receives the highest-level visible light feature map and infrared feature map output by the shared parameter backbone network, as well as the laser spectral feature vector output by the one-dimensional convolutional branch. The visible light feature, infrared feature and laser spectral feature are fused through a three-way attention mechanism to output the fused multi-scale feature. The cross-modal feature pyramid construction module, which connects the shared parameter backbone network and the cross-modal interactive fusion module respectively, receives the visible light feature maps and infrared feature maps of each level output by the shared parameter backbone network, as well as the fused high-level features output by the cross-modal interactive fusion module. It constructs and outputs the cross-modal feature pyramid through a top-down learnable weighted fusion path and a bottom-up learnable weighted fusion path. The Transformer decoder connected to the cross-modal feature pyramid construction module receives the cross-modal feature pyramid and processes it through a multi-layer decoding layer stack. Each decoding layer sequentially performs multi-head self-attention calculation, multi-head cross-attention calculation with the cross-modal feature pyramid as the key, residual connection and layer normalization. The linear classification layer and linear regression layer output the predicted class probability, bounding box coordinates and substance determination confidence of each target. By connecting the dynamic feedback and early warning module of the three-dimensional fusion oil leak detection core, an early warning signal is triggered based on the comparison result of the substance determination confidence level and the preset threshold, and the fused pseudo-color enhanced image is displayed.

[0017] Preferably, the system further includes a model training module, which is connected to the three-dimensional fusion oil leak detection core and is used to train the pipeline oil leak detection model, specifically including: The modal isolation training unit is configured to train the backbone network, one-dimensional convolutional branch and Transformer decoder with shared parameters separately using a single modality of visible light image dataset, infrared image dataset and laser echo feature sequence dataset. During training, a weighted sum of classification loss and location regression loss is used. The material property alignment training unit is configured to use paired visible light images and infrared images and corresponding laser spectral feature vectors as training data. The parameters of the one-dimensional convolution branch are fixed, and the visual features extracted by the backbone network with shared parameters are constrained to approach the laser spectral features by cosine similarity loss, thereby optimizing the backbone network parameters. The multimodal fusion training unit is configured to use pairs of visible light images, infrared images, and laser echo feature sequences to perform end-to-end training on a complete pipeline oil leak detection model, including a cross-modal interactive fusion module. A composite loss function is used to optimize all parameters, including classification loss, location regression loss, spectral physical consistency loss, and cross-modal distributed alignment loss.

[0018] The present invention has the following beneficial effects: 1. This invention provides a pipeline oil leak detection method and system based on three-modal physical-visual collaboration. By introducing laser echo feature sequences as third-modal data and setting a one-dimensional convolutional branch to encode them, the pipeline oil leak detection model can acquire the spectral physical characteristics of oil substances. Through the three-way attention mechanism in the cross-modal interactive fusion module, laser spectral features are used as queries to perform hard mask constraints on visible light and infrared image features. This enables the model to filter suspected areas in the image based on the physical properties of the substances during the feature fusion stage, reducing false detections caused by visual interference such as water accumulation, shadows, and paint patches, and improving the accuracy of the detection results. 2. This invention provides a pipeline oil leak detection method and system based on trimodal physical-visual collaboration. It employs a three-stage strategy during model training: modal isolation training, material attribute alignment training, and multimodal fusion training. In the modal isolation training stage, each branch network is trained using single-modal data, allowing each branch to independently extract the underlying features of its corresponding modality. In the material attribute alignment training stage, laser spectral features are used as a benchmark, and visible light and infrared features are constrained to align with them using cosine similarity loss, establishing a correlation between visual features and physical attributes. In the multimodal fusion training stage, paired trimodal data is used to perform end-to-end optimization of the complete model. This three-stage training strategy ensures that the model possesses robust single-modal feature extraction capabilities and a solid physical-visual correlation foundation before fusing multimodal information, solving the problems of feature competition and modal bias in multimodal data training and improving the model's robustness in complex environments. 3. This invention provides a pipeline oil leak detection method and system based on trimodal physical-visual collaboration. Through top-down and bottom-up learnable weighted fusion paths in the cross-modal feature pyramid construction module, weighted fusion parameters that can be dynamically adjusted according to ambient light, infrared temperature difference, and laser response intensity are introduced during the feature pyramid construction process. Furthermore, the top-down path transmits high-level semantic information with laser physical constraints to low-level detailed features, while the bottom-up path compensates for the semantic loss of low-level features through dilated convolution downsampling. This dynamic weighted pyramid structure enables the model to adaptively adjust the fusion ratio of trimodal features under different environmental conditions, improving the model's detection capability for multi-scale targets such as small leaks and large-area oil spills, as well as its detection stability under different lighting, climate, and detection distances. 4. This invention provides a pipeline oil leak detection method and system based on trimodal physical-visual collaboration. Through the multi-layer decoding structure in the Transformer decoder, with a learnable object query vector as input, it sequentially passes through multi-head self-attention calculation and multi-head cross-attention calculation with cross-modal feature pyramid as key value, enabling each query vector to interact with features of different levels and modalities during the decoding process. It also utilizes the laser material attribute information contained in the high-level features to gradually refine the target prediction. Finally, through the linear classification layer and regression layer, it outputs the target category, bounding box coordinates and material determination confidence, realizing end-to-end integrated output of target detection and physical attribute discrimination. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method corresponding to an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It should be understood that the step numbers used in the text are for ease of description only and are not intended to limit the order in which the steps are performed.

[0022] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0023] The terms “comprising” and “including” indicate the presence of the described feature, whole, step, operation, element and / or component, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0024] The term “and / or” refers to any combination of one or more of the associated listed items, as well as all possible combinations, and includes these combinations.

[0025] Example 1: This embodiment provides a pipeline oil leak detection method based on three-modal physical-visual collaboration, such as... Figure 1 As shown, the method includes: S1. Using a robotic dog, drone, or fixed monitoring device equipped with a visible light camera, infrared thermal imager, and laser active detector, synchronously perceive the area of ​​the pipeline to be inspected and acquire trimodal data, which includes visible light images, infrared images, and laser echo feature sequences, wherein: The visible light image is acquired by a visible light camera and output as an RGB three-channel image with the original acquisition resolution (e.g., 1920×1080 pixels). The infrared image is acquired by an infrared thermal imager and output as a single-channel image, with each pixel value being either the original temperature value or the radiation intensity value. The laser echo feature sequence is obtained by emitting laser light of a specific wavelength (e.g., 1064nm or 1550nm) into the target area using an active laser detector and receiving the returned spectral energy distribution. After analog-to-digital conversion, a one-dimensional laser echo feature sequence is obtained, with the sequence length determined by the number of sampling points (1024 sampling points in this embodiment). Furthermore, to eliminate the significant differences in imaging principles and data dimensions among different sensors, the raw trimodal data is preprocessed, including spatial and scale alignment, image normalization, and dimensional normalization, among which: The spatial and scale alignment is used to ensure that the images input to the network have a uniform dimension. This includes scaling visible light and infrared images to 1024×1024 pixels and using a pre-calibrated extrinsic parameter matrix (i.e., the rotation and translation matrix between the camera coordinate system and the lidar coordinate system) to adjust the spatial coordinates of each laser detection point corresponding to the laser echo feature sequence. Projecting onto the image coordinate system yields the corresponding pixel coordinates. For multiple laser detection points projected onto the same pixel area, the average value of their spectral characteristics is taken as the representative value of the pixel position. Through the spatial and scale alignment operation, each laser detection point establishes a correspondence with a specific pixel on the image, achieving spatial pixel-level alignment. The image normalization includes visible light image normalization and infrared image normalization. Specifically, visible light image normalization involves dividing the pixel value of each channel of the RGB image by 255 to map the pixel value range to the [0,1] interval. Infrared image normalization involves scaling the original temperature or radiation intensity value of the infrared image to the [0,1] interval through linear mapping to achieve alignment of the magnitudes of thermal field information and reflection intensity information. The dimensional normalization specifically involves subtracting the mean of the laser echo characteristic sequence from the mean and dividing by the standard deviation to obtain the normalized laser echo sequence. After the above preprocessing, aligned and normalized three-modal data are obtained, including normalized visible light image, normalized infrared image, and normalized laser spectral feature vector.

[0026] S2. Input the aligned and normalized three-modal data into the pre-built and trained pipeline oil leak detection model. The pipeline oil leak detection model includes a backbone network with shared parameters, a one-dimensional convolutional branch, a cross-modal interactive fusion module, and a Transformer decoder, wherein: S21. The backbone network with shared parameters is a weight-sharing convolutional neural network (e.g., ResNet or SwinTransformer). In this embodiment, ResNet50 is selected as the backbone network to independently process visible light images and infrared images, i.e., the two inputs share the same set of convolutional kernel parameters. For visible light images (3-channel input) and infrared images (1-channel input), the network input dimension is adapted by copying the infrared image three times in the input layer. The backbone network outputs visible light feature maps and infrared feature maps corresponding to three different feature levels. In this embodiment, the feature level is denoted as... These correspond to feature maps downsampled by 8, 16, and 32 times respectively in the backbone network, and the feature maps are denoted as... and ,in, To obtain images from visible light The first one extracted from the backbone network with shared parameters Multi-scale characteristics of the layer; To obtain from infrared images The first one extracted from the backbone network with shared parameters The multi-scale characteristics of the layers, further, This represents a bottom-up feature hierarchy; for example, in a typical architecture... The corresponding feature map is downsampled by 8 times, resulting in rich spatial details; The feature map is downsampled by 32 times, resulting in stronger semantic information. Simultaneously, the one-dimensional convolutional branch is used to encode the physical features of the normalized laser echo sequence, outputting a laser spectral feature vector. Specifically, the physical feature encoding is as follows: The laser echo feature sequence (a one-dimensional vector of length 1024) is input into a one-dimensional convolutional encoder and mapped to a fixed-length laser spectral feature vector. In this embodiment, the one-dimensional convolutional encoder consists of 3 convolutional layers (with kernel sizes of 7, 5, and 3 and a stride of 2) and 2 fully connected layers, and outputs a fixed-length 128-dimensional laser spectral feature vector. The laser spectral feature vector is normalized based on a preset oil spectral feature template to obtain the normalized laser spectral feature vector. For ease of understanding, an example of the preset oil spectral feature template is provided below: Assuming the test is targeting crude oil, a standard spectral response curve of the crude oil in the near-infrared band (900nm-1700nm) is obtained beforehand using a laboratory spectrometer. This standard spectral response curve is then sampled at 128 equally spaced wavelength points to obtain a 128-dimensional template vector. The template vector has a high dimension value corresponding to the main absorption peak near 1200nm (corresponding to the second harmonic frequency of the CH bond). The second absorption peak near 1400 nm corresponds to the second highest dimensionality (e.g., ), while the corresponding dimension value is lower in the non-absorption band (e.g. In practical processing, the 128-dimensional original vector output by the one-dimensional convolutional encoder is... With template vector Calculate the cosine similarity, and based on the cosine similarity pair Normalization is performed, expressed by the formula: ; In the formula, For normalized ; Cosine similarity; To prevent zero values ​​for small constants (such as 0.01); Through the above processing, the template matching degree is high ( Vectors that are close to 1 are enhanced, while those that differ greatly from the template are enhanced. Interference signals close to 0 are suppressed, resulting in the normalized laser spectral feature vector. ; S22. The cross-modal interactive fusion module achieves deep fusion of visible light, infrared, and laser modalities through a three-way collaborative attention mechanism and constructs a cross-modal feature pyramid. Through three-way deep semantic interaction, it utilizes the material determination capability of the laser spectrum to solve the problem of false alarms or missed alarms in visual modalities under complex lighting, shadow occlusion, or slight temperature differences. Furthermore, the cross-modal interactive fusion module includes a three-way collaborative feature interaction unit and a cross-modal feature pyramid construction unit, wherein: S221, the three-way collaborative feature interaction unit is used to perform deep collaborative interaction between cross-modal semantic information and physical attributes of the highest-level features, specifically: S2211. The highest level visible light feature extracted from the multi-scale features of the backbone network. infrared features Each through flattening operation Convert to sequence vectors to obtain visible light sequences and infrared sequence The flattening operation gives the model a global receptive field, enabling it to capture long-range spatial dependencies in the context of distant pipelines. Simultaneously, it integrates the laser spectral feature vectors... Through linear projection operation Mapped to the same dimension of laser physical sequence This enables the model to capture long-range spatial dependencies and material property constraints within the context of pipelines, expressed as a formula: ; ; ; S2212. Calculate the interrelationship between two modal features using a three-way attention mechanism, including: For each modality sequence, a learnable linear transformation matrix is ​​used. , and Generate its query, key, and value vector representations respectively, expressed as a formula: , , ; , , ; , , ; In the formula, , and These are query representations for visible light modes, infrared modes, and laser modes, respectively. , and These are the bond representations for the visible light mode, infrared mode, and laser mode, respectively. , and These are value vector representations of the visible light mode, infrared mode, and laser mode, respectively. To incorporate laser physical information as a query benchmark, laser query Adding these values ​​to the visible light query and the infrared query respectively yields the fused query, expressed as a formula: ; ; In the formula, and These are the fused visible light query and infrared query, respectively. and It is a learnable scalar parameter, initially set to 0.1, and optimized through backpropagation during training; Cross-modal cross attention is calculated. When the visible light-infrared attention output is obtained, the visible light query and infrared key and infrared value that are fused with laser physical sequence information are scaled dot product attention. This process uses the infrared thermal radiation distribution characteristics to perform semantic completion of the weak visible light texture and uses the visible light edge information to perform spatial constraint on the infrared thermal halo region. When calculating the infrared-visible light attention output, the infrared query and visible light key and visible light value that integrate laser physical sequence information are used to perform scaled dot product attention. This process uses the radiative saliency of the infrared mode to perform semantic calibration on suspected shadow interference in the visible light. The above calculation is expressed by the following formula: ; ; In the formula, Visible light-infrared attention output; Infrared-visible light attention output; Scaling factor The dimension of the key vector; This is a transpose operation; The visible-infrared attention output and the infrared-visible attention output are added element-wise, and then added element-wise to the laser physical sequence. The resulting multi-scale features are obtained through a reshaping operation, expressed as follows: ; In the formula, The resulting multi-scale features; To restore the spatial dimension reshaping operation; This is for layer normalization processing.

[0027] S222. The cross-modal scale feature fusion unit starts with the highest-level feature in the fused multi-scale features obtained in step S221, and constructs a cross-modal feature pyramid, which includes two complementary paths: top-down and bottom-up. By dynamically adjusting the feature weights of different levels and modes, and introducing the confidence level of the material properties of the laser spectrum as a physical reference for weight adjustment, the model's ability to capture multi-scale oil spill targets in complex backgrounds is improved. Specifically: S2221. First, perform top-down feature fusion, fusing features from high-level to low-level features step by step: In this embodiment, after upsampling the fused multi-scale features, the corresponding mid-layer visible light feature map extracted by the backbone network is added element by element to the infrared feature map to obtain the mid-layer visual fusion feature. The upsampled feature and the mid-layer visual fusion feature are then subjected to learnable weighted fusion and convolution and batch normalization to obtain the weighted fusion feature of the current level. Through this path, the oil leakage judgment semantic information with laser physical constraints in the high layer is transmitted to the low-layer features with fine texture. Repeat the bilinear upsampling, weighted fusion, and convolutional normalization operations, and then fuse them sequentially with the lower-level visible light feature maps and infrared feature maps extracted from the backbone network until the lowest layer is processed, obtaining a top-down weighted fused feature set. The above top-down feature fusion process is expressed by the following formula: ; ; In the formula, and They are respectively Layer weighted fusion features and The weighted fusion characteristics of the layers; for Convolution normalization operator; This is an upsampling operation; These are learnable weights; To compensate for the semantic loss that may result from upsampling low-level features in the top-down path and to further enhance the expressive power of high-level features, a bottom-up supplementary fusion is performed, specifically: The lowest-level weighted fusion feature obtained from the top-down path (which is the lowest-level weighted fusion feature in this embodiment) After performing dilated convolution downsampling with a stride of 2, it is fused with the corresponding level's cross-modal weighted fusion features using learnable weighted fusion. The fused features are then downsampled and fused with higher-level cross-modal weighted fusion features using learnable weighted fusion, and then processed through a fusion block. (Including convolution, normalization, and non-linear activation functions) processing, until all feature levels of the weighted fused features have completed downsampling, learnable weighted fusion operations, and fusion blocks, obtaining a bottom-up cross-modal feature pyramid. For ease of understanding, this path can be represented as: ; ; In the formula, and These are the feature representations obtained during the bottom-up process; For downsampling convolution; and These are learnable weights; The cross-modal interactive fusion module outputs a feature set consisting of fused features from three different scales. , serving as a feature pyramid that encodes rich cross-modal information; Preferably, in the learnable weighted fusion, the weight parameters used for fusion are... , , and It is not a fixed value; the fusion ratio is calculated in real time based on the input data by a dynamic weight generation network. Specifically, the inputs of the dynamic weight generation network include the average brightness of the visible light image (calculating the global average brightness of the visible light image), the infrared gradient magnitude (calculating the Sobel gradient map of the infrared image and taking the average), and the laser response intensity (calculating...). and The cosine similarity is calculated by concatenating the corresponding scalar values ​​of the input data and inputting them into an MLP. The MLP contains a hidden layer (3D input, 16D output, ReLU activation) and an output layer (16D input, 4D output, Sigmoid activation). Finally, it outputs four weight values ​​in the interval [0,1], which correspond to the weight parameters of the four fusion operations. When the laser response is strong, and The corresponding increase makes higher-level features contribute more; when visual conditions are good... and Increase.

[0028] S23. Preferably, during the training phase of the pipeline oil leak detection model, to avoid the risks of insufficient multimodal samples, modal bias, and unstable trimodal alignment, a three-stage strategy of modal isolation training, material property alignment training, and multimodal fusion training is adopted, wherein: The first stage is modal isolation training. In this stage, each branch network is trained using a single modal dataset, and the cross-modal interaction fusion module is bypassed (i.e., the output is directly set to zero or skipped). In this embodiment: During training of the visible light branch, the input consists only of the visible light image and its corresponding oil leak annotation box. The parameters of the shared backbone network (which only updates the visible light processing part) and the Transformer decoder are optimized. The loss function is a weighted sum of the classification loss and the location regression loss, where the classification loss is the focus loss and the location regression loss is a linear combination of the L1 loss and the GIoU loss (each with a weight of 0.5). The training lasts for 20 epochs. During infrared branch training, the input consists only of infrared images and their annotations. The shared backbone network (which only updates the part that processes infrared images) and decoder are optimized. The loss function is the same as that used during visible light branch training. The training lasts for 20 epochs. During laser branch training, the input consists only of the laser spectral feature vector and its annotation (each sample corresponds to an image region, labeled whether there is oil leakage). The one-dimensional convolutional branch and decoder are optimized. Since the laser is a global feature, the decoder only outputs the global classification of whether the target exists, without outputting the bounding box. The loss function only uses the classification loss. The training lasts for 20 epochs. Through the above-mentioned isolated training, each branch network can independently learn the feature extraction of the corresponding modality, avoiding the modality bias problem that may occur in the early stage of multimodal joint training.

[0029] The second stage is material property alignment training. In this embodiment, this stage uses paired visible light images, infrared images and corresponding laser spectral feature vectors as training data. The parameters of the one-dimensional convolution branch are fixed. The visible light high-level features and infrared high-level features extracted by the backbone network with shared parameters are respectively compared with the laser spectral feature vector to calculate the cosine similarity loss. The parameters of the backbone network with shared parameters are optimized with the cosine similarity loss and trained for 10 epochs. The third stage is multimodal fusion training. In this stage, paired visible light images, infrared images, and laser echo feature sequences are used to train the complete pipeline oil leak detection model, including the cross-modal interactive fusion module, end-to-end. A composite loss function is used for parameter optimization. The composite loss function includes classification loss, position regression loss, spectral physical consistency loss, and cross-modal distributed alignment loss. Among them, the classification loss is the focus loss, the position regression loss is a linear combination of L1 loss and generalized intersection-union loss, the spectral physical consistency loss is the cosine similarity between the image features and the laser spectral features within the detection box, and the cross-modal distributed alignment loss is the KL divergence between the distributions of visible light, infrared, and laser modal features. Preferably, in this embodiment, the balance coefficients of classification loss, position regression loss, spectral physical consistency loss, and cross-modal distributed alignment loss in this stage are set to 1.0, 1.0, 0.5, and 0.1, respectively. This stage is trained for 30 epochs with an initial learning rate of 0.0001, which decays by 0.1 every 10 epochs.

[0030] S24, The Transformer decoder receives the cross-modal feature pyramid obtained in step S222. As input, it is used to decode the cross-modal feature pyramid, wherein the Transformer decoder is composed of multiple stacked decoding layers, so as to... The specific process of decoding the cross-modal feature pyramid using a learnable object query vector as input and a Transformer decoder is as follows: For each decoding layer, Each object query vector is input into a multi-head self-attention module, which calculates... Self-attention among query vectors, output the updated... One query vector; The updated Each query vector is used as a query. The feature sequence formed by flattening and splicing the feature maps of all levels in the cross-modal feature pyramid is used as the key and value. It is input into the multi-head cross-attention module to calculate the attention weight between each query and all positional features in the feature sequence. The feature sequence is weighted and summed according to the attention weight to obtain the aggregated feature vector corresponding to each query. The aggregated feature vector is residually joined with the query vector input to the cross-attention module and then normalized to obtain the output of the decoding layer. One query vector; go through After stacking the decoding layers, the first layer will be processed. Layer output Each query vector is input into a linear classification layer and a linear regression layer, respectively. The linear classification layer outputs the target category probability for each query, and the linear regression layer outputs the target bounding box coordinates and the substance determination confidence for each query.

[0031] S3, based on the output of the Transformer decoder Each target is predicted, and the final oil leak detection result is obtained through post-processing (such as non-maximum suppression; in this embodiment, the cross-union threshold is set to 0.5, and the prediction box with the highest confidence is retained). This includes the category of the oil leak target, the bounding box coordinates, and the confidence level of the substance determination.

[0032] S4. Furthermore, to verify the effectiveness of the method described in this embodiment, a comparative experiment was conducted, specifically: This embodiment integrates a subset of pipeline scenes from the FLIR multispectral dataset as the dataset, containing 5000 aligned visible light and infrared image pairs. Each image is labeled with an oil leak area. In addition, this embodiment also uses a self-developed laser oil detection device to collect laser echo sequences corresponding to the above image pairs in actual pipeline scenes, thus forming 5000 sets of trimodal samples. In this embodiment, 4000 sets are used for model training, and 1000 sets are reserved as the test set.

[0033] In the comparative experiment, this embodiment set up two different model configurations. One is a baseline model based on existing methods, which only uses visible light and infrared dual-modal input, performs feature fusion through channel stitching, and is equipped with a conventional convolutional detection head. During training, it only uses the same 4000 sets of image pairs and does not introduce laser data. The other is the complete detection method proposed in this embodiment, which learns according to the three-stage training strategy in the method described in this embodiment, making full use of information from all modalities.

[0034] Experimental results show that the method described in this embodiment achieves significant advantages on the test set, with a mean accuracy (mAP@0.5) of 87.3%, an improvement of 9.5 percentage points compared to the baseline model's 77.8%. Particularly in small target detection, for oil leak areas smaller than 32×32 pixels, the method in this embodiment achieves a recall rate of 71.5%, an improvement of 14.2 percentage points compared to the baseline model's 57.3%. Furthermore, facing 200 special test samples in complex interference scenarios such as water accumulation and shadows, the baseline model had 47 false positives, while the method described in this embodiment only had 4 false positives, reducing the false positive rate by approximately 91.5%. When deployed on the NVIDIA Jetson AGXXavier hardware platform, the method described in this embodiment has an average inference time of only 52 milliseconds per inference, fully meeting real-time requirements.

[0035] To further verify the effectiveness of each module, this embodiment also conducted a systematic ablation experiment. When the laser branch was removed and only the bimodal input was retained (still using the three-stage training strategy), the mAP decreased to 81.2%, indicating that the physical characteristics of the laser made a significant contribution to improving the model's perception capability. If the dynamic weight fusion module was removed and a fixed weight fusion method was used instead, the mAP decreased to 83.5%, indicating that the adaptive weight mechanism can indeed optimize the fusion effect based on environmental characteristics. However, if the three-stage training was canceled and an end-to-end training method was used directly, the model exhibited significant loss oscillations in the early stages of training, with the final mAP at 84.1%, and the training time increased by 30%. These results collectively demonstrate that the three-stage training strategy proposed in this embodiment plays a crucial role in improving model stability and convergence efficiency. Overall, the trimodal collaborative detection method in this embodiment shows significant advantages in detection accuracy, anti-interference capability, and real-time inference performance.

[0036] Example 2: This embodiment provides a pipeline oil leak detection system based on a three-modal physical-visual collaborative approach. The system includes: The three-modal active sensing hardware module is used to simultaneously acquire visible light images, infrared images, and laser echo feature sequences of the pipeline area to be detected. The multi-source data preprocessing engine, connected to the tri-modal active sensing hardware module, is used to perform spatial pixel-level alignment and normalization processing on the received visible light image, infrared image and laser echo feature sequence, and output the aligned and normalized tri-modal data. The three-dimensional fusion oil leak detection core is connected to the multi-source data preprocessing engine. It contains a pipeline oil leak detection model, which includes a shared parameter backbone network, a one-dimensional convolutional branch, a cross-modal interactive fusion module, a cross-modal feature pyramid construction module, and a Transformer decoder. It is used to output oil leak detection results based on the input aligned and normalized three-modal data. The dynamic feedback and early warning module is connected to the three-dimensional fusion oil leak detection core and is used to trigger an early warning signal and display the fusion image based on the oil leak detection result.

[0037] Preferably, the pipeline oil leak detection model outputs the oil leak detection result based on the input aligned and normalized three-modal data as follows: The visible light and infrared images are extracted in parallel by a backbone network with shared parameters, and multiple levels of visible light feature maps and infrared feature maps are output. The normalized laser echo sequence is encoded by a one-dimensional convolutional branch, and the laser spectral feature vector is output. The cross-modal interactive fusion module, which connects the shared parameter backbone network and the one-dimensional convolutional branch respectively, receives the highest-level visible light feature map and infrared feature map output by the shared parameter backbone network, as well as the laser spectral feature vector output by the one-dimensional convolutional branch. The visible light feature, infrared feature and laser spectral feature are fused through a three-way attention mechanism to output the fused multi-scale feature. The cross-modal feature pyramid construction module, which connects the shared parameter backbone network and the cross-modal interactive fusion module respectively, receives the visible light feature maps and infrared feature maps of each level output by the shared parameter backbone network, as well as the fused high-level features output by the cross-modal interactive fusion module. It constructs and outputs the cross-modal feature pyramid through a top-down learnable weighted fusion path and a bottom-up learnable weighted fusion path. The Transformer decoder connected to the cross-modal feature pyramid construction module receives the cross-modal feature pyramid and processes it through a multi-layer decoding layer stack. Each decoding layer sequentially performs multi-head self-attention calculation, multi-head cross-attention calculation with the cross-modal feature pyramid as the key, residual connection and layer normalization. The linear classification layer and linear regression layer output the predicted class probability, bounding box coordinates and substance determination confidence of each target. By connecting the dynamic feedback and early warning module of the three-dimensional fusion oil leak detection core, an early warning signal is triggered based on the comparison result of the substance determination confidence level and the preset threshold, and the fused pseudo-color enhanced image is displayed.

[0038] Preferably, the system further includes a model training module, which is connected to the three-dimensional fusion oil leak detection core and is used to train the pipeline oil leak detection model, specifically including: The modal isolation training unit is configured to train the backbone network, one-dimensional convolutional branch and Transformer decoder with shared parameters separately using a single modality of visible light image dataset, infrared image dataset and laser echo feature sequence dataset. During training, a weighted sum of classification loss and location regression loss is used. The material property alignment training unit is configured to use paired visible light images and infrared images and corresponding laser spectral feature vectors as training data. The parameters of the one-dimensional convolution branch are fixed, and the visual features extracted by the backbone network with shared parameters are constrained to approach the laser spectral features by cosine similarity loss, thereby optimizing the backbone network parameters. The multimodal fusion training unit is configured to use pairs of visible light images, infrared images, and laser echo feature sequences to perform end-to-end training on a complete pipeline oil leak detection model, including a cross-modal interactive fusion module. A composite loss function is used to optimize all parameters, including classification loss, location regression loss, spectral physical consistency loss, and cross-modal distributed alignment loss.

[0039] It is worth noting that the system described in this invention and the method described in Embodiment 1 are based on the same inventive concept, and will not be repeated here.

[0040] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0041] Those skilled in the art will recognize that the units and algorithmic steps described in the embodiments disclosed herein (such as feature extraction via a shared parameter backbone network, bidirectional attention fusion, and cross-modal pyramid construction) can be implemented using electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0042] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the aforementioned method embodiment one, and will not be repeated here.

[0043] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the part of the technical solution of this application that contributes to the prior art (i.e., the cross-modal cooperative mechanism for solving modal bias and weak feature submersion) can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, inspection robot control terminal, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0044] The above description is merely an embodiment of the present invention and does not limit the scope of the patent. Those skilled in the art should understand that equivalent transformations made using the bidirectional semantic interaction and multi-scale adaptive fusion concept described in this invention, or direct / indirect applications in other related technical fields such as oil and gas monitoring and power line inspection, for specific interference scenarios such as lighting, smoke, and heat halo in pipeline inspection, are similarly included within the scope of patent protection of this invention.

Claims

1. A pipeline oil leak detection method based on three-modal physical-visual collaboration, characterized in that, The method includes: Acquire three-modal data of the pipeline region to be inspected, including visible light images, infrared images, and laser echo feature sequences; The three-modal data of the pipeline region to be detected are input into a pre-constructed and trained pipeline oil leak detection model. The pipeline oil leak detection model includes a backbone network with shared parameters, a one-dimensional convolutional branch, a cross-modal interactive fusion module, and a Transformer decoder, wherein: A backbone network with shared parameters is used to extract multi-scale features from visible light and infrared images; a one-dimensional convolutional branch is used to encode the laser echo feature sequence and output the laser spectral feature vector; the cross-modal interactive fusion module fuses the features extracted from the three modal data through a three-way attention mechanism and constructs a cross-modal feature pyramid based on the fused features; The Transformer decoder is used to decode the cross-modal feature pyramid to obtain the oil leak detection results.

2. The pipeline oil leak detection method based on three-modal physical-visual collaboration according to claim 1, characterized in that, After spatial pixel-level alignment and normalization of the three-modal data, it is input into a pre-built and trained pipeline oil leak detection model, including: Scale the visible light and infrared images to a preset size; By projecting the spatial coordinates of the laser detection points corresponding to the laser echo feature sequence onto the image coordinate system using a pre-calibrated extrinsic parameter matrix, the pixel coordinates of each laser detection point on the image are obtained. The RGB channel values ​​of a visible light image are transformed to the [0,1] range using a linear mapping. The original temperature value or radiation intensity value of each pixel in the infrared image is transformed to the [0,1] interval through linear mapping; The laser echo feature sequence is normalized to obtain the normalized laser echo sequence.

3. The pipeline oil leak detection method based on three-modal physical-visual collaboration according to claim 1, characterized in that, The method also includes a three-stage strategy in the training phase of the pipeline oil leak detection model, employing modal isolation training, material property alignment training, and multimodal fusion training, wherein: In the modal isolation training phase, visible light image datasets, infrared image datasets, and laser echo feature sequence datasets are used respectively to train the backbone network, one-dimensional convolutional branch, and Transformer decoder with shared parameters separately. During training, the weighted sum of classification loss and location regression loss is used as the loss function. In the material property alignment training stage, pairs of visible light images, infrared images, and corresponding laser spectral feature vectors are used as training data. The parameters of the one-dimensional convolution branch are fixed. The visible light high-level features and infrared high-level features extracted by the backbone network with shared parameters are respectively compared with the laser spectral feature vector to calculate the cosine similarity loss. The parameters of the backbone network with shared parameters are optimized by the cosine similarity loss. In the multimodal fusion training phase, paired visible light images, infrared images, and laser echo feature sequences are used to train the complete pipeline oil leak detection model, including the cross-modal interactive fusion module, end-to-end. A composite loss function is used for parameter optimization. The composite loss function includes classification loss, position regression loss, spectral physical consistency loss, and cross-modal distributed alignment loss. The classification loss is the focus loss, the position regression loss is a linear combination of L1 loss and generalized intersection-union loss, the spectral physical consistency loss is the cosine similarity between the image features and the laser spectral features within the detection box, and the cross-modal distributed alignment loss is the KL divergence between the distributions of visible light, infrared, and laser modal features.

4. The pipeline oil leak detection method based on three-modal physical-visual collaboration according to claim 1, characterized in that, The cross-modal interaction fusion module fuses features extracted from three modal data through a three-way attention mechanism, including: The highest-level visible light feature map and infrared feature map extracted from the multi-scale features of the backbone network are flattened along the spatial dimension and converted into visible light sequence and infrared sequence, respectively; at the same time, the laser spectral feature vector output by the one-dimensional convolution branch is mapped into a laser physical sequence of the same dimension through linear projection. Using the laser physics sequence as the query, the laser physics sequence, visible light sequence, and infrared sequence are transformed through learnable linear transformations to generate query vectors, key vectors, and value vectors, respectively. Specifically: When calculating the visible-infrared attention, the visible light query that incorporates laser physical sequence information is scaled and dot-product attention with the infrared key and infrared value to obtain the visible-infrared attention output; when calculating the infrared-visible attention, the infrared query that incorporates laser physical sequence information is scaled and dot-product attention with the visible light key and visible light value to obtain the infrared-visible attention output. The visible-infrared attention output and the infrared-visible attention output are added element-wise and then added element-wise to the laser physical sequence. The fused multi-scale features are obtained through the reshaping operation.

5. The pipeline oil leak detection method based on three-modal physical-visual collaboration according to claim 4, characterized in that, The construction of a cross-modal feature pyramid based on the fused features is as follows: Starting with the highest-level feature in the fused multi-scale features, a top-down feature fusion is performed, including: The fused multi-scale features are upsampled to obtain upsampled features; the corresponding mid-level visible light feature map and infrared feature map extracted by the backbone network are added element-wise to obtain mid-level visual fusion features; the upsampled features and mid-level visual fusion features are fused using learnable weighted fusion, followed by convolution and batch normalization to obtain the current level weighted fusion features; the upsampling, weighted fusion and convolution normalization operations are repeated, and the features are fused sequentially with the lower-level visible light feature map and infrared feature map extracted by the backbone network until the lowest level is processed to obtain a top-down weighted fusion feature set; The lowest-level weighted fusion features obtained in the top-down path are downsampled to obtain downsampled features. The downsampled features are then fused with the corresponding top-down weighted fusion features using a learnable weighted fusion method. After fusion, the features are processed by convolution and batch normalization to obtain the current layer features of the bottom-up path. The current layer features are then downsampled again and fused with the top-down weighted fusion features of the previous layer using a learnable weighted fusion method. The downsampling and weighted fusion operations are repeated until all feature layers have been processed to obtain the bottom-up cross-modal feature pyramid.

6. The pipeline oil leak detection method based on three-modal physical-visual collaboration according to claim 5, characterized in that, In the learnable weighted fusion, the weight parameters used for fusion are learnable parameters optimized through gradient backpropagation during the training of the pipeline oil leak detection model. During the inference of the pipeline oil leak detection model, the weight parameters and the features of the three-modal data are used to calculate the fusion ratio of each modality feature. The fusion ratio is dynamically determined based on the average brightness of the visible light image, the gradient amplitude of the infrared image, and the response intensity of the laser spectral feature vector in the three-modal data.

7. The pipeline oil leak detection method based on three-modal physical-visual collaboration according to claim 1, characterized in that, The Transformer decoder consists of multiple stacked decoding layers, which are based on... The specific process of decoding the cross-modal feature pyramid using a learnable object query vector as input and a Transformer decoder is as follows: For each decoding layer, Each object query vector is input into a multi-head self-attention module, which calculates... Self-attention among query vectors, output the updated... One query vector; The updated Each query vector is used as a query. The feature sequence formed by flattening and splicing the feature maps of all levels in the cross-modal feature pyramid is used as the key and value. It is input into the multi-head cross-attention module to calculate the attention weight between each query and all positional features in the feature sequence. The feature sequence is weighted and summed according to the attention weight to obtain the aggregated feature vector corresponding to each query. The aggregated feature vector is residually joined with the query vector input to the cross-attention module and then normalized to obtain the output of the decoding layer. One query vector; go through After stacking the decoding layers, the first layer will be processed. Layer output Each query vector is input into a linear classification layer and a linear regression layer, respectively. The linear classification layer outputs the target category probability for each query, and the linear regression layer outputs the target bounding box coordinates and the substance determination confidence for each query.

8. A pipeline oil leak detection system based on trimodal physical-visual collaboration, the system being used to execute the pipeline oil leak detection method based on trimodal physical-visual collaboration as described in any one of claims 1-7, characterized in that, The system includes: The three-modal active sensing hardware module is used to simultaneously acquire visible light images, infrared images, and laser echo feature sequences of the pipeline area to be detected. The multi-source data preprocessing engine, connected to the tri-modal active sensing hardware module, is used to perform spatial pixel-level alignment and normalization processing on the received visible light image, infrared image and laser echo feature sequence, and output the aligned and normalized tri-modal data. The three-dimensional fusion oil leak detection core is connected to the multi-source data preprocessing engine. It contains a pipeline oil leak detection model, which includes a shared parameter backbone network, a one-dimensional convolutional branch, a cross-modal interactive fusion module, a cross-modal feature pyramid construction module, and a Transformer decoder. It is used to output oil leak detection results based on the input aligned and normalized three-modal data. The dynamic feedback and early warning module is connected to the three-dimensional fusion oil leak detection core and is used to trigger an early warning signal and display the fusion image based on the oil leak detection result.

9. The pipeline oil leak detection system based on three-modal physical-visual collaboration according to claim 8, characterized in that, The pipeline oil leak detection model outputs the oil leak detection results based on the input aligned and normalized three-modal data as follows: The visible light and infrared images are extracted in parallel by a backbone network with shared parameters, and multiple levels of visible light feature maps and infrared feature maps are output. The normalized laser echo sequence is encoded by a one-dimensional convolutional branch, and the laser spectral feature vector is output. The cross-modal interactive fusion module, which connects the shared parameter backbone network and the one-dimensional convolutional branch respectively, receives the highest-level visible light feature map and infrared feature map output by the shared parameter backbone network, as well as the laser spectral feature vector output by the one-dimensional convolutional branch. The visible light feature, infrared feature and laser spectral feature are fused through a three-way attention mechanism to output the fused multi-scale feature. The cross-modal feature pyramid construction module, which connects the shared parameter backbone network and the cross-modal interactive fusion module respectively, receives the visible light feature maps and infrared feature maps of each level output by the shared parameter backbone network, as well as the fused high-level features output by the cross-modal interactive fusion module. It constructs and outputs the cross-modal feature pyramid through a top-down learnable weighted fusion path and a bottom-up learnable weighted fusion path. The Transformer decoder connected to the cross-modal feature pyramid construction module receives the cross-modal feature pyramid and processes it through a multi-layer decoding layer stack. Each decoding layer sequentially performs multi-head self-attention calculation, multi-head cross-attention calculation with the cross-modal feature pyramid as the key, residual connection and layer normalization. The linear classification layer and linear regression layer output the predicted class probability, bounding box coordinates and substance determination confidence of each target. By connecting the dynamic feedback and early warning module of the three-dimensional fusion oil leak detection core, an early warning signal is triggered based on the comparison result of the substance determination confidence level and the preset threshold, and the fused pseudo-color enhanced image is displayed.

10. The pipeline oil leak detection system based on three-modal physical-visual collaboration according to claim 8, characterized in that, The system also includes a model training module, which is connected to the three-dimensional fusion oil leak detection core and is used to train the pipeline oil leak detection model, specifically including: The modal isolation training unit is configured to train the backbone network, one-dimensional convolutional branch and Transformer decoder with shared parameters separately using a single modality of visible light image dataset, infrared image dataset and laser echo feature sequence dataset. During training, a weighted sum of classification loss and location regression loss is used. The material property alignment training unit is configured to use paired visible light images and infrared images and corresponding laser spectral feature vectors as training data. The parameters of the one-dimensional convolution branch are fixed, and the visual features extracted by the backbone network with shared parameters are constrained to approach the laser spectral features by cosine similarity loss, thereby optimizing the backbone network parameters. The multimodal fusion training unit is configured to use pairs of visible light images, infrared images, and laser echo feature sequences to perform end-to-end training on a complete pipeline oil leak detection model, including a cross-modal interactive fusion module. A composite loss function is used to optimize all parameters, including classification loss, location regression loss, spectral physical consistency loss, and cross-modal distributed alignment loss.

Images