Well site target detection method based on lightweight dual-band image fusion

Abstract

The invention discloses a well site target detection method based on lightweight dual-band image fusion. The method comprises the following steps: S1, collecting and preprocessing a visible light image and an infrared thermal image of a well site under special weather conditions; s2, marking the pre-processed visible light image and the pre-processed infrared thermal image; s3, establishing an end-to-end dual-band image fusion network model LAFusion based on the convolutional neural network; s4, establishing an improved lightweight target detection model based on YOLOv8; s5, pre-training the proposed dual-band image fusion network model LAFusion and the improved target detection network model through the preprocessed well site visible light image and infrared thermal image data set, directly enabling an output layer of the fusion model LAFusion to be compatible with the detection network model, and outputting a qualified model; s6, adaptively deploying the qualified model output in S5 to an edge end; and finally, an integrated well site intelligent monitoring system is constructed. According to the invention, the real-time performance of the system in a well site security scene is ensured while the detection precision is ensured.

Description

Technical Field

[0001] This invention relates to the field of target detection and image fusion technology, specifically to a well site target detection method based on lightweight dual-band image fusion. Background Technology

[0002] In the field of oil and gas extraction, well sites, as core production scenarios, rely on efficient and accurate target detection technologies for equipment operation status monitoring, personnel safety management, and abnormal risk early warning. Traditional well site target detection often uses a single visible light imaging scheme. However, well sites frequently face complex operating conditions and interference—such as loss of imaging details due to low-light environments at night, and target feature obscuring caused by blurred and superimposed images due to extreme weather (rain, snow, fog, haze, sandstorms, etc.). These issues significantly reduce the accuracy of single-band detection, making it difficult to meet the needs of precise monitoring.

[0003] Infrared images utilize thermal infrared radiation imaging. The brightness value of each pixel in the image depends on the intensity of thermal radiation energy of the target and the well site. It can also display the structural features of the target and the well site well in low-light scenes. However, infrared images generally have low resolution and cannot present detailed information about the well site. Visible light images have high resolution and contain rich details when there is sufficient light, but the resolution will decrease significantly in low-light scenes.

[0004] With the upgrading of industrial intelligence, well sites have higher requirements for the lightweight and real-time performance of detection technology: on the one hand, the computing power of edge computing equipment at well sites is limited, and complex algorithms are difficult to deploy; on the other hand, traditional multi-band fusion methods often have problems such as high data redundancy and complex fusion rules, which leads to increased detection delays and makes it impossible to respond quickly to emergency scenarios such as equipment failure and personnel violations.

[0005] Existing dual-band image fusion target detection methods can be mainly divided into three categories: First, pixel-level fusion schemes, which directly fuse pixels of visible light and infrared images using traditional algorithms such as weighted averaging and wavelet transform to generate a single fused image, which is then input into the detection model. This type of method focuses on the superposition of information at the pixel level, such as a weighted fusion strategy based on a fixed threshold. Second, feature-level fusion schemes, which use deep learning networks to extract feature maps of the two modalities separately, and then merge the features at the network backbone or neck through methods such as splicing and addition. Some methods design dedicated fusion modules, such as differential modality feature fusion modules, which achieve feature complementarity by extracting differential and common mode information. Third, decision-level fusion schemes, which first have independent models detect the dual-band images separately, and then merge the detection results using algorithms such as weighted bounding box fusion. However, the above dual-band image fusion target detection methods have significant shortcomings in well site scenarios. In terms of fusion effectiveness, pixel-level fusion, due to the use of fixed weight thresholds, is prone to the loss of equipment texture details and infrared thermal features, resulting in problems such as decreased image clarity and amplified noise. Feature-level fusion, on the other hand, often lacks adaptive mechanisms for well site conditions, making it difficult to balance the feature weights of the two modes under interference from sandstorms and haze. In terms of deployment adaptability, traditional fusion models often rely on complex network structures, causing inference delays to exceed the safety warning threshold, which cannot meet the real-time requirements of well site security scenarios. Summary of the Invention

[0006] In order to overcome the shortcomings of the existing technology, the purpose of this invention is to provide a well site target detection method based on lightweight dual-band image fusion. This detection method ensures the real-time performance of the system in well site security scenarios while guaranteeing detection accuracy.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A well site target detection method based on lightweight dual-band image fusion includes the following steps; S1. Establishing a dataset: Dual-band image acquisition and preprocessing: Collect visible light and infrared thermal images of the well site under special weather conditions, including different weather, time and well site location, and preprocess the visible light and infrared thermal images; S2. Label the preprocessed visible light image and infrared thermal image to obtain the visible light image dataset and the infrared thermal image dataset. S3. Establish an end-to-end dual-band image fusion network model LAFusion (Lightweight Attention-guided Fusion Net) based on convolutional neural networks; S4. Establish a lightweight object detection model based on YOLOv8 and make lightweight improvements to the object detection network model; S5, Fusion-Detection End-to-End Joint Training: The proposed dual-band image fusion network model LAFusion and the improved target detection network model were pre-trained using pre-processed well site visible light images and infrared thermal image datasets. The output layer of the fusion model LAFusion is directly compatible with the detection network model, and the output model is qualified. There is no need to design a dual-branch detection network, which further improves the overall size of the network. S6. Adapt and deploy the qualified model (.pth format) output by S5 to the edge; finally, build an integrated well site intelligent monitoring system to realize full-process automation from dual-band image acquisition, real-time model inference (fusion + detection) to remote control and anomaly early warning.

[0008] S1 specifically includes the following steps: S1.1 Image Acquisition: Using a binocular camera, images are acquired in different areas of the well site (such as around the wellhead, along the oil pipeline, and in the oil storage tank area) under different weather conditions, at a frequency of 10 sets of images per hour to ensure that the dataset covers the entire well site. S1.2 Image Calibration: Due to the differences in optical parameters between visible light and infrared cameras, it is necessary to calibrate the binocular camera. Using the parameters obtained from the calibration, the infrared image is mapped to the visible light image coordinate system through perspective transformation to achieve pixel-level alignment of the dual-band images and eliminate the fusion misalignment problem caused by viewing angle deviation. S1.3 Size Normalization: Considering the limited computing resources of the well site edge equipment, all preprocessed images are uniformly scaled to a resolution of 512×512, and bilinear interpolation is used to ensure that the image features are not distorted during the scaling process.

[0009] In S2, the well site dual-band images collected in S1 are standardized and labeled to generate label data that is adapted to the training of LAFusion fusion network and improved YOLOv8 model, providing accurate feature information for subsequent model training. The labeled dataset is randomly divided into training set, validation set and test set. Positive samples with unclear pixel regions are not labeled. The training set is used to fit the model by setting the parameters of the classifier and training the classification model. When combined with the validation set, multiple classifiers are fitted based on the modified parameter values. The validation set data is used to evaluate the accuracy of each model on the same dataset, and then the parameters corresponding to the best performing model are selected. After obtaining the optimal model through the training and validation sets, the test set is used to make predictions for the model to measure the performance and generalization ability of the optimal model.

[0010] The LAFusion algorithm proposed in S3, specifically, is based on a collaborative design of "hierarchical convolutional structure + adaptive attention mechanism" to achieve algorithm lightweighting while solving the problems of "feature loss, fixed weights, and computational redundancy" in traditional dual-band fusion in well site scenarios. The specific technical steps are as follows: S31, Shallow Standard Convolutional Layer: The input is an image pair consisting of the visible light image and the infrared thermal image preprocessed in S1. A 3×3 standard convolutional kernel is used for feature extraction. The core purpose of this layer is to fully preserve the basic features of the original image. It extracts detailed information about the texture of the equipment (such as valve outlines and pipe interfaces) and the outline of personnel from the visible light image, and captures the thermal radiation distribution of the equipment from the infrared thermal image. This avoids the premature loss of key features due to lightweight design and lays the foundation for subsequent fusion. S32, Intermediate Ghost Convolutional Layer: As a core component of the lightweight design, Ghost convolution is used instead of traditional standard convolution. Its principle is to significantly reduce the number of parameters and computational load while maintaining feature representation capabilities by using "basic convolution to generate core feature maps + linear transformation to generate redundant feature maps." Specifically, Ghost convolution first uses the basic convolution kernel. Generate core feature maps Then, through a 1×1 linear transformation matrix Generate s redundant feature maps Finally, the output is concatenated to obtain the same dimension as the traditional standard convolution. The calculation process is shown in formulas (1)-(2): in, The feature map is generated by the basic convolution kernel, and Conv represents the standard convolution operation. This is a visible light feature map. Infrared feature map, Based on the convolution kernel, This is a fused feature map concatenated after Ghost convolution. The number of feature maps generated by standard convolution. Here, is the number of feature maps generated by the Ghost convolution, ⊕ represents the feature map concatenation operation, and s represents the number of redundant feature maps. The number of parameters for a traditional standard convolution is... The Ghost convolution parameter count is... ; The parameter reduction ratio is The calculated parameter count is reduced by 66.7%, achieving a significant weight reduction effect.

[0011] To address the issue of significant industrial noise in well site images (such as speckles caused by dust and artifacts generated by equipment vibration), this layer simultaneously embeds a Batch Normalization (BN) layer and a ReLU activation function. Normalization suppresses noise amplification, while nonlinear activation enhances the discriminative power of effective features, ensuring that feature quality is not reduced while maintaining a lightweight design.

[0012] S33, Output Layer Standard Convolutional Layer: A 1×1 standard convolutional kernel is used to fuse the feature map output from S32. Perform dimensional adjustment and feature integration to integrate the infrared feature map With visible light feature map This layer generates a single-channel fusion feature map, and normalizes the feature values ​​to the [0,1] interval using the Sigmoid activation function, ultimately producing a fusion image that can be directly input into downstream target detection tasks. The key to this layer design is to eliminate feature redundancy through the "dimensionality reduction-integration" effect of 1×1 convolution, ensuring that the fusion image retains both visible light detail and texture and highlights infrared thermal target features, meeting the dual requirements of well site target detection for "details + thermal features".

[0013] S34, Adaptive fusion weight allocation of the CBAM attention mechanism: To address the issue of "different importance of visible light and infrared modes under different operating conditions," global average pooling is used to fuse the infrared feature maps obtained from S31 and S32. With visible light feature map The features are converted into channel feature vectors, and then the importance weights of each channel are learned through a two-layer fully connected network. High-importance channel features are given high weights, and low-importance channel features are given low weights. This ensures that the features of the thermal target are not masked by visible light noise under low light conditions. When the light intensity is sufficient, the weight of the visible light modal features is increased to highlight the texture details of the device.

[0014] S4 specifically includes: S41. Basic Framework Selection and Initial Configuration: YOLOv8-nano was chosen as the basic framework for object detection, primarily because of its lightweight nature (approximately 1.1M parameters and 2.8G FLOPs of computation), allowing for rapid adaptation to edge devices. Based on the characteristics of the well site targets, the size of the fused image obtained in S3 was fixed at 512×512, and the anchor frame size was re-clustered based on the statistical data of the measured targets at the well site. This lays the foundation for the backbone network to extract target features in S42 and for the compressed model in S43 to retain key detection capabilities.

[0015] S42. Lightweight Improvement of Backbone Network: To address the issue of high computational cost of the original YOLOv8-nano backbone network CSPDarknet selected in S41, it was replaced with MobileNetv3-small network. Through depthwise separable convolution and inverse residual structure, the network significantly reduces the computational cost while retaining the well site target feature extraction capability of anchor frame adaptation in S41. S43. Model Compression and Redundancy Removal: Based on the improvements to the backbone network in S42, secondary lightweighting is achieved through model pruning and knowledge distillation to ensure compatibility with the computing power of edge devices at the well site.

[0016] In S42, depthwise separable convolution is used to replace traditional convolution: all 3×3 standard convolutions in the original CSPDarknet are replaced with a combination of depthwise separable convolutions of "depthwise convolution + pointwise convolution"; the depthwise convolution uses a single-channel convolution kernel to process each channel of the input feature map. In the bottleneck layer of MobileNetv3-small, an inverse residual structure of "1×1 convolution for dimensionality increase → 3×3 depthwise convolution → 1×1 convolution for dimensionality reduction" is adopted. The feature expression capability is enhanced by dimensionality increase, and the number of parameters is reduced by dimensionality reduction. At the same time, an SE (squeeze-excitement) attention module is embedded after each bottleneck layer to strengthen the extraction of key features of the well site target (such as equipment outline and personnel limbs) and avoid feature loss caused by lightweighting. The calculation process is shown in equations (3)-(5): in, This represents the computational cost of a standard convolution, where K is the kernel size. Input the number of channels for the feature map received by the MobileNetv3-small network in step S42. Here, H represents the number of output channels, and W represents the height and width of the feature map, respectively. The total computational cost of depthwise separable convolution is... Because the kernel size K of the 3×3 standard convolution in the original CSPDarknet in step 3 is 3, the total computational cost will be reduced compared to standard convolution when using depthwise separable convolution. .

[0017] S43 takes the MobileNetv3-small backbone network improved from S42 as the core optimization object, and mainly includes model pruning and knowledge distillation. The model pruning adopts a dual strategy of "channel pruning + layer pruning". First, the importance weights of each convolutional layer channel in the MobileNetv3-small network in S42 are calculated through L1 regularization, and redundant channels with absolute weight values ​​less than the threshold (0.01 in this scheme) are removed. Then, the information entropy of the output feature maps of each layer of the network is analyzed, and redundant convolutional layers with information entropy lower than the threshold are removed, which ultimately further reduces the number of model parameters.

[0018] The knowledge distillation process uses the uncompressed YOLOv8-large model as the "teacher model" and the lightweight model corresponding to the improved S42 backbone network as the "student model," learning the feature distribution of the teacher model. A distillation loss function (combining classification loss, regression loss, and feature loss with weights of 0.3:0.5:0.2) allows the student model to learn the feature distribution and decision logic of the teacher model. The feature loss calculates the MSE of the corresponding feature layers in both the student and teacher models, ensuring that the lightweight model maintains at least 90% of the detection accuracy of the teacher model while reducing the number of parameters.

[0019] Specifically, the S5 fusion-detection joint training includes: S51. Using various well site dual-band image datasets from S1 and S2, including preprocessed images and labels, extract the coordinates of the target region from the labels; load the model parameters of the fusion network and detection network from S3 and S4, and adjust the hyperparameters during joint training, such as the number of iterations, weights, and initial learning rate. S52. Calculate the fusion loss using the fusion results from the S3 fusion network LAFusion. The fusion loss refers to the global pixel error, brightness, contrast, and structural consistency between the fused image and the original dual-band image. Calculate the detection loss using the prediction results from the S4 detection network YOLOv8. The detection loss refers to the error between the predicted category and the true label, as well as the positional error between the predicted detection box and the true detection box. The final joint loss function is calculated from the fusion loss and the detection loss. The optimization priority of the two is balanced by dynamic weights to achieve collaborative optimization.

[0020] In S6, the method for establishing a three-layer system of "perception-computation-application" and hardware adaptation specifically includes: S61. Hardware selection and edge sensing layer deployment: Based on the well site conditions and the requirements of the lightweight model operation, complete the hardware selection and sensing layer equipment installation to provide stable data input for S61. S62, Edge computing layer software module development: Based on the selected RK3588 edge computing platform, develop software modules adapted to the lightweight fusion model and detection model proposed by S3 and S4 to realize end-to-end processing of image acquisition, fusion, detection and data transmission. S63. Application layer platform development and function integration: Develop web / mobile application platforms to achieve data visualization, human-computer interaction and remote control, and meet the operational needs of well site management personnel.

[0021] Specifically, S61 includes: First, the Rockchip RK3588 platform was selected as the edge computing unit (equipped with an NPU, with a computing power of 6 TOPS, supporting INT8 / FP16 precision inference, which can meet the real-time operation requirements of the lightweight fusion detection network model proposed by S3 and S4). Secondly, the cameras are installed in appropriate locations according to the well site layout to avoid obstructing key areas such as wellheads, oil pipelines, and oil storage tanks; the binocular cameras are calibrated on-site, and the sensor calibration parameters preset in S1 are combined to ensure that the collected infrared and visible light images are aligned at the pixel level, providing accurate data for subsequent fusion processing.

[0022] Specifically, S62 involves converting the LAFusion fusion network (.pth format) and the improved YOLOv8 model (.pth format) saved in S5 using the RKNN Toolkit toolchain. First, the PyTorch model is exported to ONNX format, retaining the single-channel fused image output by the LAFusion fusion network in S3 and the detection boxes and class confidence information output by the improved YOLOv8 in S4, ensuring data exchange compatibility between the fusion and detection modules. Then, it is converted to the RKNN format supported by the RK3588 NPU, using INT8 quantization, improving inference speed by over 30% while maintaining an accuracy loss of ≤3%. Finally, the RKNN inference engine is deployed, and a C++ inference interface is written to automate the model loading, input data preprocessing, inference computation, and output result parsing processes.

[0023] The beneficial effects of this invention are: 1. This invention, through the collaborative design of the LAFusion lightweight dual-band fusion algorithm and the improved YOLOv8 detection model, not only solves the feature loss problem of traditional single-band detection in low-light, haze, and dusty scenes (infrared thermal features + visible light texture complementarity), but also significantly reduces the total time consumption of the fusion + detection process by relying on lightweight algorithm design and hardware adaptation optimization, while ensuring high detection accuracy. Ultimately, it achieves a dual breakthrough of "high accuracy + low latency", which can effectively identify various key targets in the well site. 2. Compared with the limitations of traditional multi-band detection schemes that rely on high-performance servers, this invention adopts a lightweight "fusion-detection" integrated network and is deployed based on the Rockchip RK3588 edge platform. This deployment scheme, which does not require complex computing power, not only significantly reduces the hardware cost of the well site detection system, but also adapts to the low-temperature environment of the well site, solving the core limitations of traditional servers such as high power consumption and poor environmental adaptability. 3. The "sensing-computing-application" three-layer system constructed by this invention realizes full-chain automation from dual-band image acquisition and real-time fusion detection to remote monitoring, alarm response, and historical data tracing; the application layer supports multi-terminal interaction on Web / mobile terminals, allowing managers to remotely view equipment status and adjust parameters without on-site supervision, greatly shortening safety early warning response time and improving the efficiency of safety production management and control. Attached Figure Description Figure 1 This is a flowchart illustrating the method proposed in this invention.

[0024] Figure 2 This is a schematic diagram of the fusion and target detection process.

[0025] Figure 3 The diagram shows the structure of the proposed fusion network LAFusion.

[0026] Figure 4 This is a diagram showing the equipment testing results. Detailed Implementation

[0027] The present invention will now be described in further detail with reference to the accompanying drawings.

[0028] This invention provides a lightweight well site target detection method based on dual-band image fusion, such as... Figure 1 As shown, it includes the following steps: S1. Establish a well site dataset, collecting visible light images and infrared thermal images, including different weather conditions, lighting conditions, distances, and different well sites. Specifically: S1.1 Image Acquisition: Using a binocular camera, images are acquired in different areas of the well site (such as around the wellhead, along the oil pipeline, and in the oil storage tank area) under different weather conditions, at a frequency of 10 sets of images per hour to ensure that the dataset covers the entire well site. S1.2 Image calibration: Due to the differences in optical parameters between visible light and infrared cameras, it is necessary to calibrate the binocular camera. Using the parameters obtained from the calibration, the infrared image is mapped to the visible light image coordinate system through perspective transformation to achieve pixel-level alignment of the dual-band images and eliminate the fusion misalignment problem caused by viewing angle deviation. S1.3 Size Normalization: Considering the limited computing resources of the well site edge equipment, all preprocessed images are uniformly scaled to a resolution of 512×512, and bilinear interpolation is used to ensure that the image features are not distorted during the scaling process; In S2, the well site dual-band images collected in S1 are standardized and labeled to generate label data that is adapted to the training of LAFusion fusion network and improved YOLOv8 model, providing accurate feature information for subsequent model training. The labeled dataset is randomly divided into training set, validation set and test set. Positive samples with unclear pixel regions are not labeled. The training set is used to fit the model by setting the parameters of the classifier and training the classification model. When combined with the validation set, multiple classifiers are fitted based on the modified parameter values. The validation set data is used to evaluate the accuracy of each model on the same dataset, and then the parameters corresponding to the best performing model are selected. After obtaining the optimal model through the training and validation sets, the test set is used to make predictions for the model to measure the performance and generalization ability of the optimal model.

[0029] The LAFusion algorithm proposed in S3, specifically, is based on a collaborative design of "hierarchical convolutional structure + adaptive attention mechanism" to achieve algorithm lightweighting while solving the problems of "feature loss, fixed weights, and computational redundancy" in traditional dual-band fusion in well site scenarios. The specific technical steps are as follows: S31, Shallow Standard Convolutional Layer: The input is an image pair consisting of the visible light image and the infrared thermal image preprocessed in S1. A 3×3 standard convolutional kernel is used for feature extraction. The core purpose of this layer is to fully preserve the basic features of the original image. It extracts detailed information about the texture of the equipment (such as valve outlines and pipe interfaces) and the outline of personnel from the visible light image, and captures the thermal radiation distribution of the equipment from the infrared thermal image. This avoids the premature loss of key features due to lightweight design and lays the foundation for subsequent fusion. S32, Intermediate Ghost Convolutional Layer: As a core component of the lightweight design, Ghost convolution is used instead of traditional standard convolution. Its principle is to significantly reduce the number of parameters and computational load while maintaining feature representation capabilities by using "basic convolution to generate core feature maps + linear transformation to generate redundant feature maps." Specifically, Ghost convolution first uses the basic convolution kernel. Generate core feature maps Then, through a 1×1 linear transformation matrix Generate s redundant feature maps Finally, the output is concatenated to obtain the same dimension as the traditional standard convolution. The calculation process is shown in formulas (1)-(2): in, The feature map is generated by the basic convolution kernel, and Conv represents the standard convolution operation. This is a visible light feature map. Infrared feature map, Based on the convolution kernel, The feature map is concatenated after Ghost convolution. The number of feature maps generated by standard convolution. Here, is the number of feature maps generated by the Ghost convolution, ⊕ represents the feature map concatenation operation, and s represents the number of redundant feature maps. The number of parameters for a traditional standard convolution is... The Ghost convolution parameter count is... The parameter quantity was reduced by a certain percentage. The calculated parameter count is reduced by 66.7%, achieving a significant weight reduction effect.

[0030] To address the issue of significant industrial noise in well site images (such as speckles caused by dust and artifacts generated by equipment vibration), this layer simultaneously embeds a Batch Normalization (BN) layer and a ReLU activation function. Normalization suppresses noise amplification, while nonlinear activation enhances the discriminative power of effective features, ensuring that feature quality is not reduced while maintaining a lightweight design.

[0031] S33, Output Layer Standard Convolutional Layer: A 1×1 standard convolutional kernel is used to fuse the feature map output from S32. Perform dimensional adjustment and feature integration to integrate the infrared feature map With visible light feature map This layer generates a single-channel fusion feature map, and normalizes the feature values ​​to the [0,1] interval using the Sigmoid activation function, ultimately producing a fusion image that can be directly input into downstream target detection tasks. The key to this layer design is to eliminate feature redundancy through the "dimensionality reduction-integration" effect of 1×1 convolution, ensuring that the fusion image retains both visible light detail and texture and highlights infrared thermal target features, meeting the dual requirements of well site target detection for "details + thermal features".

[0032] S34, Adaptive fusion weight allocation of the CBAM attention mechanism: To address the issue of "different importance of visible light and infrared modes under different operating conditions," global average pooling is used to fuse the infrared feature maps obtained from S31 and S32. With visible light feature map The features are converted into channel feature vectors, and then the importance weights of each channel are learned through a two-layer fully connected network. High-importance channel features are given high weights, and low-importance channel features are given low weights. This ensures that the features of the thermal target are not masked by visible light noise under low light conditions. When the light intensity is sufficient, the weight of the visible light modal features is increased to highlight the texture details of the device.

[0033] S4 specifically includes: S41. Basic Framework Selection and Initial Configuration: YOLOv8-nano was chosen as the basic framework for object detection. The core reason is its lightweight nature (approximately 1.1M parameters and 2.8G FLOPs of computation), allowing for rapid adaptation to edge devices. Based on the characteristics of the well site targets, the fused image size obtained in S3 was fixed at 512×512. The anchor frame size was re-clustered based on the statistical data of the measured targets at the well site, laying the foundation for the backbone network to extract target features in S42 and for the compressed model in S43 to retain key detection capabilities.

[0034] S42. Lightweight Improvement of Backbone Network: To address the issue of high computational cost of the original YOLOv8-nano backbone network CSPDarknet selected in S41, it was replaced with MobileNetv3-small network. Through depthwise separable convolution and inverse residual structure, the network significantly reduces the computational cost while retaining the well site target feature extraction capability of anchor frame adaptation in S41. S43. Model Compression and Redundancy Removal: Based on the improvements to the backbone network in S42, secondary lightweighting is achieved through model pruning and knowledge distillation to ensure compatibility with the computing power of edge devices at the well site.

[0035] In S42, depthwise separable convolution is used to replace traditional convolution: all 3×3 standard convolutions in the original CSPDarknet are replaced with a combination of depthwise separable convolutions of "depthwise convolution + pointwise convolution"; the depthwise convolution uses a single-channel convolution kernel to process each channel of the input feature map. In the bottleneck layer of MobileNetv3-small, an inverse residual structure of "1×1 convolution for dimensionality increase → 3×3 depthwise convolution → 1×1 convolution for dimensionality reduction" is adopted. The feature expression capability is enhanced by dimensionality increase, and the number of parameters is reduced by dimensionality reduction. At the same time, an SE (squeeze-excitement) attention module is embedded after each bottleneck layer to strengthen the extraction of key features of the well site target (such as equipment outline and personnel limbs) and avoid feature loss caused by lightweighting. The calculation process is shown in equations (3)-(5): in, This represents the computational cost of a standard convolution, where K is the kernel size. Input the number of channels for the feature map received by the MobileNetv3-small network in step S42. Here, H represents the number of output channels, and W represents the height and width of the feature map, respectively. The total computational cost of depthwise separable convolution is... Because the kernel size K of the 3×3 standard convolution in the original CSPDarknet in step 3 is 3, the total computational cost will be reduced compared to standard convolution when using depthwise separable convolution. .

[0036] The S43 optimization focuses on the MobileNetv3-small backbone network improved from S42, and mainly includes model pruning and knowledge distillation.

[0037] Specifically, model pruning employs a dual strategy of "channel pruning + layer pruning". First, L1 regularization is used to calculate the importance weights of each convolutional layer channel in the MobileNetv3-small network in S42, and redundant channels with absolute weight values ​​less than a threshold (0.01 in this scheme) are removed. Then, the information entropy of the output feature maps of each layer of the network is analyzed, and redundant convolutional layers with information entropy lower than the threshold are removed, ultimately further reducing the number of model parameters.

[0038] Specifically, knowledge distillation uses the uncompressed YOLOv8-large model as the "teacher model" and the lightweight model corresponding to the improved S42 backbone network as the "student model," learning the feature distribution of the teacher model. The student model learns the feature distribution and decision logic of the teacher model through a distillation loss function (a combination of classification loss, regression loss, and feature loss with weights of 0.3:0.5:0.2). The feature loss calculates the MSE of the corresponding feature layers in both the student and teacher models, ensuring that the lightweight model maintains at least 90% of the detection accuracy of the teacher model while reducing the number of parameters.

[0039] Specifically, the S5 fusion-detection joint training includes: S51. Using various well site dual-band image datasets from S1 and S2, including preprocessed images and labels, extract the coordinates of the target region from the labels. Load the model parameters of the fusion network and detection network from S3 and S4, and adjust the hyperparameters during joint training, such as the number of iterations, weights, and initial learning rate. S52. Calculate the fusion loss using the fusion results from the LAFusion fusion network in step 3. The fusion loss refers to the global pixel error, brightness, contrast, and structural consistency between the fused image and the original dual-band image. Calculate the detection loss using the prediction results from the YOLOv8 detection network in step S4. The detection loss refers to the error between the predicted category and the ground truth label, as well as the positional error between the predicted detection box and the ground truth detection box. The final joint loss function is calculated from the fusion loss and the detection loss. Co-optimization is achieved by dynamically balancing the optimization priorities of the two.

[0040] In S6, the method for establishing a three-layer system of "perception-computation-application" and hardware adaptation specifically includes: S61. Hardware selection and edge sensing layer deployment: Based on the well site conditions and the requirements of the lightweight model operation, complete the hardware selection and sensing layer equipment installation to provide stable data input for S61. S62, Edge computing layer software module development: Based on the selected RK3588 edge computing platform, develop software modules adapted to the S3 and S4 lightweight models to realize end-to-end processing of image acquisition, fusion, detection and data transmission; S63. Application layer platform development and function integration: Develop web / mobile application platforms to achieve data visualization, human-computer interaction and remote control, and meet the operational needs of well site management personnel.

[0041] Specifically, S61 includes: First, selecting the Rockchip RK3588 platform as the edge computing unit (equipped with an NPU, boasting a computing power of 6 TOPS, supporting INT8 / FP16 precision inference, and meeting the real-time operation requirements of the lightweight fusion detection network model proposed in S3 and S4). Second, installing cameras in appropriate locations according to the well site layout, avoiding obstruction of key areas such as wellheads, oil pipelines, and oil storage tanks; calibrating the binocular cameras on-site, and combining the sensor calibration parameters preset in step S1 to ensure pixel-level alignment of the acquired infrared and visible light images, providing accurate data for subsequent fusion processing.

[0042] Specifically, S62 involves converting the LAFusion fusion network (.pth format) and the improved YOLOv8 model (.pth format) saved in S5 using the RKNN Toolkit toolchain. First, the PyTorch model is exported to ONNX format, retaining the single-channel fused image output by the LAFusion fusion network in S3 and the detection bounding boxes and class confidence information output by the improved YOLOv8 in S4, ensuring data exchange compatibility between the fusion and detection modules. Then, it is converted to the RKNN format supported by the RK3588 NPU, using INT8 quantization, improving inference speed by over 30% while maintaining an accuracy loss of ≤3%. Finally, the RKNN inference engine is deployed, and a C++ inference interface is written to automate the model loading, input data preprocessing, inference computation, and output result parsing processes.

[0043] The embodiments described above are merely preferred embodiments for fully illustrating the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A well site target detection method based on lightweight dual-band image fusion, characterized in that, Includes the following steps; S1. Collect visible light images and infrared thermal images of the well site under special weather conditions, including different weather, time and well site location, and preprocess the visible light images and infrared thermal images. S2. Label the preprocessed visible light image and infrared thermal image to obtain the visible light image dataset and the infrared thermal image dataset. S3. Establish an end-to-end dual-band image fusion network model LAFusion based on convolutional neural networks; S4. Establish a lightweight object detection model based on YOLOv8 and make lightweight improvements to the object detection network model; S5. The proposed dual-band image fusion network model LAFusion and the improved target detection network model are pre-trained using the pre-processed well site visible light image and infrared thermal image dataset. The output layer of the fusion model LAFusion is directly compatible with the detection network model, and the output is a qualified model. S6. Adapt and deploy the qualified model output from S5 to the edge; finally, build an integrated well site intelligent monitoring system to realize full-process automation from dual-band image acquisition and real-time model inference to remote control and anomaly early warning.

2. The well site target detection method based on lightweight dual-band image fusion according to claim 1, characterized in that, S1 specifically includes the following steps: S1.

1. Use a binocular camera to collect data at a certain frequency in different areas of the well site under different weather conditions to ensure that the dataset covers the entire well site scene; S1.2, Calibrate the binocular camera; Using the parameters obtained from calibration, the infrared image is mapped to the visible light image coordinate system through perspective transformation, achieving pixel-level alignment of the dual-band image. S1.3 Scale all preprocessed images uniformly, and use bilinear interpolation to ensure that image features are not distorted during the scaling process.

3. The well site target detection method based on lightweight dual-band image fusion according to claim 2, characterized in that, In S2, the well site dual-band images collected in S1 are standardized and labeled to generate label data that is adapted to the training of LAFusion fusion network and improved YOLOv8 model, providing accurate feature information for subsequent model training. The labeled dataset is randomly divided into training set, validation set and test set. Positive samples with unclear pixel regions are not labeled. The training set is used to fit the model by setting the parameters of the classifier and training the classification model. When combined with the validation set, multiple classifiers are fitted based on the modified parameter values. The validation set data is used to evaluate the accuracy of each model on the same dataset, and then the parameters corresponding to the best performing model are selected. After obtaining the optimal model through the training and validation sets, the test set is used to make predictions for the model to measure the performance and generalization ability of the optimal model.

4. The well site target detection method based on lightweight dual-band image fusion according to claim 3, characterized in that, The specific steps of the LAFusion algorithm proposed in S3 are as follows: S31. Input the image pair consisting of the visible light image and the infrared thermal image after preprocessing in S1, and extract features using a 3×3 standard convolution kernel; S32. Use Ghost convolution; first use the basic convolution kernel. Generate core feature maps Then, through a 1×1 linear transformation matrix Generate s redundant feature maps Finally, the output is concatenated to obtain the same dimension as the traditional standard convolution. The calculation process is shown in formulas (1)-(2): in, The feature map is generated by the basic convolution kernel, and Conv represents the standard convolution operation. This is a visible light feature map. Infrared feature map, Based on the convolution kernel, This is a fused feature map concatenated after Ghost convolution. The number of feature maps generated by standard convolution. represents the number of feature maps generated by Ghost convolution, ⊕ represents the feature map concatenation operation, and s represents the number of redundant feature maps; S33. The fused feature map output from S32 is processed using a 1×1 standard convolutional kernel. Perform dimensional adjustment and feature integration to integrate the infrared feature map With visible light feature map The feature map is fused into a single channel and the feature values ​​are normalized to the [0,1] interval by the Sigmoid activation function, and finally a fused image that can be directly input into downstream object detection tasks is generated. S34. The infrared feature maps obtained in S31 and S32 are processed by global average pooling. With visible light feature map The features are converted into channel feature vectors, and then the importance weights of each channel are learned through a two-layer fully connected network. High-importance channel features are given high weights, and low-importance channel features are given low weights. This ensures that the features of the thermal target are not masked by visible light noise under low light conditions. When the light intensity is sufficient, the weight of the visible light modal features is increased to highlight the texture details of the device.

5. The well site target detection method based on lightweight dual-band image fusion according to claim 4, characterized in that, S4 specifically includes: S41. Select YOLOv8-nano as the basic framework for target detection. Based on the characteristics of the well site targets, fix the size of the fused image obtained in S3, and re-cluster the anchor frame size according to the statistical data of the measured targets at the well site. S42. Select the MobileNetv3-small network and achieve significant weight reduction while retaining the well site target feature extraction capability of anchor frame adaptation in S41 through depthwise separable convolution and inverse residual structure. S43. Based on the improvements to the backbone network in S42, secondary lightweighting is achieved through model pruning and knowledge distillation.

6. The well site target detection method based on lightweight dual-band image fusion according to claim 5, characterized in that, In S42, depthwise separable convolution is used to construct a combination of "depthwise convolution + pointwise convolution"; the depthwise convolution uses a single-channel convolution kernel to process each channel of the input feature map respectively. In the bottleneck layer of MobileNetv3-small, an inverse residual structure of "1×1 convolution for dimensionality increase → 3×3 depthwise convolution → 1×1 convolution for dimensionality reduction" is adopted. The feature expression capability is enhanced by dimensionality increase, and the number of parameters is reduced by dimensionality reduction. At the same time, an SE attention module is embedded after each bottleneck layer to strengthen the extraction and calculation process of key features of the well site target, as shown in equations (3)-(5): in, This represents the computational cost of a standard convolution, where K is the kernel size. Input the number of channels for the feature map received by the MobileNetv3-small network in S42. Here, H represents the number of output channels, and W represents the height and width of the feature map, respectively. The total computational cost of depthwise separable convolution is... .

7. The well site target detection method based on lightweight dual-band image fusion according to claim 6, characterized in that, S43 takes the MobileNetv3-small backbone network improved by S42 as the core optimization object, and includes model pruning and knowledge distillation. The model pruning adopts a dual strategy of "channel pruning + layer pruning". First, the importance weights of each convolutional layer channel in the MobileNetv3-small network in S42 are calculated through L1 regularization, and redundant channels with absolute weight values ​​less than a threshold are removed. Then, the information entropy of the output feature maps of each layer of the network is analyzed, and redundant convolutional layers with information entropy lower than a threshold are removed. The knowledge distillation uses the uncompressed YOLOv8-large model as the "teacher model" and the lightweight model corresponding to the S42 improved backbone network as the "student model" to learn the feature distribution of the teacher model. The student model learns the feature distribution and decision logic of the teacher model through the distillation loss function. The feature loss is calculated by the MSE of the corresponding feature layers of the student model and the teacher model.

8. The well site target detection method based on lightweight dual-band image fusion according to claim 5, characterized in that, Specifically, the S5 fusion-detection joint training includes: S51. Using various well site dual-band image datasets from S1 and S2, including preprocessed images and labels, extract the coordinates of the target region from the labels; load the model parameters of the fusion network and detection network from S3 and S4, and adjust the hyperparameters during joint training. S52. Calculate the fusion loss using the fusion results from the S3 fusion network LAFusion. The fusion loss refers to the global pixel error, brightness, contrast, and structural consistency between the fused image and the original dual-band image. Calculate the detection loss using the prediction results from the S4 detection network YOLOv8. The detection loss refers to the error between the predicted category and the true label, as well as the positional error between the predicted detection box and the true detection box. The final joint loss function is calculated from the fusion loss and the detection loss. The optimization priority of the two is balanced by dynamic weights to achieve collaborative optimization.

9. A well site target detection method based on lightweight dual-band image fusion according to claim 8, characterized in that, In S6, a three-layer system of "perception-computation-application" and hardware adaptation method are established, specifically including: S61. Hardware selection and edge sensing layer deployment: Based on the well site conditions and the requirements of the lightweight model operation, complete the hardware selection and sensing layer equipment installation to provide stable data input for S61. S62, Edge computing layer software module development: Based on the selected RK3588 edge computing platform, develop software modules adapted to the lightweight fusion model and detection model proposed by S3 and S4 to realize end-to-end processing of image acquisition, fusion, detection and data transmission. S63. Application layer platform development and function integration: Develop web / mobile application platforms to achieve data visualization, human-computer interaction and remote control, and meet the operational needs of well site management personnel.

10. A well site target detection method based on lightweight dual-band image fusion according to claim 9, characterized in that, Specifically, S61 includes: First, the Rockchip RK3588 platform was selected as the edge computing unit; Secondly, the cameras should be installed in appropriate locations according to the well site layout to avoid obstructing key areas such as wellheads, oil pipelines, and oil storage tanks. The binocular camera is calibrated on-site, and the sensor calibration parameters preset in S1 are combined to ensure that the acquired infrared and visible light images are aligned at the pixel level, providing accurate data for subsequent fusion processing; Specifically, S62 involves: converting the LAFusion fusion network and the improved YOLOv8 model saved in S5 using the RKNNToolkit toolchain; first, exporting the PyTorch model to ONNX format, retaining the single-channel fusion image output by the LAFusion fusion network in S3 and the detection boxes and class confidence information output by the improved YOLOv8 in S4, ensuring data interaction compatibility between the fusion and detection modules; then converting it to the RKNN format supported by the RK3588 NPU, using INT8 quantization; deploying the RKNN inference engine, writing a C++ inference interface, and automating the process of model loading, input data preprocessing, inference computation, and output result parsing.

Images