Infrared image gas and liquid leakage detection method and system based on deep learning

Abstract

The present invention provides a method for detecting gas and liquid leakage using infrared images based on deep learning, the method comprising the following steps: collecting infrared thermal image data in gas and liquid leakage scenarios; preprocessing the infrared thermal image data in gas and liquid leakage scenarios, marking gas and liquid leakage information on the infrared thermal image data, obtaining a training data set, and using the training data set to train a target detection network to obtain the trained target detection network weights; acquiring and preprocessing infrared thermal imaging data in a scene to be detected, loading the trained target detection network weights, and performing real-time detection on the infrared thermal imaging data in the scene to be detected input to the target detection network to obtain the detection results; and visually outputting the detection results. The present invention is based on a target detection algorithm based on deep learning, uses field data to train a neural network, and uses the characteristics of neural networks that are faster, more accurate, and more stable than traditional methods to improve the degree of automation and accuracy of detection.

Description

Technical Field

[0001] The present invention relates to a method and system for detecting gas and liquid leakage using infrared images based on deep learning, and belongs to the technical field of intelligent target detection. Background Art

[0002] In the field of target detection, the infrared target detection system combined with infrared thermal imaging technology has the advantages of all-weather, long detection distance, and no influence from atmospheric and lighting conditions. It can achieve good detection results in rain, snow, fog, night and other environments, and has broad application prospects. At the same time, infrared target detection is a basic task in the fields of infrared reconnaissance, intelligent security, and night assisted driving.

[0003] Infrared thermal imaging technology for detecting leaking gas and liquid uses a thermal imager to receive infrared radiation containing the characteristic absorption peak band of the fluid to complete imaging detection of leaking objects. Most traditional infrared target detection methods follow the detection method of visible light images, and artificially design features based on the gray value difference between the infrared target and the background to extract the shallow semantic features of the target, and use these features to identify and classify the target. In the artificially designed feature method, according to the different feature extraction methods, it can be divided into infrared target detection based on interest domain search, infrared target detection based on threshold segmentation, and infrared target detection based on statistical learning.

[0004] However, most of the existing deep learning target detection algorithms are based on convolutional neural networks, which construct high-level semantics by extracting image color, texture, geometry and other features. The blurred edges and poor contrast of infrared images will affect the semantic expression ability of the model to a certain extent, reducing the detection accuracy of the algorithm. At the same time, large-scale target detection models have high requirements for memory and computing power, and cannot meet the real-time deployment requirements in embedded devices. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies in the prior art and to propose a method and system for infrared image gas leakage and liquid leakage detection based on deep learning, which improves the speed and accuracy of infrared image gas leakage and liquid leakage detection.

[0006] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0007] In a first aspect, the present invention provides a method for detecting gas and liquid leakage using infrared images based on deep learning, comprising the following steps:

[0008] Collect infrared thermal image data in gas and liquid leakage scenarios;

[0009] Preprocess the infrared thermal image data in the gas leakage and liquid leakage scene, mark the gas leakage and liquid leakage information on the infrared thermal image data, obtain a training data set, and use the training data set to train the target detection network to obtain the trained target detection network weights;

[0010] Acquire and preprocess the infrared thermal imaging data of the scene to be detected, load the trained target detection network weights, perform real-time detection on the infrared thermal imaging data of the scene to be detected input into the target detection network, and obtain the detection results;

[0011] Visualize the detection results.

[0012] Furthermore, the method for acquiring infrared thermal image data includes: calibrating the infrared thermal imager to obtain parameter information of the infrared thermal imager, and using the infrared thermal imager to collect video data of the scene to be detected and analyzing the video data to obtain infrared thermal image data of the scene to be detected;

[0013] The infrared thermal imager is an IR-HiNet-640 device. The resolution of a single thermal imaging image obtained is 640*512, and the measurable temperature range is -22℃ to +550℃.

[0014] Furthermore, the method for preprocessing infrared thermal image data includes the following steps:

[0015] The infrared thermal image data is enhanced to reduce noise interference in the infrared thermal imaging;

[0016] Methods for image enhancement of infrared image data include morphological filtering operations;

[0017] The morphological filtering operations include image dilation, erosion and image sharpening.

[0018] Further, the method of training the target detection network using the training data set includes the following steps:

[0019] The Kmeans algorithm is used to perform cluster analysis on the training data set to obtain the sizes of anchor candidate boxes, and then they are assigned to the large, medium, and small feature scales of the target detection network.

[0020] The network weights trained with the MS COCO dataset are used as the initial weights of the target detection network, and the target detection network is trained using the training dataset to obtain the final trained network parameter weight information.

[0021] Furthermore, the Kmeans algorithm is used to perform cluster analysis on the samples to obtain anchor candidate boxes and assign them to three feature scales. The specific steps are as follows:

[0022] Step A: Initialize k anchor boxes by randomly selecting k values ​​from all the labeled bounding boxes as the initial values ​​of the k anchor boxes, where k is a set value, anchor box is the anchor candidate box, and bounding box is the detection box labeled in the dataset;

[0023] Step B: Calculate the IOU of each bounding box and anchor box respectively;

[0024] The intersection-to-union ratio is calculated as the ratio of the intersection and union of the anchor box and the bounding box:

[0025]

[0026] In the formula, ∩ means taking the intersection, ∪ means taking the union, and IOU means the intersection-union ratio;

[0027] Calculate the distance parameter d, which represents its error:

[0028] d = 1-IOU

[0029] Step C: After the calculation in the previous step, the error d(n,k) of each bounding box for each anchor box can be obtained. d(n,k) represents the error distance between the nth bounding box and the kth anchor box.

[0030] By comparing the error distance set {d(i,1), d(i,2), …, d(i,k)} of the i-th bounding box to each anchor box, the anchor box with the minimum error is selected and classified into the classification set of the anchor box with the minimum error;

[0031] Classify each bounding box and obtain the classification set of each anchor box;

[0032] Step D: For each anchor box’s bounding box in the classification set, find the width and height median of the bounding box in the classification set and use it as the new size of the anchor box;

[0033] Repeat steps BD and compare the anchor box to which the newly calculated bouding box belongs with the anchor box to which the original bouding box belongs. If they are consistent, stop. If they are inconsistent, jump to repeat steps BD and continue the calculation.

[0034] Furthermore, the target detection network is an improved YOLO v4 network, comprising the following parts:

[0035] Backbone network: used to extract feature maps of infrared image data;

[0036] Feature extraction network: used to further extract feature maps of infrared image data;

[0037] Feature fusion network: used to fuse feature maps;

[0038] Detection box selection network: used to select the detection box;

[0039] The construction method of the target detection network includes the following steps:

[0040] Drawing on the idea of ​​post-fusion of CSPNet, a new feature extraction network is used to replace the original continuous DBL convolution module; the feature maps of the two paths are extracted and fused separately, one path uses 1*1 convolution to retain low-level semantic information, and the other path obtains high-dimensional semantic features through multi-layer convolution pooling activation layers, and finally spliced ​​together through shortcuts;

[0041] A new feature fusion module is proposed to fuse feature maps of three scales to improve the accuracy and speed of the object detection network;

[0042] The input size of the target detection network is set to the original size of the infrared thermal image data.

[0043] Furthermore, the backbone network is CSPdarknet; the feature extraction network is CSPDarknet53;

[0044] The CSPDarknet includes a CSPX module; the CSPX module uses 1×1 convolution to decompose the input features into a trunk and a side branch, wherein the trunk is a residual structure of darknet, including 5 residual blocks, and the side branch is a feature transformation branch composed of 1×1 convolution, and feature fusion is achieved by cascading;

[0045] CSPDarknet uses the Mish activation function as the nonlinear unit of the network, and the expression is as follows:

[0046] f(x) = x·tanh(ln(1+e x ))

[0047] In the formula, · is the multiplication sign, x is the parameter information in the feature map, and f(x) is the output of the activation function.

[0048] Furthermore, the feature fusion module uses the FPN and ASFF feature fusion algorithms to fuse the feature maps of the three scales, specifically including the following steps:

[0049] Use large, medium and small scales to detect objects of different sizes, and fuse the feature information of the three scales;

[0050] For inconsistent channel numbers, a 1*1 convolution layer is used to compress the feature channels into the same layer.

[0051] When upsampling, transposed convolution is used for upsampling;

[0052] When downsampling, for a 1 / 2 ratio downsampling, use a 3*3 convolutional layer with a stride of 2, and modify the number of channels and resolution at the same time;

[0053] For the 1 / 4 scale, a max pooling layer with stride 2 is added after the 2-stride convolution.

[0054] Furthermore, the target detection network further includes a loss function for evaluating the degree to which the predicted value of the model is different from the true value, and the loss function includes: a bounding box regression loss function, a classification loss function, and a confidence loss function;

[0055] Among them, the classification loss function and confidence loss function use cross entropy; the regression loss function uses CIOU loss function;

[0056] The CIOU loss function takes into account the overlapping area, center point distance, and aspect ratio, where d is the Euclidean distance between the center points of the predicted box and the true box, c represents the diagonal distance, α is the weight parameter, and ν is a parameter to measure the consistency of the aspect ratio; the specific formula is as follows:

[0057]

[0058]

[0059]

[0060] In the formula, arctan is a trigonometric function, d is the Euclidean distance between the center points of the predicted box and the true box, c represents the diagonal distance; α is the weight parameter, h and h gt represents the height of the two boxes, ν is a parameter to measure the consistency of aspect ratio, and L CIOU is the loss function.

[0061] Furthermore, the detection box selection network adopts a non-maximum suppression algorithm, which uses the DIoU intersection and union value as a criterion. The process of selecting the detection box is as follows:

[0062] Get the resulting bounding box;

[0063] Select the bounding box with the highest confidence and put it into the set;

[0064] Traverse B i , calculate each B i The DIoU with the candidate box is used to suppress the candidate box using the following formula, and finally the score of the candidate box and the candidate box is obtained;

[0065]

[0066]

[0067] Among them, B i represents the i-th bounding box, S i is the confidence score obtained by the classification of the bounding box, ε is the preset threshold, M is the bounding box with the highest confidence obtained by argmax, RDIoU is the distance between the center points of the two candidate boxes, b and b gt Represents the center point of the anchor box and the target box, and c is the diagonal length of the smallest box containing the two boxes.

[0068] In a second aspect, the present invention provides an infrared image gas and liquid leakage detection system based on deep learning, comprising:

[0069] Acquisition module: used to collect infrared thermal image data in gas and liquid leakage scenarios;

[0070] Training module: used to pre-process infrared thermal image data in gas leakage and liquid leakage scenarios, mark gas leakage and liquid leakage information on infrared thermal image data, obtain training data sets, and use the training data sets to train the target detection network to obtain the weights of the trained target detection network;

[0071] Detection module: used to obtain and pre-process the infrared thermal imaging data of the scene to be detected, load the trained target detection network weights, perform real-time detection on the infrared thermal imaging data of the scene to be detected input by the target detection network, and obtain the detection results;

[0072] Output module: used to visualize the detection results.

[0073] In a third aspect, the present invention provides an infrared image gas and liquid leakage detection system based on deep learning, including a processor and a storage medium;

[0074] The storage medium is used to store instructions;

[0075] The processor is used to operate according to the instructions to execute the steps of the method described in the first aspect.

[0076] Compared with the prior art, the present invention has the following beneficial effects:

[0077] 1. The present invention is based on a deep learning target detection algorithm, uses field data to train a neural network, and uses the characteristics of neural networks that are faster, more accurate, and more stable than traditional methods to improve the degree of automation and accuracy of detection; the present invention effectively solves the problem of gas and liquid leakage detection in industrial scenarios through a deep learning method.

[0078] 2. The present invention uses an image enhancement algorithm based on morphological filtering to suppress infrared image noise and enhance partial edge information of the image.

[0079] 3. In the neck structure of the original YOLO v4, there are a large number of CBL convolution modules. The network structure of this module is changed to CSPDarknet in the present invention to reduce the amount of calculation. The feature map in the target detection network is used as input, and the features of the two paths are extracted and fused by convolution. One path uses 1*1 convolution to retain the low-level semantic information, and the other path obtains high-dimensional semantic features through multi-layer convolution pooling activation layers, and finally spliced ​​together through shortcuts.

[0080] 4. The present invention uses IR-HiNet thermal imager and excellent imaging processing circuit, which has the characteristics of small size, low power consumption, fast startup, excellent imaging quality, accurate temperature measurement, etc. HiNet-640 is an uncooled vanadium oxide infrared detector that can generate 640*512 pixel thermal images, which can make thermal imaging more accurate and still show more details at a longer distance.

[0081] 5. In order to solve the problem of multi-scale target detection in complex scenes and improve the accuracy of the model, the present invention proposes a new feature fusion network to fuse the feature information of three scales. Shallow features can retain spatial information, which is conducive to correctly locating pixels and generating more accurate target detection frames. However, as the depth of the feature layer increases, the spatial resolution of the image is constantly decreasing, and this accurate spatial information is also constantly lost. However, it has rich semantic information. For this feature inconsistency, the feature fusion network is used to learn spatial information to suppress the inconsistency of different feature maps, thereby improving the scale invariance of the feature.

[0082] 6. The network structure diagram of the feature fusion module proposed in the present invention. YOLO downsamples and outputs 1 / 8, 1 / 16, and 1 / 32 times respectively through the feature extraction of the backbone network. The shallow features can retain spatial information, which is conducive to correctly locating pixels and generating more accurate target detection frames. However, as the depth of the feature layer increases, the image spatial resolution is constantly decreasing, and this accurate spatial information is also constantly lost. However, it has rich semantic information. For this feature inconsistency, the feature fusion network is used to learn spatial information to suppress the inconsistency of different feature maps, thereby improving the scale invariance of the feature. Since the three levels of features in YOLO v4 have different resolutions and different numbers of channels, the upsampling and downsampling strategies for each scale are improved accordingly. It can be downsampled from 1 / 8 to 1 / 32, and each feature map is finally upsampled or downsampled by other feature maps and spliced ​​and fused with itself. Through this fusion method, the inconsistency of each scale is greatly suppressed. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 The infrared thermal imager sensor of the present invention;

[0084] Figure 2 This is a result diagram of using the image enhancement technology in the present invention;

[0085] Figure 3 This is a schematic diagram of the network of the improved feature extraction module in the present invention;

[0086] Figure 4 This is a network structure diagram of the feature fusion module proposed in the present invention;

[0087] Figure 5 This is the visualization result of the single-frame image detection of the present invention.

[0088] Figure 6 It is a schematic diagram of the backbone network in the present invention;

[0089] Figure 7 This is a diagram of the four-fold upsampling network structure of the feature extraction module in the present invention;

[0090] Figure 8 This is a diagram of the four-fold downsampling network structure of the feature extraction module in the present invention;

[0091] Fig. 9 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0092] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0093] Embodiment 1:

[0094] The present embodiment provides an infrared image gas leakage and liquid leakage detection method based on deep learning. In view of the characteristics of low infrared image resolution, low signal-to-noise ratio, and uneven infrared thermal images, the present invention uses image enhancement technology to reduce noise to a certain extent, smooth object contours, and enhance contrast. At the same time, a feature extraction module and a feature fusion module are proposed to improve the learning ability and detection speed of the target detection network for infrared image features. Specifically, first, an infrared thermal imager IR-HiNet-640 is used to collect video data in gas leakage and liquid leakage scenes and parse them to obtain infrared thermal image data. Secondly, the image expansion and corrosion operation is used to suppress background noise, and the image sharpening operation, namely the Laplace operator, is used to enhance the image mutation information, details, and edge information. And the annotation software is used to annotate to obtain the final data set. Step 3: Use Kmeans algorithm to cluster samples to obtain anchor candidate boxes and assign them to three feature scales. Improve the target detection network YOLO v4 network. Learn from the idea of ​​CSPNet post-fusion, propose a feature extraction module to replace the original continuous DBL convolution module to reduce network parameters and increase gradient information, and propose a new feature fusion module to fuse the feature maps of three scales to improve the accuracy of the target detection network. The regression box loss function uses CioU loss and introduces NMS-CIoU non-maximum suppression algorithm. At the same time, in order to avoid the problem of overfitting in network training and reduce the training time of the network, the weights trained by MSCOCO dataset are used as the initial weights, and the network is trained using infrared thermal imaging dataset to obtain the final training network parameter weight information. Finally, load the network weight information, read the infrared thermal imaging video stream, and in order to reduce the redundant calculation amount, modify some network content to adapt to the original size (640*512) of the original data directly input into the target detection network for real-time detection, and visualize the detection results. Compared with traditional methods of gas and liquid leakage detection and other sensor detection methods, the present invention achieves good detection results in a self-made data set.

[0095] Specifically, the method comprises the following steps:

[0096] Step 1: Use the infrared thermal imager IR-HiNet-640 to collect video data in the gas and liquid leakage scene and analyze it to obtain infrared thermal image data.

[0097] Step 2: Use morphological filtering operations such as image dilation, erosion, and image sharpening to enhance the image, reduce noise interference in infrared thermal imaging, and use annotation software to annotate the image to obtain the final data set.

[0098] Step 3: Use the Kmeans algorithm to cluster the samples to obtain anchor candidate boxes and assign them to three feature scales. Improve the target detection network YOLO v4 network, learn from the idea of ​​CSPNet post-fusion, use a new feature extraction network to replace the original continuous DBL convolution module, and propose a new feature fusion module to fuse the feature maps of the three scales to improve the accuracy and speed of the target detection network. At the same time, in order to avoid the problem of overfitting in network training and reduce the training time of the network, use the weights trained by the MS COCO dataset as the initial weights, use the infrared thermal imaging dataset to train the network, and obtain the final training network parameter weight information.

[0099] Step 4: Load the network weight information and read the infrared thermal imaging video stream. In order to reduce calculations, modify the network to input the original size (640*512) data into the target detection network for real-time detection and visualize the detection results.

[0100] Specifically, in step one, an infrared thermal imager is used to collect video data of gas leakage and liquid leakage scenes and analyze them to obtain infrared thermal image data. The infrared thermal imager uses an IR-HiNet-640 device, and the resolution of a single thermal imaging image obtained is 640*512, and the measurable temperature range is -22°C to +550°C.

[0101] Specifically, in step 2, the morphological filtering operation is to move the structural element on the original image, perform logical operations on the structural element and the image, and then output the image. Assuming that the input image A, B represents the structural element, z represents a set of real integers, and the definition of using B to dilate and erode A is:

[0102]

[0103]

[0104] As can be seen from the above formula, the dilation operation is to first image the structure element B at the origin, then displace the image on A by z, perform an OR operation on the structure element B and the covered A, and finally obtain the output result. The entire dilation operation process is similar to the convolution operation process described above. After processing the image with the dilation operation, the small breaks in the image can be turned into connected areas. The erosion operation is to translate the structure element B by z, and the output result is the set of all points z contained in A by B. The erosion operation eliminates details in the image that are smaller than the size of the structure element by controlling the size of the structure element.

[0105] Specifically, in step 2, the image sharpening operation uses the Laplace operator. The Laplace operator is a second-order differential operator in n-dimensional Euclidean space. It is an integral transform commonly used in engineering mathematics and belongs to the spatial sharpening filter operation. The differential form of the Laplace operator is:

[0106]

[0107] The form converted to filter mask is:

[0108] 0 1 0 1 -4 1 0 1 0

[0109] The mask has the same result in the four 90-degree directions of up, down, left, and right, which means it has no directionality in the 90-degree direction. In order to make the mask have this property in the 45-degree direction, the filter mask is expanded and defined as:

[0110] 1 1 1 1 -8 1 1 1 1

[0111] Specifically, in step 3, the Kmeans algorithm is used to perform cluster analysis on the samples to obtain anchor candidate boxes and assign them to three feature scales. The specific method steps are as follows:

[0112] (1) Initialize k anchor boxes by randomly selecting k values ​​from all bounding boxes as the initial values ​​of k anchor boxes, where k is 9.

[0113] (2) Calculate the intersection over union value of each bounding box and each anchor box. Iou is calculated as the ratio of the intersection and union of the anchor box and the bounding box:

[0114]

[0115] Since the larger the iou value, the better, the distance parameter d is defined to represent its error:

[0116] d = 1-IOU

[0117] (3) After the calculation in the previous step, we can get the error d(n,k) of each bounding box with respect to each anchor box. We compare the error sizes of each bounding box with respect to each anchor box {d(i,1), d(i,2), …, d(i,k)}, select the anchor box with the smallest error, and assign this bounding box to it. We perform this operation for each bounding box and finally record which bounding boxes belong to each anchor box.

[0118] (4) For the bounding boxes in each anchor box, we calculate the width and height median of these bounding boxes and use them as the new size of the anchor box. Repeat steps 2 to 4 until it is found in step 4 that the anchor box class to which all bounding boxes belong is the same as the anchor box class to which they belonged before. The final result is as follows:

[0119] Anchor box 1 Anchor box 2 Anchor box 3 feature Figure 1 (120,113) (176,207) (243,276) feature Figure 2 (32,61) (62,45) (59,119) feature Figure 3 (10,13) (16,30) (33,23)

[0120] Specifically, in step three, the feature extraction network uses CSPDarknet. CSPDarknet is a modular network composed of CSPX. The CSPX module uses 1×1 convolution to decompose the input features into the trunk and side branches. The trunk is the residual structure of darknet, which contains X residual blocks. The side branches are feature transformation branches composed of 1×1 convolutions. The purpose is to match the number of channels, and finally achieve feature fusion through cascading. CSPDarknet uses the Mish activation function as the nonlinear unit of the network. Compared with the ReLU activation function, its computational complexity is relatively large, but it shows better results than ReLU in deep neural networks such as CSPDarknet. Its characteristics are no upper limit, but a lower limit, smoothness, and non-monotonicity. The specific expression is as follows:

[0121] f(x) = x·tanh(ln(1+e x ))

[0122] Specifically, in step 3, in the neck structure of YOLO v4, there are a large number of CBL convolution modules. The input features of this structure are usually the high-level output features of CSPDarknet or the cascade output results of the path aggregation network. They have a large feature size and number of channels. Directly using 3*3 convolution to extract features will introduce a large amount of parameters and computational complexity. Therefore, drawing on the idea of ​​post-fusion in CSPNet, a new feature extraction module is designed to replace the continuous CBL module in the original network, and the features of the two paths are extracted and fused. Specifically, one path uses 1*1 convolution to retain low-level semantic information and reduce calculations. The other path obtains high-dimensional semantic features through multi-layer convolution pooling activation layers, in which the activation layer uses the Leaky-Relu activation function. Finally, they are spliced ​​together through shortcuts. This cross-connected network can obtain richer gradient information, greatly reduce the amount of calculation, and improve the reasoning speed and accuracy of the network.

[0123] Specifically, Figure 6 As shown, in step three, YOLOv4 follows the structural design of YOLOv3 and uses three scales to detect targets of different sizes, among which the feature map of 1 / 32 size has a large receptive field and rich semantic information, which is suitable for detecting large-sized targets; the feature map of 1 / 8 size has a large image resolution and a strong ability to represent geometric detail information, which is suitable for detecting small-sized targets, and the feature map of 1 / 16 size is used to detect objects of general size. In order to solve the problem of multi-scale target detection in complex scenes and improve the accuracy of the model, the present invention proposes a new feature fusion network to fuse the feature information of three scales. Shallow features can retain spatial information, which is conducive to correctly locating pixels and generating more accurate target detection frames. However, as the depth of the feature layer increases, the image spatial resolution is constantly decreasing, and this accurate spatial information is also constantly lost, but it has rich semantic information. For this feature inconsistency, the feature fusion network is used to learn spatial information to suppress the inconsistency of different feature maps, thereby improving the scale invariance of the feature.

[0124] Since the features of the three levels in YOLO v4 have different resolutions and different numbers of channels, the upsampling and downsampling strategies of each scale are improved accordingly. For the case of inconsistent channel numbers, a 1*1 convolution layer can be applied to compress the number of feature channels into the same layer.

[0125] When upsampling, Figure 7As shown, upsampling is performed by transposed convolution. Transposed convolution is a learning-based method that can better learn network parameters than linear interpolation, with less information loss and greater mapping ability. In the present invention, a transposed convolution will perform two times upsampling to improve resolution, and two transposed convolution operations are required for 1 / 32 layer to 1 / 8 layer, respectively.

[0126] When downsampling, for the 1 / 2 ratio, we first simply use a 3*3 convolution layer with a stride of 2, while modifying the number of channels and resolution. For the 1 / 4 ratio, we add a max pooling layer with a stride of 2 after the 2-step convolution. The four-fold downsampling process is as follows Figure 8 shown.

[0127] Specifically, in step 4, the network weight information is loaded, the infrared thermal imaging video stream is read, the network is modified to input the thermal imaging data of the original size (640*512) into the target detection network for real-time detection, and the detection results are visualized using visualization tools. The feature map size obtained after the image is downsampled by 1 / 32 of the backbone network is 20*16, and the feature map sizes corresponding to 1 / 8 and 1 / 16 are 80*64 and 40*32 respectively. Since the number of categories is 2, the number of channels is 21, and the calculation formula is ((2+5)*3). 2 is the number of categories, 5 contains coordinate-related information (w, h, x, y) and the presence or absence of targets, and 3 is feature maps of three different scales.

[0128] Specifically, the loss function mainly includes three parts: bounding box regression loss function, classification loss function, and confidence loss function. The classification loss function and confidence loss function use cross entropy, while the regression loss function uses CIOU loss function. The CIOU loss function takes into account geometric information such as overlapping area, center point distance, and aspect ratio, where d is the Euclidean distance between the center point of the predicted box and the true box, and c represents the diagonal distance. α is the weight parameter, and ν is a parameter to measure the consistency of the aspect ratio. The specific formula is as follows:

[0129]

[0130]

[0131]

[0132] Specifically, in the original NMS, the IoU metric is used to suppress redundant detection boxes, but since only the overlapping area is considered, it often causes incorrect suppression, especially when the bbox is included. Therefore, DIoU can be used as the NMS criterion, which not only considers the overlapping area but also the center point distance. The specific expression is as follows, where B i represents the i-th bounding box, Si is the confidence score obtained by the classification of the bounding box, ε is the preset threshold, and M is the bounding box with the highest confidence obtained by argmax. First, select the candidate box with the highest confidence and put it into the set, then traverse B i , calculate each B i The DIoU with the candidate box is used to suppress the candidate box using the following formula, and finally the candidate box consistent with NMS and the score of the candidate box are obtained.

[0133]

[0134]

[0135] like Figure 1 The following figure shows the IR-HiNet thermal imager. 1 is a high-performance infrared lens, 2 is a low-noise uncooled infrared movement, and it uses an excellent imaging processing circuit. It has the characteristics of small size, low power consumption, fast startup, excellent imaging quality, and accurate temperature measurement. HiNet-640 is an uncooled vanadium oxide infrared detector that can generate a 640*512 pixel thermal image, which can make thermal imaging more accurate and still show more details at a longer distance.

[0136] like Figure 2 As shown, it is the result diagram of image enhancement in the present invention, wherein Figure 2 Shanghe Figure 2 The left picture below is the original picture. Figure 2 The upper right side shows the result after Laplace operator enhancement. Figure 2 The lower middle and right are the results after corrosion and expansion respectively.

[0137] like Figure 3 As shown in the figure, the cross-connection fusion network designed in the present invention extracts the features of the two paths and fuses them. Specifically, one path uses 1*1 convolution to retain the low-level semantic information and reduce the calculation. The other path obtains high-dimensional semantic features through multi-layer convolution pooling activation layers, and finally splices them together through shortcuts. This cross-connection network can obtain richer gradient information. This module can replace the continuous CBL module in the original network, greatly reducing the amount of calculation and improving the reasoning speed and accuracy of the network.

[0138] like Figure 4As shown in the figure, it is the network structure diagram of the feature fusion module proposed in the present invention. YOLO downsamples and outputs 1 / 8, 1 / 16, and 1 / 32 times respectively through the feature extraction of the backbone network. We fuse the feature information of these three scales. Shallow features can retain spatial information, which is conducive to correctly locating pixels and generating more accurate target detection frames. However, as the depth of the feature layer increases, the spatial resolution of the image is constantly decreasing, and this accurate spatial information is also constantly lost. But it has rich semantic information. For this feature inconsistency, the feature fusion network is used to learn spatial information to suppress the inconsistency of different feature maps, thereby improving the scale invariance of the feature. Since the three levels of features in YOLO v4 have different resolutions and different numbers of channels, the upsampling and downsampling strategies of each scale are improved accordingly.

[0139] When upsampling, first apply a 1*1 convolution layer to compress the number of feature channels into the same layer, and then use the deconvolution method to increase the resolution separately. For downsampling at a ratio of 1 / 2, first simply use a 3*3 convolution layer with a stride of 2, and modify the number of channels and resolution at the same time. For a ratio of 1 / 4, add a maximum pooling layer with a stride of 2 after the 2-step convolution, so that it can be downsampled from 1 / 8 to 1 / 32. Each feature map is finally obtained by upsampling or downsampling other feature maps and concatenating and fusion with itself. Through this fusion method, the inconsistency of each scale is greatly suppressed.

[0140] Embodiment 2:

[0141] This embodiment provides an infrared image gas and liquid leakage detection system based on deep learning, including:

[0142] Acquisition module: used to collect infrared thermal image data in gas and liquid leakage scenarios;

[0143] Training module: used to pre-process infrared thermal image data in gas leakage and liquid leakage scenarios, mark gas leakage and liquid leakage information on infrared thermal image data, obtain training data sets, and use the training data sets to train the target detection network to obtain the weights of the trained target detection network;

[0144] Detection module: used to obtain and pre-process the infrared thermal imaging data of the scene to be detected, load the trained target detection network weights, perform real-time detection on the infrared thermal imaging data of the scene to be detected input by the target detection network, and obtain the detection results;

[0145] Output module: used to visualize the detection results.

[0146] The device of this embodiment can be used to implement the steps of the method described in the first embodiment.

[0147] Embodiment three:

[0148] The embodiment of the present invention also provides an infrared image gas and liquid leakage detection system based on deep learning, including a processor and a storage medium;

[0149] The storage medium is used to store instructions;

[0150] The processor is used to operate according to the instructions to execute the steps of the method described in embodiment 1.

[0151] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0152] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0153] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0154] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0155] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for detecting gas and liquid leakage using infrared images based on deep learning, characterized in that: The following steps are involved: Collect infrared thermal image data in gas and liquid leakage scenarios; Preprocess the infrared thermal image data in the gas leakage and liquid leakage scene, mark the gas leakage and liquid leakage information on the infrared thermal image data, obtain a training data set, and use the training data set to train the target detection network to obtain the trained target detection network weights; Acquire and preprocess the infrared thermal imaging data of the scene to be detected, load the trained target detection network weights, perform real-time detection on the infrared thermal imaging data of the scene to be detected input into the target detection network, and obtain the detection results; Visualize and output the detection results; The method of training an object detection network using a training dataset includes the following steps: The Kmeans algorithm is used to perform cluster analysis on the training data set to obtain the sizes of anchor candidate boxes, and then they are assigned to the large, medium, and small feature scales of the target detection network. Using the network weights trained with the MS COCO dataset as the initial weights of the target detection network, and using the training dataset to train the target detection network to obtain the final trained network parameter weight information; The Kmeans algorithm is used to perform cluster analysis on the samples to obtain anchor candidate boxes and assign them to three feature scales. The specific steps are as follows: Step A: Initialize k anchor boxes by randomly selecting k values ​​from all the annotated bounding boxes as the initial values ​​of the k anchor boxes, where k is a set value, the anchor box is the anchor candidate box, and the bounding box is the detection box annotated in the dataset; Step B: Calculate the IOU of each bounding box and anchor box respectively; The intersection-to-union ratio is calculated as the ratio of the intersection and union of the anchor box and the bounding box: In the formula, ∩ means taking the intersection, ∪ means taking the union, and IOU means the intersection-union ratio; Calculate the distance parameter d, which represents its error: d = 1-IOU Step C: After the calculation in the previous step, the error d(n,k) of each bounding box for each anchor box can be obtained. d(n,k) represents the error distance between the nth bounding box and the kth anchor box. By comparing the error distance set {d(i,1), d(i,2), …, d(i,k)} of the i-th bounding box to each anchor box, the anchor box with the minimum error is selected and classified into the classification set of the anchor box with the minimum error; Classify each bounding box and obtain the classification set of each anchor box; Step D: For each anchor box’s bounding box in the classification set, find the width and height median of the bounding box in the classification set and use it as the new size of the anchor box; Repeat steps BD and compare the anchor box to which the newly calculated bouding box belongs with the anchor box to which the original bouding box belongs. If they are consistent, stop. If they are inconsistent, jump to repeat steps BD and continue the calculation.

2. The infrared image gas and liquid leakage detection method according to claim 1, characterized in that: The method for acquiring infrared thermal image data includes: calibrating an infrared thermal imager to obtain parameter information of the infrared thermal imager, and using the infrared thermal imager to collect video data of a scene to be detected and analyzing the video data to obtain infrared thermal image data of the scene to be detected; The infrared thermal imager is an IR-HiNet-640 device, and the resolution of a single thermal image obtained is 640*512, and the measurable temperature range is -22℃ to +550℃; The method for preprocessing infrared thermal image data comprises the following steps: The infrared thermal image data is enhanced to reduce noise interference in the infrared thermal imaging; Methods for image enhancement of infrared image data include morphological filtering operations; The morphological filtering operations include image dilation, erosion and image sharpening.

3. The infrared image gas and liquid leakage detection method according to claim 1, characterized in that: The target detection network is an improved YOLO v4 network, which includes the following parts: Backbone network: used to extract feature maps of infrared image data; Feature extraction network: used to further extract feature maps of infrared image data; Feature fusion network: used to fuse feature maps; Detection box selection network: used to select the detection box; The construction method of the target detection network includes the following steps: Drawing on the idea of ​​post-fusion of CSPNet, a new feature extraction network is used to replace the original continuous DBL convolution module; feature maps of the two paths are extracted and fused separately, one using 1*1 convolution to retain low-level semantic information, and the other using multi-layer convolutional pooling activation layers to obtain high-dimensional semantic features, and finally spliced ​​together through shortcuts; A new feature fusion module is proposed to fuse feature maps of three scales to improve the accuracy and speed of the object detection network; The input size of the target detection network is set to the original size of the infrared thermal image data.

4. The infrared image gas and liquid leakage detection method according to claim 3 is characterized in that: The backbone network is CSPdarknet; the feature extraction network is CSPDarknet53; The CSPDarknet includes a CSPX module; the CSPX module uses 1×1 convolution to decompose the input features into a trunk and a side branch, wherein the trunk is a residual structure of darknet, including 5 residual blocks, and the side branch is a feature transformation branch composed of 1×1 convolution, and feature fusion is achieved by cascading; CSPDarknet uses the Mish activation function as the nonlinear unit of the network.

5. The infrared image gas and liquid leakage detection method according to claim 3 is characterized in that: The feature fusion module uses the FPN and ASFF feature fusion algorithms to fuse the feature maps of the three scales, specifically including the following steps: Use large, medium and small scales to detect objects of different sizes, and fuse the feature information of the three scales; For inconsistent channel numbers, a 1*1 convolution layer is used to compress the feature channels into the same layer. When upsampling, transposed convolution is used for upsampling; When downsampling, for a 1 / 2 ratio downsampling, use a 3*3 convolutional layer with a stride of 2, and modify the number of channels and resolution at the same time; For the 1 / 4 scale, a max pooling layer with stride 2 is added after the 2-stride convolution.

6. The infrared image gas and liquid leakage detection method according to claim 3 is characterized in that: The target detection network also includes a loss function for evaluating the degree to which the predicted value of the model is different from the true value, and the loss function includes: a bounding box regression loss function, a classification loss function, and a confidence loss function; Among them, the classification loss function and confidence loss function use cross entropy; the regression loss function uses CIOU loss function; The detection box selection network adopts a non-maximum suppression algorithm, and the non-maximum suppression algorithm uses the DIoU intersection and union value as a standard.

7. An infrared image gas and liquid leakage detection system based on deep learning, characterized in that: include: Acquisition module: used to collect infrared thermal image data in gas and liquid leakage scenarios; Training module: used to pre-process infrared thermal image data in gas leakage and liquid leakage scenarios, mark gas leakage and liquid leakage information on infrared thermal image data, obtain training data sets, and use the training data sets to train the target detection network to obtain the weights of the trained target detection network; Detection module: used to obtain and pre-process the infrared thermal imaging data of the scene to be detected, load the trained target detection network weights, perform real-time detection on the infrared thermal imaging data of the scene to be detected input by the target detection network, and obtain the detection results; Output module: used to visualize the detection results; The method of training an object detection network using a training dataset includes the following steps: The Kmeans algorithm is used to perform cluster analysis on the training data set to obtain the sizes of anchor candidate boxes, and then they are assigned to the large, medium, and small feature scales of the target detection network. Using the network weights trained with the MS COCO dataset as the initial weights of the target detection network, and using the training dataset to train the target detection network to obtain the final trained network parameter weight information; The Kmeans algorithm is used to perform cluster analysis on the samples to obtain anchor candidate boxes and assign them to three feature scales. The specific steps are as follows: Step A: Initialize k anchor boxes by randomly selecting k values ​​from all the annotated bounding boxes as the initial values ​​of the k anchor boxes, where k is a set value, the anchor box is the anchor candidate box, and the bounding box is the detection box annotated in the dataset; Step B: Calculate the IOU of each bounding box and anchor box respectively; The intersection-to-union ratio is calculated as the ratio of the intersection and union of the anchor box and the bounding box: In the formula, ∩ means taking the intersection, ∪ means taking the union, and IOU means the intersection-union ratio; Calculate the distance parameter d, which represents its error: d = 1-IOU Step C: After the calculation in the previous step, the error d(n,k) of each bounding box for each anchor box can be obtained. d(n,k) represents the error distance between the nth bounding box and the kth anchor box. By comparing the error distance set {d(i,1), d(i,2), …, d(i,k)} of the i-th bounding box to each anchor box, the anchor box with the minimum error is selected and classified into the classification set of the anchor box with the minimum error; Classify each bounding box and obtain the classification set of each anchor box; Step D: For each anchor box’s bounding box in the classification set, find the width and height median of the bounding box in the classification set and use it as the new size of the anchor box; Repeat steps BD, compare the anchorbox to which the newly calculated boudingbox belongs with the anchorbox to which the original bouding box belongs, and stop if they are consistent. If they are inconsistent, jump to repeat steps BD and continue the calculation.

8. An infrared image gas and liquid leakage detection system based on deep learning, including a processor and a storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 6.

Images