Visual confirmation method for leak hunting action in filling process of liquid tank truck

Abstract

The invention discloses a visual confirmation method for a leak hunting action in a filling process of a liquid tank truck, and the method comprises the steps: recognizing and confirming the leak hunting action of an operator on a crane pipe connection region based on a video collected by a camera in the filling process of the liquid tank truck; the method is used for monitoring and confirming whether personnel leak hunting operation is completed according to requirements before tank car filling. The method comprises the following steps: performing target detection by adopting a trained deep learning model, identifying a vehicle, an operation box, a crane pipe, a sprinkling can and field personnel, and confirming a crane pipe connection scene; a deep learning model is adopted for attitude estimation, and the limb action that a person holds a sprinkling can by hand and sprays soapy water near a crane pipe is recognized; by means of multi-target correlation detection, according to the relative position of the person and the crane pipe connection area, the continuous process that the person observes the bubble condition near the crane pipe is recognized.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision recognition, and in particular to a visual intelligent confirmation method for leak detection during the filling process of a liquid tank truck. Background Art

[0002] In the process of road transportation of liquid hazardous chemicals by liquid tank trucks, the filling of tank trucks is one of the key operations with special risks. Due to the physical and chemical characteristics of liquid hazardous chemicals, once a leak occurs, it may lead to serious production safety accidents, causing significant injuries and losses. Therefore, it is crucial to ensure the tightness of the connection between the tank truck and the filling equipment and the tightness of the pipeline.

[0003] For dangerous chemicals such as Class A and Class B liquids, the use of crane pipes is an important means to ensure safe production, protect the environment, and comply with regulations. The crane pipe is equipped with valves and control devices that can accurately control the flow and stop of the liquid. The operator can control it in a safe position to avoid direct contact with dangerous liquids, which improves the safety and accuracy of the operation. In general operations, after the operator connects the crane pipe, he will perform a leak detection operation on the connection area by spraying soapy water. The specific method is to spray soapy water on the connection and observe whether bubbles are generated to determine whether the connection is tight and whether the airtightness meets the standards. This detection process depends on the standardization of the personnel's operation, and there may be missing or incomplete actions. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to propose a visual intelligent confirmation method for the leak detection action during the filling process of a liquid tank truck. By processing the monitoring video data of the filling area, detecting and confirming the correct completion of the leak detection action process, it is possible to more intelligently and safely monitor personnel to complete the tank leak detection operation and effectively avoid safety accidents. This method monitors the operation area through a camera, collects the operating actions and continuous processes between the operator and the tank truck and crane in real time, and uses image recognition technology to confirm whether key targets appear in the picture. Through this automated visual monitoring method, the completion status of the leak detection action can be effectively identified and determined, the absence of the leak detection action can be automatically detected, and the safety of the operation can be improved.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A visual confirmation method for leak detection during the filling process of a liquid tank truck. During the period when personnel are performing leak detection operations on the tank truck during the filling process, the monitoring video frame rate collected by a fixed-position camera is 30 frames per second, and one frame is taken every 5 frames. For each frame of the image, image processing, target and posture detection, and scene state determination are performed based on a deep learning method. The first step is to identify the vehicle position, locate the operation box and the crane pipe target, determine that the crane pipe is connected to the interface in the tank truck operation box, and determine the position of the crane pipe connection area in the picture; the second step is to detect the posture of the person and the spray pot target in the picture, and identify the action posture of the person holding the spray pot; the third step is to identify the relative position of the person and the operation box and the crane pipe connection, and determine that the person is near the crane pipe connection; the fourth step is to identify the spraying action of the person holding the spray pot; the fifth step is to identify the continuous process of the person staying and observing at the crane pipe connection. Specifically including the following steps:

[0007] S1: Collect monitoring videos of the filling and leak detection operations under various lighting conditions, and capture scene images including vehicles, operating boxes, cranes, spray bottles, and on-site personnel as well as environmental images at a ratio of 7:1;

[0008] S2: Collect monitoring videos of the filling and leak detection operations under various lighting conditions, and capture scene images including vehicles, operating boxes, crane pipes, spray bottles, and on-site personnel as well as environmental images at a ratio of 7:1;

[0009] S3: Use the "transfer learning" method and adaptive learning rate algorithm to train the target detection model to achieve target detection and positioning of vehicles, operating boxes, crane pipes, spray bottles and on-site personnel;

[0010] S4: According to the relative position and angle between the crane pipe and the operation box, it is determined that the crane pipe has been connected to the injection port of the tank truck;

[0011] S5: Run the posture estimation model to extract the key points of the person's body, calculate the angle between the vectors of adjacent key points, and identify related actions such as holding a spray bottle and spraying soapy water;

[0012] S6: Based on the continuous detection results of the relative position of the personnel and the crane pipe and the personnel-related actions, confirm that the personnel conducted continuous observation after spraying soapy water and confirmed that the leak detection operation was completed correctly.

[0013] S7: If the three states and action processes of crane pipe connection, soapy water spraying, and continuous observation are all completed in sequence, it is determined that the personnel have correctly performed the standardized leak detection action.

[0014] The beneficial effects of the present invention are: providing a visual intelligent confirmation method for leak detection during the filling process of a liquid tank truck, and real-time monitoring of personnel performing leak detection operations on the tank body based on the action of spraying soapy water from a watering can. Based on deep learning and posture estimation, personnel and related facilities and equipment are captured, and the missing events of necessary actions in the leak detection process can be detected, the standardization of the leak detection process can be detected, a safety early warning mechanism can be constructed, safety hazards can be effectively eliminated, and the filling operation process of tank truck petrochemical products can be monitored more intelligently and safely. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of the method execution in the present invention.

[0016] Figure 2 Schematic diagram of detection scene examples and targets of the present invention. DETAILED DESCRIPTION

[0017] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings:

[0018] Figure 1 is a flowchart of the method of the present invention. Figure 1 As shown, a tank leak detection operation posture detection method according to the present invention comprises the following steps:

[0019] S1: Under various lighting conditions, 360° video of tanker safety inspection operations on site is collected from four directions without blind spots, and on-site monitoring videos of filling and leak detection operations are collected. Scene images including vehicles, operating boxes, crane pipes, spray bottles, and on-site personnel as well as environmental images are captured at a ratio of 7:1;

[0020] S2: Divide the dataset in a ratio of 8:2:1 to obtain training set, validation set, and test set for machine learning. Then perform target annotation to generate labels and use generative adversarial network technology to expand the dataset to cover small sample targets and achieve generalization effect. By introducing a dynamic partitioning strategy based on data distribution, the training set, validation set, and test set are made more representative, ensuring that each set contains enough sample diversity.

[0021] S3: Use the "transfer learning" method and adaptive learning rate algorithm to train the target detection model to achieve target detection and positioning of vehicles, operating boxes, crane pipes, spray bottles and on-site personnel;

[0022] S4: According to the relative position and angle between the crane pipe and the operation box, it is determined that the crane pipe has been connected to the injection port of the tank truck;

[0023] S5: Run the posture estimation model to extract the key points of the person's body, calculate the angle between the vectors of adjacent key points, and identify related actions such as holding a spray bottle and spraying soapy water;

[0024] S6: Based on the continuous detection results of the relative position of the personnel and the crane pipe and the personnel's related actions, confirm that the personnel have conducted continuous observation after spraying soapy water and confirm that the leak detection operation is completed correctly;

[0025] S7: If the three states and action processes of crane pipe connection, soapy water spraying, and continuous observation are all completed in sequence, it is determined that the personnel have correctly performed the standardized leak detection action.

[0026] The collecting of the leak detection operation site monitoring video and expanding the data set described in S1 includes the following sub-steps:

[0027] S11. Under different lighting conditions, collect 360° videos of safety inspection of tank trucks in chemical plants from four directions without blind spots, intercept scene images including vehicles, operating boxes, crane pipes, spray bottles, and on-site personnel and common scene images from the video stream at a ratio of 7:1, intercept 3,500 images including specific targets and 500 common scene images, and crop each image into several sub-images with a long side of 1,000 and a short side of 600;

[0028] S12. Use the incremental learning method to perform geometric transformation on the extracted initial image and combine it with the generative adversarial network GANs to expand the data set.

[0029] The specific method for geometric transformation of the image described in S12 is:

[0030] Horizontal Flip: Flip the image on the horizontal axis to generate samples with different perspectives.

[0031] Translation: Moving an image along the x-axis or y-axis in the image plane, simulating capturing the image at a different location.

[0032] Random scaling: randomly enlarge or reduce the image to increase the robustness of the model to images of different sizes;

[0033] The steps of expanding the data set by using a generative adversarial network described in S12 are as follows:

[0034] S121. Use the initial image dataset to train a GANs model. GANs consists of two neural networks: a generator and a discriminator. The task of the generator is to generate new images, while the task of the discriminator is to distinguish between generated images and real images.

[0035] S122. Use the generator to generate new image samples that are visually similar to the original dataset but have different details and variations.

[0036] S123, merging the new images generated by GANs with the original dataset (including the images expanded by geometric transformation) to form a larger dataset;

[0037] S2 divides the factory image dataset into 8:2:1 ratios to obtain training sets, validation sets, and test sets, and then annotates and labels them, including the following sub-steps:

[0038] S21, classify the data set according to the monitoring position, ambient light illumination and the number of key targets included;

[0039] S22. Divide the classified data set into a ratio of 8:2:1 to obtain a training set, a validation set, and a test set.

[0040] S23. Use semi-supervised learning for labeling. First, manually label a part of the data, then use these labeled data to train an initial model, and then use this model to automatically label the remaining data, thereby ensuring the labeling quality to a certain extent.

[0041] The training target detection model described in S3 uses the "transfer learning" method to train the YOLOv8 target detection model as follows:

[0042] S31. Design of the network structure of the posture estimation model. The "Real-Time Multi-person One-stage" RTMO is used, and the backbone network uses CSPDarknet to improve the network's learning ability and the model's reasoning speed by reducing the propagation of repeated gradient information.

[0043] The main components and structure of CSPDarknet:

[0044] Conv Layer: A standard convolutional layer that uses a convolution operation with a kernel size of 3x3, with BatchNormalization and Leaky ReLU as activation functions.

[0045] Residual Blocks: The residual blocks in ResNet are retained, and their main function is to solve the gradient vanishing problem in deep networks. Each residual block consists of two to three convolutional layers, and the input information is transmitted through skip connections.

[0046] Cross Stage Partial (CSP) Connection: The CSP structure divides the input feature map into two parts, one part directly passes through the residual block, and the other part skips the residual block and is later spliced ​​with the output feature map of the residual block. Through this division and fusion, CSP can effectively reduce the propagation of repeated gradient information, reduce the amount of calculation and improve the generalization ability of the model.

[0047] Downsampling: Downsampling is performed at the end of each stage, using a convolution or pooling operation with a stride of 2 to halve the resolution of the feature map while increasing the number of channels to extract higher-level features.

[0048] Stage Structure: CSPDarknet consists of multiple stages, each of which contains several CSP modules. As the stage goes deeper, the number of channels of the feature map gradually increases and the resolution gradually decreases.

[0049] SPP Module (Spatial Pyramid Pooling Module): An SPP module is connected at the end of the backbone network to further increase the receptive field and capture multi-scale features.

[0050] The last three feature maps are processed using a hybrid encoder to produce two spatial features with downsampling rates of 16 and 32, respectively. Each pixel in these features is mapped to a grid cell evenly distributed on the original image plane. The network head uses a double convolution block at each spatial level to generate a score and corresponding pose feature for each grid cell. These pose features are used to predict bounding boxes, keypoint coordinates, and visibility, and detailed information for 1-D heatmap prediction is generated through a dynamic coordinate classifier.

[0051] S32. Divide the data set. Use the k-fold cross-validation method to divide the data set marked in S2, that is, divide the data set into k subsets, each subset is used as a test set, and the rest is used as a training set, making full use of the data for model training and evaluation, reducing the randomness of model performance.

[0052] S33, model training and evaluation. The learning rate was tried from 0.001 and adjusted according to the change in loss during training. The batch size determines the number of samples used each time the model parameters are updated. A smaller batch size may make the training process more unstable but help escape the local minimum; a larger batch size may make the training more stable but may sacrifice some generalization ability. For a dataset of 6000 images, try using batch sizes such as 32, 64, or 128 and observe their effects on training speed and stability.

[0053] The steps for estimating the posture of a person described in S4 are as follows:

[0054] S41. Deploy the posture estimation model RTMO in the project and extract 17 key points of the human body in the video, including 0-nose, 1-left eye, 2-right eye, 3-left ear, 4-right ear, 5-left shoulder, 6-right shoulder, 7-left elbow, 8-right elbow, 9-left wrist, 10-right wrist, 11-left hip, 12-right hip, 13-left knee, 14-right knee, 15-left ankle and 16-right ankle.

[0055] S42. According to the coordinates of the key points, the limb vector is obtained, and the inner product formula of the vector is derived. The limb angle is calculated by the inverse tangent function. According to different limb angles and their degrees, the judgment conditions are set to estimate the human body movement. The human body movement is combined with the position of the person and the leak detection operation of the inspector on the tank. The specific calculation method is as follows:

[0056] Bending: Take the center point c1 of the left and right shoulders, the center point c2 of the left and right hips, and the center point c3 of the left and right knees, and calculate the vector and vector The limb angle θ is obtained according to the vector inner product formula.

[0057] Inner product formula:

[0058] Derived θ:

[0059] in, and are the upper body vector and the lower body vector respectively, and θ is the angle between the upper body and the lower body.

[0060] Judgment condition: θ<140°

[0061] Extend your arm: Take the hand key point q1, the arm center point q2, and the shoulder key point q3, and calculate the vector and vector The limb angle α is obtained according to the vector inner product formula.

[0062] Inner product formula:

[0063] Derived α:

[0064] in, and are the forearm vector and the upper arm vector respectively, and α is the angle between the forearm and the upper arm.

[0065] Judgment condition: α>90°

[0066] Spraying: The key points of the hand coincide with the target of the spray bottle, the personnel are located in the crane pipe connection area, holding the spray bottle with the arm extended forward for 3 to 5 seconds to complete the spraying action, and stay and observe for a while to confirm.

Claims

1. A method for visually confirming the leak detection action during the filling process of a liquid tank truck, characterized in that: During the leak detection operation of the tank truck during the filling process, a fixed-position camera is used to collect monitoring video; for each frame of the image, image processing, target and posture detection, and scene status judgment are performed based on deep learning methods: the first step is to identify the vehicle position, locate the operation box and the crane pipe target, determine that the crane pipe is connected to the interface in the tank truck operation box, and determine the position of the crane pipe connection area in the picture; the second step is to detect the posture of the person and the spray pot target in the picture, and identify the movement posture of the person holding the spray pot; the third step is to identify the relative position of the person and the operation box and the crane pipe connection, and determine that the person is near the crane pipe connection; the fourth step is to identify the spraying action of the person holding the spray pot; the fifth step is to identify the continuous process of the person staying and observing at the crane pipe connection.

2. A visual confirmation method for leak detection during filling of a liquid tanker according to claim 1, characterized in that: The specific steps include: S1: Collect monitoring videos of the filling and leak detection operations under various lighting conditions, and capture scene images including vehicles, operating boxes, cranes, spray bottles, and on-site personnel as well as environmental images at a ratio of 7:1; S2: Divide the data set into a ratio of 8:2:1 to obtain the training set, validation set, and test set for machine learning. Then, perform target annotation to generate labels and use generative adversarial network technology to expand the data set to cover small sample targets. S3: The target detection model is trained using the "transfer learning" method and adaptive learning rate algorithm to achieve target detection and positioning of vehicles, operating boxes, crane pipes, spray bottles and on-site personnel; S4: According to the relative position and angle between the crane pipe and the operation box, it is determined that the crane pipe has been connected to the injection port of the tank truck; S5: Run the posture estimation model, extract the key points of the person's body, calculate the angle between the vectors of adjacent key points, and identify a series of related actions including but not limited to holding a spray bottle and spraying soapy water; S6: Based on the continuous detection results of the relative position of the personnel and the crane pipe and the personnel-related actions, confirm that the personnel conducted continuous observation after spraying soapy water and confirmed that the leak detection operation was completed correctly. S7: If the three states and action processes of crane pipe connection, soapy water spraying, and continuous observation are all completed in sequence, it is determined that the personnel have correctly performed the standardized leak detection action.

3. A visual confirmation method for leak detection during filling of a liquid tanker according to claim 2, characterized in that: The step S1 of collecting video images of the production site area is to collect 360° videos of the on-site tank truck safety inspection operations from four directions under different lighting conditions, and to capture images of the vehicle, operating box, crane pipe, spray pot, on-site personnel, and ordinary scene images from the video stream in a 7:1 ratio.

4. A visual confirmation method for leak detection during filling of a liquid tanker according to claim 2, characterized in that: The "expanding the data set using generative adversarial network technology" in step S2 is to use an incremental learning method to perform geometric transformation on the extracted initial image and combine it with the generative adversarial network GANs to expand the data set; The specific methods for geometric transformation of the image include but are not limited to: (1) Horizontal flipping: flip the image on the horizontal axis to generate samples with different perspectives; (2) Translation: moving the image along the x-axis or y-axis on the image plane to simulate the capture of the image at different positions; (3) Random scaling: randomly enlarging or reducing the image to increase the robustness of the model to images of different sizes; The steps of expanding the data set by using a generative adversarial network are as follows: S121. Use the initial image dataset to train a GANs model. GANs consists of two neural networks: a generator and a discriminator. The task of the generator is to generate new images, while the task of the discriminator is to distinguish between generated images and real images. S122. Use the generator to generate new image samples that are visually similar to the original dataset but have different details and variations. S123. Merge the new images generated by GANs with the original dataset (including images expanded by geometric transformation) to form a larger dataset.

5. The visual confirmation method for leak detection during filling of a liquid tanker according to claim 2, characterized in that: The "target label generation" described in step S2 is annotated using semi-supervised learning. First, a part of the data is manually annotated, and then an initial model is trained using the annotated data. The model is then used to automatically annotate the remaining data to ensure the quality of the annotation.

6. A visual confirmation method for leak detection during filling of a liquid tanker according to claim 2, characterized in that: The training target detection model in step S3 is based on the YOLOv8 pre-training model, and uses an adaptive learning rate algorithm to dynamically adjust the learning rate according to the gradient and loss in the model training process. The loss function selects DFL Loss and CIoU Loss to improve the positioning accuracy of the detection box, adopts the transfer learning method to speed up the training model, and sets the early stopping mechanism to avoid overfitting and improve the generalization ability of the model.

7. A visual confirmation method for leak detection during filling of a liquid tanker according to claim 6, characterized in that: The training target detection model in step S3 is specifically performed as follows: S31. Attitude estimation model network structure design: The "Real-Time Multi-person One-stage" RTMO model is adopted, and the backbone network adopts CSPDarknet to improve the network's learning ability and the model's reasoning speed by reducing the propagation of repeated gradient information; S32, divide the data set: Use the k-fold cross-validation method to divide the data set marked in S2, that is, divide the data set into k subsets, each subset is used as a test set, and the rest is used as a training set, making full use of the data for model training and evaluation, reducing the randomness of model performance; S33, model training and evaluation: The learning rate was tried starting from 0.001 and adjusted according to the change in loss during training; the batch size determined the number of samples used each time the model parameters were updated, and the impact of the batch size on the training speed and stability was observed during the training process.

8. The visual confirmation method for leak detection during filling of a liquid tanker according to claim 2, characterized in that: The determination of the crane pipe connection status described in step S4 is based on the target detection and positioning of the tank truck operating box, the target detection and positioning of the crane pipe equipment, and the calculation of the relative distance and overlap degree of the positions of the two targets in the picture, so as to determine the connection status of the crane pipe at the injection port of the operating box.

9. A visual confirmation method for leak detection during filling of a liquid tanker according to claim 2, characterized in that: The posture estimation described in step S5 uses a single-stage posture estimation framework (Real-Time Multi-person One-stage) RTMO, and uses dual one-dimensional heat maps in the YOLO architecture to represent key points to seamlessly integrate coordinate classification to extract the coordinates of key points of the human body, and use these coordinates to calculate the angles and the relative positions of various parts of the human body to determine the posture action: when a person picks up a spray can, the spray can target overlaps with the hand key point; when a person sprays soapy water to detect leaks, the person is near the crane pipe connection area, holding the spray can toward the connection, and the upper arm and forearm are extended forward at an obtuse angle; The continuous observation described in step S6 is achieved by detecting the duration of time that the person holds a spray bottle and stands near the connection point of the crane pipe of the operating box, and calculating: the position of the person, the movement of holding the spray bottle, the position of the operating box and the crane pipe, which are obtained by the aforementioned target detection and posture detection methods.

10. A visual confirmation method for leak detection during filling of a liquid tanker according to claim 9, characterized in that: The specific steps of estimating the posture of a person described in step S5 are as follows: S41. Deploy the posture estimation model RTMO in the project and extract 17 key points of the human body in the video, including 0-nose, 1-left eye, 2-right eye, 3-left ear, 4-right ear, 5-left shoulder, 6-right shoulder, 7-left elbow, 8-right elbow, 9-left wrist, 10-right wrist, 11-left hip, 12-right hip, 13-left knee, 14-right knee, 15-left ankle and 16-right ankle; S42. According to the coordinates of the key points, the limb vector is obtained. The inner product formula of the vector is used to deduce the limb angle through the inverse tangent function. According to different limb angles and their degrees, the judgment conditions are set to estimate the human body movement; the human body movement is combined with the position of the person, and the leak detection operation of the inspector on the tank body; the specific calculation method is as follows: Bending: Take the center point c1 of the left and right shoulders, the center point c2 of the left and right hips, and the center point c3 of the left and right knees, and calculate the vector and vector The limb angle θ is obtained according to the vector inner product formula: Inner product formula: Derived θ: in, and are the upper body vector and the lower body vector respectively, and θ is the angle between the upper body and the lower body; Judgment condition: θ<140° Extend your arm: Take the hand key point q1, the arm center point q2, and the shoulder key point q3, and calculate the vector and vector The limb angle α is obtained according to the vector inner product formula: Inner product formula: Derived α: in, and are the forearm vector and the upper arm vector respectively, and α is the angle between the forearm and the upper arm. Judgment condition: α>90° Spraying: The key points of the hand coincide with the target of the spray bottle, the personnel are located in the crane pipe connection area, holding the spray bottle with the arm extended forward for 3 to 5 seconds to complete the spraying action, and stay and observe for a while to confirm.

Images