Visual detection method for crane pipe connection firmness confirmation action

Abstract

The invention discloses a visual detection method for a crane pipe connection firmness confirmation action, which is used for confirming that an operator pulls and drags a crane pipe according to a monitoring video acquired at a set angle in a petrochemical tank truck capacity filling process, and is used for confirming whether the crane pipe connection is firm or not and enhancing the normative management of the filling process. Firstly, video signals are extracted frame by frame to form images, then image videos are input into a trained machine learning model, a valve area and a crane pipe connector of the petrochemical tank car are detected through a target detection method, and then skeleton key point coordinates of a human body are obtained through a posture estimation human body key point detection method. Relevant information is obtained and stored in a database. On the basis of the detection target and the human body key point information, whether a person confirms the connection firmness of the crane pipe or not is judged through human body key point position change information accompanied by limb movement, and if yes, the amplitude and direction of the action are analyzed.

Description

Technical Field

[0001] The present invention designs a visual detection method for confirming the connection firmness of a crane pipe, belongs to the field of computer vision, and is an innovative application of machine learning and image processing methods in production safety management. Background Art

[0002] During the filling process of petrochemical tank trucks, since the filling operation of tank trucks is a highly dangerous operation, in order to ensure the safety of the filling process, check the firmness of the connection, prevent the leakage of the contents of the tank truck, and reduce safety hazards, the operation of pulling the crane pipe is essential. There are still irregular behaviors in the industry that require human supervision. With the development of deep learning, target detection algorithms are becoming more mature. The R-CNN algorithm proposes a basic framework for target detection. The YOLO algorithm and the SSD algorithm improve the speed and accuracy of target detection. Current research is also focused on designing more efficient network structures, such as EfficientNet and ResNeXt, etc., aiming to improve the computational efficiency and accuracy of the model; use technologies such as generative adversarial networks to generate more diverse data samples, and improve the generalization ability and robustness of the target detection model. The current yolov8 model can meet the background requirements of the present invention.

[0003] Human posture detection is an important research direction in the field of computer vision. Its goal is to identify and locate the key points of the human body from images or videos to capture the posture and movement of the human body. Through mainstream posture detection models such as OpenPose, posenet, hrnet, AlphaPose, yolo-pose, etc., the key points of the human body can be identified with a high accuracy. At the same time, we can combine the classification method of machine learning to convert the coordinate information of the key points of the human body into meaningful information for us, and then add the artificial judgment of whether it is the "holding the crane tube with both hands" posture, to help us solve the problem of identifying the "holding the crane tube with both hands" required by the present invention, and then judge the number and strength of pulling.

[0004] Motion amplitude detection is one of the technologies that is still being optimized for human posture detection. Currently, motion amplitude detection mainly relies on the relative displacement of joints in adjacent video frames. The joint point detection algorithm is used to find the corresponding joint points in the last frame and the first frame, and the displacement mean of each pair of corresponding joint points is calculated. The displacement mean Dis is normalized to obtain the motion amplitude value of the two frames. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, reduce the consumption of human supervision, and make the tank truck filling operation more standardized, the present invention proposes a visual detection method for confirming the firmness of the crane pipe connection. The crane pipe target is detected by the target detection method, and the number and amplitude of pulling are calculated by calculating the horizontal coordinate displacement of a series of shoulder key points. It is used for intelligent monitoring of workers in the filling process of petrochemical liquid tank trucks, reducing danger and saving the cost of human supervision. The operator's pulling posture is identified through the relationship between the key points of the human body, and finally the number and amplitude of pulling are calculated by calculating the mean of the displacement of the key points during the change of the pulling posture, which is conducive to the advancement of industrial supervision automation.

[0006] The technical solution adopted by the present invention is a visual detection method for confirming the connection firmness of a crane pipe, comprising the following steps:

[0007] S1: Obtain surveillance video from a camera at a set angle and obtain images frame by frame.

[0008] S2: Based on the target detection method of yolov8, the data set is annotated and the model is trained to detect the valve area of ​​the tank truck and the connector of the crane pipe, and the detection results of the detection frames with detection targets are saved to the database together with the frame number.

[0009] S3: Use the yolov8-pose model to detect the key point information of the person in each frame and store it in the database.

[0010] S4: Analyze the key point information of the human body in step 3 and the position of the detection target obtained in step 2, and extract the state frame of the person "holding the crane with both hands".

[0011] S5: Analyze the state frame of holding the crane tube, and analyze the number and strength of pulling according to a series of human shoulder coordinate data. And compare it with the set threshold to determine whether it meets the standard for confirming the firmness of the crane tube.

[0012] After the model training is completed and the judgment criteria are set, the complete video detection process includes:

[0013] S1: Detect the starting frame in the video where the crane is located in the valve area (i.e., the operator inserts the crane), and filter the target detection information and human key point information after this frame.

[0014] S2: Set a monitoring area at the location where the crane appears, that is, a rectangular frame of a reasonable size. If the human body posture in the rectangular frame meets the standard of "holding the crane with both hands", record this information in the database.

[0015] S3: For the key point information of a series of frames judged as "holding the crane pipe with both hands", since the amplitude of the action of "pulling the crane pipe" will be reflected in the left and right movement of the human shoulder, the horizontal coordinate information of the left and right shoulder key points of this series of frames is processed to find out the change of adjacent maximum and minimum values ​​of the horizontal coordinate during this period of time and the number of gradually increasing and decreasing swings, and the size and number of this change are compared with the set threshold to determine whether it is a firmness confirmation; at the same time, the key point information of a period of time judged as a firmness confirmation is processed, and the ratio of the change of the horizontal coordinates of two adjacent frames and the frame interval duration is calculated, these values ​​are added up, and compared with the set threshold to determine whether the pulling is large or small.

[0016] The present invention is to confirm that the operator has pulled the crane pipe according to the monitoring video collected at a set angle during the filling process of the petrochemical tank truck, so as to confirm whether the crane pipe is firmly connected and strengthen the standardized management of the filling process. First, the video signal is extracted into images frame by frame, and then the image video is input into the trained machine learning model. The valve area and the crane pipe connector of the petrochemical tank truck are first detected by the target detection method, and then the human body key point detection method is used to obtain the coordinates of the human body's skeletal key points, and the relevant information is obtained and stored in the database. Based on the above detection targets and human body key point information, the human body key point position change information accompanied by limb movement is used to determine whether the person has confirmed the firmness of the crane pipe connection. If so, the amplitude and direction of the action are analyzed. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a single video workflow diagram of the present invention.

[0018] Figure 2 This is an example of the key points of the "holding the crane pipe with both hands" posture.

[0019] Figure 3 These are illustrations of various angles used in the training of the "holding a crane tube with both hands" posture model.

[0020] Figure 4 This is a diagram of the shoulder swing in the "pulling" action. DETAILED DESCRIPTION

[0021] The method is described in detail below.

[0022] A visual detection method for confirming the firmness of a crane pipe connection comprises the following steps:

[0023] S1: Obtain surveillance video from a camera at a set angle and obtain images frame by frame.

[0024] It is required that the angle between the line projected by the camera and the operator on the xoy plane of the vertical three-dimensional coordinate axis and the wall on which the camera is mounted should be greater than 60°, so that the operator's body key points are clearer; the resolution of the surveillance video of the tank truck yard should not be less than 1920*1080, and the frame rate should not be less than 30.

[0025] S2: Based on the target detection method of yolov8, a data set is established, annotated, and a model is trained to detect the valve area of ​​the tank truck and the connector of the crane pipe. The detection results of the detection frames with detection targets are saved to the database together with the frame number.

[0026] The established and trained dataset should be based on on-site operation videos of the site, taking into account the physical differences of workers and the differences in shooting perspectives. 2,000 datasets containing detection targets and 2,000 datasets without detection targets should be selected. Videos shot by different workers and shooting perspectives should be selected for frame sampling and annotation, and the training set, validation set, and test set should be divided into training set, validation set, and test set in a ratio of 8:1:1. The labels of the annotated objects are the tank truck valve area and the crane pipe connector.

[0027] S3: Use the yolov8-pose model to detect the key point information of the person in each frame and store it in the database. S4: Analyze the key point information of the human body in step 3 and the position of the detection target obtained in step 2, extract the state frame of the person holding the crane pipe with both hands, expand the detection frame of the crane pipe connector to the surroundings, obtain an area, and judge the posture of the person in the area.

[0028] The posture estimation model chooses yolov8m-pose in yolo. The format of the human body key points adopted by the posture estimation model is COCO-17, taking into account the speed and accuracy required by the present invention. The saved content is the horizontal and vertical coordinates of the following key points: 0-left shoulder, 1-left elbow, 2-left wrist, 3-right shoulder, 4-right elbow, 5-right wrist.

[0029] For each frame of a detection video, where the detection result contains two targets to be detected, record:

[0030] 1. The index of the test result corresponds to the following: 0-tank truck valve area, 1-crane pipe connector.

[0031] 2. The frame number of the frame.

[0032] 3. The xywh data of the detection box coordinates of the detection result, where x, y, w, and h represent the horizontal and vertical coordinates of the upper left corner of the detection box as well as the width and height respectively.

[0033] It is determined whether the data of the crane pipe connector in each frame intersects with the detection frame of the tank car valve area, all data corresponding to the non-intersecting frames are ignored, and the remaining data are saved in the database.

[0034] S5: Analyze the state frame of holding the crane tube, and analyze the number and strength of pulling according to a series of human shoulder coordinate data. And compare it with the set threshold to determine whether it meets the standard for confirming the firmness of the crane tube.

[0035] The judgment method adopted in S5 adopts the random forest algorithm in machine learning, which requires manual establishment of training sets and validation sets. After testing and verification, the parameters with greater influencing factors can be extracted as follows: the counterclockwise rotation angle α1 formed by the left shoulder-left elbow-left wrist, the angle α2 formed by the left shoulder-left elbow and the horizontal line, the angle α3 formed by the left elbow-left wrist and the horizontal line, the counterclockwise rotation angle β1 formed by the right shoulder-right elbow-right wrist, the angle β2 formed by the right shoulder-right elbow and the horizontal line, and the angle β3 formed by the right elbow-right wrist and the horizontal line. For examples, see the attached Figure 3 , plus the judgment result of whether this frame is "holding the crane with both hands" 1 or 0, the random forest model parameters are set as follows:

[0036] n_estimators = 100 uses 100 decision trees

[0037] max_depth=10 The maximum depth of each tree is 10

[0038] min_samples_split=2 The minimum number of samples required for internal node re-division is 2min_samples_leaf=1, and the minimum number of samples required for a leaf node is 1.

[0039] A relationship model is obtained based on the angle formed by the key points and the judgment mark of whether it is the "holding the crane with both hands" posture.

[0040] S6: Using this model, the key points of the human body in the extended area of ​​the crane pipe connector in the database are converted into model parameters, and the output results are obtained by inputting the model. If the result is 1, the key point coordinates and frame number data of the shoulders are extracted for the next step of judgment.

[0041] S7: For the data extracted in the previous step, the firmness confirmation judgment and the pulling action amplitude analysis are performed, including the following two steps.

[0042] S71: The specific provisions for judging the firmness confirmation are as follows:

[0043] 1) In a period of time when the crane can be detected with both hands, assume that the key points corresponding to the left shoulder, right shoulder, and midpoint are R, M, and O, and the change value of O over a period of time is O1, O2, O 3...... The maximum and minimum values ​​are respectively max , O min , see attached Figure 4 .

[0044] 2) If the maximum and minimum values ​​of this set of data appear alternately three times or more, go to 3), otherwise the crane pipe firmness test confirmation mark is 0.

[0045] 3) If the interval between two adjacent maximum and minimum values ​​is greater than the set pulling and shaking time interval threshold T1, go to 4), otherwise the crane pipe firmness detection confirmation mark is 0.

[0046] 4) If the difference between two adjacent maximum and minimum values ​​is greater than the swing amplitude threshold value T2, it is determined that the stability confirmation of the crane pipe is completed, and the crane pipe stability detection confirmation mark is set to 1.

[0047] 5) Get the crane tube firmness confirmation mark. If it is 1, execute the next step, otherwise it is determined that the crane tube firmness confirmation was not performed during this filling process.

[0048] S72: For a frame of data that has completed the firmness confirmation, the frame number frame and the horizontal coordinates of the shoulder key points and the midpoint horizontal coordinate X O , for each set of data X O,i and frame i Calculate the difference between two adjacent frames.

[0049] ΔX o,i =X o,i -X o,i-1 , Δframe i =frame i -frame i-1 The sum of the ratios of the two is used as the standard calculation formula for judging the range of motion as follows:

[0050]

[0051] The motion amplitude X is compared with the set threshold value T3. If it is greater than, it is a large-amplitude motion; if it is less than, it is a small-amplitude motion.

[0052]

[0053]

Claims

1. A visual inspection method for confirming the firmness of a crane pipe connection, characterized in that: According to the surveillance video collected at a set angle, the video signal is extracted frame by frame into images, and then the image video is input into the trained machine learning model. The valve area and crane pipe connector of the petrochemical tank truck are first detected using the target detection method, and then the human body key point detection method is used to obtain the coordinates of the human body's skeleton key points, and the relevant information is obtained and stored in the database; The pulling posture of the operator is identified through the relationship between key points of the human body. The number and amplitude of pulling are calculated by calculating the mean value of the displacement of key points during the change of pulling posture.

2. A visual inspection method for confirming the connection firmness of a crane pipe according to claim 1, characterized in that: The horizontal and vertical coordinates of the key points are marked as follows: 0-left shoulder, 1-left elbow, 2-left wrist, 3-right shoulder, 4-right elbow, 5-right wrist.

3. A visual inspection method for confirming the connection firmness of a crane pipe according to claim 1, characterized in that: The following steps are involved: S1: Obtain surveillance video from a camera at a set angle and obtain images frame by frame; S2: Based on the target detection method of yolov8, the data set is annotated and the model is trained to detect the valve area of ​​the tank truck and the connector of the crane pipe, and the detection results of the detection frame containing the detection target are saved to the database together with the frame number; S3: Use the yolov8-pose model to detect the key point information of the person in each frame and store it in the database; S4: analyzing the key point information of the human body in step 3 and the position of the detection target obtained in step 2, and extracting the state frame of the person holding the crane with both hands; S5: Analyze the state frame of holding the crane tube, and analyze the number and strength of pulling according to a series of human shoulder coordinate data. And compare it with the set threshold to determine whether it meets the standard for confirming the firmness of the crane tube.

4. A visual inspection method for confirming the connection firmness of a crane pipe according to claim 3, characterized in that: In step S1: In order to ensure the accuracy of the coordinates of the key points of the human body, the angle between the line connecting the projections of the camera and the operator on the vertical three-dimensional coordinate axis and the wall on which the camera is mounted should be greater than 60°. The resolution of the monitoring video of the tank truck area is 1920*1080, and the frame rate is not less than 30.

5. A visual inspection method for confirming the connection firmness of a crane pipe according to claim 3, characterized in that: In the step S2, the established and trained data set should be based on the on-site operation video of the site, and the videos shot by workers of different physiques and different shooting angles should be selected for frame sampling and annotation. The training set, verification set, and test set should be divided into training set, verification set, and test set in a ratio of 8:1:

1. The labels of the annotated objects are the tank truck valve area and the crane pipe connector.

6. A visual inspection method for confirming the connection firmness of a crane pipe according to claim 1, characterized in that: Specifically include: The coordinates of the human key points described in step S4 adopt the human key point data set format of COCO-17 of yolov8-pose, including: nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees and left and right ankles. According to the angle information formed between these key points, the action of the worker holding the crane with both hands is judged through machine learning methods.

7. A visual inspection method for confirming the connection firmness of a crane pipe according to claim 3, characterized in that: The estimation of the number and strength of pulling according to the shoulder coordinate data in step S5 specifically refers to that in the data of the human body key points of all the frames judged as "holding a crane" in the previous step, the shoulder horizontal coordinate changes from small to large and then to small once, and the change amplitude exceeds the set threshold, which is recorded as one pulling; at the same time, in the shoulder horizontal coordinate change recorded as one pulling, the ratio between the sum of the absolute values ​​of the horizontal coordinate change distances frame by frame and the total duration is used to characterize the pulling strength.

8. A visual inspection method for confirming the connection firmness of a crane pipe according to claim 7, characterized in that: The angle information between the key points specifically refers to: in the crane area, the counterclockwise rotation angle formed by the left elbow-left shoulder and the left elbow-left wrist, and the counterclockwise rotation angle formed by the right elbow-right shoulder and the right elbow-right wrist. A series of video frames are intercepted to obtain the rotation angle data and add a manual judgment mark of whether it is a hand-held crane (indicated by 1 or 0). This information is combined with machine learning methods to train the model for subsequent judgment.

Images