Manual valve abnormal operation detection method based on target detection and key point tracking

Abstract

The invention relates to the technical field of target detection, and discloses a manual valve abnormal operation detection method based on target detection and key point tracking, and the method comprises the steps: collecting video image data through a camera, carrying out the target detection of an operator and the hand of the operator, and extracting a detection frame; meanwhile, key point tracking detection is conducted on spokes of a valve handle or a valve hand wheel, and whether valve operation exists or not is judged according to the overlapping relation of the hand detection frame and the valve detection frame; the clockwise or anticlockwise rotation state of the valve is determined through key point time sequence analysis, and the opening, closing and state of the valve are further determined; and finally, a judgment result is input into a logical reasoning module to judge whether valve operation is abnormal or not. Compared with the prior art, the method can accurately identify whether the valve is opened and closed and whether the operation sequence is correct or not, and can timely send out alarm information for abnormal operation so as to timely eliminate potential safety hazards.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a method for detecting abnormal operation of a manual valve based on target detection and key point tracking. Background Art

[0002] Although automated control valves have occupied a dominant position in modern industry, manually controlled valves still have important value and irreplaceable roles. In intermittent operation scenarios such as material loading and unloading, manual valve operation is usually the only way to control material transmission. In certain special environments, such as extreme conditions such as high temperature, high pressure, and strong radiation, the stability and reliability of the automated control system may be affected, while manual control valves can work normally in these environments to ensure the smooth progress of the production process. In addition, when the automated control system fails or requires emergency intervention, manual control valves are an indispensable emergency backup measure. In continuous manual valve operation, there are usually strict regulations on valve switching operations and the opening and closing sequence of different valves. Once an operation error occurs, it may lead to production accidents, damage to personnel and equipment, and other safety hazards.

[0003] However, traditional manual valve operation often relies on manual inspection and record-keeping, which consumes considerable time and manpower and prevents timely detection of operational anomalies. Remote camera monitoring technology allows managers to continuously monitor valve operations within a designated area, improving management efficiency and responsiveness. However, the need for precise capture and in-depth analysis of valve operations, coupled with the massive amount of video data generated by multiple cameras, the numerous valves, and the potential for visual fatigue from long hours of monitoring personnel, all add to the complexity and difficulty of valve operation monitoring, potentially reducing the effectiveness and accuracy of monitoring. Intelligent video surveillance can detect objects such as valves, but it cannot directly identify dynamic operations and cannot provide logical judgment on whether an operation is correct. Summary of the Invention

[0004] Purpose of the invention: In response to the problems pointed out in the background technology, the present invention discloses a method for identifying and detecting abnormal operation of manual valves based on target detection and key point tracking, which can accurately identify whether the valve is opened, closed and the operation sequence is correct, and can promptly issue alarm information for abnormal operation so as to eliminate safety hazards in time.

[0005] Technical solution: The present invention discloses a method for detecting abnormal operation of a manual valve based on target detection and key point tracking, comprising the following steps: Step 1: Collect video image data through the camera, perform target detection on the operator and the operator's hands, and extract the hand detection frame; Step 2: Perform key point tracking detection on the spokes of the valve handle or valve handwheel to obtain the valve detection frame; Step 3: Determine whether valve operation is occurring based on the overlap between the hand detection frame and the valve detection frame; determine the clockwise or counterclockwise rotation state of the valve through key point timing analysis, and further determine the valve switch operation recognition result: valve switch and its state; Step 4: Finally, determine whether there is any abnormality in the valve operation based on the judgment result.

[0006] Furthermore, in step 1, target detection is performed on the operator and the operator's hand, and a hand detection frame is extracted, as follows: Step 1.1: Build a target detection and tracking model for the operator and his hands. The target detection model is trained using the YOLO deep learning model. Step 1.2: Create a dataset of people and their hands in the manual valve operation scenario. Collect video data of the manual valve operation scenario and load the video stream into image frames to construct a dataset that includes information about the operator and his hands. Step 1.3: Use LabelMe software to annotate the operator and his hand, generate a JSON file, and then use a script to convert it into a Yolo format file. Step 1.4: Create training and test datasets from the collected images. Step 1.5: Input the obtained dataset into the YOLO deep learning model in step 1.1 to train the model for operator and hand target detection; Step 1.6: Based on the target detection results, use the CSRTTracker algorithm in the OpenCV vision library to track the human target.

[0007] Furthermore, the key point tracking detection of the spokes of the valve handle or valve handwheel in step 2 is specifically as follows: Step 2.1: Build a key point tracking detection model for the valve; Step 2.2: In the dataset image of the valve operation scene, use labelme to synchronously label the valves and their corresponding key points; Step 2.3: Keypoint detection model training. The Python script file generates the collected images into training and test datasets, which are then fed into the keypoint tracking and detection model for keypoint model training. Step 2.4: Key point tracking, using Kalman filter or optical flow method, matches the key points between consecutive frames, determines the relative position of the key points in space, and tracks the movement of the key points.

[0008] Furthermore, in step 2.2, for a handle-type valve, the key points are the two ends of the handle; for a handwheel-type valve, the key points are the two end points of the spoke.

[0009] Furthermore, the real-time monitoring of the valve opening and closing actions in step 3 specifically includes the following process: Step 3.1: Valve operation recognition: When the overlap between the hand target detection frame and the valve target detection frame reaches more than 10%, it is determined that a valve-related operation is being performed, and then the valve rotation-related calculation is triggered; Step 3.2: Extract the coordinate information of key points at continuous or appropriate intervals, use the math.atan function in the OPENCV vision library to calculate the angle, and calculate the rotation direction through the vector cross product function. A negative rotation angle indicates clockwise rotation, corresponding to the valve opening process; a positive rotation angle indicates counterclockwise rotation, corresponding to the valve closing process.

[0010] Furthermore, the determination of whether the valve operation is abnormal in step 4 is as follows: Step 4.1: Write the valve switch operation recognition results identified in step 3 into the vector group. After each writing, calculate and compare the recognition vector group with the reference vector group corresponding to the initially set valve standard operation; Step 4.2: If it is consistent with the benchmark vector group, continue the downward operation; if it is inconsistent, issue a warning signal. Beneficial effects

[0011] The present invention discloses a method for identifying and detecting abnormal operation of manual valves based on target detection and key point tracking. The method performs target detection on the operator and the operator's hands to extract the hand detection frame; performs key point tracking detection on the spokes of the valve handle or valve handwheel to obtain the valve detection frame; and determines whether the valve operation exists through the overlapping relationship between the hand detection frame and the valve detection frame. This method can achieve accurate identification of whether the valve is opened or closed and whether the operation sequence is correct, and can promptly issue alarm information for abnormal operation to eliminate safety hazards in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a flow chart of the overall structure of the present invention; Figure 2 This is the framework structure diagram of the YOLO target detection deep learning model; Figure 3 is the target detection result map; Figure 4 This is the key point detection result of the handwheel valve; Figure 5 This is the key point detection result of the handle valve; Figure 6 It is the key point tracking and valve operation detection result diagram; Figure 7 This is the alarm diagram for incorrect operation. DETAILED DESCRIPTION

[0013] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0014] The present invention discloses a method for detecting abnormal operation of a manual valve based on target detection and key point tracking, comprising the following steps: Step 1: Build a target detection and tracking model for the operator and his hands. The target detection model is trained using the relevant YOLO target detection deep learning model. Figure 2 .

[0015] 1) Capture images and build a dataset of people and their hands in scenarios involving manual valve operation. Collect video data from scenarios involving manual valve operation and load the video stream into image frames to build a dataset containing information about the operator and their hands.

[0016] 2) Dataset annotation: Use LabelMe software to annotate the outlines of the person and their hands. The annotation generates a JSON file, which is then converted into a YOLO format file using a script.

[0017] 3) The Python script file generates the collected images into a training dataset file "train.txt" and a test dataset file "test.txt".

[0018] 4) Input the dataset file obtained in the above steps into the YOLO deep learning model to train the model for operator and hand target detection.

[0019] 5) Based on the target detection results, the CSRTTracker algorithm in the OpenCV vision library is used to track the human target.

[0020] Step 2: Build a key point tracking detection model for the valve

[0021] Keypoint Annotation: In the dataset images of valve operation scenes, LabelMe was used to simultaneously annotate the valves and their corresponding keypoints. For handle-type valves, the keypoints were the ends of the handle; for handwheel-type valves, the keypoints were the center and edge of the spokes.

[0022] Keypoint detection model training. The Python script creates a training dataset file (train.txt) and a test dataset file (test.txt) from the captured images. These datasets are then fed into the YOLO deep learning model to train the keypoint model.

[0023] Keypoint tracking: Using a Kalman filter or optical flow method, keypoints between consecutive frames are matched to determine their relative positions in space, enabling accurate tracking of keypoint motion.

[0024] Step 3: Real-time monitoring of valve opening and closing actions.

[0025] 1) Valve operation recognition: When the overlap between the hand target detection frame and the valve target detection frame reaches 5% or more, it is determined that a valve-related operation is in progress, and the valve rotation-related calculation is triggered.

[0026] 2) Extract the coordinates of consecutive or appropriately spaced key points, use the math.atan function in the OpenCV vision library to calculate the angle, and calculate the rotation direction using the vector cross product function. A negative rotation angle indicates counterclockwise rotation, corresponding to the valve opening process; a positive rotation angle indicates clockwise rotation, corresponding to the valve closing process.

[0027] Step 4: Detect abnormal valve operation

[0028] 1) Write the valve operation recognition results identified in the above steps into the vector group. After each write, the recognition vector group is compared with the reference vector group corresponding to the initially set valve standard operation.

[0029] 2) If it is consistent with the benchmark vector group, the downward operation can continue; if it is inconsistent, a warning signal is issued.

[0030] Application examples:

[0031] 1. Obtain the video of the person manually operating the valve under monitoring, process the video, load it into picture frames, and annotate the people and hands in the picture frames using labelme software to build a target detection dataset for the target person and hands. After the annotated dataset is completed, the target detection model is built using the yolov8 target detection model framework. The detection results are shown in Figure 3 .

[0032] 2. Label the key points of the two types of valves. For handle valves, label the ends of the handle; for handwheel valves, label the center point and two end points of the spokes. Input the constructed dataset into the yolov8 key point detection model framework to build a key point detection and tracking model. The training results of the handwheel valve key point detection model are shown in Figure 4, the key point test results of the handle valve are shown in Figure 5 .

[0033] 3. Extract the hand detection frame and valve detection frame data in the video respectively, and determine the overlap between the two. When the overlap is greater than the set threshold (IOU>0.5), it is determined that the valve operation has begun and the key point detection and tracking model is started.

[0034] 4. After starting the key point tracking model, extract the key point position coordinates of the valve key points at different time points, including the valve axis coordinate p1 (p1[0], p1[1]) at time T1, the valve edge coordinate p2 (p2[0], p2[1]) at time T1, and the valve edge coordinate p3 (p3[0], p3[1]) at time T2. Construct the edge point and valve center point vectors at different times through the position coordinates, namely (p2[0]-p1[0], p2[1]-p1[1]) and (p3[0]-p1[0], p3[1]-p1[1]). Determine the rotation direction through vector cross product calculation. The calculation formula is: Direction=(p2[0]-p1[0])×(p3[1]-p1[1])-(p2[1]-p1[1])×(p3[0]-p1[0]) When direction>0, the rotation direction is counterclockwise and the valve is closed; when direction<0, the rotation direction is clockwise and the valve is open. The judgment result is shown in Figure 6 .

[0035] 5. In this example, the correct opening and closing sequence of the valves is valve 1 open, valve 2 open, valve 1 closed, and valve 2 closed. This information is stored in the list: [(1,O),(2,O),(1,C),(2,C)], where 1 and 2 represent valves, O represents valve open, and C represents valve closed. The valve operation status in the captured video is stored in the actual operation list and compared with the preset list. If there is a difference, an alarm message is sent. In this example, an alarm is issued for the valve sequence, and the entire screen is adjusted to dark red to warn, such as Figure 7 shown.

[0036] The above embodiments are intended only to illustrate the technical concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. They are not intended to limit the scope of protection of the present invention. Any equivalent changes or modifications made in accordance with the spirit of the present invention are intended to be covered by the scope of protection of the present invention.

Claims

1. A method for detecting abnormal operation of a manual valve based on target detection and key point tracking, characterized in that: The steps include: Step 1: Collect video image data through the camera, perform target detection on the operator and the operator's hands, and extract the hand detection frame; Step 2: Perform key point tracking detection on the spokes of the valve handle or valve handwheel to obtain the valve detection frame; Step 3: Determine whether there is valve operation through the overlapping relationship between the hand detection frame and the valve detection frame; determine the clockwise or counterclockwise rotation state of the valve through key point timing analysis, and further determine the valve switch operation recognition result, valve switch and its state; Step 4: Finally, determine whether there is any abnormality in the valve operation based on the judgment result.

2. The manual valve abnormal operation detection method based on target detection and key point tracking according to claim 1 is characterized in that: In step 1, target detection is performed on the operator and the operator's hand, and the hand detection frame is extracted, as follows: Step 1.1: Build a target detection and tracking model for the operator and his hands. The target detection model is trained using the YOLO deep learning model. Step 1.2: Establish a data set of people and their hands in the manual valve operation scene, collect video data in the manual valve operation scene, and load the video stream into image frames, so as to construct a data set including the operator and his hand information; Step 1.3: Use labelme software to annotate the operator and his hand, generate a json file, and then use a script to convert it into a yolo file; Step 1.4: Make the collected images into training data sets and test data sets; Step 1.5: Input the obtained data set into the YOLO deep learning model in step 1.1 to train the model for operator and hand target detection; Step 1.6: Based on the target detection results, use the CSRTTracker algorithm in the OpenCV vision library to track the human target.

3. The manual valve abnormal operation detection method based on target detection and key point tracking according to claim 1 is characterized in that: The key point tracking detection of the spokes of the valve handle or valve handwheel in step 2 is specifically as follows: Step 2.1: Construct a key point tracking detection model for the valve; Step 2.2: In the dataset image of the valve operation scene, use labelme to synchronously annotate the valves and their corresponding key points; Step 2.3: Key point detection model training. The Python script file makes the collected images into training data sets and test data sets, inputs them into the key point tracking detection model, and performs key point model training. Step 2.4: Key point tracking, using Kalman filter or optical flow method, to match the key points between consecutive frames, determine the relative position of the key points in space, and track the movement of the key points.

4. The manual valve abnormal operation detection method based on target detection and key point tracking according to claim 2 is characterized in that: In step 2.2, for a handle-type valve, the key points are the two ends of the handle; for a handwheel-type valve, the key points are the center point and two end points of the spoke.

5. The method for detecting abnormal operation of a manual valve based on target detection and key point tracking according to claim 1, characterized in that: The real-time monitoring of the valve opening and closing actions in step 3 specifically includes the following process: Step 3.1: Valve operation recognition: when the overlap between the hand target detection frame and the valve target detection frame reaches more than 10%, it is determined that valve-related operations are being performed, and then the valve rotation-related calculation is triggered; Step 3.2: Extract the coordinate information of key points at continuous or appropriate intervals, use the math.atan function in the OPENCV vision library to calculate the angle, and calculate the rotation direction through the vector cross product calculation function. The negative rotation angle is clockwise rotation, corresponding to the valve opening process; the positive rotation angle is counterclockwise rotation, corresponding to the valve closing process.

6. The manual valve abnormal operation detection method based on target detection and key point tracking according to claim 1 is characterized in that: In step 4, determining whether there is an abnormality in valve operation is as follows: Step 4.1: Write the valve switch operation recognition result identified in step 3 into the vector group. After each writing, calculate and compare the recognition vector group with the reference vector group corresponding to the initially set valve standard operation; Step 4.2: If it is consistent with the benchmark vector group, continue to operate downward; if it is inconsistent, issue a warning signal.

Images