Method and system for detecting the standardization of operation actions of blind plate plugging operators based on intelligent Internet of Things
Through intelligent IoT technology combined with human posture estimation and YOLO object detection algorithm, automatic detection of the operation actions of blind plate plugging workers is realized, solving the high cost and low efficiency problems caused by human judgment in the existing technology, and improving the accuracy of detection Sex and safety.
Patent Information
- Application Number
- CN202410978114.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-07-22
AI Technical Summary
The standardized inspection of existing blind plate plugging workers relies on human judgment, which is costly and inefficient, and the detection accuracy is reduced when the staff is tired, which poses safety risks.
Using an intelligent IoT method, combined with the human posture estimation model and the YOLO object detection algorithm, the status of the blind plate and wrench is detected through video recording and human body key point recognition, to determine whether the personnel operation is correct, and a detection report is generated.
The accuracy and efficiency of blind plate plugging operators' operational motion detection is improved, and the risk of safety accidents is reduced.
Smart Images

Figure CN118522077B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer technology, and in particular to a method and system for detecting the normativeness of operation of blind plate extraction and plugging operators based on intelligent Internet of Things. Background Art
[0002] Human key point estimation is a feature recognition task in the field of computer vision. It has broad application prospects in computer animation film and television production, behavior recognition, posture tracking, human activity analysis and other fields. Human key point estimation uses a given image or video to obtain the position of human joints or parts. From the perspective of deep learning, human posture estimation can be divided into two categories: single-person key point estimation and multi-person key point estimation. Human key point estimation can usually be divided into high-level abstraction (Bottom Up) and low-level pixels (Top Down). The Bottom Up human key point estimation method first finds all target points in the input image through the key point estimation model, and then uses the clustering method to classify the obtained points into different individuals. The Top Down human key point estimation method generally includes two modules: human target detection and human key point estimation, that is, first detect each human target through the target detection-related algorithm, obtain the target detection box of all individuals in the image, and then use the target detection box as input to the key point estimation model to finally obtain the key point information. Intelligent IoT based on the industrial Internet has created conditions for the realization of data-driven blind plate plugging operator operation action standardization detection algorithm.
[0003] At present, target detection algorithms are divided into two categories based on the detection method: one is "single stage" and the other is "double stage". For the "single stage" target detection algorithm, it directly adds position parameters to the label, so as to achieve the simultaneous type and position discrimination of objects in the image. For the "double stage" target detection algorithm, it adopts the sliding prediction box method, using prediction boxes of various sizes to slide on the image. After traversing the image information, it conducts a probability evaluation on the real prediction box that frames the target information, scores each prediction box, and finally determines the prediction box with the largest score as the location information of the target object and outputs it.
[0004] In the existing technology, the standardization of blind plate plugging operation is detected by manual judgment, which is costly and inefficient. The biggest problem with manual detection is that when the staff is tired, they may make errors, which greatly reduces the accuracy of the evaluation and brings great safety hazards. In the boom of industrial automation, the process and actions of blind plate plugging are evaluated by combining human posture estimation methods. This method will greatly improve the accuracy and efficiency of evaluation. Summary of the invention
[0005] In order to overcome the above problems, the purpose of the present invention is to provide a method and system for detecting the standardization of operating actions of blind plate extraction and plugging operators based on intelligent Internet of Things, which can evaluate and analyze the test actions of the personnel to be evaluated in the context of specific blind plate extraction and plugging operations.
[0006] According to the first aspect of the embodiment of this specification, a method for detecting the standardization of operation actions of blind plate plugging operators based on intelligent IoT is proposed, which is used to evaluate the standardization of blind plate plugging operation actions. The method includes the following steps:
[0007] Step S1, recording a video of an operator through a terminal device, using an improved human key point recognition algorithm to detect the person in the recorded video, and obtaining human key point information.
[0008] Step S2, perform object recognition through YOLO to detect the status of the wrench and the blind plate in the video. If the result of each frame recognition is a solid blind plate, the pipeline is in a closed and blocked state. If the result of each frame recognition is a hollow state of the throttling ring, it indicates that the pipeline is in an open state. According to whether there is a wrench in each frame of the video, the time series of the person using the wrench to operate the blind plate is obtained; according to the hollow and solid states of the blind plate, it is judged whether the person correctly sets the blind plate to the closed and blocked state, or sets the blind plate to the throttling open state.
[0009] Step S3: Draw a time series diagram corresponding to the key points of the human hand according to the key point information of the human body, and perform action recognition and judgment in the blind plate plugging operation in combination with the wrench and blind plate status. Perform action recognition and judgment in the blind plate plugging operation, that is, judge whether the personnel tighten or loosen the bolts on a single valve diagonally; draw a time series diagram corresponding to the key points of the human hand according to the key point information of the human body obtained in step S1, and obtain the four coordinate changes of the hand nodes when the personnel remove or install the bolts, and further judge whether the bolts are removed or installed diagonally.
[0010] Step S4, according to whether the operator removes or installs the bolts in the correct order when operating the blind plate plugging, and whether the operator sets the blind plate status correctly, the final test result is obtained and a test report is generated.
[0011] Wherein, step S1 further comprises the following steps:
[0012] Step S11, first, detect whether there is a person in the first frame of the video according to the YOLO algorithm. If there is a person in the first frame, the entire video will be further identified and detected. If there is no person in the first frame, the video will be determined to be invalid and no subsequent detection will be performed.
[0013] Step S12, when a person appears in the first frame, the video is divided into pictures frame by frame, and the key points of the human body and other information are detected for each frame. Here, an improved human skeleton extraction model is used, which can accurately depict the key points of the human body on the picture, and at the same time, save the position of the human body and the number of frames.
[0014] The improved human skeleton model is specifically implemented as follows:
[0015] By improving the feature extraction network (backbone) of the original human skeleton model, the feature maps of different stages of the SwinTransformer model are fused to make full use of the semantic information of human bodies at different scales and reduce the loss of detail information. 1 、F 2 、F 3 、F 4 These are the feature outputs of each stage of the SwinTransformer model. The output of each stage is downsampled to the same target size through bilinear interpolation, then integrated, and finally enters the convolutional neural network for channel number conversion to obtain the output feature F f .
[0016] Wherein, step S2 further comprises the following steps:
[0017] Step S21: Perform YOLO object recognition on the obtained video to identify the hollow and solid states of the wrench and the blind plate respectively.
[0018] Step S22, according to whether there is a wrench in each frame of the picture, the video is segmented, specifically into a bolt loosening stage, a blind plate replacement stage, and a bolt tightening stage.
[0019] Step S23, first process the blind plate replacement stage, and the next step processes the bolt loosening and bolt tightening stages. The hollow and solid states of the blind plate in each frame of the personnel operation video obtained in step S22 can be used to determine whether the operator correctly sets the blind plate to a closed and blocked state, or sets the blind plate to a throttling open state.
[0020] Wherein, step S3 further comprises the following steps:
[0021] Step S31, first extract the coordinate change information of the left hand key point, filter and derive the time series corresponding to the left hand key point, and use the recursive average filtering algorithm to perform noise reduction on the time series of the human body key point by processing the derived time series. The formula is as follows:
[0022]
[0023] Where x is the key point information value before filtering, y is the key point information value after mean filtering, and N is the size of the recursive average filtering algorithm window.
[0024] After filtering, the sliding window method is used to find the gradient corresponding to the time series. The formula is as follows:
[0025]
[0026]
[0027]
[0028] In the formula, x joint is the key point information value of the hand, N dection is the size of the window, k joint is the gradient value.
[0029] Step S32, judging the video of the bolt tightening stage. In the bolt loosening stage, in a human skeleton video, when the operator removes the bolt with a wrench, his left hand is generally fixed, and then the other hand rotates the bolt back and forth. First, the coordinate change information of the left hand key point is extracted, and the time series corresponding to the left hand key point is first filtered and derived. By processing the derived time series, a threshold is set to judge the frame number set of the operator's left hand when removing a certain bolt, and then the average value of the coordinate information corresponding to the left hand when operating each bolt is obtained, thereby obtaining the four coordinate changes of the hand node when the operator removes the bolt, and further using the machine learning support vector machine SVM to determine whether the bolt is removed diagonally.
[0030] Step S33, in the stage of tightening the bolts, the same operation as step S32 is adopted, and the average value of the coordinate information corresponding to the left and right hands when operating each bolt is obtained according to the coordinate change information of the key points of the left hand of the human body, thereby obtaining the four coordinate changes of the hand nodes of the operator when removing the bolts, and further judging whether the bolts are tightened diagonally through SVM.
[0031] Step S34, in the video, when the operator tightens the four bolts, does he tighten them diagonally and replace the blind plate? If so, it means that the operator has correctly performed the blind plate extraction and plugging operation; if not, it means that the operator has incorrectly performed the blind plate extraction and plugging operation; save the final correct or incorrect operation of the operator.
[0032] According to the second aspect of the embodiments of this specification, a system for detecting the standardization of operating actions of blind plate extraction and plugging operators based on intelligent Internet of Things is provided, the system comprising a video acquisition module, a human key point detection module, a YOLO object recognition module, a personnel operating action recognition module, and an evaluation result reporting module.
[0033] The video acquisition module is responsible for obtaining information of the evaluation action video stream according to the evaluation requirements.
[0034] The human key point detection module uses the video stream collected by the video acquisition module to detect the human key points through an improved human key point recognition algorithm, and records the time series change data and geometric center point coordinates corresponding to the left hand key points of the operator operating the blind plate extraction and blocking in the vertical and horizontal directions.
[0035] The YOLO object recognition module uses the video stream collected by the video acquisition module to detect the hollow and solid states of the wrench and the blind plate in the video, and obtains the time sequence of the person using the wrench to operate the blind plate according to whether the wrench exists in each frame of the video; judges whether the person correctly changes the blind plate from solid to hollow, or from hollow to solid according to the hollow and solid states of the blind plate; and segments the video according to whether the wrench exists in each frame of the video, specifically into the stage of loosening the bolt, the stage of replacing the blind plate, and the stage of tightening the bolt.
[0036] The personnel operation action recognition module is used to draw the time series diagram corresponding to the key points of the human hand, and combine the wrench and blind plate status to perform action judgment in the blind plate extraction and plugging operation. In the loosening bolt stage obtained by the YOLO object recognition module, in a human skeleton video, when the operator removes the bolt with a wrench, his left hand is generally fixed, and then the other hand rotates the bolt back and forth. Using the time series change data and geometric center point coordinates corresponding to the left hand key points of the blind plate extraction and plugging operator obtained after the human key point detection module, first extract the coordinate change information of the left hand key points, filter and derive the time series corresponding to the left hand key points, and process the derived time series. By setting a threshold, the frame number set of the operator's left hand in removing a bolt is judged, and then we get the average value of the left hand corresponding coordinate information when operating each bolt, thereby obtaining the four coordinate changes of the hand node when the person removes the bolt, and further judge whether to remove the bolt diagonally. The same method is used to judge in the tightening stage.
[0037] The evaluation result reporting module is used to visualize the detection results obtained by the YOLO object recognition module and the human operation action recognition module, and to prepare an evaluation result report.
[0038] Among them, the video acquisition module further includes: a task assignment and recording unit: responsible for issuing assessment task requirements to personnel and recording the assessment process of personnel; a judgment unit: used to judge whether the video information recognizes the human body. If the human body is recognized, it enters the execution unit. If the human body is not recognized, the video is considered invalid and does not enter the execution unit.
[0039] The human body key point detection module further comprises an execution unit responsible for detecting various key points of the human body, and drawing a key point image of the human body in the collected video, and at the same time, saving data such as the key point information of the human body.
[0040] Among them, the YOLO object recognition module further includes: a wrench recognition unit: responsible for identifying whether a wrench appears in each frame of the picture; a video segmentation unit: because personnel need to use a wrench when operating a blind plate, the video can be divided into a loosening bolt stage, a blind plate replacement stage, and a bolt tightening bolt stage according to the result of the wrench recognition in the video, and each stage corresponds to a continuous number of picture frames in the video; a blind plate hollow and solid recognition unit: determines whether the personnel correctly changes the blind plate from solid to hollow, or from hollow to solid, and determines whether the personnel correctly sets the blind plate to a closed blocking state, or sets the blind plate to a throttling open state; a storage unit: responsible for saving the corresponding judgment results.
[0041] Among them, the personnel operation action recognition module further includes: a data analysis unit: used to obtain the hand joint point information corresponding to the personnel operating the blind plate; the fourth coordinate change of the hand node when the personnel removes the bolt, and a judgment unit: based on the obtained fourth coordinate change of the human hand, using the support vector machine learning method, the results are divided into two categories, namely diagonal tightening and non-diagonal tightening.
[0042] Among them, the evaluation result reporting module further includes: a reading unit and a result analysis unit, the reading unit is used to read the evaluation data of the YOLO object recognition module and the personnel operation action recognition module, and the result analysis unit generates an evaluation report to facilitate personnel learning and correction.
[0043] Beneficial effects of the present invention:
[0044] The present invention is a method for detecting the standardization of operating actions of blind plate extraction and plugging operators based on intelligent Internet of Things. The achievement uses a human posture estimation model, which can effectively extract the information features of key points of the human body, and combines the machine learning support vector machine method to determine whether the personnel are tightening the bolts diagonally. At the same time, the achievement also uses the YOLO target detection algorithm to continuously detect the switch state of the blind plate. Compared with the current detection relying on sensors or traditional methods such as manual work, the invention combines the human posture estimation method and the YOLO target detection algorithm to evaluate the process and actions of blind plate extraction and plugging. This method will greatly improve the evaluation execution efficiency, improve the accuracy of the standardization detection of workers' operating actions, and greatly reduce the risk of safety accidents. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 It is a block diagram of the system principle of the present invention;
[0046] Figure 2 It is a work flow chart of the present invention;
[0047] Figure 3 It is a feature extraction module diagram of the improved human key point model of the present invention;
[0048] Figure 4 It is a flow chart of the video segmentation module of the present invention;
[0049] Figure 5 This is a flow chart of the operation action detection module of the blind plate plugging operator of the present invention;
[0050] FIG6(a) is a time series corresponding to the hand nodes of a person in an example of the present invention;
[0051] FIG6( b ) is a graph showing the slopes corresponding to the hand nodes of a person in an example of the present invention. DETAILED DESCRIPTION
[0052] In order to solve the problem of detecting the operating actions of blind plate extraction and plugging operators, the present invention provides a method and system for detecting the standardization of the operating actions of blind plate extraction and plugging operators based on intelligent Internet of Things.
[0053] The following reference Figure 1 The embodiments of the present invention will be described. The embodiments described below are examples of the present invention and are not intended to limit the scope of protection of the present invention.
[0054] The following is based on Figure 1 The present embodiment involves a system for detecting the standardization of operating actions of blind plate extraction and plugging operators based on intelligent Internet of Things.
[0055] Figure 1 What is shown is a principle block diagram of a method for detecting the standardization of operating actions of blind plate plugging operators. The system includes a video acquisition module, a human key point detection module, a YOLO object recognition module, a human operating action recognition module, and an evaluation result reporting module.
[0056] The video acquisition module is responsible for obtaining information of the evaluation action video stream according to the evaluation requirements.
[0057] The human body key point detection module detects the human body key points through the deployed improved human body posture estimation deep learning algorithm, and records the time series change data and geometric center point coordinates corresponding to the left hand key points of the operator operating the blind plate extraction and blocking in the vertical and horizontal directions.
[0058] The YOLO object recognition module detects the hollow and solid states of the wrench and the blind plate in the video, and obtains the time sequence of the person using the wrench to operate the blind plate according to whether the wrench exists in each frame of the video; determines whether the person correctly changes the blind plate from solid to hollow, or from hollow to solid according to the hollow and solid states of the blind plate; and segments the video according to whether the wrench exists in each frame, specifically into the stage of loosening the bolt, the stage of replacing the blind plate, and the stage of tightening the bolt.
[0059] The personnel operation action recognition module judges the bolt tightening stage video of step S22 respectively. In the bolt loosening stage, in a human skeleton video, when the operator removes the bolt with a wrench, his left hand is generally fixed, and then the other hand rotates the bolt back and forth. First, the coordinate change information of the left hand key point is extracted, and the time series corresponding to the left hand key point is first filtered and derived. By processing the derived time series, a threshold is set to determine the frame number set of the operator's left hand when removing a certain bolt. Then, we get the average value of the left hand corresponding coordinate information when operating each bolt, thereby obtaining the four coordinate changes of the hand node when the person removes the bolt, and further determine whether the bolt is removed diagonally.
[0060] The evaluation result reporting module is responsible for integrating the results of cloud data processing, visually displaying the relevant information, and preparing an evaluation result report.
[0061] Figure 2 It is a flowchart for explaining a method for detecting the operation actions of blind plate plugging operators when blind plate plugging operators are performing blind plate plugging operations in this embodiment. It is a method for detecting the operation actions of blind plate plugging operators based on intelligent IoT. The working process is as follows:
[0062] Step S1, obtaining video data: obtaining operation video data of blind plate extraction and plugging operators through a terminal.
[0063] In this step, the video data related to the blind plate plugging operation is first preprocessed, including image denoising, to reduce noise and improve the recognizability of video quality, thereby improving the recognition rate and accuracy of the algorithm. Common denoising methods include median filtering, which can eliminate salt and pepper noise in the video image and improve the recognizability of video quality.
[0064] Step S2: Human key point detection: Human key points are extracted from the video data at the edge.
[0065] In this step, the key point information of the person is obtained through the key point detection and recognition algorithm deployed at the edge. Since the video to be detected is a continuous action, the video file input to the key point detection and recognition algorithm can be first processed by frame extraction, thereby reducing the running time of the model; then the filtering algorithm is used to perform noise reduction processing on the key points of the human body. One of the filtering algorithms such as mean filtering, median filtering, and limiting filtering can be used to reduce time series noise, thereby improving the recognition rate and accuracy of the algorithm. According to the time series of the key points of the human body after noise reduction processing, in this example, the vertical and horizontal displacements of the key points of the left hand of the blind plate extraction operator are obtained, and the corresponding time series data and geometric center point coordinates are obtained.
[0066] Step S3, perform YOLO object recognition on the obtained video, and identify the hollow and solid states of the wrench and the blind plate respectively. The 8-shaped blind plate used is shaped like an 8-shaped character, with a blind plate at one end and a throttling ring at the other end, but the diameter is the same as the pipe diameter of the pipeline, and does not play a throttling role. The 8-shaped blind plate is easy to use. When isolation is required, the blind plate end is used, and when normal operation is required, the throttling ring end is used. It can also be used to fill the installation gap of the blind plate on the pipeline; if the result of each frame recognition is a solid blind plate, the pipeline is in a closed and blocked state, and if the result of each frame recognition is that the throttling ring is in a hollow state, it indicates that the pipeline is in an open state.
[0067] Step S4, video segmentation stage; segment the video according to whether there is a wrench in each frame of the image, specifically into the stage of loosening the bolt, the stage of replacing the blind plate, and the stage of tightening the bolt.
[0068] Step S5, detect whether the blind plate is correctly replaced; determine whether the operator has correctly replaced the blind plate based on the hollow and solid states of the blind plate in each frame of the obtained personnel operation video.
[0069] Step S6, operation action recognition. In the bolt tightening stage, in a human skeleton video, when the operator removes the bolt with a wrench, his left hand is generally fixed, and then the other hand rotates the bolt back and forth. First, the coordinate change information of the left hand key point is extracted, and the time series corresponding to the left hand key point is first filtered and derived. By processing the derived time series, a threshold is set to judge the frame number set of the operator's left hand when removing a certain bolt, and then the average value of the left hand corresponding coordinate information when operating each bolt is obtained, so as to obtain the fourth coordinate change of the hand node when the person removes the bolt, and further judge whether the bolt is removed diagonally; responsible for obtaining the hand joint point information corresponding to the person operating the blind plate; the fourth coordinate change of the hand node when the person removes the bolt, according to the fourth coordinate change of the human hand, the traditional method or the support vector machine learning method is used to divide the results into diagonal tightening and non-diagonal tightening.
[0070] Figure 3 It is used to illustrate the improved part of the model in the human key point detection model.
[0071] By improving the feature extraction network (backbone) of the original human skeleton model, the feature maps of different stages of the SwinTransformer model are fused to make full use of the semantic information of human bodies at different scales and reduce the loss of detail information. 1 、F 2 、F 3 、F 4 These are the feature outputs of each stage of the SwinTransformer model. The output of each stage is downsampled to the same target size through bilinear interpolation, then integrated, and finally enters the convolutional neural network for channel number conversion to obtain the feature F f .
[0072] Bilinear interpolation algorithm:
[0073] Feature fusion requires modifying the size of the high-level feature map first. Therefore, the upsampling method of bilinear interpolation modifies the size of the high-level feature map. The principle of the upsampling method of bilinear interpolation is as follows:
[0074] Assume that we already know that function f is in Q 11 (x 1 ,y 1 ), Q 12 (x 1 ,y 2 ), Q 21 (x 2 ,y 1 ), Q 22 (x 2 ,y 2 ) If we want to know the value of P(x, y) on function f, we first perform linear interpolation in the x direction to get f(R 1 ), f(R 2 ) is shown in the following formula:
[0075]
[0076]
[0077] Then perform linear interpolation in the y direction to obtain f(P) as shown below:
[0078]
[0079] Although the use of bilinear interpolation method for upsampling increases the amount of calculation to a certain extent, the feature map retains more useful information during the transmission process, reduces the loss of feature information during the transmission process, ensures the grayscale continuity of the feature map, and is conducive to the detection of the human skeleton.
[0080] After the improved feature extraction skeleton, the recognition accuracy of the model was increased from 87.1% to 89.4%, thereby improving the recognition accuracy of human skeleton nodes and improving the effect of detecting the operating actions of blind plate extraction and plugging operators.
[0081] Figure 4 It is a flowchart for explaining the video segmentation in the blind plate plugging operation. In step S4, the video is segmented according to whether there is a wrench in each frame of the picture. Specifically, it is divided into the loosening bolt stage, the blind plate replacement stage, and the bolt tightening stage, and the wrench information in the video frame after YOLO object detection is read.
[0082] Next, if the number of frames in which the wrench exists in the continuous images is greater than 1100, save the starting frame; if the number of frames in which the wrench does not exist in the continuous images is greater than 1100, also save the starting frame of this video frame, get all the starting frames, add the length of the video, and get the frame number index of the video segment.
[0083] Figure 5 It is a flowchart for explaining the diagonal judgment of bolts operated by personnel in the blind plate extraction and plugging operation. In step S5, in the bolt tightening stage, in a human skeleton video, when the operator removes the bolt with a wrench, his left hand is generally fixed, and then the other hand rotates the bolt back and forth. First, the coordinate change information of the left hand key point is extracted, and the time series corresponding to the left hand key point is first filtered and differentiated. By processing the time series after differentiation, the threshold is set to judge the frame number set of the operator's left hand when removing a certain bolt. Then, we get the average value of the left hand corresponding coordinate information when operating each bolt, so as to obtain the four coordinate changes of the hand node when the personnel removes the bolt, and further judge whether the bolt is removed diagonally; responsible for obtaining the corresponding hand joint point information when the personnel operate the blind plate.
[0084] Using the machine learning support vector machine method, we selected videos of people operating blind plate plugging in different scenarios, obtained the four coordinate changes of the key points of the people's hands when removing the bolts, made a data set, and used the support vector machine learning method to divide the results into diagonal tightening and non-diagonal tightening.
[0085] Machine learning support vector machine SVM judgment:
[0086] The main idea of SVM is to find the optimal hyperplane that can separate different categories of data. Specifically, SVM maps sample points to high-dimensional space during classification, transforms linearly inseparable problems in low-dimensional space into linearly separable problems in high-dimensional space, and then finds a hyperplane (decision boundary) to separate the two categories of data.
[0087] Given a training set X={(x 1 , y 1 ), …, (x N , y N )}, where x i is the input feature vector, y i is the corresponding category label.
[0088] The decision function of SVM can be expressed as:
[0089]
[0090] Where: f(x) represents the classification output of the input sample x, which is a label (such as +1 or -1), sign is the sign function, according to w T The positive and negative values of x + b are used to classify. w is the normal vector of the hyperplane, and b is the bias term.
[0091] The optimization goal of SVM is to find a hyperplane with the maximum margin, which can be expressed as: maximize 2 / ||w||, where ||w|| is the norm of w.
[0092] Figure 6(a) and Figure 6(b) are the time series graphs and slope graphs corresponding to the key points of the left and right hands of a person tightening a bolt in a specific scenario.
[0093] In this example, set the following values:
[0094] The sliding window in the median filtering algorithm is set to 3*3.
[0095] The sliding window in the denoising process performed on the time series of the key points of the human body by the average filtering algorithm is set to 10.
[0096] As shown in the following scenario, during the blind plate extraction operation, personnel need to follow the instructions and tighten the bolts on the blind plate in a certain order.
[0097] A person performs blind plate plugging operation in a blind plate plugging operation scenario. The system includes a video acquisition module, a human key point detection module, a YOLO object recognition module, a person operation action recognition module, and an evaluation result reporting module; the video acquisition module is responsible for obtaining information of the evaluation action video stream according to the evaluation requirements;
[0098] The human key point detection module detects the human key points through the deployed improved human posture estimation deep learning algorithm, and records the time series change data and geometric center point coordinates of the left hand key points of the operator operating the blind plate extraction and plugging in the vertical and horizontal directions; the YOLO object recognition module detects the hollow and solid states of the wrench and the blind plate in the video, and obtains the time series of the person using the wrench to operate the blind plate according to whether there is a wrench in each frame of the video; according to the hollow and solid states of the blind plate, it determines whether the person correctly changes the blind plate from solid to hollow, or from hollow to solid, and determines whether the person correctly sets the blind plate to the closed blocking state, or sets the blind plate to the throttling open state; according to whether there is a wrench in each frame, the video is segmented, specifically divided into the loosening bolt stage, the blind plate replacement stage, and the bolt tightening stage; the person's operating actions The recognition module judges the video of the bolt tightening stage respectively. In the bolt loosening stage, in a human skeleton video, when the operator removes the bolt with a wrench, his left hand is generally fixed, and then the other hand rotates the bolt back and forth. First, the coordinate change information of the left hand key point is extracted as shown in Figure 6(a). The time series corresponding to the left hand key point is filtered and differentiated. By processing the differentiated time series, a threshold is set to judge the frame number set of the operator's left hand when removing a certain bolt. Then, we get the average value of the corresponding coordinate information of the left hand when operating each bolt, thereby obtaining the four coordinate changes of the hand node when the operator removes the bolt, and further judging whether the bolt is removed diagonally; the evaluation result report module is responsible for integrating the results of cloud data processing, visualizing the relevant information, and making an evaluation result report.
[0099] In this example, as shown in Figure 6(a) and Figure 6(b), the blind plate plugging operator operates the first bolt from frame 0 to 160, the second bolt from frame 171 to 314, the third bolt from frame 348 to 470, and the fourth bolt from frame 489 to 645. The coordinates corresponding to each time series are (892, 987), (1061, 895), (882, 963), (1027, 834), respectively; and then sent to the machine learning classifier to determine that it is diagonal tightening.
[0100] A person operates a blind plate to remove the plug. The video acquisition module collects the complete operation video. The human key point detection module obtains the time series diagram of the hand key points. The YOLO object recognition module obtains the video segmentation result. The (14,1089) frame is the loosening bolt stage, the (1089,3814) frame is the blind plate flipping stage, and the (3814,5434) frame is the bolting stage. The result of the blind plate flipping stage is that the person changes the blind plate from solid to hollow. The person operation action recognition module recognizes the loosening bolt stage and the bolting bolt stage respectively. In the loosening bolt stage, the (78, 304) frame is the first segment, the (339, 612) frame is the second segment, the (638, 870) frame is the third segment, and the (894, 1064) frame is the fourth segment. The four coordinates obtained are [[607.3893,919.5980], [876.2465, 799.3253], [751.6123, 910.7820], [803.7829, 696.4880]], and sent to the machine learning SVM classifier to be judged as diagonal tightening. In the bolt tightening stage, [(0, 250) frame is the first segment, (647,895) frame is the second segment, (908,1193) frame is the third segment, (1213,1512) frame is the fourth segment, and the corresponding four coordinates are [[641.2344,954.160], [895.4368,781.3902], [763.716,927.689], [844.7105, 752.9783]], and sent to the machine learning SVM classifier to be judged as diagonal tightening. The evaluation result report module finally concluded that the personnel tightened the bolts diagonally when loosening them, and that they changed the blind plate from solid to hollow when flipping the blind plate, and that they tightened the bolts diagonally when tightening them.
[0101] The blind plate plugging operation intelligent detection device also includes a processor and a memory. The processor is, for example, a CPU (central processing unit), and the memory is, for example, an SSD (Solid State Drive) or a flash memory, which can store various information.
[0102] The above embodiments can be implemented by software, hardware, firmware or a combination thereof. If implemented in the form of a computer program product, the product can be composed of one or more computer instructions. When these instructions are loaded or executed on a computer, corresponding results are generated according to the processes and functions described in the embodiments of the present invention. The camera can be various types of cameras, including outdoor anti-theft cameras and mobile phone cameras. The display can be the screen of a smart phone and a tablet computer. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or can be transferred from one medium to another, such as from one website, computer, server or data center to another website, computer, server or data center. The computer-readable storage medium can be any medium that can be accessed by a computer, or it can be a storage device such as a server or data center containing multiple media. The available media can be magnetic media (such as floppy disks, hard disks, tapes), optical media (such as DVDs) or semiconductor media (such as solid-state drives Solid State Disk (SSD)), etc.
[0103] The above is only a specific implementation of the embodiment of this specification. The technical solution of the present invention is further explained through this implementation so that those skilled in the art can understand the technical concept of the present invention. The methods and terms involved that are not explicitly disclosed, such as algorithms, can be implemented by conventional technologies in the field, and no specific restrictions are imposed on them. It should be understood that the above is only a preferred embodiment of the present invention, and the patent scope of the present invention cannot be limited thereto. All equivalent structures or equivalent process changes made using the contents of the present specification and drawings, or directly or indirectly used in other related technical fields, are similarly included in the patent protection scope of the present invention.
Claims
1. A method for detecting the standardization of operation actions of blind plate plugging operators based on intelligent IoT, characterized in that: The following steps are involved: Step S1: Record the video of the operator through the terminal device, use the improved human key point recognition algorithm to detect the person in the recorded video, and obtain the human key point information; the specific implementation process is as follows: Step S11, detecting whether a person appears in the first frame of the video according to the YOLO algorithm, if a person appears in the first frame, the entire video is identified and detected, if no person appears in the first frame, the video is determined to be invalid and no subsequent detection is performed; Step S12, when a person appears in the first frame, the video is divided into pictures frame by frame, and the improved human skeleton model is used to detect the key point information of the human body for each frame, and the key points of the human body are depicted on the picture, and the position and frame number data of the human body are saved at the same time; The improved human skeleton model is specifically implemented as follows: by improving the feature extraction network backbone of the original human skeleton model, the feature maps of different stages of the SwinTransformer model are fused, and F1, F2, F3, and F4 are respectively the feature outputs of each stage of the SwinTransformer model, and the output of each stage is downsampled to the same target size through bilinear interpolation, and then integrated, and finally enters the convolutional neural network for channel number conversion to obtain output features; Step S2, perform object recognition through YOLO to detect the status of the wrench and blind plate in the video. If the result of each frame recognition is a solid blind plate, the pipeline is in a closed and blocked state. If the result of each frame recognition is a hollow state of the throttling ring, it indicates that the pipeline is in an open state. According to whether there is a wrench in each frame of the video, the time sequence of the person using the wrench to operate the blind plate is obtained; according to the hollow and solid states of the blind plate, it is judged whether the person correctly sets the blind plate to a closed and blocked state, or sets the blind plate to a throttling open state; the specific object recognition implementation process is as follows: Step S21, performing YOLO object recognition on the obtained video to respectively identify the hollow and solid states of the wrench and the blind plate; Step S22, according to whether there is a wrench in each frame of the picture, the video is segmented into a bolt loosening stage, a blind plate replacement stage, and a bolt tightening stage; Step S23, first process the blind plate replacement stage, then process the bolt loosening and bolt tightening stages, and determine whether the operator correctly sets the blind plate to a closed blocking state or a throttling open state based on the hollow and solid states of the blind plate in each frame of the personnel operation video obtained in step S21; Step S3: Draw a time series diagram corresponding to the key points of the human hand according to the key point information of the human body, and perform action recognition and judgment in the blind plate plugging operation in combination with the wrench and blind plate status; the specific implementation process is as follows: Step S31, extract the coordinate change information of the left hand key point, filter and derive the time series corresponding to the left hand key point, and use the recursive average filtering algorithm to perform noise reduction on the time series of the human body key point by processing the derived time series. The formula is as follows: Where x is the key point information value before filtering, y is the key point information value after mean filtering, and N is the size of the recursive average filtering algorithm window; After filtering, the sliding window method is used to find the gradient corresponding to the time series. The formula is as follows: In the formula, x joint is the key point information value of the hand, N dection is the size of the window, k joint is the gradient value; Step S32, judging the video of the bolt tightening stage. In the bolt loosening stage, in a human skeleton video, when the operator removes the bolt with a wrench, his left hand is fixed, and the other hand rotates the bolt back and forth; firstly, the coordinate change information of the left hand key point is extracted, and the time series corresponding to the left hand key point is first filtered and derived. By processing the derived time series, a threshold is set to judge the frame number set of the operator's left hand when removing a certain bolt, and then the average value of the coordinate information corresponding to the left hand when operating each bolt is obtained, so as to obtain the four coordinate changes of the hand node when the operator removes the bolt, and use the machine learning support vector machine SVM to determine whether the bolt is removed diagonally; Step S33: In the bolting stage, the same operation as step S32 is performed, and the average value of the coordinate information corresponding to the left hand and the right hand when operating each bolt is obtained according to the coordinate change information of the key points of the left hand of the human body, so as to obtain the four coordinate changes of the hand nodes of the operator when removing the bolt, and determine whether the bolt is screwed diagonally by SVM; Step S34: In the video, whether the operator tightens the four bolts diagonally and replaces the blind plate. If so, it means that the operator correctly performs the blind plate plugging operation; if not, it means that the operator incorrectly performs the blind plate plugging operation; Save the final correctness of the operator's operation; Step S4, according to whether the operator removes or installs the bolts in the correct order when operating the blind plate plugging, and whether the operator correctly sets the blind plate status, the test results are obtained and a test report is generated.
2. A blind plate plugging operator operation normative detection system based on intelligent IoT, used to implement the detection method described in claim 1, characterized in that: It includes video acquisition module, human key point detection module, YOLO object recognition module, human operation action recognition module, and evaluation result report module; The video acquisition module is responsible for obtaining information of the evaluation action video stream according to the evaluation requirements; The human key point detection module uses the collected video stream to detect the human key points through an improved human key point recognition algorithm, and records the time series change data and geometric center point coordinates corresponding to the left hand key points of the operator operating the blind plate extraction and plugging in the vertical and horizontal directions; The YOLO object recognition module uses the collected video stream to detect the hollow and solid states of the wrench and blind plate in the video; The personnel operation action recognition module is used to draw a time series diagram corresponding to the key points of the human hand, and to judge the action in the blind plate plugging operation in combination with the wrench and blind plate status; The evaluation result reporting module is used to visualize the detection results obtained by the YOLO object recognition module and the human operation action recognition module, and to prepare an evaluation result report.
3. The blind plate plugging operator operation normative detection system based on intelligent IoT according to claim 2 is characterized in that: The YOLO object recognition module includes the following units: Wrench recognition unit: responsible for identifying whether a wrench appears in each frame of the image; Video segmentation unit: according to the wrench recognition result in the video, the video is divided into the stage of loosening the bolt, the stage of replacing the blind plate, and the stage of tightening the bolt; Blind plate hollow and solid identification unit: judge whether the personnel correctly changes the blind plate from solid to hollow, or from hollow to solid, judge whether the personnel correctly sets the blind plate to the closed blocking state, or sets the blind plate to the throttling open state; Storage unit: responsible for saving the corresponding judgment results.
4. The blind plate plugging operator operation normative detection system based on intelligent IoT according to claim 3 is characterized in that: The personnel operation action recognition module includes the following units: Data analysis unit: used to obtain the corresponding hand joint point information when the personnel operates the blind plate, and the fourth coordinate changes of the hand nodes when the personnel remove the bolts; Judgment unit: Based on the obtained fourth coordinate changes of the human hand, the support vector machine learning method is used to divide the results into two categories, namely diagonal twisting and non-diagonal twisting.
5. According to the system for detecting the standardization of operation actions of blind plate plugging operators based on intelligent IoT in claim 4, the specific implementation process of the operator operation action recognition module is as follows: In the stage of loosening the bolts obtained by the YOLO object recognition module, in a video of a human skeleton, the operator's left hand is fixed when removing the bolts with a wrench, and the other hand rotates the bolt back and forth; the time series change data and geometric center point coordinates of the left hand key points of the blind plate extraction operator obtained after passing through the human key point detection module are used to extract the coordinate change information of the left hand key points, and the time series corresponding to the left hand key points are first filtered and differentiated. By processing the differentiated time series, the set of frames of the operator's left hand removing a bolt is judged by setting a threshold, and then the average value of the corresponding coordinate information of the left hand when operating each bolt is obtained, thereby obtaining the four coordinate changes of the hand nodes when the operator removes the bolts, and judging whether the bolts are removed diagonally. The same method is used to judge in the tightening stage.
