Machine tool operator glove wearing condition detection method

Abstract

The present application belongs to the technical field of intelligent monitoring, and particularly relates to a machine tool operator glove wearing condition detection method. The present application completes the detection of gloves in the manner of artificial neural network. In view of the problem in the prior art that only single material and single glove are detected, and the types of commonly used gloves in the factory cannot be covered, the present application classifies the gloves according to the material and color, and in order to improve the detection efficiency, the present application adds the detection of the operator's empty hand. In view of the situation that the operator leaves the gloves near the machine tool and causes false detection, the present application adds the human key point detection, and determines whether the wrist key point falls in the glove target frame, and finally obtains the determination of the worker wearing gloves.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent monitoring, and particularly relates to a detection method for glove wearing condition of machine tool operators. BACKGROUND

[0002] Modern safety theory believes that the direct cause of safety accidents is the unsafe behavior of people and the unsafe state of things. Therefore, managing and monitoring the unsafe behavior of people and the unsafe state of things is an important content of safety management. Workers' hands directly contact with adverse and harmful factors in the workplace and upper limb injuries are the most common physical trauma. Using hand protection at the right time can greatly reduce the risk of injury; hand safety cannot be separated from the wearing of gloves, and hand safety starts with gloves. In the machining workshop, 20% of the employees are injured every year, of which 52% are hand injuries, and hand safety is crucial. According to the relevant safety operation procedures, gloves cannot be worn when operating machine tools or rotating bodies, which has safety risks. The current safety behavior management cannot solve the safety accidents in human-machine cooperation. Generally speaking, human error is one of the main causes of safety accidents, and the error may be due to the carelessness or lack of awareness of workers. In addition, some automatic machining machine tools check whether the safety cover is covered by using infrared sensors. However, ordinary infrared sensors cannot detect whether workers are operating machine tools without wearing gloves.

[0003] Therefore, the monitoring and management of safety gloves in human-machine cooperation safety need to find a kind of accurate and high-reliability technology. In order to meet the requirements of safe production, it is necessary to ensure that the operator wears the corresponding gloves for operation. The traditional way of equipping security personnel for monitoring in the production workshop not only consumes a lot of manpower, but also the security personnel cannot monitor 24 hours in real time and monitor the safety of each worker. In order to ensure the safety of workers and reduce the cost caused by the increase of security personnel, a visual detection system is needed to monitor whether the workers correctly wear gloves. SUMMARY

[0004] In view of the above problems, the application adopts the artificial neural network to complete the detection of gloves. In view of the problem that the existing technology only detects single material and single gloves, which cannot cover the types of gloves commonly used in factories, the application classifies the gloves according to the material and color, and in order to improve the detection efficiency, the application adds the detection of the operator's empty hands. In view of the situation that the operator leaves the gloves near the machine tool and causes false detection, the application adds the detection of human key points, and determines whether the wrist key point falls within the glove target frame, and finally comprehensively obtains the determination of the worker wearing gloves.

[0005] The technical scheme of the application is as follows:

[0006] A detection method for glove wearing condition of machine tool operators, comprising the following steps:

[0007] S1, acquiring an image near a machine tool;

[0008] S2, judging whether a human body and a glove exist according to the obtained image, if yes, entering S3, otherwise returning to S1; wherein the specific method of judging whether a human body and a glove exist is:

[0009] detecting a human body from the acquired image, if a human body is detected, estimating a posture of the human body using a depth model, completing detection of a wrist key point and outputting (x M ,y M ,Vis M ), x M , y M being position coordinates of the wrist in human body key point detection, Vis M being joint visibility, if the visibility is lower than 0.5, it is considered that the wrist does not exist in the image, i.e. no glove is worn, and whether a glove is worn is judged; if a human body is not detected, returning to S1;

[0010] The method of judging whether a glove is worn is to use an improved YOLOv3 network to judge, the improved YOLOv3 network is to remove the output branches of large and small scales on the basis of the traditional YOLOv3 network model, corresponding to removing the output sizes of 13*13 and 52*52, constructing a single-scale YOLOv3 network architecture, and using up-sampling to fuse large-scale features with small-scale and medium-scale features, so that medium targets have the features of small targets and large targets, and after feature splicing, convolution is performed to obtain a feature map with 9 channels; the input image of the improved YOLOv3 network is divided into S*S grids, and then each grid is detected to detect the target whose center point falls within the grid; each unit cell predicts B boundary box information, including 5 elements: x, y, w, h, C confidence , wherein (x, y, w, h) respectively represent the center coordinates and the relative values of width and height of the boundary box relative to the grid; C confidence is the confidence, which reflects whether a target object is contained and the accuracy rate under the condition that a target object is contained, and is defined as:

[0011]

[0012] wherein Pr(Object) represents the probability that the center point of the object in the image falls within the current grid, represents the intersection over union of the object boundary box predicted by the current network and the real object boundary box:

[0013]

[0014] wherein, BBoxtruth BBox pred respectively represent the real object bounding box and the network predicted bounding box, and are introduced when calculating the bounding box confidence so that the network can judge the quality of the output bounding box; each network also predicts C independent object class probabilities, C being the number of object categories contained in the data set; during training, the object class probability is defined as the conditional probability Pr(C i |Object) when the network contains the object; each grid predicts multiple bounding boxes, each bounding box corresponding to a confidence, by setting a confidence threshold, performing non-maximum suppression to remove bounding boxes with confidence below the threshold, and taking the union of bounding boxes with confidence above the threshold as the prediction result;

[0015] The specific method for training the improved YOLOv3 network is: obtaining actual captured worker images and labeling them as training data, since the improved YOLOv3 network uses a set of feature maps to represent the position and category of objects in the predicted image, the output feature map with 9 channels includes target box position coordinates, target box width and height, and category information, where the category information corresponds to three different gloves and no gloves in the training data; the loss function used is:

[0016]

[0017] where, indicates whether the jth bounding box in the ith grid is responsible for predicting the object in the image, when the jth bounding box in the ith grid is responsible for predicting the object, otherwise, indicates whether there is an object center point in the image whose coordinates fall within the ith grid area, when there is an object center point falling within the ith grid area otherwise, indicates whether the jth bounding box in the ith grid is not responsible for predicting the object in the image; and the value of is opposite to that of ; (x, y, w, h) represents the size of the real object bounding box; indicates the size of the grid predicted bounding box; C and respectively represent the real bounding box confidence and the grid predicted object class probability; λ coord is used to balance the error of the bounding box size in the loss function; λ noobj is used to balance the bounding box confidence score error in the grid area that does not contain the object; the first two terms of the formula represent the regression loss, to strengthen the expression of the error of medium targets, increase and The third and fourth terms are confidence loss errors, and cross-entropy functions are adopted; the last term is a classification loss, and a cross-entropy function is selected as a loss function;

[0018] The improved YOLOv3 network after training obtains output as (x b ,y b ,w b ,w b ,C b ), w b 、h b are the width and height of the target frame output in the glove detection; x b 、y b are the center point positions of the target frame output in the glove detection, and C b is the confidence;

[0019] S3, determining whether the wrist key point falls in the vicinity of the glove target frame, specifically, if:

[0020]

[0021] is established, it is determined that the wrist key point falls in the vicinity of the glove target frame, that is, the gloves are worn, otherwise, the gloves are not worn.

[0022] Further, the three kinds of gloves in the category information are canvas gloves, white gloves and black gloves.

[0023] Further, when it is determined that the gloves are worn, the image is continuously collected within a set time to determine whether the gloves are worn, and if the proportion of wearing gloves exceeds a set threshold, an alarm is issued.

[0024] The present application has the beneficial effects that: the present application proposes a glove detection system based on an artificial neural network to solve the problem that the current glove detection system in the factory basically relies on manual restraint of the operator, uses the improved YOLOv3 network for glove detection, and simultaneously adds human key point detection, so that the accuracy of detecting gloves can be improved and misjudgment can be prevented. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 It is a glove wearing condition detection schematic diagram of the present application.

[0026] Figure 2 It is a feature fusion pyramid schematic diagram.

[0027] Figure 3 It is an improved YOLOv3 network structure schematic diagram.

[0028] Figure 4 It is a flow chart for judging whether gloves are worn.

[0029] Figure 5 The application scenario of the application is shown in the figure.

[0030] Figure 6 The effect identification diagram is shown in the figure. DETAILED DESCRIPTION

[0031] The technical solutions of the application are described below in detail with reference to the accompanying drawings and examples, so that those skilled in the art can better understand the application.

[0032] The improved YOLOv3 network is used for detecting gloves in the application. The improved YOLOv3 used in the application integrates the boundary box prediction and object classification in the traditional target detection method into one network, and uses a set of feature maps to represent the position and category of the object in the image. The input image is in the style of S*S grid, and then each grid is used to detect the target falling in the grid. Each cell predicts B boundary box information, including 5 elements: x, y, w, h, C confidence , where (x, y, w, h) respectively represent the center coordinates and the relative values of the width and height of the boundary box relative to the grid. C confidence is the confidence, which reflects whether the target object is contained and the accuracy rate under the condition that the target object is contained, and is defined as

[0033]

[0034] where Pr(Object) represents the probability that the center point of the object in the image falls in the current grid, represents the intersection over union (IOU) of the boundary box predicted by the current network and the real object boundary box, and the calculation process is as follows:

[0035]

[0036] where BBox truth and BBox pred respectively represent the real object boundary box and the network predicted boundary box. By introducing when calculating the boundary box confidence, the network can judge the quality of the output boundary box. Each network also predicts C independent object category probabilities, and C is the number of object categories contained in the data set. During the training process, the object category probability is defined as the conditional probability of the object category when the network contains the object, Pr(C i |Object). Each grid predicts multiple boundary boxes, each boundary box corresponds to a confidence, by setting the confidence threshold, performing non-maximum suppression, removing the boundary boxes with confidence lower than the threshold, and taking the union of the boundary boxes with confidence higher than the threshold as the prediction result.

[0037] YOLO v3 uses feature maps at three scales for object detection. Each feature map uses anchor boxes as prior boxes, borrowing the idea of ​​anchors from Faster R-CNN. In fact, it is a set of candidate boxes with fixed aspect ratios, which is equivalent to a set of templates. These templates are used for subsequent detection.

[0038] The improvement to the YOLO v3 network structure was based on the concept of feature fusion in the feature pyramid. Figure 2 The example shown is the concept of a feature fusion pyramid. Figure 2 The feature pyramid on the right is generated from the feature pyramid on the left. The process is as follows: First, convolution is performed on the input image to obtain layer 1. Then, convolution is performed on layer 1 to obtain layer 2. Finally, layer 4 is upsampled to the same size as layer 2. This step aims to prevent layer 2 from losing features of large objects due to convolution, allowing it to incorporate features from layer 4, which has a larger receptive field. Afterward, convolution is performed on the processed layers 2 and 4, and the result is input into layer 5. Similarly, feature fusion is performed on each corresponding layer to obtain multiple sets of feature maps for prediction. This operation can construct a deeper feature pyramid, fusing multi-layer feature information and performing predictions on different feature maps.

[0039] In this invention, the glove's position and size are relatively fixed. To more accurately predict the glove's bounding box position using YOLOv3 regression, K-Means clustering is performed on the dataset, resulting in most glove bounding boxes falling into the medium-sized bounding box category. Borrowing from the feature pyramid concept of feature fusion, convolution is performed on layer 6, fusing it with layer 5 to obtain a fused medium-sized target prediction. To prevent the network from losing features of large targets, the predictions for large targets in layer 4 are then upsampled and fused with the resulting fused medium-sized target prediction to obtain the final output. Finally, the output branches for large and small targets are removed, as are the 13×13 and 52×52 output sizes, constructing a single-scale YOLO v3 network architecture.

[0040] Drawing on the aforementioned feature fusion approach, YOLO v3 is improved by using upsampling to fuse high-level and low-level features. To better predict medium-sized targets, low-level and high-level information is further fused, allowing medium-sized targets to possess features of both small and large targets. This results in a single set of feature maps, which are then used for prediction. The improved network structure is as follows: Figure 3As shown, the input image sequentially passes through the first convolution module, the first residual module, the second residual module, the third residual module, the fourth residual module and the fifth residual module to obtain the first feature, and the output of the fourth residual module is defined as the second feature, the output of the second residual module is defined as the third feature, and the first feature passes through the second convolution module to obtain a large-scale feature; the first feature passes through the third convolution and upsampling module, is fused with the second feature, and then passes through the fourth convolution module to obtain a medium-scale feature; the output of the fourth convolution module passes through the fifth convolution and upsampling module, is fused with the third feature, and then passes through the sixth convolution module to obtain a small-scale feature; the small-scale feature passes through the seventh convolution module, the large-scale feature passes through the eighth convolution and upsampling module, and is fused with the medium-scale feature, and then passes through the ninth convolution module to obtain the final output feature map.

[0041] Compared with the traditional 3-scale output, the network is faster in predicting the target in the middle, and the detailed information after feature splicing is more abundant. In order to avoid the problem of too many parameters in the network structure caused by increasing the channel, the channel number is re-normalized to 9 after convolution after feature splicing. The dimensions of the output tensors in the network are all hyperparameters, and after training and verification, the tensors with dimensions that are multiples of 2 can guarantee better detection effect. Only the dimension of the final output tensor is 9, and the reason is that each candidate box needs to predict 9 pieces of information, that is, it needs to output the target box position (two-dimensional, that is, the horizontal coordinate and the vertical coordinate), the width and height of the target box (two-dimensional, that is, the width and the height), the confidence (one-dimensional), and the category information (four-dimensional, that is, the old canvas gloves, the white gloves, the black gloves, and the empty hands). Then, the batch regularization algorithm is introduced to speed up the network convergence, and the full convolution network structure is trained with multi-scale images to improve the generalization ability of the network.

[0042] The method for making training data is as follows: using LabelImg tool, according to the standard of VOC 2007, using a rectangular frame to mark the category information and position information of the target to be detected. The generated file is stored in the *.xml file under the Annotation folder of VOC 2007, and the image data is stored under the ImgData folder. The images in the data set are all generated by frame-by-frame screenshot of the actual video shot in the factory. The category information and position information of the target to be detected are marked on the image. After data augmentation, a total of 4526 images are obtained, including 1847 old canvas gloves, 1296 white gloves, 965 black gloves and 418 empty hands. The data set is divided into training set and test set according to the ratio of 7:3, the training set is input into the improved YOLO v3 network for training, and then the test set is used for verification. The network will be optimized in the direction of minimum loss.

[0043] Accordingly, to better suit this invention, the loss function has also been improved. In the YOLO v3 object detection algorithm, the loss function consists of three parts: regression loss, classification loss, and confidence loss. Based on the improvements to the network structure described above, the regression loss and classification loss in the YOLO v3 algorithm are optimized: the regression box error is expressed using the standard deviation of width and height; the confidence loss and classification loss are expressed using the binary cross-entropy function, which is then squared, as follows:

[0044]

[0045] in, This indicates whether the j-th bounding box within the i-th grid is responsible for predicting objects in the image. When the j-th bounding box within the i-th grid is responsible for predicting objects... on the contrary, This indicates whether the center point coordinates of an object in the image fall within the i-th grid region. If the center point of an object falls within the i-th grid region... on the contrary, This indicates whether the j-th bounding box within the i-th grid is not responsible for predicting objects in the image. and The values ​​of (x, y, w, h) are opposite; (x, y, w, h) represents the actual bounding box size of the object. Indicates the grid prediction bounding box size; C and λ represents the true bounding box confidence and the grid-predicted object class probability, respectively. coord It is mainly used to balance the boundary size error ratio in the loss function, where λ coord =5; λ noobj This is primarily used to balance the confidence score error of bounding boxes within mesh regions that do not contain objects. Since foreground objects are relatively few in the image, the confidence scores of most bounding boxes are constrained to 0. Therefore, appropriately reducing this error is beneficial for faster mesh convergence and improves the network's recall. Here, λ... noobj =0.5. The first two terms of the formula represent the regression loss. To strengthen the expression of error for moderate targets, an additional term is added before this term. and The coefficients are: the third and fourth terms are the confidence loss error, which are optimized into the cross-entropy function to transform the confidence problem into a binary classification problem and accelerate the convergence of training; the last term is the classification loss, which selects the cross-entropy function as the loss function to reduce the classification of target objects to a binary classification problem and avoid the problem of missed detection caused by multiple overlapping targets.

[0046] After training the network as described above, the final output of the network is: (x b ,yb ,w b ,w b ,C b ), respectively, representing the coordinates of the center of the target frame, size, and confidence. The detection accuracy is evaluated by using mAP, and the larger the mAP, the better the detection effect. mAP is equal to the average accuracy (AP value), and the AP value is the area of the R-P curve surrounded by the recall rate (Recall) and the accuracy (Precision); using the Python project file MAP-Master, the mAP value of the improved network is shown in Table 1. After testing, the detection frame rate can reach 30fps, meeting the real-time requirement.

[0047] Table 1 mAP values of different targets

[0048] Class Old canvas gloves White gloves Black gloves Empty hands mAP 98.052％ 97.397％ 94.396％ 91.684％

[0049] Then MediaPipe is introduced to detect human key points. Using MediaPipe to detect human key points can be divided into the following steps:

[0050] 1. Preprocessing: using OpenCV related functions, the input image is normalized and color space converted so that it can be sent to the model for processing.

[0051] 2. Human detection: in the preprocessed image, a machine learning model (such as a deep learning model) is used to detect the human body in the image. In MediaPipe, the already trained model BlazePose is used for human detection.

[0052] 3. Key point detection: once the human body is detected, another deep learning model is used to detect the key points of the human body. These key points are important landmarks of the human body, such as head, shoulder, elbow, wrist, knee, ankle, etc.

[0053] 4. Pose estimation: after detecting the key points of the human body, another deep learning model is used to estimate the pose of the human body. This model can predict the relationship between the key points of the human body, such as the distance between the shoulder and the ankle, the angle between the elbow and the wrist, etc. These relationships can be used to calculate the pose of the human body, such as the direction and posture of the body.

[0054] 5. Output results: finally, the three-dimensional coordinates of each key point of the human body and the visibility of each coordinate point (x M ,y M ,Vis M ) are output. Visibility means whether the key point is clearly displayed, and in this scheme, only those with a visibility higher than 0.5 are displayed.

[0055] After combining the output information of the glove detection and the human key point detection (x b ,y b ,w b ,h b ,C b ), (x M ,y M ,Vis M ), comprehensive judgment is carried out. The judgment method is as shown in Figure 4 :

[0056] The method for determining whether the wrist key point falls in the vicinity of the target frame of the glove is as follows:

[0057]

[0058] wherein w b , h b are the width and height of the target frame output in the glove detection; x b , y b are the center point positions of the target frame output in the glove detection; x M , y M are the position coordinates of the wrist in the human key point detection.

[0059] In actual application, the specific connection mode and the functions of various components are as shown in Figure 4 .

[0060] Embodiment

[0061] In this example, the training data is obtained by shooting 720p, 30fps machine tool working condition videos by a handheld camera. The video types include videos of workers wearing old canvas gloves, white gloves (brand new canvas gloves and white line gloves), black gloves and no gloves (empty hands). Then all the videos are imported into PotPlayer, and the videos are intercepted according to the method of intercepting every 10 frames, obtaining 4526 images, of which 1847 are old canvas gloves, 1296 are white gloves, 965 are black gloves and 418 are empty hands.

[0062] After that, all images are imported into LabelImg for image annotation. According to the training requirements of YOLO, the VOC 2007 format is used for annotation, and the class information and position information of the target to be detected are marked using a rectangular box. The generated file is stored in the *.xml file under the Annotation folder of VOC 2007, and the image data is stored under the JPEGImages folder. Then, according to the 7:3 ratio, the dataset is divided into training set and test set. The classified test set file name is stored under ImageSets. In order to realize the image at different times, data enhancement is performed on the test set, that is, the contrast, brightness is adjusted, and the image is mirror flipped.

[0063] The example is based on Windows 10 operating system, realized by Python language programming, and uses PyTorch as a deep learning framework. The batch size in training is set to 20, and the epoch is set to 2000. After the image is input into the network, the specific operation is as follows:

[0064] Taking an image with an input height of 416, a width of 416, and a channel number of 3 as an example, after inputting it into the improved YOLOv3 network, the following operations will be performed.

[0065] 1. First, a convolution operation is performed, in which the size of the convolution kernel is 3, and an image with a height of 416, a width of 416, and a channel number of 32 (hereinafter referred to as 416*416*32) is output.

[0066] 2. The 416*416*32 image obtained in step 1 is input into a residual block (Residual Block). Each residual block involves convolution, batch normalization processing, LeakyReLU, etc. An image with a size of 208*208*64 is output.

[0067] 3. The 208*208*64 image obtained in step 2 is input into two consecutive residual blocks to obtain an image with a size of 104*104*128.

[0068] 4. The 104*104*128 image obtained in step 3 is input into 8 consecutive residual blocks to obtain an image with a size of 52*52*256.

[0069] 5. The 52*52*256 image obtained in step 4 is input into 8 residual blocks to obtain an image with a size of 26*26*512.

[0070] 6. The 26*26*256 image obtained in step 5 is input into 4 consecutive residual blocks to obtain an image with a size of

[0071] 13*13*1024 image.

[0072] 7. The 13*13*1024 image obtained in step 6 is convolved to obtain a 13*13*1024 image.

[0073] 8. The 13*13*1024 image obtained in step 7 is convolved with a 3*3 kernel and a 1*1 kernel to obtain a 13*13*27 image.

[0074] 9. The 13*13*1024 image obtained in step 7 is convolved and upsampled to obtain a 26*26*256 image.

[0075] 10. The 26*26*256 image obtained in step 9 is fused with the 26*26*512 image obtained in step 5 to obtain a 26*26*768 image.

[0076] 11. The 26*26*768 image obtained in step 10 is convolved to obtain a 26*26*256 image containing more details.

[0077] 26*26*256 image.

[0078] 12. The 26*26*256 image obtained in step 11 is convolved with a 3*3 kernel and a 1*1 kernel to obtain a 26*26*27 image.

[0079] 13. The 26*26*256 image obtained in step 10 is convolved and upsampled to obtain a 52*52*128 image.

[0080] 14. The 52*52*128 image obtained in step 13 is fused with the 52*52*256 image obtained in step 4 to obtain a 52*52*384 image.

[0081] 15. The 52*52*384 image obtained in step 14 is convolved to obtain a 52*52*128 image.

[0082] 16. The 52*52*128 image obtained in step 15 is convolved with a 3*3 kernel and a 1*1 kernel to obtain a 52*52*27 image.

[0083] 17. The 52*52*27 image obtained in step 16 is convolved with a 1*1 kernel to obtain a 26*26*27 image.

[0084] 18. Feature fusion is performed on the 26*26*27 image obtained in step 17 and the 26*26*27 image obtained in step 12 to obtain a 26*26*54 image.

[0085] 19. Convolution and up-sampling are performed on the 13*13*27 image obtained in step 8 to obtain a 26*26*27 image. Feature fusion is performed on the 26*26*27 image and the 26*26*54 image obtained in step 18 to obtain a 26*26*81 image.

[0086] 20. After the 26*26*81 image obtained in step 19 is subjected to a 3*3 convolution kernel and a 1*1 convolution, a final output 26*26*27 image is obtained, which is taken as the final output result of the network.

[0087] Taking the image input and output of the first residual block as an example, the calculation involved is:

[0088] F (1) = Conv(F (0) , K 1 )

[0089] wherein Conv represents convolution calculation, F (0) represents a 416*416 32-dimensional tensor formed by the size and channel number of the input image, K 1 represents a filter with a size of 3, a step of 2, and a padding of 1. F (1) represents a 208*208 64-dimensional tensor formed by the size and channel number of the output image.

[0090] F (2) = Conv(F (1) , K 2 )

[0091] wherein Conv represents convolution calculation, F (1) represents a 208*208 32-dimensional tensor formed by the size and channel number of the input image, K 2 represents a filter with a size of 1, a step of 1, and a padding of 0. F (2) represents an image composed of a 208*208 32-dimensional tensor.

[0092] F (3) = BN(F (2) )

[0093] wherein BN represents batch normalization processing (Batch Normalization), F (2) represents the size and channel number of the input image of this batch, and F (3) represents the size and channel number of the output image of this batch.

[0094]

[0095] where LeakyRuLU represents a Leaky ReLU activation function, F (3) represents the size and channel number of the input image in this batch, F (4) represents the size and channel number of the output image. When the input feature is less than 0, the activation function can properly preserve the feature information. Therefore, when the output value of a certain neuron is less than 0, the LeakyReLU back propagation gradient is not 0, avoiding the problem that the output feature value is less than 0, the neuron dies, and the network cannot be trained, caused by ReLU as an activation function.

[0096] F (5) = Conv(F (4) , K 3 )

[0097] where Conv represents a convolution calculation, F (4) represents a 208*208 32-dimensional tensor formed by the size and channel number of the input image, K 3 represents a filter with a size of 3, a step of 1, and a padding of 1. F (5) represents an output image composed of a 208*208 64-dimensional tensor.

[0098] F (6) = BN(F (5) )

[0099] where BN represents a batch normalization process, F (5) represents the size and channel number of the input image in this batch, F (6) represents the size and channel number of the output image.

[0100] F (7) = LeakyRuLU(F (6) )

[0101] where LeakyRuLU represents a Leaky ReLU activation function, F (6) represents the size and channel number of the input image in this batch, F (7) represents the size and channel number of the output image.

[0102] Taking the upsampling operation in the 9th step as an example, an input of a 13*13 1024-dimensional tensor is inserted with a point around the 13*13 pixels, so as to form a 26*26 image. Generally, the nearest neighbor pixel interpolation method is used to fill the new pixels:

[0103]

[0104] wherein represents rounding down.

[0105] Take the fusion in step 10 as an example, the fusion means adding two input tensors to obtain a higher dimensional tensor, that is, input a 256-dimensional tensor and a 512-dimensional tensor to finally obtain a 768-dimensional tensor, wherein the size of the matrix does not change.

[0106] Take the matrix composed of the input image (take a 9*9*1 matrix as an example) as follows:

[0107]

[0108] Randomly take a filter with a size of 3, a step of 1, and a padding of 1:

[0109]

[0110] The operation of convolution (generally identified as ) is to slide the filter window with a step and apply it, multiply the corresponding elements of the input elements of the filter at each position, and then sum. Then save this result to the corresponding position of the output. Perform this process on all positions to obtain the output of the convolution operation. Take the upper left 3*3 part as an example, because the padding is 1, it is necessary to pad outside the matrix (the number filled is generally 0).

[0111]

[0112] After the 9*9 matrix is subjected to the convolution operation, the following is obtained:

[0113]

[0114] The output matrix is subjected to batch normalization (in actual operation, batch normalization needs to be processed on the entire batch, and the size of the batch is generally a multiple of 2, and here the batch number is taken as an example) as follows:

[0115] 21. Calculate the mean of each column (take the first column as an example):

[0116]

[0117] μ = [102.667, 165.444, 156.778, 144.333, 143.222, 157.000, 169.000, 168.778, 129.444]

[0118] 22. Center each column in the matrix:

[0119] F μ(0, 0) = 92 - 102.667 = -10.667

[0120]

[0121] 23. Calculate the variance and standard deviation of each column:

[0122]

[0123]

[0124] σ 2 = [903.778, 1101.802, 815.062, 1340.889, 152.395, 760.444, 1911.333, 2836.840, 695.136]

[0125] σ = [30.063, 33.193, 28.549, 36.618, 12.345, 27.576, 43.719, 53.262, 26.365]

[0126] 24. Standardization:

[0127]

[0128]

[0129] 25. Take the scaling ratio as 1, scale and take the translation as 0, so the output is:

[0130]

[0131] Perform the operation of LeakyReLU on the output matrix, that is, judge each value in the output matrix. If it is greater than 0, output the original value; if it is less than zero, output 0.1*original value.

[0132]

[0133] The network will repeat the calculation according to the above process until it reaches the maximum training number or reaches a lower loss. In this process, the filter is updated, and finally the appropriate filter parameters are formed. After inputting an image, the network will feedback the position coordinates of the gloves contained in it, the target frame size and confidence (x b , y b , w b , h b , C b ). And combined with the wrist position coordinates (x M , y M , Vis M ) given by MediaPipe to judge. Take xb = 216, y b = 204, w b = 30, h b = 20; x M = 201, y M = 200.

[0134]

[0135] Therefore, it is determined that the operator wears gloves. As shown in Figure 6 , the wrist key point falls within the target box.

[0136] The actual application of this example is shown in Figure 1 , if the final return wears gloves, record the operator in this operation has a violation, and return 1; if not wearing gloves, return 0. In order to prevent the accuracy of the system from being affected by the sudden intrusion of other personnel, the worker taking off the gloves and misjudgment, this scheme gives some fault tolerance to the time of the worker wearing gloves, that is, collect the images within 2s, count the cases of the worker wearing gloves. If the proportion of the worker wearing gloves within 2s reaches 80%, the development board will issue an alarm sound; if the proportion is not reached, no alarm will be given. For the operator who appears to have violated the rules, the development board will also upload the machine number and record the time when the violation operation occurs.

Claims

1. A machine tool operator glove wearing condition detection method characterized by, The method comprises the following steps: S1, acquiring an image near the machine tool; S2, judging whether there is a human body and a glove according to the obtained image, if yes, entering S3, otherwise returning to S1; wherein the specific method of judging whether there is a human body and a glove is: Human body detection is performed on the acquired image, if human body is detected, pose of the human body is estimated using a depth model, detection of wrist key points is completed and (x M ,y M ,Vis M ) is outputted, x M and y M are position coordinates of the wrist in human body key point detection, Vis M is joint visibility, if the visibility is lower than 0.5, it is considered that the wrist joint does not exist in the image, and whether a glove is worn is judged; If no human body is detected, return to S1; The method for judging whether gloves are worn is to use an improved YOLOv3 network for judgment, the improved YOLOv3 network is based on the traditional YOLOv3 network model, removes the output branches of large and small scales, corresponds to removing the output sizes of 13*13 and 52*52, constructs a single-scale YOLOv3 network architecture, and uses up-sampling to fuse large-scale features with small-scale features and medium-scale features, so that the medium target has the features of small targets and large targets, and after feature splicing, convolution is performed again to obtain a feature map with 9 channels; the input image of the improved YOLOv3 network is divided into S*S grids, and then each grid is detected to detect the target whose center point falls in the grid; each cell predicts B bounding box information, including 5 elements: x, y, w, h, C confidence , wherein (x, y, w, h) respectively represent the relative values of the center coordinates and the width and height values of the bounding box relative to the grid; C confidence is the confidence, which reflects whether a target object is contained and the accuracy under the condition that a target object is contained, and is defined as: where Pr(Object) represents the probability that the center point of an object in the image falls within the current grid, representing the intersection over union of the predicted object bounding box of the current network and the real object bounding box: wherein, BBox truth and BBox pred respectively represent the real object bounding box and the network predicted bounding box, and the makes the network judge the quality of the output bounding box; each network also predicts C independent object class probabilities, C being the number of object categories contained in the dataset; during the training process, the object class probability is defined as the conditional probability Pr(C i |Object) when the network contains the object; each grid predicts multiple bounding boxes, each of which corresponds to a confidence value; by setting a confidence threshold, non-maximum suppression is performed to remove bounding boxes with confidence values below the threshold, and the union of bounding boxes with confidence values above the threshold is taken as the prediction result; The specific method of training the improved YOLOv3 network is: obtaining the actually photographed worker image and labeling it as training data, because the improved YOLOv3 network uses a set of feature maps to represent the position and category of objects in the predicted image at the same time, the output feature map with 9 channels includes target frame position coordinates, target frame width and height, and category information, wherein the category information corresponds to three different gloves and no gloves in the training data, wherein the classification method of the three different gloves is according to the material or color; the loss function adopted is: wherein, represents whether the jth bounding box in the ith grid is responsible for predicting the object in the image, when the jth bounding box in the ith grid is responsible for predicting the object, on the contrary, represents whether there is an object center point coordinate in the image falling in the ith grid area, when there is an object center point falling in the ith grid area on the contrary, represents whether the jth bounding box in the ith grid is not responsible for predicting the object in the image; and the value of is opposite to that of (x, y, w, h) represents the real object bounding box size; represents the grid prediction bounding box size; C and respectively represent the real bounding box confidence and the grid prediction object category probability; λ coord is used to balance the bounding box size error ratio in the loss function; λ noobj is used to balance the bounding box confidence score error in the grid area not containing the object; the first two terms of the formula represent the regression loss, in order to strengthen the error expression of the medium target, the coefficients of and and are increased before the term; the third and fourth terms are the confidence loss error, and the cross entropy function is adopted; the last term is the classification loss, and the cross entropy function is selected as the loss function; The trained improved YOLOv3 network obtains outputs of (x b ,y b ,w b ,w b ,C b ) after inputting an image b ,h b are the width and height of the target frame output in the glove detection; x b ,y b are the center point positions of the target frame output in the glove detection, and C b is the confidence. S3, judging whether the wrist key point falls near the glove target frame, specifically, if: If yes, it is determined that the wrist key point falls near the glove target frame, that is, wearing gloves, otherwise, not wearing gloves.

2. The machine operator glove wearing condition detection method according to claim 1, characterized by, The three gloves in the category information are respectively canvas gloves, white gloves and black gloves.

3. The machine operator glove donning condition detection method according to claim 1, characterized by, After determining that gloves are worn, continue to collect images within a set time to judge whether gloves are worn, and if the proportion of wearing gloves exceeds the set threshold, an alarm is issued.

Images