A workshop personnel dangerous behavior detection method under complex environment

Abstract

The present application belongs to the field of computer vision safety monitoring, in particular to a workshop personnel dangerous behavior detection method under complex environment, specifically, skeleton joint information is collected and pretreated; reliability is calculated, and the wrong skeleton joint is reconstructed; a human three-dimensional space-time topology graph is constructed based on human joint data; the three-dimensional topology graph is trained by neural network to obtain a dangerous behavior detection model; the constructed human three-dimensional space-time topology graph is imported into the trained behavior recognition model for behavior detection. Compared with the prior art, in the data set preprocessing, the skeleton data of non-human bodies such as product equipment is filtered out, the space-time graph convolution network is run to extract the space-time dimension characteristics of personnel behavior, and the accurate detection of personnel behavior is realized; through reliability calculation, the wrong data is identified, and the human joint is reconstructed, thereby improving the effect of personnel behavior detection under the shielding scene in the workshop.

Description

Technical Field

[0001] This invention belongs to the field of computer vision safety monitoring, and in particular, it is a method for detecting dangerous behaviors of workers in complex environments. Background Technology

[0002] With the continuous development of industrialization, the level of intelligence and digitalization in workshop production activities has increased accordingly. Currently, frequent safety accidents in workshops pose a serious threat to the personal safety of workshop personnel and factory production activities. Analysis of the causes of these accidents reveals that dangerous behavior by workshop personnel is one of the key factors. However, manual monitoring is costly, and with numerous monitoring points in the workshop, supervisors are prone to oversights due to visual fatigue. Therefore, relying on traditional monitoring methods to prevent dangerous behavior is inefficient and has limited effectiveness. Thus, researching intelligent detection methods for workshop personnel behavior is of great significance for ensuring safe production in enterprises. With the rapid development of computer vision technology, research on workshop personnel behavior detection has made some progress. However, current behavior detection still faces some problems. In the complex environment of a workshop, there is a large accumulation of equipment or products. When workers enter the work scene, depth sensors may collect noisy personnel feature data due to occlusion by equipment or products, thereby reducing the performance of the behavior detection model. This paper proposes a method for detecting dangerous behavior of workshop personnel in complex environments based on computer vision technology. The specific content is as follows. Summary of the Invention

[0003] The purpose of this invention is to provide a method for detecting dangerous behaviors of personnel in complex workshop environments. This method is based on computer vision technology and can identify dangerous behaviors of personnel in complex workshop environments.

[0004] The technical solution to achieve the purpose of this invention is: a method for detecting hazardous behaviors of workers in complex environments, the specific steps of which are as follows:

[0005] Step 1: Collect skeleton joint information and perform preprocessing;

[0006] Step 2: Reliability calculation, reconstructing erroneous skeleton joints;

[0007] Step 3: Construct a three-dimensional spatiotemporal topology map of the human body based on human joint data;

[0008] Step 4: Train a neural network using the collected correct key point information to create a 3D topology map, thereby obtaining a dangerous behavior detection model;

[0009] Step 5: Import the three-dimensional spatiotemporal topology map of the human body constructed in Step 3 into the behavior recognition model trained in Step 4 for behavior detection.

[0010] Compared with the prior art, the present invention has the following significant advantages:

[0011] (1) This invention proposes a behavior detection method suitable for complex workshop environments. In complex environments, the personnel feature data obtained by depth sensors are noisy or incomplete. Human joint reconstruction technology is used to improve and optimize the erroneous joint data, thereby improving the performance of the behavior detection model in occluded scenes in the workshop.

[0012] (2) This invention uses skeleton data for behavior detection, reducing the time for data processing during behavior detection; in the preprocessing of the dataset, skeleton data of non-human bodies such as products and equipment are filtered out, improving the efficiency of behavior detection; the runtime spatiotemporal graph convolutional network effectively extracts the spatiotemporal dimension features of human behavior, achieving accurate detection of human behavior. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating a method for detecting hazardous behaviors of workshop personnel in a complex environment, according to an embodiment of the present invention.

[0014] Figure 2 A schematic diagram of the joints of the human skeleton;

[0015] Figure 3 This is a network diagram for behavior detection.

[0016] Figure 4 This is a diagram illustrating the search for incorrect key points. Detailed Implementation

[0017] The present invention will now be described in further detail with reference to the accompanying drawings.

[0018] The present invention provides a method for detecting hazardous behaviors of workers in complex environments in workshops, comprising the following steps:

[0019] Step 1: Collect skeleton joint information and perform preprocessing;

[0020] Step 1.1: Deploy depth vision sensors (Microsoft Kinect V2) at key workstations in the workshop, installing two depth sensors at each workstation; and collect the coordinate information of the joint points of the human skeleton in the workshop.

[0021] Skeletal joint information of human behavior was collected using depth vision sensors (Microsoft Kinect V2) deployed in the workshop. The dataset was acquired at a rate of 30 frames per second, with each frame containing 25 skeletal joints of the human body (as shown in the attached image). Figure 2 The meaning of each key point is shown in Table 1. The absolute coordinates of key point i are represented as (x...). i ,yi ,z i ), i = 1, 2…25. The origin of this coordinate system is located at the center of the Kinect infrared camera. The X-axis is to the left along the direction of the camera's illumination, the Y-axis is to the up along the direction of the camera's illumination, and the Z-axis is to the right along the direction of the camera's illumination.

[0022] Table 1 Meaning of each joint point

[0023]

[0024]

[0025] Step 1.2: Perform preprocessing operations on the collected dataset;

[0026] (1) Filtering of non-human skeletal joint data such as product equipment

[0027] Depth vision sensors sometimes mistakenly collect data on the skeletal joints of non-human objects (structurally similar to human skeletal joints). Therefore, it is necessary to remove this misidentified skeletal data. The method is to select the average displacement value of 25 node coordinates between adjacent frames, calculated as shown in the following formula:

[0028]

[0029] Wherein, the absolute coordinates of key point i in frame t are defined as (x... ti ,y ti ,z ti K represents the average displacement of the 25 node coordinates at frame t and frame t+1. The threshold is set to 0.01m. When K is less than this value, the data is filtered out.

[0030] (2) Coordinate transformation

[0031] The coordinates of the 3D skeleton joints in the absolute coordinate system are represented using a relative coordinate system. For ease of processing, the mid-hip (node ​​1) is chosen as the origin of the relative coordinate system, and all joints are transformed. The specific method is shown in the following formula:

[0032]

[0033] The coordinates (x, y) of the absolute coordinate system of the joint i i ,y i ,z i After transformation, the coordinates (x') in the relative coordinate system are obtained. i ,y' i ,z' i ).

[0034] Step 2: Reliability calculation, reconstructing faulty skeleton joints;

[0035] The Kinect depth vision sensor was used to collect the coordinate information of the human skeleton joints in the workshop. The human joints were searched using a tree method. The skeleton data classification algorithm based on joint constraints was used to calculate the reliability of the collected joint data and classify the joint data. Kalman filtering was used to re-predict the incorrect skeleton joint information, and the nodes predicted by Kalman filtering were corrected with reference to the constraint of bone length, so as to reconstruct the incorrect joints.

[0036] Step 2.1: Based on the node coordinate information obtained in Step 1.1, a tree-like approach is used to search for the joints and retrieve the key points.

[0037] First, identify the erroneous joints. Based on the human body's joint structure information, use a tree-based search, selecting the left shoulder (node ​​5), right shoulder (node ​​9), left hip (node ​​13), right hip (node ​​17), and neck (node ​​3) as the root nodes. Figure 4 The diagram illustrates the search for faulty joints, with arrows indicating the search direction. Taking the left shoulder (node ​​5) as an example, the search direction is left shoulder (node ​​5) → left elbow (node ​​6) → left wrist (node ​​7) → left hand (node ​​8). These joints are searched one by one using this method. The meaning of each joint is shown in Table 1.

[0038] Step 2.2: Reliability calculation to identify faulty nodes;

[0039] Because products or equipment may obstruct the worker's body, the joint data acquired by the Kinect sensor contains some erroneous data. To process this erroneous data, it is necessary to classify and differentiate the joint data. Erroneous joint data is generally judged from two aspects: first, the shaking of the body joint points, i.e., changes in movement speed; second, abnormal joint positions, such as certain parts of the body suddenly becoming longer or shorter, leading to body incoordination. Based on these two aspects, a skeleton data classification algorithm based on joint constraints is proposed. The joint reliability is mainly calculated through two parts: (1) reliability calculation based on joint movement speed; (2) reliability calculation based on bone length.

[0040] (1) Reliability calculation based on joint motion velocity:

[0041] When the Kinect sensor cannot accurately estimate the position of a keypoint due to occlusion, that keypoint will exhibit jitter. The degree of jitter at a particular node can be determined by calculating the speed of joint movement. Let the three-dimensional coordinates (x, y, z) of keypoint i in frame t and frame t+1 be... ti ,yti ,z ti ), (x (t+1)i ,y (t+1)i ,z (t+1)i Therefore, the displacement S of the joint can be calculated by the following formula:

[0042]

[0043] Since Kinect's frame rate is 30 frames per second, the time interval t between each frame is 1 / 30 second, and the joint motion velocity v is calculated by the following formula:

[0044]

[0045] Referring to the normal running speed of an adult as 10km / h, we set the speed threshold to 12km / h. If the speed of a certain joint exceeds the threshold after calculating the speed of the joint, the corresponding joint at this moment can be identified as abnormal joint data, that is, the joint data needs to be reconstructed.

[0046] (2) Reliability calculation based on bone length:

[0047] The human skeleton is equivalent to a hinge mechanism. The length of the skeleton should be a constant during movement. The length of the skeleton is represented by the Euclidean distance between two adjacent joints, as shown in the following formula:

[0048]

[0049] Where i and j represent the labels of human joints and the length of bones. i_j This represents the length of adjacent joints i and j.

[0050] The distance between each skeletal point when the entire body of the person is tracked by the Kinect depth sensor in the first frame is used as the reference length.

[0051] Define the total number of joints connected to joint i as S. joint_num Let f be the line segment between the connected joints, and let d be the ratio of the length of line segment f to the reference length in frame t. f (t) is calculated by the following formula:

[0052]

[0053] In the formula l f_std Let l be the reference length of line segment f. f (t) represents the actual length of line segment f in frame t.

[0054] The reliability of a joint based on bone length is determined by the average of the proportions of the differences between the joint segments connected to it and the reference length, as shown in the following formula:

[0055]

[0056] D(t) measures the difference in bone length between joint point i and its neighboring points. In this paper, the threshold for bone length change between consecutive frames is set to 30%. If the value of D(t) is greater than the threshold, it indicates that the skeleton joint information of this frame is abnormal data and needs to be reconstructed.

[0057] Step 2.3: Reconstruct the erroneous joint data using Kalman filtering;

[0058] Step 2.3.1: Using the erroneous joint coordinates (X1, Y1, Z1) obtained in Step 2.2, perform prediction using Kalman filtering. The predicted joint coordinates are:

[0059] Step 2.3.2: Using the bone reference length obtained in Step 2.2 as a constraint, adjust the joint coordinates predicted in Step 2.3.1. Adjustments are made, with the faulty node as its child node and its predecessor node as its parent node (X2, Y2, Z2). Since the estimated bone length between the node position and its parent node remains unchanged, the bone length l between the two nodes is referenced in the reliability based on bone length. s_std Therefore, it is estimated that the joint should be located in a circle with the parent node's position coordinates (X2, Y2, Z2) as the center and a radius of l. f_std On the sphere, the constraint equations are as follows:

[0060] (X2-X) 2 +(Y2-Y) 2 +(Z2-Z) 2 =l f_std 2

[0061] Step 2.3.3: Select the spherical surface relative to the estimated joint position The point with the closest Euclidean distance is used as the optimized estimate of the joint position. That is, first establish The equation of the straight line in space (X2,Y2,Z2) is shown below:

[0062]

[0063] Solving the two formulas together, we get... The result is:

[0064]

[0065]

[0066]

[0067] Selection and The solution with the minimum Euclidean distance is used as the optimized joint position.

[0068] Step 3: Construct a three-dimensional spatiotemporal topology map of the human body based on human joint data;

[0069] The spatiotemporal topology of the human body's 3D skeleton is formed by concatenating and stitching together the reconstructed erroneous joints from step 2 with the originally correct human joints. The 3D skeleton spatiotemporal topology consists of a set of joints and a set of edges, as shown in the following equation:

[0070] G = (V, E)

[0071] Where G is the spatiotemporal topology of the human three-dimensional skeleton, and V is the set of skeleton joint data in a relative coordinate system, V = {v ti |t=1,2,…,T,i=1,2,…,N}, where t represents the t-th frame of the skeleton node data, i represents the index of the joint (as shown in Table 1), and N=25, representing 25 skeleton joints; while the edge set E={E}, consisting of the spatial and temporal edges of the skeleton, is... S E T}, where E S ={v ti v tj |(i,j)∈H} represents the skeletal edges where joints are naturally connected in the skeleton graph space, H is the set of naturally connected joint pairs in the human body, and E T ={v ti v (t+1)i} is the time edge connecting the same skeleton joint in frame t and frame t+1, representing the dynamic spatial position change of a skeleton joint over time.

[0072] Step 4: Train a neural network using the collected correct key point information to create a 3D topology map, thereby obtaining a dangerous behavior detection model;

[0073] Step 4.1: Define the types of hazardous behaviors designed according to workshop requirements;

[0074] The hazardous behavior categories are designed according to the workshop requirements. These behaviors include normal behaviors and hazardous behaviors. Normal behaviors include handling and walking. Hazardous behaviors include running, jumping, leaning against products or equipment, using communication equipment, smoking, entering the equipment area, throwing objects, and spinning.

[0075] Step 4.2: Preprocess the node coordinate information obtained in Step 1.2, assuming that the information collected during model training is correct joint point information; then, according to the operation in Step 3, construct a three-dimensional topology map of the human body from the preprocessed joint position data.

[0076] Step 4.3: Use the three-dimensional topology map of the human body constructed in Step 4.2 to train the neural network and obtain the dangerous behavior detection model;

[0077] Based on the 3D spatiotemporal topology map of the human body constructed from the joints in step 4.2, it is fed into a spatiotemporal graph convolutional network (ST-GCN). This network consists of nine stacked spatiotemporal graph convolutional modules. The first to third layers have 64 output channels, the fourth to sixth layers have 128 output channels, and the seventh to ninth layers have 256 output channels. Then, a global average pooling layer is added after the network to map all the different sample features and reduce the network parameters. Finally, a Softmax (normalized exponential function) classifier is used to predict the classification results.

[0078] During training, the total training epochs were set to 100, and the batch size was set to 64. The learning rate was set to 0.1 for the first 30 epochs, 0.01 for epochs 31 to 60, and 0.001 for epochs 61 to 100. The cross-entropy loss function was used, and the Adam optimizer was used. The model was saved after training was completed.

[0079] Step 5: Import the three-dimensional spatiotemporal topology map of the human body constructed in Step 3 into the behavior recognition model trained in Step 4 for behavior detection.

[0080] Using a constructed 3D topological map of the human body as input, features are extracted using spatiotemporal graph convolution. By loading the model trained in step 4, behavior recognition can be performed on this input. A threshold of 0.8 is set; if the behavior detection network determines that the behavior belongs to a predefined dangerous behavior category and the confidence level is higher than the set threshold, then the behavior is judged as dangerous.

[0081] The specific hazardous behavior detection process of this invention is as follows: The Kinect depth sensor continuously collects skeletal joint point position information from personnel. These coordinate nodes are searched using a tree-based approach. The coordinate points are then classified based on the reliability of joint velocity and the reliability of bone length. When a joint point is identified as an incorrect joint point, a Kalman filter is used to re-predict the erroneous node, and the predicted joint point is corrected using the constraint of bone length. A three-dimensional spatiotemporal topology map of the human body is constructed using the correct joint points and the optimized erroneous nodes. This map is then loaded into a behavior detection model for behavior detection, enabling the detection of hazardous behaviors of personnel in complex workshop environments.

Claims

1. A method for detecting hazardous behaviors of workers in complex environments, characterized in that, The specific steps are as follows: Step 1: Collect skeleton joint information and perform preprocessing; Step 2: Perform reliability calculations based on joint motion velocity and bone length. When the joint motion velocity exceeds the velocity threshold or the average change ratio of bone length relative to the reference length exceeds the length threshold, erroneous joints are identified. Kalman filtering is used to predict the location of erroneous joints. With the bone reference length as a constraint, the predicted joints are limited to a sphere with the parent node as the center and the bone reference length as the radius. The point on the sphere with the closest Euclidean distance to the predicted position is selected to complete the reconstruction of the erroneous joint. Step 3: Construct a three-dimensional spatiotemporal topology map of the human body based on human joint data; Step 4: Train a neural network using the collected correct key point information to create a 3D topology map, thereby obtaining a dangerous behavior detection model; Step 5: Import the three-dimensional spatiotemporal topology map of the human body constructed in Step 3 into the behavior recognition model trained in Step 4 for behavior detection.

2. The method for detecting hazardous behaviors of workshop personnel in complex environments according to claim 1, characterized in that, Step 1 includes the following steps: Step 1.1: Deploy depth vision sensors at key workstations in the workshop and collect coordinate information of the joint points of the human skeleton on site. Define joints The absolute coordinates are represented as , The human body contains 25 skeletal joints; Step 1.2: Perform preprocessing operations on the collected dataset; (1) Filtering of non-human skeletal joint data for product equipment The skeleton data of objects misidentified as people is removed by selecting the average displacement value of 25 node coordinates between adjacent frames, calculated as shown in the following formula: ； Among them, defining the key points The absolute coordinates of the t-th frame are K represents the average displacement of the 25 node coordinates at frame t and frame t+1. The threshold is set to 0.01m. When K is less than this value, the data is filtered out. (2) Coordinate transformation The coordinates of the 3D skeleton joints in the absolute coordinate system are represented using a relative coordinate system. For ease of processing, the mid-hip is chosen as the origin of the relative coordinate system, and all joints are transformed. The specific method is shown in the following formula: ； Key points coordinates in the absolute coordinate system The coordinates are obtained after transformation to a relative coordinate system. .

3. The method for detecting hazardous behaviors of workshop personnel in complex environments according to claim 1, characterized in that, Step 2 includes the following specific steps: Step 2.1: Based on the node coordinate information obtained in Step 1.1, a tree-like approach is used to search for joints and retrieve key points; among them, the left shoulder, right shoulder, left hip, right hip, and neck are used as root nodes; Step 2.2, reliability calculation, identify faulty key points; Step 2.3: For the erroneous key points, reconstruct the erroneous key points using Kalman filtering.

4. The method for detecting hazardous behaviors of workshop personnel in complex environments according to claim 1, characterized in that, Step 4 includes the following specific steps: Step 4.1: Define the types of hazardous behaviors designed according to workshop requirements; Step 4.2: Based on the node coordinate information obtained in Step 1.2, perform preprocessing, assuming that the obtained joint point information is correct, and then construct a three-dimensional topology map of the human body based on the preprocessed joint position data according to the operation in Step 3. Step 4.3: Use the three-dimensional topology map of the human body constructed in step 4.2 to train the neural network and obtain the dangerous behavior detection model; A 3D topological map of the human body is fed into a spatiotemporal graph convolutional network. This network consists of nine stacked spatiotemporal graph convolutional modules. Each graph convolutional block performs both spatial and temporal convolutions to extract behavioral features. The first to third layers have 64 output channels, the fourth to sixth layers have 128 output channels, and the seventh to ninth layers have 256 output channels. A global average pooling layer is then added to the end of the network to map the features of all different samples and reduce the number of network parameters. Finally, a normalized exponential function classifier is used to predict the classification results.

5. The method for detecting hazardous behaviors of workshop personnel in complex environments according to claim 3, characterized in that, The reliability calculation in step 2.2 includes two forms: (1) Reliability calculation based on joint motion velocity; Set up key points The three-dimensional coordinates of the t-th frame and the (t+1)-th frame , Therefore, the displacement of the joints Calculated by the following formula: ； Because Kinect captures frames at a rate of 30 frames per second, the time interval between each frame... for Seconds, joint movement speed It is calculated by the following formula: ； The normal running speed for adults is We set the speed threshold to If the speed of a certain joint exceeds the threshold after the joint speed calculation is performed, the corresponding joint at this moment is judged as abnormal joint data, that is, the joint data needs to be reconstructed. (2) Reliability calculation based on bone length; The human skeleton is equivalent to a hinge mechanism. The length of the skeleton should be a constant during movement. The length of the skeleton is represented by the Euclidean distance between two adjacent joints, as shown in the following formula: ； in and The labels representing the joints of the human body, bones Indicates adjacent joints and Length; The distance between each skeletal point when the entire body of a person is tracked by the Kinect depth sensor in the first frame is used as the reference length. Definition and joint The total number of connected joints is The line segment between the connected joints is In the frame midline segment The difference ratio between the length and the reference length Calculated by the following formula: ； In the formula For line segments Reference length, For line segments In frame The actual length in the text; The reliability of a joint based on bone length is determined by the average of the proportions of the differences between all its connected joint segments and the reference length, as shown in the following formula: ； in To measure key points Based on the degree of difference in bone length between adjacent points, this paper sets a threshold of 30% for bone length variation between consecutive frames. If the value is greater than the threshold, it indicates that the skeleton joint information of this frame is abnormal data and needs to be reconstructed.

6. The method for detecting hazardous behaviors of workshop personnel in complex environments according to claim 3, characterized in that, The specific steps of step 2.3 are as follows: Step 2.3.1, Define the incorrect joint location Predicted key point locations ; Step 2.3.2: Using the bone reference length obtained in Step 2.2 as a constraint, the joint coordinates predicted in Step 2.3.1 are... Make adjustments; make the faulty node the child node, and the node preceding the faulty node the parent node. Since the estimated node position and the bone length between its parent node remain unchanged, the bone length of the two nodes is referenced in the reliability based on bone length. Therefore, the joint should be estimated to be at the position of the parent node. With center at and radius at, On the sphere, the constraint equations are as follows: ； Step 2.3.3: Select the position on the spherical surface relative to the estimated joint location. The point with the closest Euclidean distance is used as the optimized estimate of the joint position. .

7. The method for detecting hazardous behaviors of workshop personnel in complex environments according to claim 1, characterized in that, The spatiotemporal topology of the human body is formed by connecting and stitching together the erroneous joints reconstructed in step 2 and the original correct human joints. The spatiotemporal topology graph of the 3D skeleton consists of a set of joints and a set of edges, as shown in the following equation: ； Wherein, G is the spatiotemporal topology of the human three-dimensional skeleton. It is a set of skeleton joint point data in a relative coordinate system. , The first skeleton node data Frame, where i represents the sequence number of the joint. This represents 25 skeleton joints; while the edge set consists of the spatial and temporal edges of the skeleton. ,in This represents the bone edges where the joints in the skeleton diagram space are naturally connected. It is a set of joint pairs that are naturally connected in the human body. It is the first Frame and the The time edge connecting the same skeleton joint of a frame represents the dynamic spatial position change of a skeleton joint over time.

Images