A method and system for intelligently identifying and warning unsafe behaviors of workers

Abstract

The present application relates to the field of safe production technology, and provides a method and system for intelligent identification and early warning of unsafe behaviors of workers. In this method, based on the acquired multi-channel video image data of the workers' construction site, the working area of ​​the workers in each channel of video image data is calibrated; the unsafe behaviors of the target objects in the calibrated working area are identified; wherein the unsafe behaviors include dangerous postures, illegal behaviors, and illegal wearing; based on the identified unsafe behaviors of the target objects, based on the time information and spatial information of the target objects in the multi-channel video image data, early warning information is pushed to the target objects. In this way, the unsafe behaviors of workers in the production environment can be accurately identified, and early warning information can be pushed to the targets in a targeted manner from the time and space levels, effectively preventing the occurrence of disasters and accidents.

Description

Technical Field

[0001] The present application relates to the field of safe production technology, and in particular to a method and system for intelligently identifying and warning of unsafe behaviors of workers. Background Art

[0002] Workers face various safety risks during the production process, and unsafe operation behaviors of workers may lead to serious accidents and cause great harm. Timely identification and early warning of unsafe behaviors of workers are of great significance. Traditional safety management often relies on manual inspection and monitoring, which is inefficient, has limited coverage, and cannot monitor workers' behavior in real time.

[0003] At present, the identification of unsafe behaviors of workers is mainly based on identification methods such as sensors and cameras, but these methods usually require a lot of computing resources and complex algorithm support, resulting in extremely high costs and are not suitable for use in large-scale applications. In addition, existing identification methods often only cover some worker behaviors, such as: no helmet, no safety belt, etc., and the identification ability for more complex and diverse unsafe behaviors (such as: falling, falling, etc.) is weak, and cannot fully meet the safety precautions in industrial production.

[0004] Therefore, there is an urgent need to provide a technical solution to the above-mentioned deficiencies in the prior art. Summary of the invention

[0005] The purpose of this application is to provide a method and system for intelligently identifying and warning of unsafe behaviors of workers, so as to solve or alleviate the problems existing in the above-mentioned prior art.

[0006] In order to achieve the above objectives, this application provides the following technical solutions:

[0007] The present application provides a method for intelligently identifying and warning of unsafe behaviors of workers, including: step S101, based on multiple channels of video image data of workers' construction sites, calibrating the working area of ​​workers in each channel of the video image data; step S102, identifying unsafe behaviors of target objects in the calibrated working area; wherein the unsafe behaviors include dangerous postures, illegal behaviors, and illegal wearing; step S103, based on the identified unsafe behaviors of the target objects, and based on the time information and spatial information of the target objects in the multiple channels of the video image data, pushing warning information to the target objects.

[0008] Preferably, in step S102, human body detection is performed on the workers in the video image data based on the constructed target detection model; based on the human skeleton point recognition model, skeleton point recognition is performed on the target object in the working area detected by the human body to obtain the key skeleton points of the target object; based on the spatiotemporal graph neural network model, the dangerous posture of the target object is identified according to the temporal topological graph data composed of the key skeleton points of the target object.

[0009] Preferably, in step S102, target object detection is performed on the objects in the video image data based on the constructed target detection model; according to the detection result of the target object, based on the position matching method, the target object in the working area is matched with the target object for the illegal behavior and / or the illegal wearing; in response to the successful matching of the illegal behavior and / or the illegal wearing, the Bayesian probability of the illegal behavior and / or the illegal wearing is calculated based on the Bayesian network to determine whether the behavior and / or wearing of the target object is illegal.

[0010] Preferably, according to the detection result of the target object, based on the position matching method, the target object in the working area is matched with the target object for the violation, including: determining whether the detected target object is located in the upper limb area of ​​the target object according to the upper limb bending angle of the target object; in response to the target object being located in the upper limb area of ​​the target object, determining that the violation match between the target object and the target object is successful.

[0011] Preferably, the performing the illegal wearing matching between the target object and the target object in the working area includes: in response to the absence of a protective device at a preset part of the target object, determining that the illegal wearing matching between the target object and the target object is successful.

[0012] Preferably, in response to the successful matching of the illegal behavior and / or the illegal wearing, based on the Bayesian network, the Bayesian probability of the illegal behavior and / or the illegal wearing is calculated to determine whether the behavior and / or the wearing of the target object is illegal, including: in response to the successful matching of the illegal behavior and / or the illegal wearing, based on the node state in the corresponding pre-built dynamic analysis Bayesian network structure, determining the corresponding node probability in the dynamic analysis Bayesian network structure; according to the corresponding node probability in the dynamic analysis Bayesian network structure, calculating the Bayesian probability of the illegal behavior and / or the illegal wearing to determine whether the behavior and / or the wearing of the target object is illegal.

[0013] Preferably, a worker tracking model established based on three-dimensional Kalman filtering is used to automatically track and number the target objects in multiple channels of video image data; and only one warning message is pushed to the target objects with the same number based on the time information and spatial information of the target objects appearing in the multiple channels of video image data.

[0014] Preferably, based on the push log of the warning information pushed to the target object, in response to the warning information not being pushed within a preset interval, the warning information is pushed to the target object again.

[0015] An embodiment of the present application also provides an intelligent identification and early warning system for unsafe behaviors of workers, including: an area calibration unit, configured to calibrate the working area of ​​the workers in each channel of the video image data based on multiple channels of video image data acquired at the workers' construction site; a behavior recognition unit, configured to identify the unsafe behaviors of the target objects in the calibrated working area; wherein the unsafe behaviors include: dangerous postures, illegal behaviors and illegal wearing; and an early warning push unit, configured to push early warning information to the target object based on the identified unsafe behaviors of the target object and the time information and spatial information of the target object in the multiple channels of the video image data.

[0016] Beneficial effects:

[0017] In the intelligent identification and early warning method for unsafe behaviors of workers provided in the embodiment of the present application, firstly, the working area of ​​the workers in each video image data is calibrated according to the multi-channel video image data of the workers' construction site; then, the unsafe behaviors of the target objects in the calibrated working area are identified; finally, according to the unsafe behaviors of the identified target objects, based on the time information and space information of the target objects in the multi-channel video image data, early warning information is pushed to the target objects. In this way, the unsafe behaviors of workers in the production environment can be accurately identified, and early warning information can be pushed to the targets in a targeted manner from the time and space level, effectively preventing the occurrence of disasters and accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. Among them:

[0019] Figure 1 A flowchart of a method for intelligently identifying and warning unsafe behaviors of workers provided according to some embodiments of the present application;

[0020] Figure 2A schematic diagram of a Bayesian network structure for dynamic analysis of workers using mobile phones (smoking) in a work area according to some embodiments of the present application;

[0021] Figure 3 A schematic diagram of analysis results of a Bayesian network for dynamic analysis of workers using mobile phones (smoking) in a work area according to some embodiments of the present application;

[0022] Figure 4 A schematic diagram of a Bayesian network structure for dynamic analysis of a worker not wearing protective equipment in a work area according to some embodiments of the present application;

[0023] Figure 5 A schematic diagram of analysis results of a Bayesian network for dynamic analysis of workers not wearing protective equipment in a work area according to some embodiments of the present application;

[0024] Figure 6 A schematic diagram of the structure of a system for intelligently identifying and warning of unsafe behaviors of workers provided according to some embodiments of the present application. DETAILED DESCRIPTION

[0025] The present application will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments. Each example is provided by way of explanation of the present application and does not limit the present application. In fact, it will be clear to those skilled in the art that modifications and variations may be made in the present application without departing from the scope or spirit of the present application. For example, a feature shown or described as a part of an embodiment may be used in another embodiment to produce yet another embodiment. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the embodiments of the present invention should belong to the scope of protection of the embodiments of the present invention.

[0026] Traditional safety management that relies on manual inspection and monitoring has problems such as low efficiency, limited coverage, and inability to monitor workers' behavior in real time. Based on this, this application proposes an intelligent identification and early warning method for unsafe behaviors of workers, which is used to accurately identify unsafe behaviors of workers in production environments, such as Figures 1 to 5 As shown, the early warning method includes:

[0027] Step S101: based on the acquired multi-channel video image data of the workers' construction site, calibrate the workers' working area in each channel of video image data.

[0028] In this application, the real-time streaming protocol (RTSP) and multi-threading technology are used to obtain the video images of the construction site collected by n network cameras in real time and in parallel (where n is a positive integer), and the video image frame of the n cameras at time t is obtained. 1-t,…,frame n-t After unifying the parameters (including the length, width, number of channels, RGB encoding format, etc. of a single frame image), they are integrated into an image batch. t Among them, the image combination batch t It is a matrix of n×l×w×3, where n is the number of network cameras, l is the length of a single frame image, w is the width of a single frame image, and the number of channels of a single frame image is 3.

[0029] The work area (areas where mobile phones and smoking are prohibited) is calibrated from the video surveillance images to determine whether the workers are in the work area. The work area can be calibrated by drawing a series of anchor points in the video surveillance image (video image data) through the anchor frame mechanism. The area enclosed by these anchor points is the work area. When the construction scene changes or the camera shooting angle changes, the area calibration needs to be re-performed. For example, define the frame t is the video image at time t, which is considered as a H×W pixel matrix, where H and W are the height and width of the video image respectively. A series of anchor points (a 1 ,a 2 ,...,a n ),in,

[0030] a n =Point(x n ,y n )

[0031] In the formula, x n ,y n Respectively represent the horizontal and vertical coordinates of the anchor point in the video image. 1 ,a 2 ,...,a n ) The area enclosed by the frame is the working area.

[0032] The calibration of the working area can also be carried out in a real scene by placing specific markers at various fixed points in the working area and detecting the position of the markers in the video image (a 1 ,a 2 ,...,a n ), the area enclosed by the locations of the landmarks is the working area.

[0033] Specifically, the calibration result of the working area in the video image is shown as follows:

[0034] S workspace =Polygon(a 1 ,a 2 ,…,a n )

[0035] Among them, (a 1 ,a 2 ,...,a n ) is a series of calibration anchor points, S workspace It is the area enclosed by the calibration anchor points.

[0036] Step S102: Identify unsafe behaviors of target objects in the calibrated working area.

[0037] In this application, human body and object detection are performed through a target detection model to effectively identify whether the target object has unsafe behaviors, where unsafe behaviors include dangerous postures, illegal behaviors, and illegal wearing.

[0038] When performing dangerous posture detection, firstly, human detection of workers in video image data is performed based on the constructed target detection model. In this application, transfer learning is used to fine-tune the target detection model, improve the target detection accuracy of the target detection model in production site scenes, and effectively solve the problem of high misrecognition rate caused by factors such as poor lighting conditions, camera deployment angles, and cluttered construction site scenes.

[0039] When using transfer learning to tune the target detection model (Yolov5m model), in the initial testing phase (the first phase), an open source pre-trained target detection model is used and deployed in a real production site scenario. The results of the target detection model recognizing human bodies are saved locally, and the results of model misidentification are selected through manual review. In the model iteration and tuning phase (the second phase), transfer learning technology and negative feedback mechanism are introduced, and the open source pre-trained target detection model is used as the basic model. A small batch of real scene annotation data is obtained through labeling. The misidentification results of the model in the testing phase are used as negative sample data for fine-tuning the model to eliminate interference from factors such as poor lighting conditions, camera deployment angles, and cluttered scene backgrounds, thereby improving the recognition accuracy of the target detection model in real scenes.

[0040] Specifically, first, collect a small batch of training data in the field scene, use the LabelImg labeling tool to generate the necessary labels and bounding boxes, and save them in the yolo label format. The labeled image data is divided into a data set, a validation set, and a test set in a ratio of 8:1:1; then, use the pre-trained weights of the Yolov5m model on the COCO data set for transfer learning; at the same time, Epoch is set to 100, and 100 rounds of training are performed; Batch-size is set to 16; Img-size is set to 640*640; and the initial learning rate (Ir0) is set to 0.01. After the model training is completed, the result weight with the best training effect is selected, and the model is evaluated on the test set.

[0041] In the deployment reasoning phase (the third phase), the iteratively tuned target detection model is deployed in a real construction scene, and the human detection results are obtained based on the video image data. Specifically, the tuned model is deployed in a real construction scene to obtain the target detection results. The results include: the position anchor of the detected human body in the image, the bounding box bounding box, and the confidence score confidence_score. By setting the confidence threshold confidence_threshold, the results with a confidence score lower than the confidence threshold are discarded. By setting the overlap threshold IOU_threshold, it is determined whether two adjacent detection results are the same object. If the IOU value between the bounding boxes of the two recognition results is greater than the overlap threshold IOU_threshold, the two detection results are regarded as the same person (object) body, and the detection result with a lower confidence score confidence_score is discarded.

[0042] After the human body detection is performed on the workers in the video image data through the target detection model and the human body detection results are obtained, the skeleton points of the target objects in the working area obtained by the human body detection are recognized based on the human skeleton point recognition model to obtain the key skeleton points of the target objects. In this application, 18 human key skeleton point topological data are obtained through the open source human key skeleton point detection model. The names of the key skeleton points are shown in Table 1. Only when part of the human body enters the picture or the human body is blocked by objects, the detected human key skeleton points may be missing. Table 1 is as follows:

[0043] Table 1 Key bone points of the human body

[0044]

[0045]

[0046] After identifying and detecting the 18 key human skeleton points of the target object (target worker), the dangerous posture of the target object is identified based on the time series topological data composed of the key skeleton points of the target object based on the spatiotemporal graph neural network model. In other words, the time series data composed of 18×t key skeleton points within time t is analyzed using the spatiotemporal graph neural network model to identify the dangerous posture of the target object.

[0047] In this application, five types of human behavior postures can be identified through the open source spatiotemporal graph neural network model, including three normal postures: walking, standing, and sitting, and two dangerous postures: falling and lying. In the process of worker posture recognition, a transfer learning technique similar to that used in the target detection model for human body detection is used to optimize the human body posture recognition performance of the spatiotemporal graph neural network model in specific scenarios in three stages, effectively solving the problem of inaccurate human body posture recognition caused by factors such as camera deployment angle, background interference, and object occlusion, enhancing the accuracy of spatiotemporal graph convolution model recognition, and reducing the misrecognition rate.

[0048] In a specific example, based on the human body detection results, the Open-Pose model is used to identify 18 key human skeleton points of the target object. Specifically, the temporal skeleton point topology data composed of 18×t key skeleton points within time t is input into the LSTM-STGCN model, and the spatial topological relationship features of the key skeleton points of five different postures are extracted using the STGCN model features. The temporal features of the key skeleton points of five different postures are extracted using the LSTM model, and a fully connected network is used to construct a feature fusion module to fuse the above key skeleton point spatial topological relationship features and key skeleton point temporal features, and finally output the posture of the target object and the probability of the posture.

[0049] At the same time, the constructed object detection model is used to detect the objects in the video image data. Similar to human body detection, through object detection, the position of objects such as mobile phones and cigarettes in the video image can be obtained, including the center point of the target object and the length and width of the rectangular frame in the video image.

[0050] In order to determine whether workers have committed illegal behaviors such as smoking or using mobile phones in the work area, in this application, according to the detection result of the target object, based on the position matching method, the target object in the work area is matched with the target object. Specifically, according to the bending angle of the upper limb of the target object, it is determined whether the detected target object is located in the upper limb area of ​​the target object; when the target object is located in the upper limb area of ​​the target object, it is determined that the illegal behavior matching between the target object and the target object is successful.

[0051] In this application, the positions of the three key skeletal points of the upper limbs (shoulder joint, elbow joint and wrist joint) in the video image are composed of three fixed points to form a triangle. The upper limb bending angle of the target object is obtained by calculating the cosine of the angle at which the elbow joint is located; then, combined with the calibrated working area, the position and border of the target object, and the position and border of the target object (mobile phone, cigarette, etc.), it is determined whether the target object uses a mobile phone, smokes, or other illegal behaviors in the working area.

[0052] According to the 18 key bone points of the human body, the human body is divided into 4 regions, namely the head, chest, upper limbs and lower limbs, corresponding to the detection and calibration in the video image. head It is expressed as:

[0053]

[0054] In the formula, S head The schematic area representing the human head area in the video image, Represented by bone points Here, the bone point For example, The position of the left eye of the human body in the video image can be expressed as:

[0055]

[0056] in, They represent the horizontal and vertical coordinates of the left eye of the human body in the video image respectively. The identification methods of other skeleton points are the same and will not be described here one by one.

[0057] The target subject's chest area S chest It is expressed as:

[0058]

[0059] S chest Represents the schematic area of ​​the human chest area in the video image, Polygon For the bone point Enclosed area.

[0060] Upper limb area S upper_limbs It is expressed as:

[0061]

[0062] in, A schematic area representing an upper limb area in a video image; Represented by bone points The line segment is the schematic area of ​​the left upper limb of the human body in the video image. Identify by bone point The connected line segment is the schematic area of ​​the right upper limb of the human body in the video image, and the union of the two is the schematic area of ​​the upper limb area of ​​the human body in the video image.

[0063] Lower limb area S Lower_limbs It is expressed as:

[0064]

[0065] Among them, S Lower_limbs Represents the schematic area of ​​the lower limb area in the video image; Line Represented by bone points The line segment is the schematic area of ​​the left lower limb of the human body in the video image; Line Represented by bone points The connected line segment is the schematic area of ​​the right lower limb of the human body in the video image; the union of the two is the schematic area of ​​the lower limb area of ​​the human body in the video image.

[0066] In order to determine the violation of the personnel, it is necessary to calculate the bending angle of the upper limb of the human body. Taking the left upper limb of the person as an example, the bending angle of the left upper limb can be obtained by the bone point The specific calculation formula is as follows:

[0067]

[0068] Among them, θ is the three bone points of the left upper limb The angle of the left upper limb is the bending angle; Represents bone points Position vector in the video image; Represents bone points The position vector in the video image.

[0069] When the target object in the working area successfully matches the target object's violation behavior, the node state in the Bayesian network structure of the dynamic analysis of the violation behavior is determined based on the node state in the Bayesian network structure of the dynamic analysis of the violation behavior. Then, based on the corresponding node probability in the Bayesian network structure of the dynamic analysis of the violation behavior, the Bayesian probability of the violation behavior is calculated to determine whether the target object's behavior is in violation. In other words, when the violation behavior is successfully matched, that is, the detection frame of the target object is located in the upper limb area of ​​the target object, it is judged that the target object may have violated the law in the target area, such as smoking and using a mobile phone; then, the Bayesian probability of the violation behavior is calculated based on the Bayesian network, and based on the Bayesian probability of the violation behavior, it is finally determined whether the target object's behavior is in violation.

[0070] Specifically, a Bayesian network for dynamic analysis of unsafe behaviors of workers in the work area is constructed to identify unsafe behaviors of workers in the work area. The Bayesian network structure for dynamic analysis of workers using mobile phones (smoking) in the work area is as follows: Figure 2As shown, the Bayesian network structure for dynamic analysis of personnel using mobile phones (smoking) in the work area consists of 8 nodes, 5 of which are parent nodes: parent node A for judging whether the personnel is located in the designated work area, parent node B for judging whether the mobile phone (cigarette) is located in the left upper limb area of ​​the human body, parent node C for judging whether the mobile phone (cigarette) is located in the right upper limb area of ​​the human body, parent node D for judging the bending angle of the left upper limb of the human body, and parent node E for judging the bending angle of the right upper limb of the human body; two intermediate nodes: intermediate node F for judging whether the personnel is using a mobile phone (smoking) with his left hand, intermediate node G for judging whether the personnel is using a mobile phone (smoking) with his right hand, and child node R for judging whether the personnel is in the work area and committing the unsafe behavior of using a mobile phone (smoking).

[0071] Correspondingly, the node information of the dynamic analysis Bayesian network structure of workers using mobile phones (smoking) in the work area is shown in Table 2. Table 2 is as follows:

[0072] Table 2 Bayesian network node information for dynamic analysis of workers using mobile phones (smoking) in the work area

[0073]

[0074]

[0075] In this application, the Demster-Schaefer theory is used to determine the conditional probability tables (CPTs) of all nodes in different node states to represent the probability of each node existing in two states (V1 or V2) according to the state of its parent node, so as to significantly enhance the credibility and accuracy of the Bayesian network. Among them, the probability distribution of different nodes determined based on the Demster-Schaefer theory is shown in Tables 3 to 10.

[0076] Table 3. Probability distribution of nodes at all levels for “A (whether the personnel are in the designated work area)”

[0077]

[0078] Table 4. Probability distribution of each level of the node “B (whether the mobile phone (cigarette) is located in the left upper limb area of ​​the human body)”

[0079]

[0080] Table 5. Probability distribution of each level of the node “C (whether the mobile phone (cigarette) is located in the right upper limb area of ​​the human body)”

[0081]

[0082] Table 6. Probability distribution of each level of the node “D (whether the bending angle of the left upper limb of the human body is greater than 90 degrees)”

[0083]

[0084] Table 7. Probability distribution of each level of the node “E (Is the bending angle of the right upper limb of the human body greater than 90 degrees)”

[0085]

[0086] Table 8. Probability distribution of nodes at all levels for “F (the person is using a mobile phone (smoking) with his left hand)” under the influence of “B (whether the mobile phone (cigarette) is located in the left upper limb area of ​​the human body)” and “D (whether the bending angle of the left upper limb of the human body is greater than 90 degrees)”

[0087]

[0088] Table 9. Probability distribution of nodes at all levels for “G (the person is using a mobile phone (smoking) with his right hand)” under the influence of “C (whether the mobile phone (cigarette) is located in the right upper limb area of ​​the human body)” and “E (whether the bending angle of the right upper limb of the human body is greater than 90 degrees)”

[0089]

[0090] Table 10. “R (whether the personnel use mobile phones (smoking) in the work area)” in “A (whether the personnel are in the designated work area)”, “F (the personnel are using mobile phones (smoking) with their left hand)” and

[0091] Under the influence of "G (the person is using a mobile phone (smoking) with his right hand)", the probability distribution of nodes at all levels

[0092]

[0093] Based on the dynamic analysis of the Bayesian network structure of workers using mobile phones (smoking) in the work area, the workers' illegal behaviors in the work area are inferred and their final unsafe behavior results are calculated, such as Figure 3 As shown, in the Bayesian network analysis results of the dynamic analysis of workers using mobile phones (smoking) in the work area, the probability of the target object using mobile phones (smoking) in the work area is 36%, while the probability of not having this behavior is 64%.

[0094] Similarly, for workers who violate the rules by not wearing work clothes, helmets or other protective gear in the work area, human body detection and target object (protective gear) detection must also be carried out in the designated work area. The detection method is the same as that for the above-mentioned violations and will not be repeated here.

[0095] According to the detection result of the target object, the target object and the target object in the working area are matched for illegal wearing based on the position matching method. Similarly, the anchor frame mechanism can be used to identify whether the target object is wearing a protective device. Through target object detection, if there is no protective device at the preset position of the target object, it is determined that the illegal wearing match between the target object and the target object is successful. In other words, through target object detection, it is determined whether the preset position (such as the head) of the target object is wearing a protective device. When no protective device is detected (such as the target object's head is not wearing a protective device), or when a protective device is detected but the protective device is not in the preset position of the target object (for example, the protective device is detected on the target object's hand, not in the head area), it is considered that the illegal wearing match between the target object and the target object is successful.

[0096] When the target object in the working area successfully matches the violation behavior of the target object, the corresponding node probability in the dynamic analysis Bayesian network structure of the violation wear is determined based on the node state in the constructed dynamic analysis Bayesian network structure of the violation wear. Then, based on the corresponding node probability in the dynamic analysis Bayesian network structure of the violation wear, the Bayesian probability of the violation wear is calculated to determine whether the behavior of the target object is in violation.

[0097] Specifically, a Bayesian network for dynamic analysis of unsafe behaviors of personnel not wearing protective equipment in the work area is constructed, such as Figure 4 As shown, the Bayesian network for dynamic analysis of the unsafe behavior of the person not wearing protective devices in the work area consists of 8 nodes, including 5 parent nodes: parent node A for judging whether the person is located in the designated work area, parent node H for judging whether the protective device is located in the head area of ​​the human body, and parent node I for judging whether the device is located in the chest area of ​​the human body; 2 intermediate nodes: intermediate node J for judging whether the person is not wearing a protective device on the head, and intermediate node K for judging whether the person is not wearing a protective device on the chest; and child node Y for judging whether the person has an unsafe behavior of not wearing a protective device in the work area.

[0098] Correspondingly, the node information of the Bayesian network for dynamic analysis of unsafe behaviors of personnel not wearing protective equipment in the work area is shown in Table 11. Table 11 is as follows:

[0099] Table 11 Bayesian network node information for dynamic analysis of workers not wearing protective equipment in the work area

[0100]

[0101] Similarly, the Demster-Schafluor theory is used to determine the unsafe behavior of personnel not wearing protective equipment in the work area. The conditional probability tables (CPTs) of all nodes in the dynamic analysis Bayesian network under different node states are used to represent the probability of each node existing in two states (V1 or V2) according to the state of its parent node, so as to significantly enhance the credibility and accuracy of the Bayesian network. Among them, the probability distribution of different nodes determined based on the Demster-Schafluor theory is shown in Tables 12 to 14.

[0102] Table 12. Probability distribution of each level of the node “H (whether the head protective device is located above the head area of ​​the human body)”

[0103]

[0104]

[0105] Table 13. Probability distribution of nodes at all levels under the influence of “I (whether the person wears protective equipment on the head)” and “H (whether the protective equipment is located above the head area of ​​the human body)”

[0106] Table 14. Probability distribution of nodes at all levels under the influence of "Y (whether personnel are in the work area without wearing protective equipment)" and "A (whether personnel are in the designated work area)" and "I (whether personnel are wearing protective equipment on their heads)"

[0107]

[0108] Based on the Bayesian network structure, the unsafe behavior of personnel not wearing protective equipment in the work area is dynamically analyzed to infer the illegal wearing of workers in the work area and calculate the final unsafe behavior results, such as Figure 5 As shown in the Bayesian network analysis results of the dynamic analysis of unsafe behaviors of personnel not wearing protective devices in the work area, the probability of the target object's unsafe behavior of wearing protective devices in violation of regulations in the work area is 29%, while the probability of the absence of such behavior is 71%.

[0109] Step S103: push warning information to the target object based on the identified unsafe behavior of the target object and the time information and space information of the target object in the multi-channel video image data.

[0110] In this application, based on the recognition results of the unsafe behavior of the target object, the unsafe behavior of workers in different camera fields of view is collaboratively analyzed. Prevent duplicate push of warning information at the spatial level: Use worker re-identification technology to automatically track and number the workers, ensuring that only one warning message is pushed for the same worker's unsafe behavior in a certain area. Specifically, based on the worker tracking model of the three-dimensional Kalman filter, the target objects in the multi-channel video image data are automatically tracked and numbered.

[0111] First, define a state space model for describing worker motion, describe the worker's motion state, including information such as the worker's position and speed, and assume that the worker moves in a uniform linear motion in three-dimensional space, as shown in the following formula:

[0112]

[0113] Among them, x t ,y t represents the position of the worker at time t, Represents the speed of the worker in the x and y directions at time t.

[0114] Then, based on the worker's current state and dynamic model, the state transition equation is used to predict the worker's state at the next moment (predicted position), and the uncertainty of the state at the next moment is estimated. In other words, a state transition model is established to describe the state change of the worker within a time step (i.e., from time t to time t+1). Specifically, the state transition model is:

[0115] x t+1 =Fx t +Bu t +w t

[0116] Where F is the state transfer matrix, which represents the dynamic characteristics of the system; Β is the control matrix, u t is the control input (usually a zero vector in worker tracking), w t is the process noise. Here, the process noise is Gaussian noise with zero mean, and the covariance matrix is ​​Q. Among them, the state transfer matrix F is:

[0117]

[0118] Where Δt is the time step.

[0119] In this application, the observation equation is used to describe the observation value z at time t measured from the sensor (camera): t With the state value x t The relationship between:

[0120] z t =Hxt +v t

[0121] Where H is the measurement matrix, which represents the mapping from state space to observation space; v t is the observation noise. Here, the observation noise is zero-mean Gaussian noise, and the covariance matrix is ​​R; the observation matrix H is:

[0122]

[0123] It should be noted that the observation matrix indicates that the sensor can only observe the position x and y, but cannot directly observe the speed.

[0124] Next, the IOU (Intersection over Union) between the actual position and predicted position of the worker in the video image data is calculated, the state estimation and state uncertainty are updated, and the actual position information is integrated into the state estimation. Then, through alternating predictions and updates, the state estimation is continuously iterated and optimized to obtain a more accurate motion state of the worker, realize automatic tracking of the worker, and generate the worker's number in the video surveillance image.

[0125] Specifically, the Kalman filter is used to estimate the current state to predict the state at the next moment, where the predicted state is expressed as:

[0126]

[0127] In the formula, is the state at time t predicted based on the state at time t-1, is the a posteriori state estimate at time t-1; Β is the control matrix, u t is the control input. In worker tracking, Β, u t Typically the zero vector.

[0128] According to the formula:

[0129] K t =P t|t-1 H T (HP t|t-1 H T +R) -1

[0130] Calculate the Kalman gain; where K t is the Kalman gain matrix, H T is the transposed matrix of the observation matrix, and R is the covariance matrix of the observation noise.

[0131] Then, according to the formula:

[0132]

[0133] Update the state estimate; where, is the posterior state estimate at time t, z t is the observation vector at time t, is the predicted observed value.

[0134] According to the formula:

[0135] P t|t =(IK t H)P t|t-1

[0136] Update the error covariance; where P t|t is the posterior error covariance matrix at time t, and I is the identity matrix.

[0137] During the worker tracking process, the Kalman filter continuously predicts the worker's position and speed and updates it based on the observation data provided by the sensor (camera), thereby achieving real-time tracking of the worker. In this process, the state at the previous moment is used to predict the state at the current moment, and the predicted state is corrected using the observation value at the current moment, thereby improving the accuracy of the estimation.

[0138] The collaborative analysis of workers' unsafe behaviors is reflected in the time level to prevent repeated push of warning information: that is, according to the time information and spatial information of the target object appearing in the multi-channel video image data, only one warning information is pushed to the target object with the same number, and a historical warning push information data table is constructed to record the worker number, regional location, warning time, unsafe behavior type and other information (push log) of the historical warning push. When the warning information pushed to the target object is not pushed within the preset interval time, the warning information is pushed to the target object again according to the push log. That is to say, if the warning information is still not processed after a period of time (gap_time), the second warning information is pushed. In this way, on the basis of accurately identifying the unsafe behaviors of workers in the production environment, the warning information is pushed to the target in a targeted manner from the time and space levels to effectively prevent the occurrence of disasters and accidents.

[0139] like Figure 6 As shown, the embodiment of the present application also provides a worker unsafe behavior intelligent identification and early warning system, including:

[0140] The area calibration unit 601 is configured to calibrate the working area of ​​the workers in each channel of video image data according to the acquired multiple channels of video image data of the workers' construction site;

[0141] The behavior recognition unit 602 is configured to recognize unsafe behaviors of the target object in the calibrated working area; wherein the unsafe behaviors include dangerous postures, illegal behaviors, and illegal wearing;

[0142] The warning push unit 603 is configured to push warning information to the target object based on the identified unsafe behavior of the target object and the time information and space information of the target object in the multi-channel video image data.

[0143] In this application, a cross-process data queue Quene is established to facilitate data transmission between different processes and batch the processed images. t The data is put into the data queue Quene. The target detection model, skeleton point recognition model, and spatiotemporal graph neural network model retrieve data from the data queue Quenc and perform batch reasoning on the unsafe behavior of the target object.

[0144] In the data queue Quene, the maximum queue length is L, which means that the data queue Quene can store L image combination batches. t , the image combination batch that was stored earlier in the data queue Quenc t It is called first and deleted from the data queue Quene after being called. In addition, there is a length self-check mechanism in the data queue Quenc. Before performing the "save" operation on the data queue Quene, it will automatically check whether the length of the current data queue Quene is greater than or equal to the maximum queue length L; if so, the data in the current data queue Quene is cleared to ensure that the data queue Quene will not overflow.

[0145] In this application, the embedded GPU computing unit (Nvidia Jetson NX development kit) is used as the hardware basis, and model compression, multi-process operation, GPU acceleration and other methods are adopted to perform software and hardware collaborative optimization of the intelligent identification system and device for unsafe behaviors of workers, which greatly reduces the hardware cost while ensuring the efficient and stable operation of the model and algorithm.

[0146] Use model compression technology to reduce model complexity, compress and prune the neural network model, and encapsulate and compile it into a neural network operator to reduce the complexity of the neural network, thereby reducing the model's consumption of computing power. Specifically, the PyTorch version of the YOLOv5m model weight yolov5m.pt file is converted into a model file yolov5m.engine suitable for running the TensorRT reasoning framework through model compression.

[0147] For early warnings that require inference predictions from multiple video surveillance channels, multi-threading technology is used to improve computing resource utilization and ensure efficient and stable operation of models and algorithms. Set up 1 main process, multiple model inference processes, and 1 collaborative analysis process, where each model inference process is responsible for the inference task of one video surveillance channel. Use cross-process communication technology to transmit the inference recognition results of the model inference process to the collaborative analysis process for spatiotemporal collaborative analysis and push early warnings to ensure efficient and stable operation of the worker unsafe behavior recognition model and algorithm.

[0148] Use GPU acceleration tools to accelerate the operation of neural network models at the underlying hardware level, thereby improving the performance of neural network operations. Specifically, compile the TensorRT accelerated reasoning toolkit specifically for the underlying GPU driver version and Cuda version. And based on the neural network parameters of the model, compile and generate neural network operators specifically, run S4.1 to obtain a model file suitable for the TensorRT reasoning framework, thereby accelerating the operation of the neural network model and improving the performance of neural network operations.

[0149] The intelligent identification and early warning system for unsafe behaviors of workers provided in the embodiments of the present application can implement the steps and processes of the intelligent identification and early warning method for unsafe behaviors of workers described in any of the above embodiments, and achieve the same technical effects, which will not be repeated here one by one.

[0150] In the description of the present invention, the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" etc. mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0151] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for intelligent identification and early warning of unsafe behaviors of workers, characterized in that: include: Step S101, based on the acquired multiple channels of video image data of the workers' construction site, calibrate the workers' working area in each channel of the video image data; Step S102, identifying unsafe behaviors of the target object in the calibrated work area; wherein the unsafe behaviors include dangerous postures, illegal behaviors and illegal wearing; the dangerous postures include falling and lying flat, and the illegal behaviors include smoking in the work area and using mobile phones in the work area; Step S103: according to the identified unsafe behavior of the target object, based on the time information and space information of the target object in the multiple channels of video image data, push warning information to the target object; In step S102, based on the constructed target detection model, human body detection is performed on the workers in the video image data; based on the human skeleton point recognition model, skeleton point recognition is performed on the target object in the working area detected by the human body to obtain key skeleton points of the target object; based on the spatiotemporal graph neural network model, the dangerous posture of the target object is identified according to the time series topological graph data composed of the key skeleton points of the target object; Based on the constructed target detection model, target object and target object detection are respectively performed on the human body and the object in the video image data; according to the detection result of the target object, based on the position matching method, the target object in the working area is matched with the target object for the illegal behavior and / or the illegal wear; in response to the successful matching of the illegal behavior and / or the illegal wear, the Bayesian probability of the illegal behavior and / or the illegal wear is calculated based on the Bayesian network to determine whether the behavior and / or the wear of the target object is illegal; Wherein, the matching of the target object and the target object in the working area for the violation behavior based on the position matching method according to the detection result of the target object includes: determining whether the detected target object is located in the upper limb area of ​​the target object according to the bending angle of the upper limb of the target object; in response to the target object being located in the upper limb area of ​​the target object, determining that the violation behavior matching between the target object and the target object is successful; The performing the illegal wearing matching between the target object and the target object in the working area includes: in response to the absence of a protective device at a preset part of the target object, determining that the illegal wearing matching between the target object and the target object is successful; In response to the successful matching of the illegal behavior and / or the illegal wearing, the Bayesian probability of the illegal behavior and / or the illegal wearing is calculated based on the Bayesian network to determine whether the behavior and / or the wearing of the target object is illegal, including: in response to the successful matching of the illegal behavior and / or the illegal wearing, based on the node state in the corresponding pre-built dynamic analysis Bayesian network structure, the corresponding node probability in the dynamic analysis Bayesian network structure is determined; according to the corresponding node probability in the dynamic analysis Bayesian network structure, the Bayesian probability of the illegal behavior and / or the illegal wearing is calculated to determine whether the behavior and / or the wearing of the target object is illegal.

2. The method for intelligent identification and early warning of unsafe behaviors of workers according to claim 1 is characterized in that: Step S103 includes: A worker tracking model established based on a three-dimensional Kalman filter is used to automatically track and number the target objects in the multiple channels of video image data; According to the time information and space information of the target object appearing in the multiple channels of video image data, only one warning message is pushed to the target object with the same number.

3. The method for intelligent identification and early warning of unsafe behaviors of workers according to claim 2 is characterized in that: Based on the push log of the warning information pushed to the target object, in response to the warning information not being pushed within a preset interval time, the warning information is pushed to the target object again.

4. An intelligent identification and early warning system for unsafe behaviors of workers, characterized in that: include: An area calibration unit is configured to calibrate a working area of ​​a worker in each channel of video image data based on the acquired multiple channels of video image data of the worker's construction site; A behavior recognition unit is configured to recognize unsafe behaviors of the target object in the calibrated work area; wherein the unsafe behaviors include: dangerous postures, illegal behaviors and illegal wearing; the dangerous postures include falling and lying flat, and the illegal behaviors include smoking in the work area and using mobile phones in the work area; The behavior recognition unit is further configured to perform human body detection on the workers in the video image data based on the constructed target detection model; perform skeleton point recognition on the target object in the working area detected by the human body based on the human skeleton point recognition model to obtain key skeleton points of the target object; and identify the dangerous posture of the target object based on the time-series topological graph data composed of the key skeleton points of the target object based on the spatiotemporal graph neural network model; Based on the constructed target detection model, target object and target object detection are respectively performed on the human body and the object in the video image data; according to the detection result of the target object, based on the position matching method, the target object in the working area is matched with the target object for the illegal behavior and / or the illegal wear; in response to the successful matching of the illegal behavior and / or the illegal wear, the Bayesian probability of the illegal behavior and / or the illegal wear is calculated based on the Bayesian network to determine whether the behavior and / or the wear of the target object is illegal; Wherein, the matching of the target object and the target object in the working area for the violation behavior based on the position matching method according to the detection result of the target object includes: determining whether the detected target object is located in the upper limb area of ​​the target object according to the bending angle of the upper limb of the target object; in response to the target object being located in the upper limb area of ​​the target object, determining that the violation behavior matching between the target object and the target object is successful; The performing the illegal wearing matching between the target object and the target object in the working area includes: in response to the absence of a protective device at a preset part of the target object, determining that the illegal wearing matching between the target object and the target object is successful; In response to the successful matching of the illegal behavior and / or the illegal wearing, based on the Bayesian network, calculating the Bayesian probability of the illegal behavior and / or the illegal wearing to determine whether the behavior and / or the wearing of the target object violates the law, including: in response to the successful matching of the illegal behavior and / or the illegal wearing, based on the node state in the corresponding pre-constructed dynamic analysis Bayesian network structure, determining the corresponding node probability in the dynamic analysis Bayesian network structure; calculating the Bayesian probability of the illegal behavior and / or the illegal wearing according to the corresponding node probability in the dynamic analysis Bayesian network structure to determine whether the behavior and / or the wearing of the target object violates the law; The warning push unit is configured to push warning information to the target object based on the identified unsafe behavior of the target object and on the time information and space information of the target object in the multiple channels of video image data.

Images