A method and system for detecting and analyzing construction safety incidents

Abstract

Embodiments of the present application provide a method and system for detecting and analyzing construction safety events, relating to the technical field of construction safety. The method includes: obtaining a first key frame sequence set of a construction site monitoring video; detecting whether there is a safety event according to the first key frame sequence set through a trained safety event detection model, where the trained safety event detection model is used to represent the mapping relationship between key frames and safety events; if there is a safety event, obtaining a second key frame sequence set of the construction site monitoring video; and obtaining the event state corresponding to the safety event based on state machine rules, the trained safety event detection model, and the second key frame sequence set. The present application can improve the problem that the detection results of safety detection at the current construction site are repeatedly sent and it is difficult to support different types of construction scenarios, achieving the effect of reducing the repeated sending of construction site detection results and supporting safety detection for different types of construction scenarios.

Description

Technical Field

[0001] Embodiments of the present application relate to the technical field of construction safety, and in particular, to a method and system for detecting and analyzing construction safety events. Background Art

[0002] In many working scenarios, such as construction sites, docks, oil fields, coal mines, and power base stations, due to reasons such as low safety awareness of workers and easy falling of objects, there are quite a few accidents caused by not wearing labor protection supplies (such as safety helmets and reflective vests), not extinguishing cigarette butts, and building collapses every year. Therefore, in order to effectively reduce the potential harm to personnel, it is necessary to conduct real-time safety detection in the above-mentioned places.

[0003] In the process of implementing the present invention, the inventors found that the detection results of safety detection in current construction sites are repeatedly sent and it is difficult to support different types of construction scenarios. Summary of the Invention

[0004] Embodiments of the present application provide a method and system for detecting and analyzing construction safety events, which can improve the problem that the detection results of safety detection in current construction sites are repeatedly sent and it is difficult to support different types of construction scenarios.

[0005] In the first aspect of the present application, a method for detecting and analyzing construction safety events is provided, including:

[0006] Obtaining a first key frame sequence set of a construction site monitoring video;

[0007] According to the first key frame sequence set, detecting whether there is a safety event through a trained safety event detection model; the safety event includes human events and natural events, the human events include not wearing labor protection supplies and / or human left fire sources, the natural events include burning and / or building collapse, and the trained safety event detection model is used to represent the mapping relationship between key frames and safety events;

[0008] If there is the safety event, obtaining a second key frame sequence set of the construction site monitoring video, and the extraction frequency of the second key frame is greater than that of the first key frame;

[0009] Based on state machine rules, the trained safety event detection model, and the second key frame sequence set, obtaining the event state corresponding to the safety event, and the event state includes not started / ended state, possible occurrence state, determined occurrence state, and possible end state.

[0010] By adopting the above technical solutions, a first key frame sequence set of the construction site monitoring video is obtained; then, based on the first key frame sequence set, whether a safety event exists is detected through the trained safety event detection model; if a safety event exists, a second key frame sequence set of the construction site monitoring video is obtained; subsequently, based on the state machine rules, the trained safety event detection model, and the second key frame sequence set, the event state corresponding to the safety event is obtained; in summary, by adopting the trained safety event detection model, safety detection can be supported for different types of construction scenarios, and at the same time, the safety event and the event state corresponding to the safety event can be obtained, which can reduce the repeated transmission of detection results; it can improve the problem that the detection results of safety detection in the current construction site are repeatedly transmitted and it is difficult to support different types of construction scenarios, and achieve the effect of reducing the repeated transmission of construction site detection results and supporting safety detection for different types of construction scenarios.

[0011] In a possible implementation manner, the state machine rules include:

[0012] The first state machine rule is that when the safety event is first detected, the event state of the safety event is converted from the not started / ended state to the possible occurrence state;

[0013] The second state machine rule is that in the possible occurrence state, when the safety event is continuously detected within a first preset time, the event state of the safety event is converted from the possible occurrence state to the determined occurrence state;

[0014] The third state machine rule is that in the determined occurrence state, when the safety event is not detected at the beginning, the event state of the safety event is converted from the determined occurrence state to the possible end state;

[0015] The fourth state machine rule is that in the possible end state, when the safety event is not detected continuously within a second preset time, the event state of the safety event is converted from the possible end state to the not started / ended state.

[0016] In a possible implementation manner, the obtaining of the event state corresponding to the safety event based on the state machine rules, the trained safety event detection model, and the second key frame sequence set includes:

[0017] According to the second key frame sequence set, the safety event is detected through the trained safety event detection model; when the safety event is first detected, according to the first state machine rule, the event state corresponding to the safety event is obtained;

[0018] When the security event is continuously detected within the possible occurrence state and exceeding the first preset time, according to the second state machine rule, obtain the event state corresponding to the security event;

[0019] When the determined occurrence state starts and the security event is not detected, according to the third state machine rule, obtain the event state corresponding to the security event;

[0020] When in the possible end state and the security event is not detected continuously within the second preset time, according to the fourth state machine rule, obtain the event state corresponding to the security event.

[0021] In a possible implementation manner, it further includes:

[0022] When the security event is the human event, track the personnel in the second key frame sequence set based on the target tracking algorithm.

[0023] In a possible implementation manner, it further includes:

[0024] When the event state corresponding to the security event is the possible occurrence state, perform face recognition on the personnel in the second key frame sequence set based on the face detection model and the face database, and obtain the face recognition information;

[0025] Push the security event and the face recognition information.

[0026] In a possible implementation manner, the method for obtaining the face database includes:

[0027] Collect the face pictures and personnel IDs of the personnel at the construction site;

[0028] Input the face picture into the face detection model, and output the one-dimensional vector of the face picture;

[0029] Store the personnel ID and the one-dimensional vector in the vector database to obtain the face database, where the data ID in the face database is the personnel ID, and the face feature value in the face database is the one-dimensional vector.

[0030] In a possible implementation manner, it further includes:

[0031] For the security event and the event state corresponding to the security event other than when the security event is the human event and the event state corresponding to the security event is the possible occurrence state, directly push the security event and the event state corresponding to the security event.

[0032] In a possible implementation manner, before obtaining the first key frame sequence set of the construction site monitoring video, it further includes:

[0033] Obtain the video stream of the monitoring video at the construction site, and decode the video stream into image frames.

[0034] In the second aspect of the present application, a method for training a safety event detection model is provided, including:

[0035] Obtain a set of historical key frame sequences of different types of construction sites with labels; the set of historical key frame sequences includes key frame sequences with labels of not wearing labor protection supplies, human left fire sources, burning, and / or building collapse; train a safety event detection model according to the set of historical key frame sequences to obtain a trained safety event detection model.

[0036] In the third aspect of the present application, a detection and analysis system for construction safety events is provided, including: a monitoring device, an edge computing device, and a background service device;

[0037] The monitoring device is connected to the edge computing device and is used to generate the video stream of the monitoring video at the construction site;

[0038] The edge computing device is used to read the video stream, decode the video stream, and extract the set of key frame sequences; it is also used to obtain the safety events at the construction site and the event status corresponding to the safety events according to the set of key frame sequences of the video stream;

[0039] The background service device is connected to the edge computing device and is used to push the safety events and the event status corresponding to the safety events, and perform face recognition on the human events in the safety events.

[0040] It should be understood that the content described in the summary of the invention is not intended to limit the key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become easily understood through the following description. Brief Description of the Drawings

[0041] In combination with the drawings and referring to the following detailed description, the above and other features, advantages, and aspects of the embodiments of the present application will become more obvious. In the drawings, the same or similar reference numerals represent the same or similar elements, where:

[0042] Figure 1 Shows the structural diagram of a detection and analysis system for construction safety events in an embodiment of the present application;

[0043] Figure 2 Shows the flowchart of a detection and analysis method for construction safety events in an embodiment of the present application;

[0044] Figure 3Shows a schematic diagram of the state machine rules in the embodiments of the present application;

[0045] Figure 4 Shows a flowchart of a method for training a security event detection model in the embodiments of the present application. Detailed implementation manners

[0046] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application.

[0047] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments provided in the present application without making creative efforts belong to the scope of protection of the present application. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for those of ordinary skill in the art related to the content disclosed in the present application, some design, manufacturing or production changes made on the basis of the technical content disclosed in the present application are only conventional technical means and should not be understood that the content disclosed in the present application is insufficient.

[0048] Referring to "embodiments" in the present application means that specific features, structures or characteristics described in connection with the embodiments can be included in at least one embodiment of the present application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those of ordinary skill in the art explicitly and implicitly understand that the embodiments described in the present application can be combined with other embodiments without conflict.

[0049] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the ordinary meanings understood by those with ordinary skills in the technical field to which this application belongs. The words such as "a", "an", "one", "the" and the like involved in this application do not indicate a quantity limitation and may represent a singular or plural number. The terms "include", "comprise", "have" and any variations thereof involved in this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may further include unlisted steps or units, or may further include other steps or units inherent to these processes, methods, products or devices. The words such as "connect", "be connected", "couple" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "plurality" involved in this application means greater than or equal to two. "And / or" describes the association relationship of associated objects and indicates that three relationships may exist. For example, "A and / or B" may represent: A exists alone, A and B exist simultaneously, and B exists alone. The terms "first", "second", "third" and the like involved in this application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0050] A method for detecting and analyzing construction safety incidents provided by an embodiment of this application can be applied to the field of construction safety technology.

[0051] Nowadays, in many working scenarios, such as construction sites, docks, oil fields, coal mines, and power stations, due to reasons such as low safety awareness of workers and easy falling of objects, quite a few accidents caused by not wearing safety helmets occur every year. Based on the above safety incidents, it can be seen that in order to effectively reduce the potential harm to personnel, it is necessary to detect the situation of workers wearing safety helmets in real time at these places.

[0052] At the same time, as construction continues to develop towards deep foundations, high-rise buildings, and complex projects, coupled with characteristics such as long cycles and high labor intensity, various construction safety accidents continue to occur. For example, as a frequently occurring accident that has long existed in the construction process, namely the collapse accident, its occurrence ratio shows an increasing trend year by year. Based on the above safety incidents, it can be seen that it is very important to give early warnings for collapse accidents in construction sites.

[0053] In addition, fires caused by human - left ignition sources (such as unextinguished cigarette butts) and burning in construction sites are also common. As is well - known, the best time to extinguish a fire is the initial stage, that is, within ten - odd minutes after the fire breaks out. At this time, the burning area is not large and the smoke flow speed is relatively slow, which is the most favorable time for extinguishing the fire and also the most favorable period for the safe evacuation of personnel. Therefore, an efficient and accurate fire recognition and alarm system plays a crucial role. Based on the above - mentioned safety incidents, it can be seen that a smoking detection algorithm can be applied to construction sites to detect in real - time whether there is a smoking behavior among the personnel in the video scene, generate a warning event immediately when someone is detected smoking, and a deep - learning method combined with human postures can also be used, which can not only identify objects but also support the behavior recognition of smoking actions, improving the recognition ability and achieving the effects of fast detection speed and low computing cost. At the same time, by combining the fire monitoring and alarm system with intelligent video analysis, the video image information can be automatically analyzed and recognized without manual intervention, and abnormal smoke and early signs of fire in the monitored area can be detected in a timely manner, giving early warnings in the fastest and best way, effectively assisting firefighters in dealing with fire crises and minimizing false alarms and missed alarms.

[0054] However, in the current solutions, the existing detection technologies are only applicable to single or simple scenarios. While they can only repeatedly send simple detection results, the face detection is also relatively slow. In other words, the detection results of the current safety detection in construction sites are repeatedly sent and it is difficult to support different types of construction scenarios. To solve this technical problem, the embodiments of this application provide a detection and analysis system for construction safety incidents.

[0055] Figure 1 The structure diagram of a detection and analysis system for construction safety incidents in the embodiments of this application is shown. Refer to Figure 1 In this embodiment, a detection and analysis system for construction safety incidents includes: a monitoring device 101, an edge - computing device 102 connected to the monitoring device, and a back - end service device 103 connected to the edge - computing device.

[0056] The monitoring device 101 can be various cameras installed in the construction site, which is used to generate the video stream of the construction site monitoring video.

[0057] The edge - computing device 102 connected to the monitoring device 101 internally deploys a trained safety - incident detection model.

[0058] After reading the video stream of the construction site monitoring video in the monitoring device 101, the edge computing device 102 can directly decode the video stream into a set of image frame sequences with timestamps. Subsequently, the edge computing device 102 can perform dynamic key frame extraction on the set of image frame sequences with timestamps, and use the extracted key frame sequence set as input to send to the trained security event detection model deployed inside the edge computing device 102. Subsequently, the trained security event detection model will detect the security events in the key frame sequence set, that is, the security events at the construction site.

[0059] Among them, the basic model used by the trained security event detection model is the PP-YOLO model. The process of deploying the trained security event detection model into the edge computing device 102 includes: using the training samples of the training model (that is, the historical key frame sequence sets of different types of construction sites with labels) to perform transfer training on the PP-YOLO model and output a model in paddle format. Subsequently, export the model in ONNX format, then convert the ONNX format model into a TensorRT engine file of half-precision floating-point type, and finally deploy the TensorRT engine file (that is, the trained security event detection model) into the edge computing device 102.

[0060] An internal state machine (event state machine) is also set in the edge computing device 102, which can obtain the event state corresponding to the security event at the construction site after detecting the security event through the trained security event detection model, that is, maintain the event state of the security event. Subsequently, the edge computing device 102 sends the security event and the event state corresponding to the security event to the background service device 103.

[0061] Among them, the state machine includes a natural event state machine and a human event state machine.

[0062] When the security event detected by the trained security event detection model is a natural event, the edge computing device 102 will directly send the security event to the natural event state machine, and at the same time adjust the time interval for extracting dynamic key frames (that is, the extraction frequency of key frames), and perform state conversion on the event state of the security event according to the state machine rules.

[0063] When the security event detection model after training detects that the security event is a human event, the edge computing device 102 will first track the personnel in the key frame sequence set based on a target tracking algorithm (such as Deep Simple Online Realtime Tracking, i.e., Deep SORT algorithm), and at the same time adjust the time interval for extracting dynamic key frames (i.e., the extraction frequency of key frames), and then send the security event to the human event state machine, and perform state transition on the event state of the security event according to the state machine rules.

[0064] A face detection model (such as Insight Face model) and a face database (such as vector database milvus) are deployed inside the background service device 103, mainly for face recognition.

[0065] After receiving the security event and the event state corresponding to the security event, the background service device 103 will judge the event type of the security event and the state type of the event state.

[0066] When the security event is a human event and the event state corresponding to the security event is the possible occurrence state, the background service device 103 will perform face recognition on the personnel in the key frame sequence set based on the face detection model and the face database, and push the obtained face recognition information and the security event to the corresponding business system connected to the detection and analysis system of the construction safety event.

[0067] For security events and the event states corresponding to security events other than when the security event is a human event and the event state corresponding to the security event is the possible occurrence state, they will be directly pushed to the corresponding business system connected to the detection and analysis system of the construction safety event.

[0068] In the embodiment of the present application, for the detection and analysis system of the construction safety event, the method of combining edge computing and background service can reduce the consumption of computing and network resources; realizing eventization based on multi-target tracking, face recognition and state machine can reduce repeated alarms; face comparison based on vector database can speed up face recognition; using a single model to implement multiple scenarios based on the security event detection model can support scene configuration at the same time. In addition, the detection and analysis system of the construction safety event uses TensorRT to achieve model acceleration, which can not only reduce the resources consumed by the operation, but also enable the algorithm to run on the edge computing device 102 with less computing power.

[0069] Specifically, a method combining edge computing and back-end services is used to reduce computing and network resource consumption. Among them, the security time detection model is deployed in the edge computing device 102, and the face detection model is deployed on the back-end service device. This means that many controls will be implemented through local devices without being handed over to the cloud, and the processing process will be completed at the local edge computing layer. This will undoubtedly greatly improve the processing efficiency and reduce the load on the cloud. At the same time, since it is closer to the user, it can also provide a faster response to the user and solve the requirements at the edge side.

[0070] In addition, all face recognition cameras are regarded as a whole, that is, multiple cameras at the construction site are linked for face recognition. Based on this whole, linkage defense and big data analysis are carried out, and early warnings are given according to the set rules (i.e., the warning rules set according to actual needs). For example, if a certain person or a person from a certain department should not appear in a certain area, then in this area, a certain person or a person from a certain department is equivalent to being on the blacklist, and a warning will be given as long as they appear. Another example is that the frequency of appearance of a certain group of people in a certain area can also be set, and a warning will be given if the number of appearances in a day or a month is higher than the set value.

[0071] In addition, according to different scenarios, events are classified, and common points are extracted. Using the configuration method can reduce the development workload while increasing the support for multiple scenarios. Based on object tracking and finite state machines, the detection result events are evented to reduce duplicate sending. At the same time, according to different state transitions, appropriate pictures or videos are sent as evidence.

[0072] The above is an introduction to the embodiment of the detection and analysis system for construction safety events. The following further illustrates the solution of the present application through the embodiment of the detection and analysis method for construction safety events.

[0073] Figure 2 The flowchart of a detection and analysis method for construction safety events in an embodiment of the present application is shown. Refer to Figure 2 In this embodiment, a detection and analysis method for construction safety events includes:

[0074] Step S201: Obtain a first key frame sequence set of the monitoring video of the construction site.

[0075] Step S202: According to the first key frame sequence set, use the trained safety event detection model to detect whether there is a safety event; the safety event includes human events and natural events, the human events include not wearing labor protection supplies and / or humanly leaving open flames, and the natural events include burning and / or building collapse. The trained safety event detection model is used to represent the mapping relationship between the key frame and the safety event.

[0076] Step S203: If there is such a safety event, obtain a second key frame sequence set of the construction site monitoring video, and the extraction frequency of the second key frame is greater than that of the first key frame.

[0077] Step S204: Based on the state machine rules, the trained safety event detection model, and the second key frame sequence set, obtain the event state corresponding to the safety event, where the event state includes unstarted / ended state, possible occurrence state, definite occurrence state, and possible ending state.

[0078] By adopting the above technical solutions, obtain a first key frame sequence set of the construction site monitoring video; then, according to the first key frame sequence set, use the trained safety event detection model to detect whether there is a safety event; if there is a safety event, obtain a second key frame sequence set of the construction site monitoring video; subsequently, based on the state machine rules, the trained safety event detection model, and the second key frame sequence set, obtain the event state corresponding to the safety event; in summary, by adopting the trained safety event detection model, it is possible to support safety detection for different types of construction scenarios, and at the same time obtain the safety event and the event state corresponding to the safety event, which can reduce the repeated transmission of detection results; it can improve the problem that the detection results of safety detection in the current construction site are repeatedly transmitted and it is difficult to support different types of construction scenarios, and achieve the effect of reducing the repeated transmission of construction site detection results and supporting safety detection for different types of construction scenarios.

[0079] In step S201, the construction site includes but is not limited to construction sites, docks, oil fields, coal mines, and power base stations. The first key frame sequence set of the construction site monitoring video can be obtained by an edge computing device according to a preset first extraction frequency. The first key frame sequence set can be that the edge computing device directly performs dynamic key frame extraction on the image frames according to the first extraction frequency. The first key frame sequence set can also be that the edge computing device, after performing dynamic key frame extraction on the image frames, extracts the first key frame sequence set from the dynamic key frames according to the first extraction frequency.

[0080] In the embodiment of the present application, the first extraction frequency is the default key frame extraction frequency of the construction safety event detection and analysis system, that is, the construction safety event detection and analysis system defaults to extract one frame every m seconds and fixedly extracts key frames. For example, set the value of m to 2, that is, the construction safety event detection and analysis system will extract key frames at a fixed frequency of one frame every 2 seconds.

[0081] In the embodiment of the present application, the first extraction frequency can be modified by changing the configuration parameters according to the actual situation. By performing extraction on the dynamic key frames, the computing resource consumption of the edge computing device can be reduced.

[0082] In step S202, the trained security event detection model is used to represent the mapping relationship between key frames and security events. That is, the first key frame sequence set of the construction site monitoring video obtained is input into the trained security event detection model. Through the trained security event detection model, it can be detected whether there is a security event in the first key frame sequence set and the event type of the security event.

[0083] In the embodiment of the present application, the event types of security events include human events and natural events.

[0084] Among them, human events include not wearing labor protection supplies and / or leaving a human-caused ignition source. Not wearing labor protection supplies includes personnel who do not wear safety helmets, reflective vests, and / or labor protection shoes in the key frame. Leaving a human-caused ignition source includes personnel who smoke and / or discard unextinguished cigarette butts in the key frame.

[0085] For example, if there is a person not wearing a safety helmet in the key frame of the first key frame sequence set, through the trained security event detection model, it can be detected that there is a human event in the key frame of the first key frame sequence set, and it can be detected that the human event is not wearing labor protection supplies.

[0086] Natural events include burning and / or building collapse. Burning includes the presence of flames and smoke generated by burning in the key frame. Building collapse includes the presence of a collapsed building in the key frame.

[0087] For example, if there is a collapsed building in the key frame of the first key frame sequence set, through the trained security event detection model, it can be detected that there is a natural event in the key frame of the first key frame sequence set, and it can be detected that the human event is building collapse.

[0088] In step S203, when it is detected that there is a security event in the key frame of the first key frame sequence set, the second key frame sequence set of the construction site monitoring video is obtained through the edge computing device according to the preset second extraction frequency. The second key frame sequence set can be that the edge computing device directly performs dynamic key frame extraction on the image frames according to the second extraction frequency. The second key frame sequence set can also be that the edge computing device extracts the second key frame sequence set from the dynamic key frames again according to the second extraction frequency after performing dynamic key frame extraction on the image frames.

[0089] In the embodiment of the present application, the extraction frequency of the second key frame is greater than the extraction frequency of the first key frame, that is, the preset second extraction frequency is greater than the preset first extraction frequency. For example, if the first extraction frequency is set to extract a frame at a fixed frequency of every 2 seconds, then the second extraction frequency will be set to extract a frame for every key frame in order to be used for subsequent tracking of security events.

[0090] In the embodiment of the present application, when the event status of a security event is the ended status, the extraction frequency of key frames will be restored from the second extraction frequency to the first extraction frequency, and the extraction time interval of key frames will be restored to the default value.

[0091] In step S204, the state machine rule is implemented relying on the state machine (event state machine) set inside the edge computing device.

[0092] In the embodiment of the present application, the state machine set inside the edge computing device is a finite state machine. For natural events, event types are not distinguished in single monitoring, so there is one state machine for each type of event. Therefore, a global state machine is used in implementation. For human events, different people are concerned in single monitoring, so there is one state machine for each person. Therefore, a local state machine is used in implementation, that is, it is created according to the occurrence of an event corresponding to a certain person and destroyed when the event ends.

[0093] In the embodiment of the present application, the event status corresponding to the security event obtained according to the security event detection model and the second key frame sequence set trained according to the state machine rule will also be converted as the event progresses. The state machine rule is the condition for the conversion of the event status corresponding to the security event and the triggered operation.

[0094] In some embodiments, the state machine rule includes:

[0095] The first state machine rule is that when the security event is first detected, the event status of the security event is converted from the not started / ended status to the possible occurrence status.

[0096] The second state machine rule is that when the security event is continuously detected within the first preset time in the possible occurrence status, the event status of the security event is converted from the possible occurrence status to the definite occurrence status.

[0097] The third state machine rule is that when the security event is not detected at the beginning in the definite occurrence status, the event status of the security event is converted from the definite occurrence status to the possible end status.

[0098] The fourth state machine rule is that when the security event is not detected continuously within the second preset time in the possible end status, the event status of the security event is converted from the possible end status to the not started / ended status.

[0099] Figure 3 Shows a schematic diagram of the state machine rule in the embodiment of the present application. See Figure 3, when a security event is detected for the first time, the event status corresponding to the security event will be changed from the not-started / ended status to the possible-occurrence status. When the event status of the security event is changed, an event push will be triggered, that is, the current event status and the key frame (image) will be pushed to the background service device, and video recording will start.

[0100] If security events are continuously detected for more than the first preset time, that is, security events are continuously detected for more than n seconds, the event status corresponding to the security event will be changed to the definite-occurrence status. After the event status of the security event is changed, an event push will be triggered, that is, the current event status and the recorded video of n seconds will be pushed to the background service device, and video recording will end.

[0101] Among them, for natural events, the key frame extraction time interval will be restored to the default initial value (i.e., the first extraction frequency). For human events, the key frame extraction time interval will be restored to the default initial value (i.e., the first extraction frequency) only when there is no human event state machine currently.

[0102] It should be noted that if security events cannot be continuously detected for n seconds, the event status corresponding to the security event will return to the not-started / ended status. At this time, no event push will be triggered, and video recording will end.

[0103] In the embodiment of the present application, the first preset time (i.e., n) is defaulted to 10. The first preset time can be modified by changing the configuration parameters according to the actual situation.

[0104] In the embodiment of the present application, when the event status of the security event is the definite-occurrence status, if no security event is detected at the beginning, the event status corresponding to the security event will be changed to the possible-end status. The change of the event status corresponding to the security event will not trigger an event push, but the key frame extraction time interval will be adjusted to extract every frame (i.e., the key frame extraction frequency will be adjusted to the second extraction frequency).

[0105] In the embodiment of the present application, when the event status of the security event is the possible-end status, if no security event is detected within the continuous second preset time, that is, no security event is detected for continuous t seconds, the event status corresponding to the security event will be changed to the not-started / ended status, and at the same time, an event push will be triggered, and the current event status corresponding to the security event and the key frame (image) will be pushed to the background service device. If the security event is not continuously not detected for t seconds, the event status corresponding to the security event will return to the definite-occurrence status.

[0106] It should be noted that determining the transition from the occurrence state to the possible end state and the transition from the possible end state to the occurrence state will both trigger the modification of the key frame extraction time interval. For natural events, the key frame extraction time interval will be restored to the default initial value (i.e., the first extraction frequency), while for human events, the key frame extraction time interval will be restored to the default initial value (i.e., the first extraction frequency) only when there is no human event state machine currently.

[0107] In the embodiment of the present application, the second preset time (i.e., t) is defaulted to 10. The second preset time can be modified by changing the configuration parameters according to the actual situation.

[0108] In some embodiments, step S204 includes: step A1 - step A5.

[0109] Step A1: Detect the safety event according to the second key frame sequence set through the trained safety event detection model.

[0110] Step A2: Under the condition of initially detecting the safety event, obtain the event state corresponding to the safety event according to the first state machine rule.

[0111] Step A3: Under the condition of continuously detecting the safety event in the possible occurrence state and exceeding the first preset time, obtain the event state corresponding to the safety event according to the second state machine rule.

[0112] Step A4: Under the condition of initially not detecting the safety event in the determined occurrence state, obtain the event state corresponding to the safety event according to the third state machine rule.

[0113] Step A5: Under the condition of continuously not detecting the safety event within the second preset time in the possible end state, obtain the event state corresponding to the safety event according to the fourth state machine rule.

[0114] In the embodiment of the present application, for the convenience of understanding, taking the safety event as not wearing labor protection supplies in human events as an example, step S204 will be described.

[0115] In the embodiment of the present application, according to the second key frame sequence set, the trained safety event detection model is used to detect not wearing labor protection supplies. Under the condition of initially detecting not wearing labor protection supplies, according to the first state machine rule, the event state corresponding to not wearing labor protection supplies is converted from the never started / ended state to the possible occurrence state at this time, that is, the event state corresponding to not wearing labor protection supplies obtained at this time is that not wearing labor protection supplies is in the possible occurrence state.

[0116] In an embodiment of the present application, when it is possible that labor protection articles are not worn and it is continuously detected that labor protection articles are not worn within a first preset time, according to the second state machine rule, the event state corresponding to the non-wearing of labor protection articles is converted from the possible occurrence state to the definite occurrence state at this time, that is, the event state corresponding to the non-wearing of labor protection articles obtained at this time is that the non-wearing of labor protection articles is in the definite occurrence state.

[0117] In an embodiment of the present application, when the non-wearing of labor protection articles is in the definite occurrence state and it starts to be not detected that labor protection articles are not worn, according to the third state machine rule, the event state corresponding to the non-wearing of labor protection articles is converted from the definite occurrence state to the possible end state at this time, that is, the event state corresponding to the non-wearing of labor protection articles obtained at this time is that the non-wearing of labor protection articles is in the possible end state.

[0118] In an embodiment of the present application, when the non-wearing of labor protection articles is in the possible end state and it is continuously not detected that labor protection articles are not worn within a second preset time, according to the fourth state machine rule, the event state corresponding to the non-wearing of labor protection articles is converted from the possible end state to the not started / ended state at this time, that is, the event state corresponding to the non-wearing of labor protection articles obtained at this time is that the non-wearing of labor protection articles is in the not started / ended state.

[0119] In an embodiment of the present application, if during the above process, it is not continuously detected that labor protection articles are not worn within the first preset time and it is not continuously not detected that labor protection articles are not worn within the second preset time, corresponding event state conversions will also be made according to the corresponding state machine rules at this time.

[0120] In some embodiments, the method further includes: step S205.

[0121] Step S205: When the safety event is the human event, track the personnel in the second key frame sequence set based on the target tracking algorithm.

[0122] In an embodiment of the present application, after the trained safety event detection model detects a human event, the edge computing device will start the processing method for the human event, that is, first track the personnel in the key frame sequence set based on the target tracking algorithm, and at the same time adjust the time interval for extracting dynamic key frames, and then send the safety event to the human event state machine, and perform state conversion on the event state of the safety event according to the state machine rules.

[0123] In some embodiments, the method further includes: step S206 - step S207.

[0124] Step S206: When the event status corresponding to the security event is the possible occurrence status, perform face recognition on the personnel in the second key frame sequence set based on the face detection model and the face database to obtain face recognition information; Step S207: Push the security event and the face recognition information.

[0125] In an embodiment of the present application, after the background service device receives the security event and the event status corresponding to the security event sent by the edge computing device, it judges the event type of the security event and the status type of the event status.

[0126] When the background service device receives a human event and the event status corresponding to the human event is the possible occurrence status, it will perform face recognition on the personnel in the second key frame sequence set based on the face detection model and the face database to obtain face recognition information. For the convenience of subsequent processing, the background service device will attach the face recognition information to the human event and push the human event and the face recognition information together to the corresponding business system connected to the detection and analysis system of the construction safety event.

[0127] In some embodiments, the method for obtaining the face database includes: Step B1 - Step B3.

[0128] Step B1: Collect the face pictures and personnel IDs of the personnel at the construction site.

[0129] Step B2: Input the face picture into the face detection model and output the one-dimensional vector of the face picture.

[0130] Step B3: Store the personnel ID and the one-dimensional vector in the vector database to obtain the face database. The data ID in the face database is the personnel ID, and the face feature value in the face database is the one-dimensional vector.

[0131] In an embodiment of the present application, the face detection model can use the Insight Face model, but is not limited to this model. The face recognition model does not need to be trained separately and can directly use the pre-trained model.

[0132] In an embodiment of the present application, collect the face pictures of the personnel at the construction site. Each person can collect 1 to 3 pictures and use these pictures as input. Output the one-dimensional vector through the face detection model. Use the personnel ID as the data ID and the output one-dimensional vector as the face feature value, and store them in the vector database milvus. At this time, the vector database can be used as the face database for subsequent face comparison and recognition.

[0133] In some embodiments, the method further includes: Step S208.

[0134] Step S208: For the security events other than those where the security event is a human event and the event status corresponding to the security event is the likely - to - occur status, and the event status corresponding to the security event, directly push the security event and the event status corresponding to the security event.

[0135] In the embodiments of the present application, for other types of security events and the event status corresponding to the security events other than human events where the event status corresponding to the human event is the likely - to - occur status, they will be directly pushed to the corresponding business systems connected to the detection and analysis system of construction safety events.

[0136] In some embodiments, before step S201, it includes: step C.

[0137] Step C: Obtain the video stream of the construction site monitoring video and decode the video stream into image frames.

[0138] In the embodiments of the present application, the video stream of the construction site monitoring video can be a real - time video stream, and based on the real - time video stream, handle the security events occurring at the current and future construction sites. The video stream of the construction site monitoring video can also be a non - real - time video stream, such as a historical video stream, and then based on the historical video stream, conduct accident analysis on the security events occurring at the historical construction sites.

[0139] In summary, a detection and analysis method for construction safety events mainly focuses on the safety detection of construction site events. First, according to whether a person is included in the event, the events are divided into human events and natural events. For example, human events include not wearing a safety helmet and smoking, and natural events include burning and building collapse. According to the characteristics of burning and building collapse events (that is, the location basically no longer changes after the event occurs, and the state duration after the event occurs is relatively long), it is known that this type of event does not require long - term tracking and ID recognition. Therefore, for this type of event, only event pushing is required after detection. According to the characteristics of not wearing a safety helmet and smoking events (that is, the responsibility needs to be assigned to a person, and it is very likely to move within the monitoring area), it is known that this type of event requires tracking to prevent repeated alarms and at the same time requires face ID recognition. Therefore, for this type of event, face recognition is required after detection, and then event pushing is carried out, which helps with subsequent supervision and automatic warning.

[0140] It can be seen that to ensure the safety of personnel in the operation areas of the construction project field, through various cameras installed at the construction site in cooperation with AI video processing and face recognition technologies, events to be detected are continuously monitored in real time, the front-end video image data is analyzed and mined, and identification and alarm services for safety risk events such as personnel, environment, equipment, and materials are provided, so as to achieve all-round real-time monitoring of people, machines, materials, methods, and environment, and meet the management needs of personnel safety and property safety in different construction site application scenarios. Real-time monitoring of whether labor workers wear safety helmets and reflective vests can effectively prevent accidental accidents caused by negligence, avoid potential safety accidents, and improve supervision efficiency.

[0141] It should be further noted that the above-mentioned method for detecting and analyzing construction safety events can be applied not only to construction safety events such as collapses, fires, not wearing safety helmets, and entry of non-site personnel into the construction site, but also to construction safety events such as falling objects from heights, object strikes, mechanical injuries, lifting injuries, vehicle injuries, and unauthorized demolition of safety facilities.

[0142] The above is an introduction to the embodiment of the method for detecting and analyzing construction safety events. The following is a further description of the solution of the present application through the embodiment of the training method of the safety event detection model.

[0143] Figure 4 The flowchart of the training method of the safety event detection model in the embodiment of the present application is shown. Refer to Figure 4 In this embodiment, the training method of the safety event detection model includes:

[0144] Step S401: Obtain a historical key frame sequence set of different types of construction sites with labels; the historical key frame sequence set includes key frame sequence sets with labels of not wearing labor protection supplies, human left fire sources, burning, and / or building collapse.

[0145] Step S402: Train the safety event detection model according to the historical key frame sequence set to obtain a trained safety event detection model.

[0146] In the embodiment of the present application, by searching for images of not wearing safety helmets, smoking, burning, and building collapse in the relevant recorded videos at the construction site, and then collecting images of the same type from the Internet, the scene is not limited to the construction site. Then, all the images are annotated, that is, the main part of the event is framed with a rectangle, and then its type is annotated to obtain a historical key frame sequence set of different types of construction sites with labels.

[0147] Specifically, enclose the person not wearing a safety helmet entirely and label it with the label of not wearing labor protection supplies and the label of human event; enclose the person smoking entirely and label it with the label of human left fire source and the label of human event; enclose the flames and smoke generated by burning entirely and label it with the label of burning and the label of self-heating event; enclose the collapsed building entirely and label it with the label of building collapse and the label of self-heating event.

[0148] In the embodiments of the present application, mosaic, color change, flipping, scaling, and shearing processing can also be used to enhance the historical key frame sequence set to generate a processed historical key frame sequence set.

[0149] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0150] It should be understood that although the steps in the flowchart of the accompanying drawings are shown in sequence according to the indication of the arrows, these steps do not necessarily have to be executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps is not strictly limited in order, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages. These sub-steps or stages do not necessarily have to be executed at the same moment, but can be executed at different moments, and their execution order does not necessarily have to be sequential, but can be executed alternately or in turn with at least a part of other steps or sub-steps or stages of other steps.

[0151] The above are only some embodiments of the present application. It should be pointed out that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. A method for detecting and analyzing construction safety incidents, characterized in that, Including: Obtaining a first set of key frame sequences of the monitoring video at the construction site; According to the first set of key frame sequences, using the trained safety event detection model to detect whether there is a safety event; the safety event includes human events and natural events, the human events include not wearing labor protection supplies and / or human left-behind fire sources, and the natural events include burning and / or building collapse. The trained safety event detection model is used to represent the mapping relationship between key frames and safety events. Among them, the state machine includes a natural event state machine and a human event state machine; for natural events, there is a state machine for each type of event, and a global state machine is used in implementation; for human events, for different people concerned in the monitoring, each person will have a state machine, and a local state machine is used in implementation. The local state machine is created according to the occurrence of the event corresponding to a certain person and destroyed when it ends. If there is the safety event, obtaining a second set of key frame sequences of the monitoring video at the construction site, and the extraction frequency of the second key frame is greater than that of the first key frame; when the safety event is the human event, tracking the personnel in the second set of key frame sequences based on the target tracking algorithm; The safety event detection model is trained by the following method: Obtaining a historical set of key frame sequences of different types of construction sites with labels; the historical set of key frame sequences includes key frame sequences with labels of not wearing labor protection supplies, human left-behind fire sources, burning, and / or building collapse; training the safety event detection model according to the historical set of key frame sequences to obtain a trained safety event detection model; Based on the state machine rules, the trained safety event detection model, and the second set of key frame sequences, obtaining the event state corresponding to the safety event, and the event state includes not started / ended state, possible occurrence state, determined occurrence state, and possible end state; When the event state of the safety event is the ended state, the extraction frequency of the key frame will be restored from the second extraction frequency to the first extraction frequency, and the extraction time interval of the key frame will be restored to the default value.

2. The method according to claim 1, characterized in that, The state machine rules include: The first state machine rule is that when the safety event is first detected, the event state of the safety event is converted from the not started / ended state to the possible occurrence state; The second state machine rule is that in the possible occurrence state, when the safety event is continuously detected within the first preset time, the event state of the safety event is converted from the possible occurrence state to the determined occurrence state; The third state machine rule is that in the determined occurrence state, when the safety event is not detected at the beginning, the event state of the safety event is converted from the determined occurrence state to the possible end state; the fourth state machine rule is that in the possible end state, when the safety event is not detected continuously within the second preset time, the event state of the safety event is converted from the possible end state to the not started / ended state.

3. The method according to claim 2, wherein Obtaining the event status corresponding to the security event based on the state machine rules, the trained security event detection model, and the second key frame sequence set includes: Detecting the security event through the trained security event detection model according to the second key frame sequence set; Under the condition of initially detecting the security event, obtaining the event status corresponding to the security event according to the first state machine rule; Under the condition of continuously detecting the security event in the possible occurrence state and exceeding the first preset time, obtaining the event status corresponding to the security event according to the second state machine rule; Under the condition of the determined occurrence state and not detecting the security event at the beginning, obtaining the event status corresponding to the security event according to the third state machine rule; Under the condition of the possible end state and not detecting the security event continuously within the second preset time, obtaining the event status corresponding to the security event according to the fourth state machine rule.

4. The method according to claim 1, characterized in that, It further includes: When the event status corresponding to the security event is the possible occurrence state, performing face recognition on the personnel in the second key frame sequence set based on the face detection model and the face database to obtain face recognition information; pushing the security event and the face recognition information.

5. The method according to claim 4, wherein The method for obtaining the face database includes: collecting face pictures and personnel IDs of the personnel at the construction site; inputting the face pictures into the face detection model to output one-dimensional vectors of the face pictures; Storing the personnel ID and the one-dimensional vector in a vector database to obtain the face database, where the data ID in the face database is the personnel ID, and the face feature value in the face database is the one-dimensional vector.

6. The method according to claim 4, characterized in that, It further includes: For the security event and the event status corresponding to the security event except that the security event is a human event and the event status corresponding to the security event is the possible occurrence state, directly pushing the security event and the event status corresponding to the security event.

7. The method according to claim 1, wherein Before obtaining the first key frame sequence set of the monitoring video of the construction site, it further includes: Obtaining the video stream of the monitoring video of the construction site and decoding the video stream into image frames.

8. A detection and analysis system for construction safety incidents, characterized in that, It includes: Monitoring device, edge computing device, and background service device; The monitoring device, connected to the edge computing device, is used to generate the video stream of the monitoring video of the construction site; The edge computing device is used to read the video stream, decode the video stream, and extract a set of key frame sequences; it is also used to obtain the first set of key frame sequences of the construction site monitoring video; according to the first set of key frame sequences, it detects whether there is a safety event through a trained safety event detection model; the safety event includes human events and natural events, the human event includes not wearing labor protection supplies and / or leaving a human-caused fire source, and the natural event includes burning and / or building collapse. The trained safety event detection model is used to represent the mapping relationship between key frames and safety events. Among them, the state machine includes a natural event state machine and a human event state machine; for natural events, there is one state machine for each type of event, and a global state machine is used in implementation; for human events, for different people concerned in the monitoring, each person will have a state machine, and a local state machine is used in implementation. The local state machine is created when the event corresponding to a certain person occurs and destroyed when it ends; if there is the safety event, obtain the second set of key frame sequences of the construction site monitoring video, and the extraction frequency of the second key frame is greater than that of the first key frame; the safety event detection model is trained through the following method: obtain the historical key frame sequence sets of different types of construction sites with labels; the historical key frame sequence sets include key frame sequence sets with labels of not wearing labor protection supplies, leaving a human-caused fire source, burning, and / or building collapse; according to the historical key frame sequence sets, train the safety event detection model to obtain a trained safety event detection model; based on the state machine rules, the trained safety event detection model, and the second set of key frame sequences, obtain the event state corresponding to the safety event, and the event state includes not started / ended state, possible occurrence state, determined occurrence state, and possible end state; when the event state of the safety event is the ended state, the extraction frequency of the key frame will be restored from the second extraction frequency to the first extraction frequency, and the extraction time interval of the key frame will be restored to the default value. The background service device is connected to the edge computing device and is used to push the safety event and the event state corresponding to the safety event. When the safety event is the human event, track the personnel in the second set of key frame sequences based on the target tracking algorithm.

Images