Construction behavior warning method and device, electronic equipment and computer readable medium

Abstract

The embodiment of the invention discloses a construction behavior warning method and device, electronic equipment and a computer readable medium. A specific embodiment of the method comprises the following steps: controlling monitoring equipment in a target area to collect a real-time video as a to-be-detected video at a top view angle; analyzing the to-be-detected video to obtain a to-be-detected video frame sequence; performing target detection processing on each to-be-detected video frame in the to-be-detected video frame sequence to generate target detection information; determining a to-be-detected person image sequence set and a to-be-detected device image sequence set; performing personnel behavior recognition processing on each to-be-detected personnel image sequence to generate behavior detection information of each construction personnel; performing equipment behavior recognition processing on each to-be-detected equipment image sequence to generate behavior detection information of each construction equipment; and controlling alarm equipment in the target area to perform alarm operation. According to the embodiment, the missed detection rate or the false detection rate of abnormal behaviors can be reduced, so that behavior warning is performed in time, and the construction safety is improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a construction behavior alarm method, device, electronic device, and computer-readable medium. Background Art

[0002] Behavior recognition is a technology used to identify and detect individual behaviors from videos or real-time images. It is widely used in various business fields. In construction scenarios, abnormal behavior alarms based on behavior recognition are common monitoring measures. At present, when abnormal behavior alarms are issued, the usual method is to perform behavior recognition on the collected images or videos in single-frame units, and then issue an alarm for the identified abnormal behavior.

[0003] However, when using the above method to issue abnormal behavior alarms, the following technical problems often occur: in construction scenarios, abnormal behaviors are usually continuous actions, and abnormal behavior detection is performed in units of single-frame images without taking into account the continuity of the action, resulting in a high missed detection rate or a high false detection rate for abnormal behaviors, which in turn leads to untimely abnormal behavior alarms or erroneous alarms.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0005] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] Some embodiments of the present disclosure propose a construction behavior alarm method, device, electronic device, and computer-readable medium to solve one or more of the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide a construction behavior alarm method, the method comprising: controlling a monitoring device in a target area to collect real-time video as a video to be detected from a top-down perspective, wherein the target area is a construction area; parsing the video to be detected to obtain a video frame sequence to be detected, wherein the video frames to be detected in the video frame sequence to be detected correspond to bit sequence information; performing target detection processing on each video frame to be detected in the video frame sequence to be detected to generate target detection information; determining a set of image sequences of personnel to be detected and a set of image sequences of equipment to be detected based on the generated target detection information and the video frame sequence to be detected; performing personnel behavior recognition processing on each image sequence of personnel to be detected in the image sequence set of personnel to be detected according to a pre-trained personnel behavior detection model to generate each construction personnel behavior detection information; performing equipment behavior recognition processing on each equipment image sequence to be detected in the equipment image sequence set to generate each construction equipment behavior detection information; and controlling the alarm device in the target area to perform an alarm operation based on the each construction personnel behavior detection information and the each construction equipment behavior detection information.

[0008] In a second aspect, some embodiments of the present disclosure provide a construction behavior alarm device, the device comprising: a control unit, configured to control a monitoring device in a target area to collect real-time video as a video to be detected from a top-down perspective, wherein the target area is a construction area; a parsing unit, configured to parse the video to be detected to obtain a sequence of video frames to be detected, wherein the video frames to be detected in the sequence of video frames to be detected correspond to bit sequence information; a target detection unit, configured to perform target detection processing on each video frame to be detected in the sequence of video frames to be detected to generate target detection information; a determination unit, configured to determine, based on the generated target detection information and the sequence of video frames to be detected, Determine a set of image sequences of personnel to be detected and a set of image sequences of equipment to be detected; a personnel behavior recognition unit is configured to perform personnel behavior recognition processing on each personnel image sequence to be detected in the above-mentioned set of personnel image sequences to be detected according to a pre-trained personnel behavior recognition model, so as to generate each construction personnel behavior recognition information; an equipment behavior recognition unit is configured to perform equipment behavior recognition processing on each equipment image sequence to be detected in the above-mentioned set of equipment image sequences to be detected, so as to generate each construction equipment behavior detection information; an alarm unit is configured to control the alarm device in the above-mentioned target area to perform an alarm operation according to each construction personnel behavior detection information and each construction equipment behavior detection information.

[0009] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0010] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0011] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: the construction behavior alarm method of some embodiments of the present disclosure can reduce the missed detection rate or false detection rate of abnormal behavior, thereby timely issuing behavior alarms and improving construction safety. Specifically, the reasons for the high missed detection rate or false detection rate of abnormal behavior and the untimely behavior alarm are: in the construction scene, abnormal behavior is usually a continuous action, and abnormal behavior detection is performed in units of single-frame images, without considering the continuity of the action, resulting in a high missed detection rate or false detection rate of abnormal behavior, which in turn leads to untimely abnormal behavior alarms or alarm errors. Based on this, the construction behavior alarm method of some embodiments of the present disclosure, first, controls the monitoring equipment in the target area to collect real-time video as the video to be detected under a bird's-eye view. Among them, the above-mentioned target area is a construction area. Thus, a wider construction area can be covered by a bird's-eye view, reducing the visual blind spots of construction personnel and construction equipment in the video to be detected. Then, the above-mentioned video to be detected is parsed and processed to obtain a video frame sequence to be detected. Among them, the video frame to be detected in the above-mentioned video frame sequence to be detected corresponds to bit sequence information. Thus, the video frame to be detected can be parsed into a frame sequence, thereby retaining the positional information of the frame and the continuity of the action in the video frame. Afterwards, target detection processing is performed on each video frame to be detected in the above-mentioned video frame sequence to be detected to generate target detection information. Thus, the target type of each active target in the frame to be detected can be identified, thereby classifying the construction personnel and construction equipment. Then, according to the generated target detection information and the above-mentioned video frame sequence to be detected, the image sequence set of the personnel to be detected and the image sequence set of the equipment to be detected are determined. Thus, the dynamic behavior of the construction personnel and the equipment state of the construction equipment can be effectively classified, which is convenient for subsequent corresponding identification processing. Secondly, according to the pre-trained personnel behavior detection model, each image sequence of the personnel to be detected in the above-mentioned image sequence set of the personnel to be detected is subjected to personnel behavior recognition processing to generate each construction personnel behavior detection information. Thus, by identifying the behavior sequence contained in the continuous frames, the missed detection rate and the false detection rate of the target behavior can be reduced. Then, each image sequence of the equipment to be detected in the above-mentioned image sequence set of the equipment to be detected is subjected to equipment behavior recognition processing to generate each construction equipment behavior detection information. Thus, the equipment status and equipment behavior of each construction equipment contained in the image sequence of the equipment to be detected can be detected. Finally, according to the above-mentioned detection information of the behavior of each construction personnel and the above-mentioned detection information of the behavior of each construction equipment, the alarm device in the above-mentioned target area is controlled to perform an alarm operation. Thus, through the continuity of the video frame sequence and the processing flow of target detection, target behavior detection, equipment status and equipment behavior detection of the frame sequence, the high missed detection rate and false detection rate caused by ignoring the continuity of actions and the interlacing of multiple devices in the construction scene can be effectively reduced, so that abnormal behavior alarms can be performed in real time, thereby improving construction safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0013] Figure 1 is a flow chart of some embodiments of the construction behavior alarm method according to the present disclosure;

[0014] Figure 2 is a schematic diagram of the structure of some embodiments of the construction behavior warning device according to the present disclosure;

[0015] Figure 3 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0017] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0018] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0019] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0020] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0021] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0022] Figure 1The process 100 of some embodiments of the construction behavior alarm method according to the present disclosure is shown. The construction behavior alarm method comprises the following steps:

[0023] Step 101, controlling the monitoring equipment in the target area to collect real-time video as the video to be detected from a bird's-eye view.

[0024] In some embodiments, the execution subject (e.g., computing device) of the construction behavior alarm method can control the monitoring device in the target area to collect real-time video as the video to be detected from a bird's-eye view through a wired connection or a wireless connection. The target area is the construction area. The execution subject can be a terminal server. The monitoring device can be a surveillance camera.

[0025] It should be noted that the above-mentioned wireless connection methods may include but are not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or developed in the future.

[0026] Step 102: parse the video to be detected to obtain a frame sequence of the video to be detected.

[0027] In some embodiments, the above-mentioned execution subject can parse the above-mentioned video to be detected to obtain a sequence of video frames to be detected. Among them, the video frames to be detected in the above-mentioned sequence of video frames to be detected correspond to bit sequence information. The above-mentioned bit sequence information can be the bit number of the corresponding video frame to be detected in the sequence of video frames to be detected. In practice, the above-mentioned execution subject can parse the above-mentioned video to be detected frame by frame through relevant library functions (such as the cv2.VideoCapture() function in the OpenCV library in Python) or a video encoder (such as the Ffmpeg video encoder) to obtain a sequence of video frames to be detected.

[0028] Step 103 : performing target detection processing on each to-be-detected video frame in the to-be-detected video frame sequence to generate target detection information.

[0029] In some embodiments, the execution subject may perform target detection processing on each video frame to be detected in the sequence of video frames to be detected to generate target detection information. The target detection information includes each anchor frame information and the target type corresponding to each anchor frame information. The anchor frame information may characterize the image range marked by the corresponding anchor frame. The anchor frame position information includes the anchor frame center position coordinates, the anchor frame length and the anchor frame width. The target type corresponding to each anchor frame information characterizes the type of the object to be detected marked by the anchor frame in the video frame to be detected. The target type may be a text label. The target type may be a construction personnel type and a construction equipment type. The construction equipment type characterizes the construction equipment of the non-road mobile mechanical equipment type. The non-road mobile mechanical equipment type includes: excavation mechanical equipment type, earth-moving mechanical equipment type, compaction mechanical equipment type and engineering lifting mechanical equipment type. The excavation mechanical equipment type may be an excavator type. The earth-moving mechanical equipment type may be an earth-moving machine type and a bulldozer type. The compaction mechanical equipment type may be a road roller type. The above-mentioned types of engineering lifting machinery and equipment may include but are not limited to crane types and boom types.

[0030] In some optional implementations of some embodiments, the execution subject may perform target detection processing on each to-be-detected video frame in the to-be-detected video frame sequence through the following steps to generate target detection information:

[0031] In the first step, for each video frame to be detected in the above-mentioned video frame sequence to be detected, the following target detection steps are performed:

[0032] The second sub-step is to perform format conversion processing on the above-mentioned video frame to be detected to obtain a converted video frame. In practice, the above-mentioned execution subject can convert the above-mentioned video frame to be detected into a preset image format through a related library function (for example, the cv2.cvtColor() function in the OpenCV library in Python) to obtain a converted video frame. As an example, the above-mentioned preset image format can be an RGB image format.

[0033] The third sub-step is to perform image enhancement processing on the converted video frame to obtain an enhanced video frame. In practice, the execution subject may perform image enhancement processing on the converted video frame using a preset image enhancement algorithm to obtain an enhanced video frame. As an example, the preset image enhancement algorithm may be, but is not limited to, a Gaussian filtering algorithm and a histogram equalization algorithm.

[0034] The fourth sub-step is to perform pixel normalization processing on the enhanced video frame to obtain a processed video frame. In practice, the execution subject may perform image normalization processing on each pixel in the enhanced video frame to obtain a processed video frame. The image normalization may be Min-Max normalization.

[0035] The fifth sub-step is to input the processed video frame into a pre-trained multi-target detection model to obtain target detection information. The target detection information includes each anchor frame information and the target type corresponding to each anchor frame information. The multi-target detection model can be a neural network model that takes the processed video frame as input and takes the target detection information as output. As an example, the multi-target detection model can be a pre-trained YOLOv5s model. The pre-training can be to train the YOLOv5s model using a labeled construction scene image set and a Focaler-IoU loss function. The construction scene image in the construction scene image set can be an image taken in a construction scene, containing at least one construction worker or at least one construction equipment matching the construction equipment type.

[0036] Step 104: Determine a set of image sequences of persons to be detected and a set of image sequences of devices to be detected based on the generated target detection information and the video frame sequence to be detected.

[0037] In some embodiments, the execution subject can determine the set of image sequences of personnel to be detected and the set of image sequences of equipment to be detected based on the generated target detection information and the video frame sequence to be detected. In practice, one or more construction personnel to be detected who are performing continuous actions can be represented by one of the image sequences of behavior to be detected in the set of image sequences of behavior to be detected. One or more construction machinery equipment can be represented by one of the image sequences of equipment to be detected in the set of image sequences of equipment to be detected.

[0038] In some optional implementations of some embodiments, the execution subject may determine a set of image sequences of persons to be detected and a set of image sequences of devices to be detected according to the generated target detection information and the video frame sequence to be detected through the following steps:

[0039] In the first step, for each video frame to be detected in the above-mentioned video frame sequence to be detected, the following classification steps are performed:

[0040] In the first sub-step, in response to determining that the target detection information corresponding to the above-mentioned video frame to be detected includes a target type representing construction workers, the above-mentioned video frame to be detected is inserted as an image of the person to be detected into a pre-created image of the person to be detected to update the sequence of images of the person to be detected.

[0041] In the second sub-step, in response to determining that the target detection information corresponding to the above-mentioned video frame to be detected includes a target type representing construction equipment, the above-mentioned video frame to be detected is inserted as an image of the device to be detected into a pre-created image sequence of the device to be detected to update the image sequence of the device to be detected.

[0042] The second step is to perform sequence segmentation on the image sequence of persons to be detected according to the updated sequence information corresponding to the image sequence of the behavior to be detected, and obtain a set of image sequences of persons to be detected. In practice, the above-mentioned execution subject can perform sequence segmentation on the image sequence of persons to be detected according to the continuity of the sequence information corresponding to each image of the person to be detected in the image sequence of persons to be detected, and obtain each image sequence of persons to be detected as a set of image sequences of the behavior to be detected. For example, the sequence of sequence information corresponding to the above-mentioned image sequence of persons to be detected is (1, 2, 3, 5, 6, 7, 8, 9, 12, 13, 14, 25, 26, 27). The above-mentioned execution subject can segment the three images of the behavior to be detected whose corresponding sequence information is 1, 2, and 3 into one image sequence of persons to be detected. The five images of persons to be detected whose corresponding sequence information is 5, 6, 7, 8, and 9 are segmented into one image sequence of the behavior to be detected.

[0043] The third step is to perform bit sequence segmentation on the image sequence of the device to be detected according to the bit sequence information corresponding to the updated image sequence of the device to be detected, and obtain a set of image sequences of the device to be detected. In practice, the above-mentioned execution subject can perform bit sequence segmentation on the image sequence of the device to be detected according to the continuity of the bit sequence information corresponding to each device to be detected image in the image sequence of the device to be detected, and obtain each device to be detected image sequence as a set of image sequences of the device to be detected. The specific implementation steps can refer to the implementation steps of "performing bit sequence segmentation on the image sequence of the person to be detected to obtain a set of image sequences of the person to be detected", which will not be repeated here.

[0044] Step 105 , based on the pre-trained personnel behavior detection model, perform personnel behavior recognition processing on each of the personnel image sequences to be detected in the personnel image sequence set to generate each construction personnel behavior detection information.

[0045] In some embodiments, the execution entity may perform personnel behavior recognition processing on each of the to-be-detected personnel image sequences in the to-be-detected personnel image sequence set according to a pre-trained personnel behavior detection model to generate behavior detection information for each construction worker.

[0046] In the process of adopting the above technical solution to solve the problems mentioned in the background technology, the following problems are often accompanied: when performing behavior recognition on continuous video frames containing multiple targets to be detected, since the multiple targets to be detected contained in the image are in motion, when some of the targets to be detected are obscured, multiple targets to be detected in the previous and next frames will be mismatched, resulting in a high missed detection rate or a high false detection rate for abnormal behaviors.

[0047] In the face of the above technical problems, the inventors decided to adopt the following solutions:

[0048] In some optional implementations of some embodiments, the execution subject may perform personnel behavior recognition processing on each of the image sequences of personnel to be detected in the above-mentioned set of image sequences of personnel to be detected according to a pre-trained personnel behavior detection model through the following steps to generate behavior detection information of each construction worker:

[0049] In the first step, for each behavior image sequence to be detected in the above-mentioned behavior image sequence set to be detected, the following behavior recognition steps are performed:

[0050] The first sub-step is to perform the following steps for each behavior image to be detected in the above-mentioned behavior image sequence to be detected:

[0051] Sub-step 1: According to each anchor frame information included in the target detection information corresponding to the above-mentioned behavior image to be detected, the above-mentioned behavior image to be detected is cropped to obtain each construction worker image. In practice, for each anchor frame information in the various anchor frame information included in the target detection information corresponding to the above-mentioned behavior image to be detected, the above-mentioned execution subject can crop according to the image range represented by the above-mentioned anchor frame information (i.e., with the coordinates of the center position of the anchor frame as the center, the length of the anchor frame as the cropping length, and the width of the anchor frame as the cropping width) to obtain an image containing the construction workers as the construction worker image.

[0052] Sub-step two, through a pre-constructed multi-scale feature extraction network, feature extraction processing is performed on the above-mentioned each construction worker image to generate each construction worker appearance feature vector. Among them, the above-mentioned multi-scale feature extraction network can be a neural network that takes the image as input and the extracted feature vector as output, and is used to extract the multi-scale features of the image. In practice, the above-mentioned execution entity can input the above-mentioned each construction worker image into the multi-scale feature extraction network in turn to generate each construction worker appearance feature vector. As an example, the above-mentioned multi-scale feature extraction network can be a residual network or a densely connected neural network. Thus, the appearance features of the construction workers contained in the construction worker images can be extracted through the multi-scale feature extraction network.

[0053] The second sub-step is to cluster the appearance feature information of each construction worker generated above to determine a set of appearance feature information groups of the construction workers. In practice, the execution subject may cluster the appearance feature information of each construction worker generated above by a preset clustering algorithm to determine a set of appearance feature information groups of the construction workers. As an example, the clustering algorithm may be a hierarchical clustering algorithm. Thus, the appearance feature vectors of the construction workers belonging to the same construction worker may be divided into the same group by a clustering algorithm to obtain a set of appearance feature information groups of the construction workers.

[0054] The third sub-step is to sort the images of the construction workers according to the above-mentioned set of the construction worker appearance feature vector groups to obtain a set of construction worker image sequences. In practice, for each of the above-mentioned construction worker appearance feature vector groups in the set of the construction worker appearance feature vector groups, the above-mentioned execution subject may first determine the individual construction worker images corresponding to the above-mentioned construction worker appearance feature vector group as a construction worker image group. Then, the above-mentioned execution subject may sort the individual construction worker images included in the above-mentioned construction worker image group according to the order of the corresponding individual behavior images to be detected in the above-mentioned behavior image sequence to obtain a construction worker image sequence. Afterwards, the obtained individual construction worker image sequences may be determined as a set of construction worker image sequences.

[0055] The fourth sub-step is, for each construction worker image sequence in the above-mentioned construction worker image sequence set, inputting the above-mentioned construction worker image sequence into a pre-trained construction worker behavior detection model to generate construction worker behavior detection information. The above-mentioned construction worker behavior detection information may include construction worker behavior type and confidence. The above-mentioned construction worker behavior type may be a text label. The above-mentioned construction worker behavior type may include but is not limited to: a type of not wearing safety protection equipment and a type of being in the construction equipment operation area. The above-mentioned type of not wearing safety protection equipment can represent that the corresponding construction worker is not wearing safety protection equipment. The above-mentioned safety protection equipment may be a safety helmet and a safety vest. The above-mentioned type of being in the construction equipment operation area can represent that the corresponding construction worker is within a preset range of the construction equipment. The above-mentioned preset range may be 3m. The above-mentioned target behavior detection model may be a neural network model with a construction worker image sequence as input and construction worker behavior recognition information as output. The above-mentioned construction worker behavior detection model may include a depth prediction model and a behavior detection model. The above-mentioned depth prediction model may be a neural network model with each behavior image to be detected in the above-mentioned behavior image sequence to be detected as input and the corresponding depth image as output. The above behavior detection model may be a temporal neural network that takes a sequence of construction worker images as input and behavior detection information as output. The above behavior detection information may include the construction worker behavior type and confidence. As an example, the above-mentioned depth prediction model can be a DPT (Dense Predictive Transformer) model. The above-mentioned behavior detection model can be a Transformer model or a recurrent neural network model. In practice, first, the above-mentioned execution subject can input the above-mentioned behavior image sequence to be detected and the above-mentioned construction personnel image sequence into the above-mentioned depth prediction model and the above-mentioned behavior detection model respectively, and obtain the depth image sequence and the behavior detection information respectively. Then, for each depth image in the above-mentioned depth image sequence, the above-mentioned execution subject can determine the three-dimensional distance between each target pixel in each depth image (the horizontal and vertical coordinates of the target pixel and the depth information) to determine the distance between the construction personnel represented by the above-mentioned construction personnel image sequence and the construction equipment existing in the above-mentioned behavior image to be detected. The above-mentioned each target pixel can be each image pixel corresponding to the center position of the construction personnel represented by the above-mentioned construction personnel image sequence in the above-mentioned depth image and the center position of the construction equipment. The center position of the above-mentioned construction personnel and the center position of the construction equipment can be determined by the center position of each anchor frame included in the target detection information corresponding to the behavior image to be detected corresponding to the above-mentioned depth image. Finally, the above-mentioned execution subject can determine the construction personnel behavior detection information through the above-mentioned behavior detection information and the determined pixel distance.

[0056] The above technical solution and its related contents, as an inventive point of the embodiment of the present disclosure, solve the technical problem that "when performing behavior recognition on continuous video frames containing multiple targets to be detected, since the multiple targets to be detected contained in the image are in motion, when some of the targets to be detected are covered, multiple targets to be detected in the previous and next frames will be mismatched, resulting in a high missed detection rate or a high false detection rate for abnormal behaviors". The factors that lead to a high missed detection rate or a high false detection rate for abnormal behaviors are often as follows: when performing behavior recognition on continuous video frames containing multiple targets to be detected, since the multiple targets to be detected contained in the image are in motion, when some of the targets to be detected are covered, multiple targets to be detected in the previous and next frames will be mismatched. If the above factors are solved, the effect of reducing the missed detection rate and false detection rate of abnormal behaviors can be achieved. In order to achieve this effect, in the present application, an image containing a single construction worker is obtained by cropping, and the appearance features of each construction worker are extracted by feature extraction. Then, the extracted appearance features of the construction workers are clustered, so that the appearance features belonging to the same construction worker in the image sequence are divided into the same construction worker appearance feature vector group. Next, the images of the construction workers corresponding to each group of construction worker appearance feature vectors are sorted according to the order of the image sequence of the personnel to be detected, and a sequence of construction worker images that can characterize the continuous actions of the construction workers is obtained. Therefore, even when the target to be detected is obscured in several consecutive frames, the images of the construction workers can still be matched continuously through feature extraction and feature clustering, thereby reducing the matching errors of multiple targets to be detected in the previous and next frames, and further reducing the missed detection rate or false detection rate of abnormal behavior.

[0057] Step 106 , performing equipment behavior recognition processing on each of the equipment image sequences to be detected in the equipment image sequence set to generate each construction equipment behavior detection information.

[0058] In some embodiments, the execution entity may perform equipment behavior recognition processing on each of the equipment image sequences to be detected in the set of equipment image sequences to be detected, so as to generate each construction equipment behavior detection information.

[0059] In some optional implementations of some embodiments, the execution subject may perform device behavior recognition processing on each device image sequence to be detected in the device image sequence set to be detected through the following steps to generate each construction device behavior detection information, including:

[0060] In the first step, for each image sequence of the device to be detected in the above-mentioned image sequence set of the device to be detected, the following steps are performed:

[0061] The first sub-step is to crop the above-mentioned image sequence of the equipment to be detected to generate each construction equipment image sequence. Among them, the equipment image in the generated construction equipment image sequence only contains a single construction equipment, and the construction equipment contained in each equipment image in the generated construction equipment image sequence is the same construction equipment. It should be noted that the implementation method of "cropping the above-mentioned image sequence of the equipment to be detected to generate each construction equipment image sequence" can refer to the implementation method of "generating a set of construction personnel image sequences", which will not be repeated here.

[0062] The second sub-step is to perform target behavior detection on each of the above-mentioned equipment image sequences to obtain construction equipment behavior detection information.

[0063] In the process of adopting the above technical solutions to solve the problems mentioned in the background technology, the following problems are often accompanied: in construction scenarios, there are many types of violations of mechanical equipment, and the determination of violations also depends on background information (such as weather conditions, etc.). The rate of missed detection or false detection of abnormal behaviors of mechanical equipment only through images is high, which leads to untimely or erroneous alarms of abnormal behaviors.

[0064] In the face of the above technical problems, the inventors decided to adopt the following solutions:

[0065] In some optional implementations of some embodiments, the execution subject may perform target behavior detection on the equipment image sequence to obtain construction equipment behavior detection information through the following steps:

[0066] The first step is to determine the construction area identification information corresponding to the monitoring device identification information of the monitoring device as the target construction area identification information. The construction area identification information represents the target area. The monitoring device identification information may be a device serial number representing the monitoring device. The construction area identification information may be a character string representing the target area. In practice, the construction area identification information corresponds to the construction area identification information to represent the corresponding relationship between the monitoring device and the construction area monitored by the monitoring device.

[0067] The second step is to control the wind speed sensor associated with the construction area identification information to test the current wind speed as wind speed information. The default unit of the wind speed information may be "m / s".

[0068] The third step is to load the target construction equipment behavior detection model. The target construction equipment behavior recognition model is a construction equipment behavior recognition model trained using a construction equipment behavior data set corresponding to the construction equipment image sequence. The target construction equipment behavior detection model includes a construction equipment operation status recognition model and a construction equipment behavior detection model. The construction equipment operation status recognition model may have the same network structure as the behavior detection model. The construction equipment behavior detection model may have the same network structure as the multi-target detection model. The construction equipment behavior data set may be a set of annotated images of construction equipment that is taking pictures against the background of the construction area and is performing operating behaviors. For example, the target type corresponding to the construction equipment image sequence may be an excavator type, and the loaded target construction equipment behavior detection model may be a construction equipment behavior recognition model trained using an excavator equipment behavior data set.

[0069] In the fourth step, the construction equipment image sequence is input into the construction equipment operation status recognition model included in the target construction equipment behavior recognition model to generate construction equipment operation status information. The construction equipment operation status information may include a construction equipment operation status label and a confidence level. For example, the construction equipment operation status label may be "operating" or "not operating".

[0070] In the fifth step, in response to determining that the generated construction equipment operation status information represents that the corresponding construction equipment is running, the above-mentioned construction equipment image sequence, the target type corresponding to the above-mentioned construction equipment image sequence and the above-mentioned construction area identification information are sent to the monitoring terminal to receive the target construction equipment identification information. The above-mentioned monitoring terminal can be a terminal used to monitor construction projects, control various construction area monitoring equipment, and construction equipment. The above-mentioned target construction equipment identification information can be the equipment serial number of the construction equipment contained in the above-mentioned construction equipment image sequence or the identification information that can represent the above-mentioned construction equipment. In practice, after receiving the above-mentioned construction equipment image sequence, the target type corresponding to the above-mentioned construction equipment image sequence and the above-mentioned construction area identification information, the above-mentioned monitoring terminal can determine the target construction area and the various equipment identification information corresponding to each construction equipment in the target construction area through the construction area identification information, and select the construction equipment identification information that matches the construction equipment contained in the construction equipment image sequence as the target construction equipment identification information and send it to the above-mentioned execution subject.

[0071] Step 6: Control the equipment camera associated with the target construction equipment identification information to capture the image inside the target construction equipment as the image to be detected inside the equipment. The target construction equipment is the construction equipment represented by the target construction equipment identification information, and the equipment camera is a built-in camera of the target construction equipment. The camera can be a camera.

[0072] The seventh step is to control the device card reader associated with the target construction equipment identification information to read the operator identity information. The device card reader is a built-in card reader of the target construction equipment. In practice, the execution subject can control the device card reader associated with the target construction equipment identification information to read the operator employee card to obtain the operator identity information. The operator identity information can be the operator employee number or the operator ID number. The operator employee card can be an IC card with the operator identity information entered.

[0073] In the eighth step, in response to determining that the operator identity information is empty or the operator identity information does not meet the preset permission conditions, generate a construction equipment behavior type that represents the unauthorized operation type. In practice, when the device card reader does not read the operator's employee card or fails to read, the operator identity information is empty. The preset permission condition may be that the operator identity information is the same as one of the permission identity information in the pre-built permission identity information set. The construction equipment behavior type may be a label that represents the equipment behavior of the corresponding construction equipment. For example, the construction equipment behavior type generated to represent the unauthorized operation type may be "the operator has no driving authority."

[0074] In the ninth step, the image to be detected in the above-mentioned device is input into the behavior detection model included in the pre-trained construction personnel behavior detection model to generate driver behavior detection information. The above-mentioned driver behavior detection information may include the driver behavior type and the corresponding confidence level. The above-mentioned driver behavior type may include but is not limited to the driver not wearing a seat belt type and the normal type.

[0075] In the tenth step, in response to determining that the construction worker behavior type included in the driver behavior detection information indicates that the driver is not wearing a seat belt, the construction worker behavior type included in the driver behavior detection information is determined as a construction equipment behavior type.

[0076] In step 11, the equipment image sequence is input into the construction equipment behavior detection model included in the target construction equipment behavior detection model to generate construction equipment behavior detection information. The construction equipment behavior detection information may include the construction equipment behavior type and the corresponding confidence level. The construction equipment behavior type includes but is not limited to earth backfill type, earth excavation type and object lifting type.

[0077] In step 12, in response to determining that the type of construction equipment corresponding to the equipment image sequence is the first type of construction equipment, the following steps are performed:

[0078] The first sub-step is to update the type of the construction equipment behavior detection information in response to determining that the construction equipment behavior type included in the above-mentioned construction equipment behavior detection information is a backfilling behavior type and the above-mentioned wind speed information is greater than or equal to the first preset wind speed. Among them, the above-mentioned first category of construction equipment types may include excavation machinery equipment types, earth-moving machinery equipment types and compaction machinery equipment types. The above-mentioned first preset wind speed may be the lowest level 4 wind speed. The above-mentioned level 4 wind speed may be 5.5m / s. In practice, the above-mentioned execution entity may update the construction equipment behavior type label representing "backfilling" included in the above-mentioned construction equipment behavior detection information to a construction equipment behavior type label representing "backfilling under level 4 wind speed".

[0079] The second sub-step is, in response to determining that the construction equipment behavior type included in the construction equipment behavior detection information is an earthwork excavation behavior type and the wind speed information is greater than or equal to the first preset wind speed, updating the type of the construction equipment behavior detection information. In practice, the execution entity may update the construction equipment behavior type label representing "earthwork excavation" included in the construction equipment behavior detection information to a construction equipment behavior type label representing "earthwork excavation under a wind speed of level 4".

[0080] In the thirteenth step, in response to determining that the type of construction equipment corresponding to the above-mentioned equipment image sequence is the second type of construction equipment, the type of construction equipment behavior included in the above-mentioned construction equipment behavior detection information is the object lifting behavior type, and the above-mentioned wind speed information is greater than or equal to the second preset wind speed, the type of construction equipment behavior detection information is updated. Among them, the above-mentioned second type of construction equipment includes engineering lifting machinery equipment type. The above-mentioned second preset wind speed can be the lowest level six wind speed. The above-mentioned level four wind speed can be 10.8m / s. In practice, the above-mentioned execution entity can update the construction equipment behavior type label representing "object lifting" included in the above-mentioned construction equipment behavior detection information to a construction equipment behavior type label representing "object lifting under level six wind speed".

[0081] In the fourteenth step, the updated construction equipment behavior detection information, the generated various construction equipment behavior types and the construction equipment operation status information are determined as the construction equipment behavior detection information.

[0082] The above technical scheme and its related contents, as an inventive point of an embodiment of the present disclosure, solve the technical problem that "in construction scenarios, there are many types of mechanical equipment violations, and the determination of violations also depends on background information (such as weather conditions, etc.). The rate of missed detection or false detection of abnormal behaviors of mechanical equipment only through images is high, which leads to untimely alarms of abnormal behaviors or erroneous alarms". The factors that lead to a high rate of missed detection or false detection of abnormal behaviors are often as follows: In construction scenarios, there are many types of mechanical equipment violations, and the determination of violations also depends on background information (such as weather conditions, etc.). The rate of missed detection or false detection of abnormal behaviors of mechanical equipment only through images is high. If the above factors are solved, the effect of reducing the missed detection rate and false detection rate of abnormal behaviors can be achieved. In order to achieve this effect, this application introduces background information and uses the corresponding target equipment behavior detection model for each different type of mechanical equipment, so that the violations of various types of construction equipment can be effectively identified, thereby reducing the missed detection rate or false detection rate of behavior.

[0083] Step 107, according to the behavior detection information of each construction worker and the behavior detection information of each construction equipment, the alarm equipment in the target area is controlled to perform an alarm operation.

[0084] In some embodiments, the above-mentioned execution entity can control the alarm equipment in the target area to perform alarm operations based on the behavior detection information of each construction worker and the behavior detection information of each construction equipment.

[0085] In some optional implementations of some embodiments, the execution subject may control the alarm device in the target area to perform an alarm operation according to the behavior detection information of each construction worker and the behavior detection information of each construction equipment through the following steps:

[0086] In the first step, for each construction worker behavior detection information in the above-mentioned construction worker behavior detection information, the following steps are performed:

[0087] In the first sub-step, in response to determining that the construction worker behavior type included in the above-mentioned construction worker behavior detection information is a target construction worker behavior type, the above-mentioned construction worker behavior detection information is determined as the target construction worker behavior detection information, and the above-mentioned target construction worker behavior types include: a type of not wearing safety protection equipment and a type of being in the construction equipment operation area.

[0088] In the second sub-step, the image sequence of persons to be detected corresponding to the above target construction personnel behavior detection information is determined as the target construction personnel image sequence.

[0089] In the second step, for each piece of construction equipment behavior detection information in the above-mentioned construction equipment behavior detection information, the following steps are performed:

[0090] In the first sub-step, in response to determining that the construction equipment behavior type included in the construction equipment behavior detection information is the target construction equipment behavior type, the construction equipment behavior detection information is determined as the target construction equipment behavior detection information. The target construction equipment behavior types include the driver unauthorized operation type, the driver not wearing a seat belt type, the earthwork transportation type under a level 4 wind speed, the earthwork backfill type under a level 4 wind speed, and the tower lifting type under a level 6 wind speed.

[0091] In the second sub-step, the image sequence of the equipment to be detected corresponding to the above-mentioned target construction equipment behavior detection information is determined as the target construction equipment image sequence.

[0092] In the third step, the determined behavior detection information of each target construction worker, the image sequence of each target construction worker, the behavior detection information of each target mechanical equipment and the image sequence of each target construction equipment are determined as construction alarm information.

[0093] The fourth step is to display the above construction alarm information on the associated display screen.

[0094] Step 5: Control the associated sound and light alarms in the target area to perform an alarm operation. In practice, the execution subject may control the associated sound and light alarms in the target area to perform sound and light alarms. The sound and light alarms may start a warning light and play a preset warning sound effect.

[0095] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: the construction behavior alarm method of some embodiments of the present disclosure can reduce the missed detection rate or false detection rate of abnormal behavior, thereby timely issuing behavior alarms and improving construction safety. Specifically, the reasons for the high missed detection rate or false detection rate of abnormal behavior and the untimely behavior alarm are: in the construction scene, abnormal behavior is usually a continuous action, and abnormal behavior detection is performed in units of single-frame images, without considering the continuity of the action, resulting in a high missed detection rate or false detection rate of abnormal behavior, which in turn leads to untimely abnormal behavior alarms or alarm errors. Based on this, the construction behavior alarm method of some embodiments of the present disclosure, first, controls the monitoring equipment in the target area to collect real-time video as the video to be detected under a bird's-eye view. Among them, the above-mentioned target area is a construction area. Thus, a wider construction area can be covered by a bird's-eye view, reducing the visual blind spots of construction personnel and construction equipment in the video to be detected. Then, the above-mentioned video to be detected is parsed and processed to obtain a video frame sequence to be detected. Among them, the video frame to be detected in the above-mentioned video frame sequence to be detected corresponds to bit sequence information. Thus, the video frame to be detected can be parsed into a frame sequence, thereby retaining the positional information of the frame and the continuity of the action in the video frame. Afterwards, target detection processing is performed on each video frame to be detected in the above-mentioned video frame sequence to be detected to generate target detection information. Thus, the target type of each active target in the frame to be detected can be identified, thereby classifying the construction personnel and construction equipment. Then, according to the generated target detection information and the above-mentioned video frame sequence to be detected, the image sequence set of the personnel to be detected and the image sequence set of the equipment to be detected are determined. Thus, the dynamic behavior of the construction personnel and the equipment state of the construction equipment can be effectively classified, which is convenient for subsequent corresponding identification processing. Secondly, according to the pre-trained personnel behavior detection model, each image sequence of the personnel to be detected in the above-mentioned image sequence set of the personnel to be detected is subjected to personnel behavior recognition processing to generate each construction personnel behavior detection information. Thus, by identifying the behavior sequence contained in the continuous frames, the missed detection rate and the false detection rate of the target behavior can be reduced. Then, each image sequence of the equipment to be detected in the above-mentioned image sequence set of the equipment to be detected is subjected to equipment behavior recognition processing to generate each construction equipment behavior detection information. Thus, the equipment status and equipment behavior of each construction equipment contained in the image sequence of the equipment to be detected can be detected. Finally, according to the above-mentioned detection information of the behavior of each construction personnel and the above-mentioned detection information of the behavior of each construction equipment, the alarm device in the above-mentioned target area is controlled to perform an alarm operation. Thus, through the continuity of the video frame sequence and the processing flow of target detection, target behavior detection, equipment status and equipment behavior detection of the frame sequence, the high missed detection rate and false detection rate caused by ignoring the continuity of actions and the interlacing of multiple devices in the construction scene can be effectively reduced, so that abnormal behavior alarms can be performed in real time, thereby improving construction safety.

[0096] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a construction behavior alarm device. These device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the construction behavior warning device can be specifically applied to various electronic devices.

[0097] like Figure 2 As shown, the construction behavior alarm device 200 of some embodiments includes: an acquisition unit 201, a parsing unit 202, a target detection unit 203, a determination unit 204, a personnel behavior recognition unit 205, an equipment behavior recognition unit 206 and an alarm unit 207. The control unit 201 is configured to control the monitoring equipment in the target area to collect real-time video as a video to be detected from a top-down perspective, wherein the target area is a construction area; the parsing unit 202 is configured to parse the video to be detected to obtain a video frame sequence to be detected, wherein the video frames to be detected in the video frame sequence to be detected correspond to bit sequence information; the target detection unit 203 is configured to perform target detection processing on each video frame to be detected in the video frame sequence to be detected to generate target detection information; the determination unit 204 is configured to determine the set of personnel image sequences to be detected and A set of image sequences of equipment to be detected; the personnel behavior recognition unit 205 is configured to perform personnel behavior recognition processing on each personnel image sequence to be detected in the above-mentioned set of personnel image sequences to be detected according to a pre-trained personnel behavior recognition model, so as to generate each construction personnel behavior recognition information; the equipment behavior recognition unit 206 is configured to perform equipment behavior recognition processing on each equipment image sequence to be detected in the above-mentioned set of equipment image sequences to be detected, so as to generate each construction equipment behavior detection information; the alarm unit 207 is configured to control the alarm device in the above-mentioned target area to perform an alarm operation according to the above-mentioned each construction personnel behavior detection information and the above-mentioned each construction equipment behavior detection information.

[0098] It is understandable that the units recorded in the construction behavior warning device 200 are similar to those in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the construction behavior warning device 200 and the units included therein, and will not be described in detail here.

[0099] Reference below Figure 3 , which shows a structural schematic diagram of an electronic device 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0100] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory 302 or a program loaded from a storage device 308 into a random access memory 303. Various programs and data required for the operation of the electronic device 300 are also stored in the random access memory 303. The processing device 301, the read-only memory 302, and the random access memory 303 are connected to each other via a bus 304. An input / output interface 305 is also connected to the bus 304.

[0101] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0102] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the read-only memory 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0103] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0104] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0105] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: controls the monitoring equipment in the target area to collect real-time video as the video to be detected from a top-down perspective, wherein the target area is a construction area; parses the video to be detected to obtain a sequence of video frames to be detected, wherein the video frames to be detected in the sequence of video frames to be detected correspond to bit sequence information; performs target detection processing on each video frame to be detected in the sequence of video frames to be detected to generate target detection information; determines a set of image sequences of personnel to be detected and a set of image sequences of equipment to be detected based on the generated target detection information and the sequence of video frames to be detected; performs personnel behavior recognition processing on each image sequence of personnel to be detected in the set of image sequences of personnel to be detected according to a pre-trained personnel behavior detection model to generate each construction personnel behavior detection information; performs equipment behavior recognition processing on each equipment image sequence to be detected in the set of equipment image sequences to be detected to generate each construction equipment behavior detection information; controls the alarm device in the target area to perform an alarm operation based on the each construction personnel behavior detection information and the each construction equipment behavior detection information.

[0106] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0107] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0108] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor including an acquisition unit, a parsing unit, a target detection unit, a determination unit, a personnel behavior recognition unit, a device behavior recognition unit and an alarm unit. Among them, the names of these units do not constitute a limitation on the units themselves in certain circumstances, for example, the acquisition unit may also be described as "a unit that controls the monitoring equipment in the target area to collect real-time video as the video to be detected from a bird's-eye view".

[0109] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0110] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A construction behavior alarm method, comprising: Controlling monitoring equipment in a target area to collect real-time video as a video to be detected from a bird's-eye view, wherein the target area is a construction area; Parsing the video to be detected to obtain a sequence of video frames to be detected, wherein the video frames to be detected in the sequence of video frames to be detected correspond to bit sequence information; Performing target detection processing on each to-be-detected video frame in the to-be-detected video frame sequence to generate target detection information; Determine a set of image sequences of persons to be detected and a set of image sequences of devices to be detected according to the generated target detection information and the sequence of video frames to be detected; According to the pre-trained personnel behavior detection model, each of the to-be-detected personnel image sequences in the to-be-detected personnel image sequence set is subjected to personnel behavior recognition processing to generate behavior detection information of each construction worker; Performing equipment behavior recognition processing on each of the equipment image sequences to be detected in the equipment image sequence set to be detected, so as to generate each construction equipment behavior detection information; According to the behavior detection information of each construction worker and the behavior detection information of each construction equipment, the alarm equipment in the target area is controlled to perform an alarm operation.

2. The method according to claim 1, wherein: The performing target detection processing on each to-be-detected video frame in the to-be-detected video frame sequence to generate target detection information includes: For each to-be-detected video frame in the to-be-detected video frame sequence, the following target detection steps are performed: Performing format conversion processing on the video frame to be detected to obtain a converted video frame; Performing image enhancement processing on the converted video frame to obtain an enhanced video frame; Performing pixel normalization processing on the enhanced video frame to obtain a processed video frame; The processed video frames are input into a pre-trained multi-target detection model to obtain target detection information, wherein the target detection information includes each anchor frame information and a target type corresponding to each anchor frame information, the corresponding target types include construction personnel types and construction equipment types, the construction equipment types represent construction equipment of non-road mobile machinery and equipment types, and the non-road mobile machinery and equipment types include: excavation machinery and equipment types, earth-moving machinery and equipment types, compaction machinery and equipment types and engineering lifting machinery and equipment types.

3. The method according to claim 2, wherein: The step of determining a set of image sequences of persons to be detected and a set of image sequences of devices to be detected based on the generated target detection information and the sequence of video frames to be detected includes: For each to-be-detected video frame in the to-be-detected video frame sequence, the following classification steps are performed: In response to determining that the target type representing construction personnel exists among the target types included in the target detection information corresponding to the video frame to be detected, inserting the video frame to be detected as a person to be detected image into a pre-created person to be detected image sequence to update the person to be detected image sequence; In response to determining that the target type representing the construction equipment exists among the target types included in the target detection information corresponding to the video frame to be detected, inserting the video frame to be detected as a device to be detected image into a pre-created device to be detected image sequence to update the device to be detected image sequence; According to each bit sequence information corresponding to the updated image sequence of the behavior to be detected, the image sequence of the person to be detected is segmented in bit sequence to obtain a set of image sequences of the person to be detected; According to each bit sequence information corresponding to the updated device image sequence to be detected, the device image sequence to be detected is segmented in bit sequence to obtain a set of device image sequences to be detected.

4. The method according to claim 3, wherein: The performing equipment behavior recognition processing on each of the equipment image sequences to be detected in the equipment image sequence set to generate each construction equipment behavior detection information includes: For each device image sequence to be detected in the device image sequence set to be detected, the following steps are performed: Performing cropping processing on the image sequence of the equipment to be detected to generate each construction equipment image sequence, wherein the equipment image in the generated construction equipment image sequence only includes a single construction equipment, and the construction equipment included in each equipment image in the generated construction equipment image sequence is the same construction equipment; For each equipment image sequence in the equipment image sequences, target behavior detection is performed on the equipment image sequence to obtain construction equipment behavior detection information.

5. The method according to claim 4, wherein: The controlling the alarm device in the target area to perform an alarm operation according to the behavior detection information of each construction worker and the behavior detection information of each construction equipment includes: For each construction worker behavior detection information in the various construction worker behavior detection information, the following steps are performed: In response to determining that the construction worker behavior type included in the construction worker behavior detection information is a target construction worker behavior type, determining the construction worker behavior detection information as target construction worker behavior detection information, wherein the target construction worker behavior type includes: a type of not wearing safety protection equipment and a type of being in a construction equipment operation area; Determine the to-be-detected personnel image sequence corresponding to the target construction personnel behavior detection information as the target construction personnel image sequence; For each piece of construction equipment behavior detection information in the individual pieces of construction equipment behavior detection information, the following steps are performed: In response to determining that the construction equipment behavior type included in the construction equipment behavior detection information is a target construction equipment behavior type, determining the construction equipment behavior detection information as target construction equipment behavior detection information, wherein the target construction equipment behavior type includes a driver without authority operation type, a driver not wearing a seat belt type, an earthwork transportation type under a level 4 wind speed, an earthwork backfill type under a level 4 wind speed, and a tower lifting type under a level 6 wind speed; Determine the to-be-detected equipment image sequence corresponding to the target construction equipment behavior detection information as the target construction equipment image sequence; Determine the determined behavior detection information of each target construction worker, the image sequence of each target construction worker, the behavior detection information of each target mechanical equipment, and the image sequence of each target construction equipment as construction alarm information; Displaying the construction alarm information on an associated display screen; Control the associated sound and light alarms in the target area to perform alarm operations.

6. A construction behavior warning device, comprising: A control unit is configured to control the monitoring equipment in the target area to collect real-time video as the video to be detected from a top-down perspective, wherein the target area is a construction area; A parsing unit is configured to parse the video to be detected to obtain a sequence of video frames to be detected, wherein the video frames to be detected in the sequence of video frames to be detected correspond to bit sequence information; an object detection unit, configured to perform object detection processing on each to-be-detected video frame in the to-be-detected video frame sequence to generate object detection information; A determination unit is configured to determine a set of image sequences of persons to be detected and a set of image sequences of devices to be detected according to the generated information of each target detection and the sequence of video frames to be detected; A personnel behavior recognition unit is configured to perform personnel behavior recognition processing on each of the to-be-detected personnel image sequences in the to-be-detected personnel image sequence set according to a pre-trained personnel behavior recognition model, so as to generate behavior recognition information of each construction worker; An equipment behavior recognition unit is configured to perform equipment behavior recognition processing on each of the equipment image sequences to be detected in the set of equipment image sequences to be detected, so as to generate each construction equipment behavior detection information; The alarm unit is configured to control the alarm device in the target area to perform an alarm operation according to the behavior detection information of each construction worker and the behavior detection information of each construction equipment.

7. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images