Alarm methods, equipment and media based on engineering construction safety monitoring video

Abstract

The embodiment of the present application discloses an alarm method, device and medium based on engineering construction safety monitoring video. A specific implementation of the method includes: in response to receiving multimodal monitoring data sent by a monitoring device, performing object behavior recognition on the monitoring image sequence included in the multimodal monitoring data to generate an object behavior recognition result; in response to determining that there is a safety problem in the standard object behavior recognition text representation, generating an alarm prompt information; for the monitoring video included in the multimodal monitoring data, executing the following processing steps: performing frame processing on the monitoring video; performing anomaly detection on each monitoring video frame image in the monitoring video frame image sequence; in response to the presence of an image anomaly detection result that meets the abnormal condition in the image anomaly detection result group, generating an abnormal prompt information, and controlling the voice alarm terminal to alarm. This implementation can improve the safety management level and efficiency of the construction site and reduce the occurrence of accidents.

Description

Technical Field

[0001] The present application relates to the field of construction monitoring, and specifically to an alarm method, equipment and medium based on engineering construction safety monitoring video. Background Art

[0002] With the acceleration of urbanization, the scale and complexity of construction projects are increasing, and the safety management of construction sites is becoming more and more important. Traditional manual inspections can only provide basic safety protection, which cannot meet the intelligent management needs of modern construction sites and easily lead to omissions in construction safety monitoring. Summary of the invention

[0003] The content of this application is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this application is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0004] Some embodiments of the present application propose an alarm method, a computer device and a computer-readable storage medium based on engineering construction safety monitoring video to solve the technical problems mentioned in the above background technology section.

[0005] In a first aspect, some embodiments of the present application provide an alarm method based on engineering construction safety monitoring video, the method comprising: in response to receiving multimodal monitoring data sent by monitoring equipment at a target engineering construction site, performing the following processing steps on a monitoring image sequence included in the multimodal monitoring data: performing object behavior recognition on the above monitoring image sequence to generate an object behavior recognition result; generating a standard object behavior recognition text based on the above object behavior recognition result; in response to determining that the above standard object behavior recognition text represents the existence of a safety problem, generating an alarm prompt information, and sending the above alarm prompt information to an associated alarm terminal; performing the following processing steps on the monitoring video included in the multimodal monitoring data: performing frame processing on the above monitoring video to generate a monitoring video frame image sequence; performing anomaly detection on each monitoring video frame image in the above monitoring video frame image sequence to generate an image anomaly detection result and obtain an image anomaly detection result group; in response to determining that there is an image anomaly detection result that meets an abnormal condition in the above image anomaly detection result group, generating anomaly prompt information, and controlling the associated voice alarm terminal to issue an alarm.

[0006] In a second aspect, the present application also provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the method described in any implementation of the first aspect is implemented.

[0007] In a third aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the method described in any implementation of the first aspect is implemented.

[0008] The above-mentioned various embodiments of the present application have the following beneficial effects: safety monitoring of the construction site can be performed based on the monitoring video or monitoring image sequence in the multimodal monitoring data. Thus, the behavior that will occur at the construction site can be known. In the case where it is determined that there is a safety problem, a prompt message can be generated and sent to the corresponding alarm terminal. In this way, it is possible to prevent potential safety hazards in advance, eliminate safety problems or avoid the object from performing the behavior that is about to occur, thereby reducing or avoiding the occurrence of potential safety hazards, improving the safety management level and efficiency of the construction site, reducing the occurrence of accidents, facilitating communication and task coordination between staff, providing emergency rescue functions, ensuring the safety of workers, improving the quality of work, and meeting the needs of different construction sites at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present application will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar figure labels represent the same or similar elements. It should be understood that the drawings are schematic and that the components and elements are not necessarily drawn to scale.

[0010] Figure 1 It is a flowchart of some embodiments of the alarm method based on engineering construction safety monitoring video according to the present application;

[0011] Figure 2 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present application. DETAILED DESCRIPTION

[0012] Embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0013] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other.

[0014] It should be noted that the concepts such as "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0015] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0016] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0017] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0018] Figure 1 1 is a flowchart of some embodiments of the alarm method based on engineering construction safety monitoring video according to the present application. 100 of some embodiments of the alarm method based on engineering construction safety monitoring video according to the present application is shown. The alarm method based on engineering construction safety monitoring video includes the following steps:

[0019] Step 101, in response to receiving multimodal monitoring data sent by monitoring equipment at a target engineering construction site, the following processing steps are performed on a monitoring image sequence included in the multimodal monitoring data:

[0020] Step 1011: perform object behavior recognition on the monitoring image sequence to generate an object behavior recognition result.

[0021] In some embodiments, the execution subject (e.g., a computing device) of the alarm method based on the engineering construction safety monitoring video may perform object behavior recognition on the above-mentioned monitoring image sequence to generate an object behavior recognition result. For example, object behavior recognition may refer to an image recognition operation for identifying whether there is an abnormal target object / abnormal event in the monitoring image sequence. The object behavior recognition result may represent the abnormal target object / abnormal event in the monitoring image sequence, and the recognition result of the safety level of the identified abnormal target object / abnormal event. The abnormal target object / abnormal event may include: alarm objects (objects in the construction site, such as exposed steel bars), alarm personnel, and alarm events. For example, the above-mentioned execution subject may perform object behavior recognition on the above-mentioned monitoring image sequence through a pre-trained object behavior recognition model to generate an object behavior recognition result. For example, the object behavior recognition model may refer to: a YOLO (You Only Look Once) model. The target engineering construction site may refer to a site currently under construction. The multimodal monitoring data may include a monitoring image sequence and a monitoring video. The monitoring device may include a monitoring camera and a photographing device. The monitoring camera may be used to shoot a monitoring video of the construction site. The photographing device may be used to shoot a monitoring image sequence of the construction site at time intervals. The higher the safety level, the more dangerous it is.

[0022] In an actual application scenario, the execution subject may perform object behavior recognition on the surveillance image sequence through the following steps to generate an object behavior recognition result:

[0023] In the first step, each surveillance image in the above surveillance image sequence is input into a pre-trained construction surveillance alarm image recognition model to generate a construction surveillance alarm image recognition sub-result and obtain a set of construction surveillance alarm image recognition sub-results. For example, the construction surveillance alarm image recognition model may refer to a pre-trained neural network model that takes surveillance images as input and takes construction surveillance alarm image recognition sub-results as output. The construction surveillance alarm image recognition sub-result may indicate whether there is an alarm event in the surveillance image. An alarm event may refer to an event such as a fire or collapse at a construction site. For example, the construction surveillance alarm image recognition model may be a YOLO (You Only Look Once) model.

[0024] In the second step, each surveillance image in the above surveillance image sequence is input into a pre-trained object behavior abnormality recognition model to generate an object behavior abnormality recognition sub-result and obtain an object behavior abnormality recognition sub-result set. The object behavior abnormality recognition model may be a pre-trained neural network model that takes surveillance images as input and takes object behavior abnormality recognition sub-results as output. For example, the object behavior abnormality recognition model may be a target detection model. The object may refer to a user or an animal. The object behavior abnormality recognition sub-result may refer to whether there are intruders / animals at the construction site, and whether there are behaviors that damage the construction site.

[0025] The third step is to merge the construction monitoring alarm image recognition sub-result set and the object behavior abnormality recognition sub-result set into an object behavior recognition result.

[0026] The construction monitoring alarm image recognition model can be trained through the following steps:

[0027] The first step is to obtain a construction monitoring image information set. The construction monitoring image information in the above-mentioned construction monitoring image information set includes: annotation labels, intervention labels and alarm monitoring image feature information sets. The construction monitoring image information also includes construction alarm images. The annotation label may refer to a label that annotates abnormal areas in the construction alarm image. The intervention label may refer to a label that performs intervention processing on the abnormal area. For example, when the annotation label represents protruding steel bars (for example, steel bars inserted horizontally on the road), the intervention label may be a label for removing the steel bars. The alarm monitoring image feature information set may include: feature information such as alarm object color and alarm object material.

[0028] The second step is to train the annotation label classification network and the intervention label classification network according to the above-mentioned construction monitoring image information set. Among them, the above-mentioned annotation label classification network and the above-mentioned intervention label classification network both correspond to alarm monitoring image feature sequences. For example, first, the above-mentioned execution subject can use the annotation labels included in each construction monitoring image information in the construction monitoring image information set as sample labels, and use the intervention labels and alarm monitoring image feature information set included in each construction monitoring image information in the construction monitoring image information set as input features, and train the annotation label classification model according to the above-mentioned construction monitoring image information set. For example, the annotation label classification model can be LightGBM. There is no limitation on the model type of the annotation label classification model. Then, the various alarm monitoring image features used for judgment by the annotation label classification model in the classification process can be combined in sequence to obtain the alarm monitoring image feature sequence corresponding to the above-mentioned annotation label classification model.

[0029] For another example, the intervention labels included in each construction monitoring image information in the construction monitoring image information set can be used as sample labels, and the annotation labels and alarm monitoring image feature information set included in each construction monitoring image information in the construction monitoring image information set can be used as input features, and the intervention label classification model can be obtained by training according to the above construction monitoring image information set. For example, the intervention label classification model can be LightGBM. There is no limitation on the model type of the intervention label classification model. Then, the various alarm monitoring image features used for judgment in the classification process of the intervention label classification model can be combined in sequence to obtain the alarm monitoring image feature sequence corresponding to the above intervention label classification model.

[0030] The third step is to determine the alarm monitoring image intervention causal feature set according to the alarm monitoring image feature sequence corresponding to the above-mentioned annotated label classification network and the alarm monitoring image feature sequence corresponding to the above-mentioned intervention label classification network. The above-mentioned execution subject can determine the intersection of the alarm monitoring image feature sequence corresponding to the above-mentioned annotated label classification network and the alarm monitoring image feature sequence corresponding to the above-mentioned intervention label classification network as the alarm monitoring image intervention causal feature set.

[0031] The fourth step is to train the initial construction monitoring alarm image recognition model according to the alarm monitoring image intervention causal feature set and the construction monitoring image information set to obtain a trained construction monitoring alarm image recognition model.

[0032] The fourth step may include the following sub-steps:

[0033] The first sub-step is to add the intervention feature to the above alarm monitoring image intervention causal feature set to update the alarm monitoring image intervention causal feature set. The intervention feature may be a feature corresponding to the intervention label.

[0034] The second sub-step is to generate a construction monitoring image training sample set based on the training set corresponding to the above-mentioned construction monitoring image information set and the updated alarm monitoring image intervention causal feature set. Each construction monitoring image training sample in the above-mentioned construction monitoring image training sample set includes a label and each alarm monitoring image feature information corresponding to the alarm monitoring image intervention causal feature set, and the alarm monitoring image feature information corresponding to the intervention feature is the intervention label. For each construction monitoring image information in the training set, the above-mentioned execution entity can remove each alarm monitoring image feature information in the above-mentioned construction monitoring image information except for each alarm monitoring image feature information corresponding to the updated alarm monitoring image intervention causal feature set, and obtain the construction monitoring image training sample set.

[0035] The third sub-step is to perform the following training steps based on the construction monitoring image training sample set:

[0036] 1. Input each construction monitoring image training sample in at least one construction monitoring image training sample in the construction monitoring image training sample set into the embedding layer included in the initial construction monitoring alarm image recognition model to obtain each word vector of the alarm monitoring image intervention causal feature set corresponding to each construction monitoring image training sample in at least one construction monitoring image training sample. The embedding layer can be used to generate a word vector (Embedding) of the alarm monitoring image feature information. One wall intervention causal feature can correspond to one word vector.

[0037] 2. Generate a predicted annotation label corresponding to each construction monitoring image training sample in at least one construction monitoring image training sample based on each word vector corresponding to each construction monitoring image training sample in at least one construction monitoring image training sample and the initial construction monitoring alarm image recognition model. The above-mentioned execution subject can input the above-mentioned each word vector into the model structure layer after the embedding layer in the initial construction monitoring alarm image recognition model to obtain the predicted annotation label corresponding to the above-mentioned construction monitoring image training sample. Here, there is no limitation on the model structure layer after the embedding layer in the initial construction monitoring alarm image recognition model.

[0038] 3. Compare the predicted annotation labels and the response annotation labels corresponding to each construction monitoring image training sample in at least one construction monitoring image training sample to obtain a comparison result. The execution subject may determine whether the predicted annotation labels and the annotation labels corresponding to each construction monitoring image training sample are the same, so as to generate the accuracy of each predicted annotation label corresponding to each construction monitoring image training sample as the comparison result.

[0039] 4. In response to determining that the comparison result meets the optimization condition, the initial construction monitoring alarm image recognition model is determined as the trained construction monitoring alarm image recognition model. The above optimization condition may be that the accuracy is greater than a preset value.

[0040] In this way, existing real data can be used without the need for assumptions, reducing model complexity and the risk of overfitting and improving the accuracy of causal inference.

[0041] The object behavior abnormality recognition model can be trained through the following steps:

[0042] The first step is to obtain a set of object behavior prediction image training samples, wherein the object behavior prediction image training samples include: sample object behavior prediction image information and sample object behavior abnormality results.

[0043] The second step is to determine the initial object behavior anomaly recognition model, wherein the initial object behavior anomaly recognition model includes: an initial aggregation model, an initial perception model, an initial enhancement model, and an initial recognition convolution model.

[0044] Here, the initial aggregation model may be a neural network model that takes sample object behavior prediction image information as input and takes initial object behavior prediction image aggregation information as output. For example, the initial aggregation model may be a TCA (Temporal Context Aggregation) model.

[0045] The initial perception model may be a neural network model that takes the initial object behavior prediction image aggregation information as input and takes the initial object behavior prediction image perception information as output. For example, the initial perception model may be an MLP (Multilayer Perceptron) model.

[0046] The initial enhancement model may be a neural network model that takes the initial object behavior prediction image perception information as input and takes the initial object behavior prediction image enhancement information as output. For example, the initial enhancement model may be a PEL (Presupposition Evidence Logic, enhanced prompt) model.

[0047] The initial recognition convolution model may be a neural network model that takes the initial object behavior prediction image enhancement information as input and takes the initial object behavior abnormality result as output. For example, the initial recognition convolution model may be a CC (causal convolution) model.

[0048] The third step is to select a target object behavior prediction image training sample from the object behavior prediction image training sample set. An object behavior prediction image training sample can be randomly selected from the object behavior prediction image training sample set as the target object behavior prediction image training sample.

[0049] In the fourth step, the sample object behavior prediction image information included in the target object behavior prediction image training sample is input into the initial aggregation model to obtain initial object behavior prediction image aggregation information.

[0050] In the fifth step, the initial object behavior prediction image aggregation information is input into the initial perception model to obtain the initial object behavior prediction image perception information.

[0051] In the sixth step, the initial object behavior prediction image perception information is input into the initial enhancement model to obtain the initial object behavior prediction image enhancement information.

[0052] In the seventh step, the initial object behavior prediction image enhancement information is input into the initial recognition convolutional model to obtain the initial object behavior abnormality result.

[0053] In the eighth step, based on a preset recognition loss function, the recognition loss value between the initial object behavior abnormality result and the corresponding sample object behavior abnormality result is determined. For example, the recognition loss function may be: mean square error loss function (MSE), hinge loss function (SVM), cross entropy loss function (CrossEntropy), 0-1 loss function, absolute value loss function, log loss function, square loss function, exponential loss function, etc.

[0054] In a ninth step, in response to determining that the recognition loss value is less than or equal to a preset recognition loss value, the initial object behavior abnormality recognition model is determined as a trained object behavior abnormality recognition model.

[0055] Thus, a relatively accurate object behavior abnormality result can be obtained by training the object behavior abnormality recognition model including the initial aggregation model, the initial perception model, the initial enhancement model, and the initial recognition convolution model. Thus, the abnormal object behavior can be more accurately identified through the relatively accurate object behavior abnormality result. Therefore, alarm processing can be performed on some abnormal object behaviors.

[0056] Step 1012: Generate standard object behavior recognition text based on the above object behavior recognition results.

[0057] In some embodiments, the execution subject may generate a standard object behavior recognition text based on the object behavior recognition result. For example, the execution subject may use a natural language processing toolkit (NLTK) to combine and process the text description data of each recognition sub-result in the object behavior recognition result, thereby generating a standard object behavior recognition text. Among them, NLTK can be used for tasks such as text processing, annotation, grammatical parsing, semantic analysis, and text classification.

[0058] Step 1013, in response to determining that the standard object behavior recognition text representation has a security problem, generating an alarm prompt information, and sending the alarm prompt information to an associated alarm terminal.

[0059] In some embodiments, the execution subject may generate warning information in response to determining that the standard object behavior recognition text representation has a safety problem, and send the warning information to an associated warning terminal. The warning terminal may refer to a terminal used to warn the construction site. For example, the voice alarm terminal may be a speaker. The warning information may include prompt information for abnormal events at the construction site.

[0060] As an example, the execution subject may generate warning information based on the standard object behavior recognition text and the predicted security level in response to the predicted security level indicating a security problem. For example, the multimodal language model may refer to a GPT model.

[0061] Step 102: Perform the following processing steps on the surveillance video included in the multimodal surveillance data:

[0062] Step 1021, performing frame processing on the above surveillance video to generate a surveillance video frame image sequence.

[0063] In some embodiments, the execution subject may perform frame processing on the surveillance video to generate a surveillance video frame image sequence. For example, a corresponding frame extraction interval may be generated according to a preset frame number processing speed. According to the preset frame extraction interval, surveillance video frame images in the surveillance video are extracted frame by frame to obtain a surveillance video frame image sequence.

[0064] Step 1022: Perform anomaly detection on each surveillance video frame image in the surveillance video frame image sequence to generate an image anomaly detection result, thereby obtaining an image anomaly detection result group.

[0065] In some embodiments, the execution entity may perform anomaly detection on each surveillance video frame image in the surveillance video frame image sequence to generate an image anomaly detection result and obtain an image anomaly detection result group.

[0066] In an actual application scenario, the execution subject may perform anomaly detection on each surveillance video frame image in the surveillance video frame image sequence through the following steps:

[0067] In the first step, each target image in each preset construction image to be detected is identified as a target image set. Each construction image to be detected may be a construction image in which a target alarm object has been detected in the past. The target image may refer to an alarm object in a construction image. For example, an image segmentation algorithm may be used to identify each target image in each construction image to be detected as a target image set. For example, an image segmentation algorithm may include: a threshold segmentation method, an edge segmentation method, a region segmentation method, and a morphological segmentation method.

[0068] In the second step, according to the preset alarm image detection requirement information, at least one target image to be detected is selected from the above target image set as the group of images to be detected for alarm. The alarm image detection requirement information may refer to the object information that needs to be detected for alarm, and may include information description of the alarm object to be detected, and may include alarm information descriptions such as alarm objects, alarm persons, and alarm events. That is, at least one target image to be detected that meets the alarm image detection requirement information may be selected from the target image set as the group of images to be detected for alarm.

[0069] The third step is to perform feature extraction processing on each of the above-mentioned detection images to be alarmed in the detection image group to generate features of the detection images to be alarmed, and obtain a feature group of the detection images to be alarmed. The feature extraction model can be used to perform feature extraction processing on each of the above-mentioned detection images to be alarmed in the detection image group to generate features of the detection images to be alarmed, and obtain a feature group of the detection images to be alarmed. The feature extraction model is trained with the residual network (ResNet) as the backbone network, and the model is trained using a large number of objects of different categories, different shapes, different materials, different colors, and human bodies. The model has the ability to extract the features of an object.

[0070] In the fourth step, each target sub-image in the above surveillance video frame image is detected as a construction target sub-image set, wherein each construction target sub-image has a rectangular frame. For example, using the open world object detection technology, each target sub-image in the above surveillance video frame image is detected as a construction target sub-image set. That is, one target sub-image represents one object to be detected.

[0071] The fifth step is to perform feature extraction processing on each construction target sub-image in the above-mentioned construction target sub-image set to generate construction target sub-image features and obtain a construction target sub-image feature group. The feature extraction model can be used to perform feature extraction processing on each construction target sub-image in the above-mentioned construction target sub-image set to generate construction target sub-image features and obtain a construction target sub-image feature group.

[0072] Step 6: For each construction target sub-image feature in the construction target sub-image feature group, determine the similarity between the construction target sub-image feature and each image feature to be detected in the image feature group to be detected, and obtain a similarity group. The similarity between the construction target sub-image feature and each image feature to be detected in the image feature group to be detected can be determined by the cosine similarity formula to obtain a similarity group.

[0073] In the seventh step, the construction target sub-image features in the construction target sub-image feature group are sorted according to the obtained similarity groups to obtain a construction target sub-image feature sequence. That is, first, the maximum similarity corresponding to each construction target sub-image feature is determined, and then the construction target sub-image features can be sorted in descending order according to the maximum similarity corresponding to each construction target sub-image feature to obtain a construction target sub-image feature sequence.

[0074] Step 8: For each construction target sub-image feature in the construction target sub-image feature sequence, perform the following processing steps:

[0075] 1. Determine the similarity group corresponding to the above construction target sub-image features as the target similarity group.

[0076] 2. According to the target similarity group, the image features to be detected to be alarmed in the image feature group to be detected to be alarmed are sorted to obtain a sequence of image features to be detected to be alarmed. That is, the image features to be detected to be alarmed in the image feature group to be detected to be alarmed can be sorted from large to small according to the target similarities included in the target similarity group to obtain a sequence of image features to be detected to be alarmed.

[0077] 3. Select the first target number of image features to be detected to be alarmed from the above-mentioned image feature sequences to be detected to be alarmed as candidate image feature sequences.

[0078] 4. Determine the feature similarity between the target image feature to be detected to be an alarm and the remaining image features to be detected to be an alarm, and obtain a feature similarity group, wherein the sequence number of the target image feature to be detected to be an alarm in the sequence of image features to be detected to be an alarm is the same as the sequence number of the construction site target sub-image feature in the sequence of target sub-image features in the construction site.

[0079] 5. According to the above feature similarity groups, select the first target number of image features to be alarmed from the remaining image features to be alarmed as the candidate image feature group to be alarmed.

[0080] 6. Based on the candidate image feature sequence and the candidate image feature group to be detected to be an alarm, determine the similarity between the construction target sub-image feature and the target image feature to be detected to be an alarm as the target feature similarity.

[0081] For example, the similarity between the construction target sub-image feature and the target to-be-alarmed detection image feature can be determined by the following formula:

[0082]

[0083] Among them, s(k i , fi ) represents the construction target sub-image feature k i and the target image feature f i The similarity between them. y represents the number of targets. f j ″ represents the candidate image feature to be detected for alarm in the candidate image feature group to be detected for alarm. j ′ represents the candidate image feature in the candidate image feature sequence.

[0084] In the ninth step, the construction target sub-image feature sequence is reordered according to the similarity of each target feature to obtain a sorted construction target sub-image feature sequence as the target construction sub-image feature sequence. That is, the construction target sub-image feature sequence can be sorted in descending order from large to small according to the similarity of each target feature to obtain a sorted construction target sub-image feature sequence as the target construction sub-image feature sequence.

[0085] In the tenth step, each target construction sub-image feature in the target construction sub-image feature sequence that meets the target detection condition is combined into an image anomaly detection result. The target detection condition may be: the target feature similarity corresponding to the target construction sub-image feature is greater than or equal to a preset similarity.

[0086] Thus, the target to be detected is automatically selected: just upload the image containing the target to be detected, the algorithm can segment the target, just click to select the target to be detected, and then automatically fill in the rectangular box and extract features. This improves the detection efficiency of the alarm object.

[0087] Step 1023, in response to determining that there is an image anomaly detection result that meets the abnormal condition in the above-mentioned image anomaly detection result group, generate abnormal prompt information, and control the associated voice alarm terminal to issue an alarm.

[0088] In some embodiments, the execution subject may generate abnormal prompt information and control the associated voice alarm terminal to issue an alarm in response to determining that there is an image abnormality detection result that satisfies the abnormality condition in the image abnormality detection result group. The voice alarm terminal may refer to a terminal used to issue an alarm on the construction site. For example, the voice alarm terminal may be a speaker. The abnormal condition may be: the image abnormality detection result indicates that there is an abnormal target object / abnormal event on the construction site.

[0089] Furthermore, a multimodal language model is used to mark the above-mentioned standard object behavior recognition text to generate marked object behavior recognition text.

[0090] In some embodiments, the execution subject may use a multimodal language model to mark the standard object behavior recognition text to generate a marked object behavior recognition text. The marked object behavior recognition text is used to highlight the object behavior and potential safety events. For example, the data representing the object behavior and potential safety events in the standard object behavior recognition text may be marked.

[0091] Furthermore, feature extraction is performed on the above-mentioned marked object behavior identification text to generate an object behavior identification text feature vector, and security level prediction is performed on the above-mentioned object behavior identification text feature vector.

[0092] In some embodiments, the execution subject may perform feature extraction on the marked object behavior recognition text to generate an object behavior recognition text feature vector, and perform security level prediction on the object behavior recognition text feature vector. A text feature extraction model may be used to perform feature extraction on the marked object behavior recognition text to generate an object behavior recognition text feature vector. For example, the text feature extraction model may be a Bert model.

[0093] In an actual application scenario, the execution subject can predict the security level of the object behavior recognition text feature vector through the following steps:

[0094] First, determine the vector similarity between the object behavior recognition text feature vector and each object behavior vector in a pre-built object behavior vector database, wherein the object behavior vector and the corresponding security level identifier are stored in the object behavior vector database.

[0095] Second, the object behavior vector with the highest vector similarity is determined as the target object behavior vector.

[0096] Third, the security level corresponding to the target object behavior vector is determined as the predicted security level of the object behavior recognition text feature vector.

[0097] Figure 2 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. The computer device may be a terminal.

[0098] like Figure 2 As shown, the computer device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0099] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any alarm method based on the engineering construction safety monitoring video.

[0100] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0101] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any alarm method based on the engineering construction safety monitoring video.

[0102] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 2 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0103] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0104] In one embodiment, the processor is used to run a computer program stored in a memory to implement the following steps: in response to receiving multimodal monitoring data sent by a monitoring device at a target engineering construction site, the following processing steps are performed on a monitoring image sequence included in the multimodal monitoring data: object behavior recognition is performed on the monitoring image sequence to generate an object behavior recognition result; a standard object behavior recognition text is generated based on the object behavior recognition result; in response to determining that the standard object behavior recognition text indicates that there is a safety problem, an alarm prompt message is generated, and the alarm prompt message is sent to an associated alarm terminal; the following processing steps are performed on the monitoring video included in the multimodal monitoring data: frame processing is performed on the monitoring video to generate a monitoring video frame image sequence; anomaly detection is performed on each monitoring video frame image in the monitoring video frame image sequence to generate an image anomaly detection result and obtain an image anomaly detection result group; in response to determining that there is an image anomaly detection result that meets an abnormal condition in the image anomaly detection result group, an abnormality prompt message is generated, and an associated voice alarm terminal is controlled to issue an alarm.

[0105] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the alarm method based on engineering construction safety monitoring video of the present application.

[0106] The computer-readable storage medium may be an internal storage unit of the computer device in the aforementioned embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., provided on the computer device.

[0107] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0108] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An alarm method based on engineering construction safety monitoring video, the method comprising: In response to receiving multimodal monitoring data sent by monitoring equipment at a target engineering construction site, the following processing steps are performed on a monitoring image sequence included in the multimodal monitoring data: Performing object behavior recognition on the surveillance image sequence to generate an object behavior recognition result; Generating standard object behavior recognition text according to the object behavior recognition result; In response to determining that the standard object behavior recognition text representation has a security problem, generating an alarm prompt information, and sending the alarm prompt information to an associated alarm terminal; For the surveillance videos included in the multimodal surveillance data, the following processing steps are performed: Performing frame processing on the surveillance video to generate a surveillance video frame image sequence; Performing anomaly detection on each surveillance video frame image in the surveillance video frame image sequence to generate an image anomaly detection result, thereby obtaining an image anomaly detection result group; In response to determining that there is an image abnormality detection result that meets the abnormality condition in the image abnormality detection result group, generating abnormality prompt information, and controlling the associated voice alarm terminal to issue an alarm; Using a multimodal language model, the standard object behavior recognition text is marked to generate a marked object behavior recognition text, wherein the marked object behavior recognition text is used to highlight the object behavior and safety hazard events; Extracting features from the marked object behavior recognition text to generate an object behavior recognition text feature vector, and predicting a security level for the object behavior recognition text feature vector; The step of generating warning information in response to determining that the standard object behavior recognition text representation has a security problem includes: In response to the predicted security level indicating that there is a security problem, generating warning information through the multimodal language model according to the standard object behavior recognition text and the predicted security level; The step of predicting the security level of the object behavior recognition text feature vector includes: Determining a vector similarity between the object behavior recognition text feature vector and each object behavior vector in a pre-constructed object behavior vector database, wherein the object behavior vector and a corresponding security level identifier are stored in the object behavior vector database; The object behavior vector with the highest vector similarity is determined as the target object behavior vector; Determining the security level corresponding to the target object behavior vector as the predicted security level of the object behavior recognition text feature vector; The performing object behavior recognition on the surveillance image sequence to generate an object behavior recognition result includes: Inputting each monitoring image in the monitoring image sequence into a pre-trained construction monitoring alarm image recognition model to generate a construction monitoring alarm image recognition sub-result, thereby obtaining a construction monitoring alarm image recognition sub-result set; Inputting each surveillance image in the surveillance image sequence into a pre-trained object behavior abnormality recognition model to generate an object behavior abnormality recognition sub-result, thereby obtaining an object behavior abnormality recognition sub-result set; Merging the construction monitoring alarm image recognition sub-result set and the object behavior abnormality recognition sub-result set into an object behavior recognition result; Among them, the object behavior abnormality recognition model is trained through the following steps: Obtaining an object behavior prediction image training sample set, wherein the object behavior prediction image training sample includes: sample object behavior prediction image information and sample object behavior abnormality results; Determining an initial object behavior anomaly recognition model, wherein the initial object behavior anomaly recognition model includes: an initial aggregation model, an initial perception model, an initial enhancement model, and an initial recognition convolution model; Selecting a target object behavior prediction image training sample from an object behavior prediction image training sample set; Inputting the sample object behavior prediction image information included in the target object behavior prediction image training sample into the initial aggregation model to obtain initial object behavior prediction image aggregation information; Inputting the aggregated information of the initial object behavior prediction image into the initial perception model to obtain the initial object behavior prediction image perception information; Inputting the initial object behavior prediction image perception information into the initial enhancement model to obtain the initial object behavior prediction image enhancement information; Inputting the initial object behavior prediction image enhancement information into the initial recognition convolution model to obtain the initial object behavior abnormality result; Based on a preset recognition loss function, determining a recognition loss value between an initial object behavior abnormality result and a corresponding sample object behavior abnormality result; In response to determining that the recognition loss value is less than or equal to a preset recognition loss value, determining the initial object behavior abnormality recognition model as a trained object behavior abnormality recognition model; The step of performing anomaly detection on each surveillance video frame image in the surveillance video frame image sequence to generate an image anomaly detection result includes: Identify each target image in each preset construction image to be detected as a target image set; According to preset alarm image detection requirement information, selecting at least one target image to be detected from the target image set as a group of images to be detected for alarm; Performing feature extraction processing on each detection image to be alarmed in the detection image to be alarmed group to generate detection image to be alarmed features, thereby obtaining a detection image to be alarmed feature group; Detecting each target sub-image in the monitoring video frame image as a construction target sub-image set; Performing feature extraction processing on each construction target sub-image in the construction target sub-image set to generate construction target sub-image features and obtain a construction target sub-image feature group; For each construction target sub-image feature in the construction target sub-image feature group, determining the similarity between the construction target sub-image feature and each to-be-alarmed detection image feature in the to-be-alarmed detection image feature group, to obtain a similarity group; According to each obtained similarity group, each construction target sub-image feature in the construction target sub-image feature group is sorted to obtain a construction target sub-image feature sequence; For each construction target sub-image feature in the construction target sub-image feature sequence, the following processing steps are performed: Determine the similarity group corresponding to the construction target sub-image feature as the target similarity group; According to the target similarity group, the individual detection image features to be alarmed in the detection image feature group to be alarmed are sorted to obtain a detection image feature sequence to be alarmed; Selecting the first target number of image features to be detected to be alarmed from the image feature sequence to be detected to be alarmed as a candidate image feature sequence; Determine the feature similarity between the target image feature to be detected by alarm and the remaining features of each image to be detected by alarm, and obtain a feature similarity group; According to the feature similarity group, selecting the first target number of the image features to be alarmed from the remaining image features to be alarmed as the candidate image features to be alarmed group; Determine, based on the candidate image feature sequence and the candidate to-be-alarmed detected image feature group, the similarity between the construction target sub-image feature and the target to-be-alarmed detected image feature as the target feature similarity; Reordering the construction target sub-image feature sequence according to the similarity of each target feature to obtain a sorted construction target sub-image feature sequence as the target construction sub-image feature sequence; The target construction sub-image features satisfying the target detection condition in the target construction sub-image feature sequence are combined into an image anomaly detection result.

2. The alarm method based on engineering construction safety monitoring video according to claim 1 is characterized in that: Before inputting each monitoring image in the monitoring image sequence into a pre-trained construction monitoring alarm image recognition model to generate a construction monitoring alarm image recognition sub-result, the method further includes: Acquire a construction monitoring image information set, wherein the construction monitoring image information in the construction monitoring image information set includes: a label, an intervention label, and an alarm monitoring image feature information set; According to the construction monitoring image information set, training a label classification network and an intervention label classification network, wherein both the label classification network and the intervention label classification network correspond to an alarm monitoring image feature sequence; Determining an alarm monitoring image intervention causal feature set according to the alarm monitoring image feature sequence corresponding to the annotated label classification network and the alarm monitoring image feature sequence corresponding to the intervention label classification network; The initial construction monitoring alarm image recognition model is trained according to the alarm monitoring image intervention causal feature set and the construction monitoring image information set to obtain a trained construction monitoring alarm image recognition model.

3. A computer device, wherein: The computer device comprises a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 2 are implemented.

4. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 2 are implemented.

Images