Iot-based intelligent construction site monitoring method and system

Abstract

The embodiment of the application provides a kind of wisdom construction site monitoring method and system based on Internet of Things, under the condition of the correlation of first, second construction site behavior abnormal characteristics, using the abnormal correlation information generated by correlation characteristic analysis model, the correlation characteristic vector of first construction site behavior abnormal characteristics and target abnormal knowledge characteristics can be accurately determined, whereby the association information of wisdom construction site monitoring video stream and target construction site abnormal event label, whereby the first template monitoring video stream of first construction site category does not need to limit training label data, it can realize the construction site abnormal event label determination of wisdom construction site monitoring video stream of first construction site category, can improve the derivative efficiency of abnormal label analysis of wisdom construction site monitoring video stream between different construction site categories of monitoring video stream.

Description

Technical Field

[0001] This application relates to the field of Internet of Things (IoT) technology, and more specifically, to a smart construction site monitoring method and system based on IoT. Background Technology

[0002] Smart construction sites refer to the establishment of an interconnected, collaborative, intelligent, and scientifically managed information ecosystem for construction projects using information technology. This involves data mining and analysis of monitoring data collected through the Internet of Things (IoT) to provide predictions of abnormal behavior trends and expert contingency plans, achieving visualized and intelligent management of construction projects. This improves the level of information technology in project management and gradually realizes green and ecological construction. Therefore, timely control of abnormal events at construction sites is necessary to ensure safety, construction quality, and project progress. However, in related technologies, the efficiency of abnormal event tag analysis and the derivation of abnormal tags from different construction site categories in smart construction site monitoring video streams is relatively slow. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a smart construction site monitoring method and system based on the Internet of Things.

[0004] According to a first aspect of this application, an Internet of Things (IoT)-based smart construction site monitoring method is provided, applied to an IoT-based smart construction site monitoring system, the method comprising:

[0005] The first construction site behavior anomaly features are obtained by using the first construction site anomaly analysis model to obtain the first construction site monitoring video stream. The smart construction site monitoring video stream is a monitoring video stream of the first construction site category. The first construction site anomaly analysis model is obtained by iteratively updating the model based on the first template monitoring video stream of the first construction site category. The smart construction site monitoring video stream is generated by monitoring through IoT monitoring devices deployed in the construction site.

[0006] Obtain target anomaly knowledge features for the target construction site anomaly event tag. The target anomaly knowledge features are used to express the target construction site anomaly event tag, and the target anomaly knowledge features include behavioral anomaly features of the monitoring video stream of the second construction site category under the target construction site anomaly event tag.

[0007] Using the abnormal correlation information between the target abnormal knowledge features and the second construction site behavior abnormal features generated by the correlation feature analysis model, the correlation feature vector between the first construction site behavior abnormal features and the target abnormal knowledge features is determined, and the correlation feature vector distribution is generated. The second construction site behavior abnormal features are obtained by the second construction site abnormal analysis model from the second template monitoring video stream. The second template monitoring video stream is the monitoring video stream of the second construction site category. The model convergence control process of the second construction site abnormal analysis model is the same as that of the first construction site abnormal analysis model.

[0008] Based on the associated feature vector distribution, the construction site abnormal event labels of the smart construction site monitoring video stream are determined.

[0009] In one possible implementation of the first aspect, before obtaining the first construction site behavior anomaly features of the smart construction site monitoring video stream using the first construction site anomaly analysis model, the method further includes:

[0010] Using a second construction site anomaly analysis model that satisfies the model convergence condition, the second construction site behavior anomaly features are extracted from the second template monitoring video stream. The second template monitoring video stream has template annotation data, which includes the prior construction site anomaly event labels of the second template monitoring video stream. The prior construction site anomaly event labels include the target construction site anomaly event labels.

[0011] Obtain the abnormal knowledge features corresponding to the prior construction site abnormal event label. The abnormal knowledge features are used to express the prior construction site abnormal event label, and the abnormal knowledge features include the behavioral abnormal features of the monitoring video stream of the second construction site category under the prior construction site abnormal event label.

[0012] Based on the second construction site behavior anomaly features and the anomaly knowledge features, a template association feature vector is obtained. The template association feature vector includes the second construction site behavior anomaly features and the anomaly knowledge features. The training annotation data of the template association feature vector includes: annotation data characterizing whether the prior construction site anomaly event label is associated with the prior construction site anomaly event label.

[0013] By using the correlation feature analysis of the initial model weight parameters, the abnormal knowledge features generated by the model and the abnormal behavior features of the second construction site are analyzed to determine the correlation feature vector of the second construction site behavior abnormal features and abnormal knowledge features in the template correlation feature vector;

[0014] Based on the distribution of associated feature vectors and the training labeled data, the model weight information of the associated feature analysis model is iteratively updated until the associated feature analysis model meets the model convergence requirements. The model weight information includes the weight information corresponding to the abnormal association information.

[0015] In one possible implementation of the first aspect, before extracting the second construction site behavior anomaly features from the second template monitoring video stream using the second construction site anomaly analysis model that satisfies the model convergence condition, the method further includes:

[0016] The first template monitoring video stream is obtained, and the first construction site anomaly analysis model with initialized model weight parameters is used to project the first template monitoring video stream onto the target feature matching bitmap to generate the first estimated construction site behavior anomaly features.

[0017] Based on the first estimated abnormal characteristics of construction site behavior, the first process output data corresponding to the model convergence control process of the first construction site anomaly analysis model is generated. Based on the first process output data, the model weight information of the first construction site anomaly analysis model is iteratively updated to generate a first construction site anomaly analysis model that meets the model convergence conditions.

[0018] The second template monitoring video stream is obtained, and the second construction site anomaly analysis model with initialized model weight parameters is used to project the second template monitoring video stream onto the target feature matching bitmap to generate the second estimated construction site behavior anomaly features.

[0019] Based on the second estimated abnormal characteristics of construction site behavior, the second process output data corresponding to the model convergence control process of the second construction site anomaly analysis model is generated. Based on the second process output data, the model weight information of the second construction site anomaly analysis model is iteratively updated to generate a second construction site anomaly analysis model that meets the model convergence conditions.

[0020] In one possible implementation of the first aspect, obtaining the anomaly knowledge features corresponding to the prior site anomaly event tags includes:

[0021] Transform prior site anomaly event labels into initial feature representation vectors in a rule-based manner;

[0022] Feature enhancement is performed on the second template monitoring video stream to generate an enhanced monitoring video stream. The third construction site behavior anomaly features of the enhanced monitoring video stream are extracted using the second construction site anomaly analysis model.

[0023] Obtain estimated learning features, which include the third construction site behavior anomaly features and the initial feature representation vector. The estimated learning label data of the estimated learning features includes the enhanced feature segments in the second template monitoring video stream.

[0024] Using an anomaly knowledge extraction model, the abnormal behavior features of the third construction site and the initial feature representation vector are projected, and the enhanced feature segments in the enhanced monitoring video stream are determined based on the projection results, and estimated enhanced feature segments are generated.

[0025] Based on the estimated enhanced feature fragments and the estimated learned label data, the model weight information of the anomaly knowledge extraction model is iteratively updated to generate the iteratively updated anomaly knowledge extraction model.

[0026] The initial feature representation vector of the prior construction site abnormal event label is projected onto the iteratively updated abnormal knowledge extraction model to generate the abnormal knowledge features of the prior construction site abnormal event label.

[0027] In one possible implementation of the first aspect, the anomaly knowledge extraction model includes a projection function and an estimation function, the projection function having projection parameters; the acquisition of estimated learning features includes:

[0028] The abnormal behavior features of the third construction site and the initial feature representation vector are fused to generate a fused representation vector, which is then used as the estimated learning feature.

[0029] The method of using anomaly knowledge extraction model to project the abnormal behavior features of the third construction site and the initial feature representation vector, and determining the enhanced feature segments in the enhanced monitoring video stream based on the projection results, and generating estimated enhanced feature segments, includes:

[0030] The fused representation vector is projected using the projection parameters of the projection function to generate a projected representation vector;

[0031] The projection representation vector is analyzed using the estimation function to determine the enhanced feature segments in the enhanced surveillance video stream and generate estimated enhanced feature segments.

[0032] The step of projecting the initial feature representation vector of the prior construction site anomaly event label onto the iteratively updated anomaly knowledge extraction model to generate the anomaly knowledge features of the prior construction site anomaly event label includes:

[0033] Based on the nodes of the initial feature representation vector in the fused representation vector, the projection target parameters of the initial feature representation vector are obtained from the projection parameters;

[0034] The initial feature representation vector is projected based on the projection target parameters to generate anomalous knowledge features of prior construction site anomalous event labels.

[0035] In one possible implementation of the first aspect, the number of target site abnormal event tags is multiple;

[0036] The abnormal correlation information between the target abnormal knowledge features and the second construction site behavior abnormal features generated by the correlation feature analysis model is used to determine the correlation feature vector between the first construction site behavior abnormal features and the target abnormal knowledge features, and to generate a correlation feature vector distribution, including:

[0037] Using the correlation feature analysis model, the abnormal correlation information between the target abnormal knowledge features and the second construction site behavior abnormal features corresponding to each target construction site abnormal event label is generated. The correlation feature vector between the first construction site behavior abnormal feature and each target abnormal knowledge feature is determined. The correlation confidence of the smart construction site monitoring video stream under each target construction site abnormal event label is generated. The correlation confidence represents the construction site abnormal event label of the smart construction site monitoring video stream and is the probability value of the target construction site abnormal event label.

[0038] The step of determining the construction site anomaly event labels of the smart construction site monitoring video stream based on the associated feature vector distribution includes:

[0039] The target construction site anomaly event label with the highest association confidence is determined as the construction site anomaly event label of the smart construction site monitoring video stream.

[0040] In one possible implementation of the first aspect, the association feature analysis model includes at least one prediction network, with each prediction network corresponding to a site anomaly event label.

[0041] The abnormal correlation information between the target abnormal knowledge features and the second construction site behavioral abnormal features corresponding to each target construction site abnormal event label generated by the correlation feature analysis model is used to determine the correlation feature vector between the first construction site behavioral abnormal features and each target abnormal knowledge feature, and to generate the correlation confidence of the smart construction site monitoring video stream under each target construction site abnormal event label, including:

[0042] The first construction site behavior anomaly features and the target anomaly knowledge features are loaded into the prediction network corresponding to the target construction site anomaly event label;

[0043] By utilizing the abnormal correlation information between the target anomaly knowledge features and the second construction site behavior anomaly features provided by each prediction network, the correlation feature vector between the target anomaly knowledge features and the first construction site behavior anomaly features is determined, and the correlation confidence of the smart construction site monitoring video stream under each target construction site anomaly event label is generated.

[0044] According to a second aspect of this application, a server is provided, the server including a machine-readable storage medium and a processor, the machine-readable storage medium storing machine-executable instructions, and the server implementing the aforementioned IoT-based smart construction site monitoring method when the processor executes the machine-executable instructions.

[0045] According to a third aspect of this application, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when the computer-executable instructions are executed, the aforementioned smart construction site monitoring method based on the Internet of Things is implemented.

[0046] Based on the above aspects, this application utilizes a first construction site anomaly analysis model to obtain first construction site behavioral anomaly features from a smart construction site monitoring video stream. The smart construction site monitoring video stream is a monitoring video stream of the first construction site category. The first construction site anomaly analysis model is obtained through iterative updates based on the first template monitoring video stream of the first construction site category. It also obtains target anomaly knowledge features for target construction site anomaly event tags. These target anomaly knowledge features are used to express the target construction site anomaly event tags, and include behavioral anomaly features of the monitoring video stream of the second construction site category under the target construction site anomaly event tags. Using the anomaly association information between the target anomaly knowledge features and the second construction site behavioral anomaly features generated by the association feature analysis model, it determines the association feature vector between the first construction site behavioral anomaly features and the target anomaly knowledge features, generating an association feature vector distribution. The second construction site behavioral anomaly features are obtained by the second construction site anomaly analysis model extracting the second template monitoring video stream. The second template monitoring video stream is a monitoring video stream of the second construction site category. The model convergence control process of the second construction site anomaly analysis model is the same as that of the first construction site anomaly analysis model. Based on the association feature vector distribution, it determines the working parameters of the smart construction site monitoring video stream. The anomaly event labeling method, where the convergence control process of the second construction site anomaly analysis model is the same as that of the first construction site anomaly analysis model, allows for a linear mapping relationship between the anomaly features of construction site behavior under the first and second construction site categories. The target anomaly knowledge features include the anomaly features of the monitoring video stream of the second construction site category under the target construction site anomaly event label. This enables the anomaly association information generated by the correlation feature analysis model to accurately express the relationship between the target anomaly knowledge features of the target construction site anomaly event label and the anomaly features of the second construction site behavior. Therefore, under the condition of correlation between the first and second construction site behavior anomaly features, the anomaly association information generated by the correlation feature analysis model can accurately determine the correlation feature vector between the first construction site behavior anomaly features and the target anomaly knowledge features. Thus, the correlation information between the smart construction site monitoring video stream and the target construction site anomaly event label can be obtained without restricting the first template monitoring video stream of the first construction site category to have training label data. This improves the derivation efficiency of anomaly label analysis of smart construction site monitoring video streams between monitoring video streams of different construction site categories. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A flowchart illustrating the IoT-based smart construction site monitoring method provided in this application embodiment;

[0049] Figure 2 This paper shows a schematic diagram of the component structure of a server for implementing the above-described IoT-based smart construction site monitoring method, provided in an embodiment of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn according to actual occupancy. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed in order or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add at least one other operation to the flowchart, or remove at least one operation from the flowchart.

[0051] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments that can be obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0052] Figure 1 The diagram illustrates a flowchart of an IoT-based smart construction site monitoring method according to an embodiment of this application. It should be understood that in other embodiments, the order of some steps in the IoT-based smart construction site monitoring method of this embodiment can be interchanged according to actual needs, or some steps can be omitted or deleted. The detailed steps of this IoT-based smart construction site monitoring method are described below.

[0053] Step S101: Use the first construction site anomaly analysis model to obtain the first construction site behavior anomaly features of the smart construction site monitoring video stream. The smart construction site monitoring video stream is a monitoring video stream of the first construction site category. The first construction site anomaly analysis model is obtained by iteratively updating the model based on the first template monitoring video stream of the first construction site category.

[0054] In an alternative implementation, the first site category is different from the second site category described below, but there are no specific limitations.

[0055] The first construction site anomaly analysis model and the second construction site anomaly analysis model described below in this embodiment can be obtained by training any existing neural network model. They can mine abnormal features of construction site behavior from the monitoring video stream. The model convergence control process of the first construction site anomaly analysis model and the second construction site anomaly analysis model is the same, but the model convergence control process is not limited.

[0056] In an alternative implementation, the first construction site anomaly analysis model and the second construction site anomaly analysis model can be the same model, thus their model convergence control processes are naturally identical. Alternatively, the first and second construction site anomaly analysis models can be two independent but identical models. The identical convergence control processes of the first and second models allow for a linear mapping relationship between the construction site behavior anomaly features parsed by the first model from the monitoring video stream of the first construction site category (i.e., the first construction site behavior anomaly features) and the construction site behavior anomaly features parsed by the second model from the monitoring video stream of the second construction site category (i.e., the second construction site behavior anomaly features). Therefore, in subsequent embodiments, the anomaly correlation information between the target anomaly knowledge features and the second construction site behavior anomaly features learned by the correlation feature analysis model can be applied to the correlation feature vector analysis of the first construction site behavior anomaly features and the target anomaly knowledge features, achieving training and labeling of construction site anomaly event tags across construction site categories.

[0057] In an alternative implementation, the step "obtaining the first construction site behavior anomaly features of the smart construction site monitoring video stream using the first construction site anomaly analysis model" may include:

[0058] By utilizing the vector projection function of the first construction site anomaly analysis model, the first construction site behavior anomaly features of the smart construction site monitoring video stream are obtained.

[0059] The abnormal behavior features of construction sites extracted from the monitoring video streams of the second construction site category using the second construction site anomaly analysis model can be called second construction site behavior anomaly features.

[0060] In an alternative implementation, the step "using the vector projection function of the first construction site anomaly analysis model to obtain the first construction site behavior anomaly features of the smart construction site monitoring video stream" may include:

[0061] Using the vector projection function of the first construction site anomaly analysis model, the semantic features of video frames in the smart construction site monitoring video stream are obtained;

[0062] Based on the semantic features of all video frames in the smart construction site monitoring video stream, the first construction site behavior anomaly feature of the smart construction site monitoring video stream is generated.

[0063] Among them, the semantic features of video frames can be fused according to the order of sentences in the smart construction site monitoring video stream to generate the first construction site behavior anomaly features.

[0064] Step S102: Obtain the target anomaly knowledge features of the target construction site anomaly event tag. The target anomaly knowledge features are used to express the target construction site anomaly event tag, and the target anomaly knowledge features include the behavioral anomaly features of the monitoring video stream of the second construction site category under the target construction site anomaly event tag.

[0065] In an alternative implementation, the target anomaly knowledge features can be extracted based on Label Embedding. For example, a surveillance video stream of a second construction site category with training label data (i.e., template annotation data containing prior construction site anomaly event labels) can be acquired, and anomaly knowledge features of prior construction site anomaly event labels can be generated using label embedding technology.

[0066] In an alternative implementation, the anomaly knowledge features of a construction site anomaly event tag include shared features learned from the construction site behavior anomaly features of the monitoring video stream of the construction site anomaly event tag (features obtained using the aforementioned construction site anomaly analysis model). That is, the behavior anomaly features of the monitoring video stream of the second construction site category under the target construction site anomaly event tag, including the target anomaly knowledge features, can be understood as the shared behavior anomaly features of the monitoring video stream of the second construction site category under the target construction site anomaly event tag.

[0067] In an alternative implementation, the scheme for obtaining the anomaly knowledge features corresponding to prior site anomaly event tags may include:

[0068] Transform prior site anomaly event labels into initial feature representation vectors in a rule-based manner;

[0069] Feature enhancement is performed on the monitoring video stream of the second template monitoring video stream to generate an enhanced monitoring video stream. Using the second construction site anomaly analysis model, the third construction site behavior anomaly features of the enhanced monitoring video stream are extracted.

[0070] The estimated learning features are obtained, which include the abnormal behavior features of the third construction site and the initial feature representation vector. The estimated learning label data of the estimated learning features includes the enhanced feature segments in the second template monitoring video stream.

[0071] Using an anomaly knowledge extraction model, the abnormal behavior features of the third construction site and the initial feature representation vector are projected, and the enhanced feature segments in the enhanced monitoring video stream are predicted and enhanced based on the projection results, generating estimated enhanced feature segments.

[0072] Based on the estimated enhanced feature fragments and the estimated learned label data, the anomaly knowledge extraction model is updated until the anomaly knowledge extraction model meets the model convergence condition.

[0073] An anomaly knowledge extraction model is used to project the initial feature representation vector of prior construction site anomaly event labels to generate anomaly knowledge features of prior construction site anomaly event labels.

[0074] In an alternative implementation, the step "adjusting the anomaly knowledge extraction model based on the estimated enhanced feature fragments and the estimated learned label data until the anomaly knowledge extraction model meets the model convergence condition" may include:

[0075] Based on the estimated enhanced feature fragments and the estimated learned label data, the abnormal knowledge extraction model is adjusted;

[0076] Return to the execution step "Enhance the features of the monitoring video stream of the second template monitoring video stream to generate an enhanced monitoring video stream, and use the second construction site anomaly analysis model to extract the third construction site behavior anomaly features of the enhanced monitoring video stream" until the anomaly knowledge extraction model meets the model convergence condition.

[0077] In an alternative implementation, the anomaly knowledge extraction model may include a projection function and an estimation function, the projection function having projection parameters. These projection parameters can be characterized by a projection matrix.

[0078] In an alternative implementation, the specific process of obtaining the estimated learning features may include: fusing the abnormal behavior features of the third construction site and the initial feature representation vector to generate a fused representation vector, and using the fused representation vector as the estimated learning features.

[0079] In an alternative implementation, the step "using the anomaly knowledge extraction model to project the abnormal behavior features of the third construction site and the initial feature representation vector, predicting the enhanced feature segments in the enhanced monitoring video stream based on the projection results, and generating estimated enhanced feature segments" may include: projecting the fused representation vector using the projection parameters of the projection function to generate a projected representation vector; analyzing the projected representation vector using an estimation function to predict the enhanced feature segments in the enhanced monitoring video stream and generating estimated enhanced feature segments.

[0080] In an alternative implementation, the step "projecting the initial feature representation vector of the prior construction site anomaly event label using the anomaly knowledge extraction model to generate anomaly knowledge features of the prior construction site anomaly event label" may include:

[0081] Based on the nodes of the initial feature representation vector in the fused representation vector, the projection target parameters of the initial feature representation vector are obtained from the projection parameters.

[0082] Based on the projection target parameters, the initial feature representation vector is projected to generate anomalous knowledge features of prior site anomaly event labels.

[0083] Step S103: Using the abnormal correlation information between the target abnormal knowledge features and the second construction site behavior abnormal features generated by the correlation feature analysis model, determine the correlation feature vector between the first construction site behavior abnormal features and the target abnormal knowledge features, and generate the correlation feature vector distribution. The second construction site behavior abnormal features are obtained by the second construction site abnormal analysis model from the second template monitoring video stream. The second template monitoring video stream is the monitoring video stream of the second construction site category. The model convergence control process of the second construction site abnormal analysis model is the same as that of the first construction site abnormal analysis model.

[0084] The correlation feature analysis model in this embodiment is obtained by iteratively updating the model based on the abnormal behavior features of the second construction site in the second template monitoring video stream and the abnormal knowledge features of the prior construction site abnormal event tags in the second template monitoring video stream.

[0085] The abnormal knowledge features of the prior construction site abnormal event tags in the second template monitoring video stream can be determined using the steps for determining abnormal knowledge features in the above embodiments.

[0086] Anomaly association information can accurately express the anomaly association information of the second construction site behavior anomaly features of the second template monitoring video stream of the target anomaly knowledge features and the target construction site anomaly event tags. Since there is a linear mapping relationship between the first construction site behavior anomaly features and the second construction site behavior anomaly features, the anomaly association information can be applied to analyze the association feature vector between the first construction site behavior anomaly features and the target anomaly knowledge features.

[0087] In an alternative implementation, the model update process for the association feature analysis model is described below. The training of the association feature analysis model is performed before step S101. The training process of the association feature analysis model includes:

[0088] Step S201: Using the second construction site anomaly analysis model that satisfies the model convergence condition, extract the second construction site behavior anomaly features from the second template monitoring video stream. The second template monitoring video stream has template annotation data, which includes the prior construction site anomaly event labels of the second template monitoring video stream. The prior construction site anomaly event labels include the target construction site anomaly event labels.

[0089] The anomaly analysis model for the second construction site was obtained by iteratively updating the model based on the monitoring video stream samples of the second construction site category.

[0090] Step S202: Obtain the abnormal knowledge features corresponding to the prior construction site abnormal event label. The abnormal knowledge features are used to express the prior construction site abnormal event label, and the abnormal knowledge features include the behavioral abnormal features of the monitoring video stream of the second construction site category under the prior construction site abnormal event label.

[0091] The system can store projection parameters corresponding to prior site anomaly event labels. In step S202, these projection parameters can be used to project the prior site anomaly event labels into anomaly knowledge features.

[0092] Step S203: Based on the abnormal behavior features and abnormal knowledge features of the second construction site, obtain the template association feature vector. The template association feature vector includes the abnormal behavior features and abnormal knowledge features of the second construction site. The training annotation data of the template association feature vector includes: annotation data representing whether the prior construction site abnormal event label is associated with the prior construction site abnormal event label.

[0093] Step S204: Analyze the abnormal association information between the abnormal knowledge features generated by the model and the abnormal behavior features of the second construction site using the association feature analysis of the initial model weight parameters, and determine the association feature vector of the abnormal behavior features and abnormal knowledge features of the second construction site in the template association feature vector;

[0094] Step S205: Based on the distribution of the associated feature vectors and the training labeled data, adjust the model weight information of the associated feature analysis model until the model convergence requirement of the associated feature analysis model is met. The model weight information includes the weight information corresponding to the abnormal association information.

[0095] As the number of iterations of the correlation feature analysis model for the initial model weight parameters increases, the abnormal correlation information between the abnormal knowledge features generated by the model and the abnormal behavior features of the second construction site will be continuously updated, and the accuracy of this abnormal correlation information will continue to improve.

[0096] In an alternative implementation, the second site anomaly analysis model can also be obtained through real-time model iteration updates, and before step S201, it may further include:

[0097] The first template monitoring video stream is obtained, and the first construction site anomaly analysis model with initialized model weight parameters is used to project the first template monitoring video stream onto the target feature matching bitmap to generate the first estimated construction site behavior anomaly features.

[0098] Based on the first estimated abnormal characteristics of the construction site behavior, the first process output data corresponding to the model convergence control process of the first construction site anomaly analysis model is generated. The model weight information of the first construction site anomaly analysis model is adjusted according to the first process output data to generate the first construction site anomaly analysis model that meets the model convergence conditions.

[0099] The second template monitoring video stream is obtained, and the second construction site anomaly analysis model with initialized model weight parameters is used to project the second template monitoring video stream onto the target feature matching bitmap to generate the second estimated construction site behavior anomaly features.

[0100] Based on the second estimated abnormal characteristics of the construction site behavior, the second process output data corresponding to the model convergence control process of the second construction site anomaly analysis model is generated. The model weight information of the second construction site anomaly analysis model is adjusted according to the second process output data to generate a second construction site anomaly analysis model that meets the model convergence conditions.

[0101] Step S104: Based on the distribution of associated feature vectors, determine the labels of abnormal events in the smart construction site monitoring video stream.

[0102] In an alternative implementation, the number of target site abnormal event tags is at least two; step S103 may specifically include:

[0103] Using the correlation feature analysis model, the abnormal correlation information between the target abnormal knowledge features corresponding to each target construction site abnormal event label and the second construction site behavioral abnormal features is generated. The correlation feature vectors of the first construction site behavioral abnormal features and each target abnormal knowledge feature are determined. The correlation confidence of the smart construction site monitoring video stream under each target construction site abnormal event label is generated. The correlation confidence represents the probability value that the construction site abnormal event label of the smart construction site monitoring video stream is the target construction site abnormal event label. In an alternative implementation, the maximum correlation confidence indicates that the probability value that the construction site abnormal event label of the smart construction site monitoring video stream is the target construction site abnormal event label is the maximum.

[0104] In an alternative implementation, step S104 may specifically include: determining the target construction site anomaly event label with the highest association confidence, which is the construction site anomaly event label of the smart construction site monitoring video stream.

[0105] Among them, the label of the target construction site abnormal event with the highest correlation confidence can also be determined by the correlation feature analysis model.

[0106] In an alternative implementation, the structure of the correlation feature analysis model includes at least one prediction network, with each prediction network corresponding to a construction site anomaly event label. The step "using the anomaly correlation information generated by the correlation feature analysis model, which is the target anomaly knowledge feature corresponding to each target construction site anomaly event label and the second construction site behavior anomaly feature, to determine the correlation feature vector between the first construction site behavior anomaly feature and each target anomaly knowledge feature, and to generate the correlation confidence of the smart construction site monitoring video stream under each target construction site anomaly event label" may include: loading the first construction site behavior anomaly feature and the target anomaly knowledge feature into the prediction network corresponding to the target construction site anomaly event label; using the anomaly correlation information provided by each prediction network, which is the target anomaly knowledge feature and the second construction site behavior anomaly feature, to determine the correlation feature vector between the target anomaly knowledge feature and the first construction site behavior anomaly feature, and to generate the correlation confidence of the smart construction site monitoring video stream under each target construction site anomaly event label.

[0107] The prediction network can include a fully connected layer, which can load the fused abnormal behavior features of the first construction site and the abnormal knowledge features of the target onto the fully connected layer. The fully connected layer then evaluates the associated feature vectors of the abnormal behavior features of the first construction site and the abnormal knowledge features of the target.

[0108] In an alternative implementation, the first site behavior anomaly features and the target anomaly knowledge features can be fused and loaded into a fully connected layer.

[0109] Based on the above steps, the first construction site anomaly analysis model is used to obtain the first construction site behavior anomaly features of the smart construction site monitoring video stream. The smart construction site monitoring video stream is a monitoring video stream of the first construction site category. The first construction site anomaly analysis model is obtained by iteratively updating the model based on the first template monitoring video stream of the first construction site category. The target anomaly event label of the target construction site is obtained. The target anomaly knowledge features are used to express the target construction site anomaly event label, and the target anomaly knowledge features include the behavior anomaly features of the monitoring video stream of the second construction site category under the target construction site anomaly event label. Using the anomaly association information between the target anomaly knowledge features and the second construction site behavior anomaly features generated by the association feature analysis model, the association feature vector between the first construction site behavior anomaly features and the target anomaly knowledge features is determined, and an association feature vector distribution is generated. The second construction site behavior anomaly features are obtained by the second construction site anomaly analysis model from the second template monitoring video stream. The second template monitoring video stream is a monitoring video stream of the second construction site category. The model convergence control process of the second construction site anomaly analysis model is the same as that of the first construction site anomaly analysis model. Based on the association feature vector distribution, the construction site anomalies of the smart construction site monitoring video stream are determined. Event tags, where the convergence control process of the second construction site anomaly analysis model is the same as that of the first construction site anomaly analysis model, can establish a linear mapping relationship between the construction site behavior anomaly features under the first and second construction site categories. The target anomaly knowledge features include the behavior anomaly features of the monitoring video stream of the second construction site category under the target construction site anomaly event tag. This allows the anomaly association information generated by the correlation feature analysis model to accurately express the relationship between the target anomaly knowledge features of the target construction site anomaly event tag and the second construction site behavior anomaly features. Therefore, under the condition of the correlation between the first and second construction site behavior anomaly features, the anomaly association information generated by the correlation feature analysis model can accurately determine the correlation feature vector between the first construction site behavior anomaly features and the target anomaly knowledge features. Thus, the association information between the smart construction site monitoring video stream and the target construction site anomaly event tag can be obtained without restricting the first template monitoring video stream of the first construction site category to have training label data. This can improve the derivation efficiency of the anomaly tag analysis of the smart construction site monitoring video stream between monitoring video streams of different construction site categories.

[0110] Figure 2 An IoT-based smart construction site monitoring system 100 is schematically illustrated and can be used to implement the various embodiments described in this application.

[0111] In one embodiment, Figure 2An Internet of Things (IoT)-based smart construction site monitoring system 100 is shown. The IoT-based smart construction site monitoring system 100 has at least one processor 102, a control module (chipset) 104 coupled to at least one of the processors 102, a memory 106 coupled to the control module 104, a non-volatile memory (NVM) / storage device 108 coupled to the control module 104, at least one input / output device 110 coupled to the control module 104, and a network interface 112 coupled to the control module 106.

[0112] The processor 102 may include at least one single-core or multi-core processor, and may include any combination of general-purpose processors or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, the IoT-based smart construction site monitoring system 100 can function as a server device such as a gateway as described in the embodiments of this application.

[0113] In some embodiments, the IoT-based smart construction site monitoring system 100 may include at least one computer-readable medium (e.g., memory 106 or NVM / storage device 108) having instructions 114 and at least one processor 102 combined with the at least one computer-readable medium and configured to execute the instructions 114 to implement the module and thus perform the actions described in this disclosure.

[0114] In one embodiment, the control module 104 may include any suitable interface controller to provide any suitable interface to at least one of the processors 102 and / or any suitable device or component communicating with the control module 104.

[0115] The control module 104 may include a memory controller module to provide an interface to the memory 106. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0116] Memory 106 can be used, for example, to load and store data and / or instructions 114 for an IoT-based smart construction site monitoring system 100. In one embodiment, memory 106 may include any suitable volatile memory, such as suitable DRAM. In some embodiments, memory 106 may include double data rate type 4 synchronous dynamic random access memory (DDR4FbRAM).

[0117] In one embodiment, the control module 104 may include at least one input / output controller to provide an interface to the NVM / storage device 108 and (at least one) input / output device 110.

[0118] For example, NVM / storage device 108 may be used to store data and / or instructions 114. NVM / storage device 108 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (at least one) non-volatile storage device (e.g., at least one hard disk drive (HDD), at least one optical disc (CD) drive, and / or at least one digital universal optical disc (DVD) drive).

[0119] NVM / storage device 108 may include storage resources that are physically part of a device installed on the IoT-based smart site monitoring system 100, or that can be accessed by the device without being part of it. For example, NVM / storage device 108 may be accessed via a network through at least one input / output device 110.

[0120] At least one input / output device 110 provides an interface for the IoT-based smart construction site monitoring system 100 to communicate with any other suitable device. The input / output device 110 may include communication components, input components, sensor components, etc. A network interface 112 provides an interface for the IoT-based smart construction site monitoring system 100 to communicate via at least one network. The IoT-based smart construction site monitoring system 100 can wirelessly communicate with at least one component of a wireless network based on any standard and / or protocol of at least one wireless network standard, such as accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof.

[0121] In one embodiment, at least one of the processors 102 may be logically packaged with at least one controller (e.g., a memory controller module) of the control module 104. In one embodiment, at least one of the processors 102 may be logically packaged with at least one controller of the control module 104 to form a system-in-package (SiD). In one embodiment, at least one of the processors 102 may be logically integrated with at least one controller of the control module 104 on the same die. In one embodiment, at least one of the processors 102 may be logically integrated with at least one controller of the control module 104 on the same die to form a system-on-a-chip (SoC).

[0122] In various embodiments, the IoT-based smart construction site monitoring system 100 may be, but is not limited to, terminal devices such as servers, desktop computing devices, or mobile computing devices (e.g., laptops, handheld computing devices, tablets, netbooks, etc.). In various embodiments, the IoT-based smart construction site monitoring system 100 may have more or fewer components and / or different architectures. For example, in some embodiments, the IoT-based smart construction site monitoring system 100 includes at least one camera, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0123] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0124] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchain. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0125] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0126] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in at least one script in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborative files (e.g., a file that stores at least one module, subroutine, or code section).

[0127] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0128] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

[0129] The present application has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present application. Therefore, the content of this specification should not be construed as a limitation of the present application.

Claims

1. A smart construction site monitoring method based on Internet of Things, characterized in that, Applied to a server, the method comprises: acquiring first construction site behavior abnormal features of a smart construction site monitoring video stream by using a first construction site abnormal analysis model, the smart construction site monitoring video stream being a monitoring video stream of a first construction site category, the first construction site abnormal analysis model being obtained by model iterative updating according to a first template monitoring video stream of the first construction site category, wherein the smart construction site monitoring video stream is generated by monitoring through an Internet of Things monitoring device arranged in a field construction site; acquiring target abnormal knowledge features of a target construction site abnormal event label, the target abnormal knowledge features being used to express the target construction site abnormal event label, and the target abnormal knowledge features including behavior abnormal features of a monitoring video stream of a second construction site category under the target construction site abnormal event label; generating abnormal correlation information of the target abnormal knowledge features and second construction site behavior abnormal features by using a correlation feature analysis model, determining a correlation feature vector of the first construction site behavior abnormal features and the target abnormal knowledge features, and generating a correlation feature vector distribution, the second construction site behavior abnormal features being obtained by extracting a second template monitoring video stream by a second construction site abnormal analysis model, the second template monitoring video stream being a monitoring video stream of a second construction site category, and the second construction site abnormal analysis model having the same model convergence control process as the first construction site abnormal analysis model; determining a construction site abnormal event label of the smart construction site monitoring video stream based on the correlation feature vector distribution; before the acquiring first construction site behavior abnormal features of a smart construction site monitoring video stream by using a first construction site abnormal analysis model, the method further comprises: extracting second construction site behavior abnormal features from the second template monitoring video stream by using a second construction site abnormal analysis model that meets a model convergence condition, the second template monitoring video stream having template annotation data, the template annotation data including a prior construction site abnormal event label of the second template monitoring video stream, and the prior construction site abnormal event label including the target construction site abnormal event label; acquiring abnormal knowledge features corresponding to the prior construction site abnormal event label, the abnormal knowledge features being used to express the prior construction site abnormal event label, and the abnormal knowledge features including behavior abnormal features of a monitoring video stream of a second construction site category under the prior construction site abnormal event label, specifically including: transforming the prior construction site abnormal event label into an initial feature representation vector in a regularized expression mode; performing feature enhancement on the second template monitoring video stream to generate an enhanced monitoring video stream, and extracting third construction site behavior abnormal features of the enhanced monitoring video stream by using the second construction site abnormal analysis model; acquiring estimated learning features, the estimated learning features including the third construction site behavior abnormal features and the initial feature representation vector, and estimated learning label data of the estimated learning features including enhanced feature segments in the second template monitoring video stream; projecting the third construction site behavior abnormal features and the initial feature representation vector by using an abnormal knowledge extraction model, determining the enhanced feature segments in the enhanced monitoring video stream according to a projection result, and generating estimated enhanced feature segments. Based on the estimated enhanced feature segment and the estimated learning label data, iteratively update the model weight information of the abnormal knowledge extraction model to generate an iteratively updated abnormal knowledge extraction model; Use the iteratively updated abnormal knowledge extraction model to project the initial feature representation vector of the prior construction site abnormal event label to generate an abnormal knowledge feature of the prior construction site abnormal event label. 2.The Internet-of-Things based smart construction site monitoring method according to claim 1, characterized in that, Before the first construction site behavior abnormal feature of the intelligent construction site monitoring video stream is acquired by using the first construction site abnormal analysis model, the method further comprises: According to the second construction site behavior abnormal feature and the abnormal knowledge feature, a template association feature vector is acquired, the template association feature vector comprises the second construction site behavior abnormal feature and the abnormal knowledge feature, and training label data of the template association feature vector comprises label data representing whether the prior construction site abnormal event label is associated with the prior construction site abnormal event label; Abnormal association information of the abnormal knowledge feature and the second construction site behavior abnormal feature generated by using the association feature analysis model with the initialized model weight parameter is used to determine the association feature vector of the second construction site behavior abnormal feature and the abnormal knowledge feature in the template association feature vector; According to the association feature vector distribution and the training label data, the model weight information of the association feature analysis model is iteratively updated until the association feature analysis model meets the model convergence requirement, and the model weight information comprises weight information corresponding to the abnormal association information. 3.The Internet-of-Things based smart construction site monitoring method according to claim 1, characterized in that, Before the second construction site behavior abnormal feature of the second template monitoring video stream is extracted by using the second construction site abnormal analysis model meeting the model convergence condition, the method further comprises: A first template monitoring video stream is acquired, and the first template monitoring video stream is projected into a target feature matching bitmap by using the first construction site abnormal analysis model with the initialized model weight parameter to generate a first estimated construction site behavior abnormal feature; According to the first estimated construction site behavior abnormal feature, first process output data corresponding to a model convergence control process of the first construction site abnormal analysis model is generated, and the model weight information of the first construction site abnormal analysis model is iteratively updated according to the first process output data to generate the first construction site abnormal analysis model meeting the model convergence condition; A second template monitoring video stream is acquired, and the second template monitoring video stream is projected into the target feature matching bitmap by using the second construction site abnormal analysis model with the initialized model weight parameter to generate a second estimated construction site behavior abnormal feature; According to the second estimated construction site behavior abnormal feature, second process output data corresponding to a model convergence control process of the second construction site abnormal analysis model is generated, and the model weight information of the second construction site abnormal analysis model is iteratively updated according to the second process output data to generate the second construction site abnormal analysis model meeting the model convergence condition. 4.The Internet-of-Things based smart construction site monitoring method according to claim 1, characterized in that, The abnormal knowledge extraction model comprises a projection function and an estimation function, the projection function has a projection parameter; and the estimated learning feature comprises: The third construction site behavior anomaly feature and the initial feature representation vector are fused to generate a fused representation vector, and the fused representation vector is used as an estimated learning feature; The third construction site behavior anomaly feature and the initial feature representation vector are projected by using the anomaly knowledge extraction model, and the enhanced feature segment in the enhanced monitoring video stream is determined according to the projection result to generate an estimated enhanced feature segment, including: The projection parameters of the projection function are used to project the fused representation vector to generate a projected representation vector; The projected representation vector is analyzed by using the estimation function to determine the enhanced feature segment in the enhanced monitoring video stream to generate an estimated enhanced feature segment; The initial feature representation vector of the prior construction site anomaly event label is projected by using the iteratively updated anomaly knowledge extraction model to generate an anomaly knowledge feature of the prior construction site anomaly event label, including: Based on the node of the initial feature representation vector in the fused representation vector, the projection target parameter of the initial feature representation vector is obtained from the projection parameter; The initial feature representation vector is projected according to the projection target parameter to generate an anomaly knowledge feature of the prior construction site anomaly event label. 5.The Internet-of-Things based smart construction site monitoring method according to claim 1, characterized in that, The number of target construction site anomaly event labels is multiple; The anomaly association information between the target anomaly knowledge feature and the second construction site behavior anomaly feature generated by using the association feature analysis model is used to determine the association feature vector of the first construction site behavior anomaly feature and the target anomaly knowledge feature to generate an association feature vector distribution, including: The anomaly association information between the target anomaly knowledge feature corresponding to each target construction site anomaly event label and the second construction site behavior anomaly feature generated by using the association feature analysis model is used to determine the association feature vector of the first construction site behavior anomaly feature and each target anomaly knowledge feature to generate the association confidence of the intelligent construction site monitoring video stream under each target construction site anomaly event label, and the association confidence represents the probability value of the construction site anomaly event label of the intelligent construction site monitoring video stream as the target construction site anomaly event label; The construction site anomaly event label of the intelligent construction site monitoring video stream is determined based on the association feature vector distribution, including: The target construction site anomaly event label with the maximum association confidence is determined as the construction site anomaly event label of the intelligent construction site monitoring video stream. 6.The IoT-based smart construction site monitoring method of claim 5, wherein, The association feature analysis model includes at least one prediction network, and one prediction network corresponds to one construction site anomaly event label; The anomaly association information between the target anomaly knowledge feature corresponding to each target construction site anomaly event label and the second construction site behavior anomaly feature generated by using the association feature analysis model is used to determine the association feature vector of the first construction site behavior anomaly feature and each target anomaly knowledge feature to generate the association confidence of the intelligent construction site monitoring video stream under each target construction site anomaly event label, including: The first construction site behavior anomaly feature and the target anomaly knowledge feature are loaded into the prediction network corresponding to the target construction site anomaly event label. The target abnormal knowledge features and the second abnormal behavior features of the construction site are associated by using the abnormal association information provided by each prediction network, to determine an association feature vector of the target abnormal knowledge features and the first abnormal behavior features of the construction site, and to generate an association confidence of the intelligent construction site monitoring video stream under each target abnormal event label of the construction site.

7. A computer readable storage medium characterized in that, The computer readable storage medium stores a program which, when executed by the processor, implements the intelligent construction site monitoring method based on the Internet of Things according to any one of claims 1-6.

8. A computer program product, characterised in that, The computer readable storage medium stores a program which, when executed by the processor, implements the intelligent construction site monitoring method based on the Internet of Things according to any one of claims 1-6.

9. An Internet of Things-based smart construction site monitoring system, characterized in that, The computer readable storage medium stores a program which, when executed by the processor, implements the intelligent construction site monitoring method based on the Internet of Things according to any one of claims 1-6.

Images