Engineering management intelligent monitoring system and method based on multi-mode perception

Abstract

The invention relates to an engineering management intelligent monitoring system and method based on multi-mode perception. The method comprises the following steps: acquiring personnel identification data and equipment sensing data corresponding to a construction site; performing time-space synchronization processing on the personnel identification data and the equipment sensing data to obtain a field condition fusion feature vector; reasoning the future operation condition of the construction site according to the site condition fusion feature vector to obtain future operation prediction data of construction; performing alarm adaptive analysis on the construction future operation prediction data to obtain safety evidence chain analysis data of the construction site; and performing construction safety monitoring on the construction site according to the safety evidence chain analysis data to obtain risk monitoring data of the construction site. By adopting the method, the engineering quality can be effectively guaranteed.

Description

Technical Field

[0001] This application relates to the field of intelligent monitoring technology, and in particular to an intelligent monitoring system and method for engineering management based on multimodal perception. Background Technology

[0002] In traditional engineering management, safety monitoring, material management, and equipment status monitoring at construction sites mainly rely on manual inspections, which suffers from problems such as low efficiency, easy omissions, and untimely response, resulting in the inability to effectively guarantee project quality. Summary of the Invention

[0003] Therefore, it is necessary to provide a method, system, and device for intelligent monitoring of engineering management based on multimodal perception that can effectively guarantee the quality of engineering projects, addressing the aforementioned technical problems.

[0004] Firstly, this application provides an intelligent monitoring method for engineering management based on multimodal perception, including:

[0005] Acquire personnel identification data and equipment sensor data corresponding to the construction site;

[0006] The personnel identification data and the equipment sensor data are processed in a spatiotemporal synchronization manner to obtain a fusion feature vector of the on-site situation.

[0007] Based on the fusion of feature vectors according to the site conditions, the future operation of the construction site is inferred to obtain prediction data of the future operation of the construction.

[0008] An alarm-adaptive analysis is performed on the predicted data of future construction operations to obtain safety evidence chain analysis data of the construction site.

[0009] Based on the safety evidence chain analysis data, construction safety monitoring is carried out at the construction site to obtain risk monitoring data for the construction site.

[0010] Secondly, this application also provides an intelligent monitoring system for engineering management based on multimodal perception, the system comprising: a terminal and a computer device;

[0011] The terminal is used to acquire personnel identification data and equipment sensor data corresponding to the construction site, and send the personnel identification data and equipment sensor data to the computer device.

[0012] The computer device is used to perform spatiotemporal synchronization processing on the personnel identification data and the device sensor data to obtain a fusion feature vector of the scene situation.

[0013] The computer device is used to fuse feature vectors based on the site conditions, infer the future operation of the construction site, and obtain prediction data for the future operation of the construction site.

[0014] The computer equipment is used to perform alarm adaptive analysis on the construction future operation prediction data to obtain the safety evidence chain analysis data of the construction site.

[0015] The computer equipment is used to perform construction safety monitoring on the construction site based on the security evidence chain analysis data, and to obtain risk monitoring data of the construction site.

[0016] Thirdly, this application also provides an intelligent monitoring device for engineering management based on multimodal perception, comprising:

[0017] The construction data acquisition module is used to acquire personnel identification data and equipment sensor data corresponding to the construction site.

[0018] The construction data fusion module is used to perform spatiotemporal synchronization processing on the personnel identification data and the equipment sensor data to obtain a fusion feature vector of the site conditions.

[0019] The construction status prediction module is used to fuse feature vectors based on the site conditions, infer the future operation of the construction site, and obtain construction future operation prediction data.

[0020] The construction safety prediction module is used to perform alarm adaptive analysis on the construction future operation prediction data to obtain the safety evidence chain analysis data of the construction site.

[0021] The construction safety monitoring module is used to monitor the construction site based on the safety evidence chain analysis data, and obtain the risk monitoring data of the construction site.

[0022] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any step of an intelligent monitoring method for engineering management based on multimodal perception.

[0023] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any step of an intelligent monitoring method for engineering management based on multimodal perception.

[0024] The aforementioned intelligent monitoring method, system, and device for engineering management based on multimodal perception comprehensively acquires personnel identification data and equipment sensor data from the construction site. It then performs high-precision spatiotemporal synchronous fusion processing on this multi-source heterogeneous data to construct a fusion feature vector that comprehensively reflects the dynamic state of the construction site. Based on this, the fusion feature vector is used to intelligently infer and predict the future operating state of the construction site, forming forward-looking prediction data for future construction operations. Furthermore, by implementing alarm-adaptive analysis on the prediction data, a safety evidence chain analysis data is constructed, including potential risk factors, triggering conditions, and event chains, enabling in-depth exploration and tracing of safety hazards. Finally, based on this safety evidence chain, continuous construction safety monitoring is conducted at the construction site, outputting risk monitoring data and early warning information in real time. This significantly improves the risk perception capability and safety response efficiency of the construction site, effectively ensuring project quality and guaranteeing safe, controllable, and scientific management throughout the entire construction process. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is an application environment diagram of an intelligent monitoring method for engineering management based on multimodal perception in one embodiment;

[0027] Figure 2 This is a flowchart illustrating an intelligent monitoring method for engineering management based on multimodal perception in one embodiment.

[0028] Figure 3 This is a flowchart illustrating the first method for obtaining prediction data of future construction operations in one embodiment;

[0029] Figure 4 This is a flowchart illustrating the second method for obtaining prediction data of future construction operations in one embodiment;

[0030] Figure 5 This is a flowchart illustrating a third method for obtaining prediction data of future construction operations in one embodiment;

[0031] Figure 6 This is a flowchart illustrating a method for obtaining secure chain of evidence analysis data in one embodiment.

[0032] Figure 7 This is a flowchart illustrating the second method for obtaining secure chain of evidence analysis data in one embodiment;

[0033] Figure 8 This is a flowchart illustrating a method for fusing feature vectors based on on-site conditions in one embodiment.

[0034] Figure 9 This is a structural block diagram of an intelligent monitoring device for engineering management based on multimodal perception in one embodiment;

[0035] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0037] This application provides an intelligent monitoring method for engineering management based on multimodal perception, which can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on other network servers. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0038] In one exemplary embodiment, such as Figure 2 As shown, an intelligent monitoring method for engineering management based on multimodal perception is provided, which is then applied to... Figure 1 Taking the server in the example, the explanation includes the following steps 202 to 210. Wherein:

[0039] Step 202: Obtain personnel identification data and equipment sensor data corresponding to the construction site.

[0040] Step 204: Perform spatiotemporal synchronization processing on personnel identification data and equipment sensor data to obtain a fusion feature vector of the on-site situation.

[0041] Step 206: Based on the site conditions, fuse feature vectors to infer the future operation of the construction site and obtain prediction data for the future operation of the construction.

[0042] Step 208: Perform alarm adaptive analysis on the future operation prediction data of the construction site to obtain safety evidence chain analysis data of the construction site.

[0043] Step 210: Based on the safety evidence chain analysis data, conduct construction safety monitoring at the construction site to obtain risk monitoring data for the construction site.

[0044] Personnel identification data can be information related to the identity of personnel at the construction site, collected and identified through facial recognition cameras, identity verification systems, or wearable sensors. It typically includes facial image feature vectors, work badge IDs, identity authentication information, current location information, and work status, and is used to verify the identity of construction personnel, track their behavior, and manage their access.

[0045] Among them, the equipment sensing data can be real-time operating status information collected by various sensors installed on the construction equipment, including but not limited to material identification sensors, including RFID readers, for identifying and tracking the status of construction materials; image recognition cameras and infrared thermometers for monitoring on-site image information and equipment temperature status; and vibration sensors for detecting abnormal operating conditions of the equipment.

[0046] Among them, the spatiotemporal synchronization processing can be a process of uniformly aligning sensor data from different sources, times, and locations. It mainly includes two parts: time synchronization and spatial coordinate mapping. Time synchronization uses a high-precision clock protocol (such as PTP) to unify the timestamps of each data. Spatial synchronization maps all data to a unified three-dimensional BIM model coordinate system through device calibration and coordinate transformation, thereby ensuring the spatiotemporal consistency and correlation of subsequent analysis.

[0047] Among them, the on-site situation fusion feature vector can be a unified representation vector generated by feature extraction, modality alignment and cross-modal fusion mechanism (such as Transformer network and attention mechanism) after completing multimodal data acquisition and spatiotemporal synchronization. It can simultaneously encode personnel behavior characteristics, equipment status information, material location and environmental parameters.

[0048] Among them, the future operation status can be the result of predicting the behavior trends, state changes or potential events of various elements at the construction site in the future time period based on a time evolution model (such as a spatiotemporal graph neural network) given the current site status. It is mainly used to assess the dynamic evolution process of upcoming violations, equipment failures, construction conflicts or safety risks.

[0049] Among them, the construction future operation prediction data can be structured data output that reflects the operation trend of the construction site within a certain time range in the future, calculated by intelligent reasoning models. It includes the predicted status of each node (such as personnel, equipment, and materials), risk path evolution, abnormal trajectory identification, etc., and is the basic data source for realizing early warning and proactive control.

[0050] Among them, alarm adaptive analysis means that after obtaining future prediction data, the system does not rely on fixed rules or thresholds, but dynamically judges the severity and credibility of risk events based on mechanisms such as abnormal behavior trajectory extraction, modal consistency verification, causal chain reasoning and expert system collaboration, and automatically generates interpretable alarm logic and response suggestions, which has robustness and self-learning capabilities.

[0051] Among them, the security evidence chain analysis data can be a set of structured multidimensional evidence information related to the event that is automatically generated by the system after the alarm event is triggered. It usually includes video clips of violations, the identity and trajectory of responsible personnel, status records of equipment involved, snapshots of on-site environmental data and their precise location in BIM space, which are used for risk auditing, accountability tracing and regulatory compliance evidence storage.

[0052] Construction safety monitoring can be an automated safety management process implemented in real time at the construction site based on predictive data and safety evidence chains. It includes a series of linked control actions such as early warning of high-risk behaviors, emergency equipment shutdown control, operation permission restriction, and heat map visual prompts, aiming to prevent personnel injury, equipment damage and safety accidents during construction.

[0053] Among them, risk monitoring data can be quantifiable information about the current risk situation continuously output by the system during the construction safety monitoring process, including real-time risk indicators, violation event flow, response control records, alarm frequency statistics and their spatiotemporal distribution, etc., which are used for dynamic risk assessment, safety situation modeling and safety management optimization decision-making for construction projects.

[0054] Specifically, data is collected in real time through various types of sensing devices deployed at the construction site. These devices include facial recognition cameras, image recognition cameras, RFID readers, infrared thermometers, and vibration sensors. Personnel identification data is obtained by matching facial images with work badge information, while equipment sensing data includes equipment operating status, temperature distribution, and vibration characteristics. All types of sensors complete data acquisition and pre-encoding processing through edge computing nodes and encapsulate the data into a unified format for transmission.

[0055] The system's computing module controls the edge nodes of the system to perform high-precision time alignment and spatial registration of personnel identification data and equipment sensor data. For time synchronization, the PTP protocol is used in conjunction with a sliding window interpolation algorithm to ensure that the timestamp accuracy of the sensor data is controlled at the millisecond level. For spatial registration, a laser calibration device is used to map the sensor positions to the BIM model coordinate system. Various features are transformed into vector representations through modal feature extraction algorithms, and a unified fusion feature vector of the site conditions is generated through a cross-modal attention mechanism and a gating fusion strategy.

[0056] Based on the site conditions, feature vectors are fused to construct a spatiotemporal graph structure containing construction personnel, equipment, and materials as nodes. Edge connections between nodes are established based on information such as spatial distance between personnel and equipment, interaction frequency, and logical associations. Subsequently, within a preset time window, the feature changes of each node at consecutive time steps and the graph structure snapshot are combined to form a time series graph input, which is fed into a spatiotemporal graph convolutional network (ST-GCN) for inference. This model aggregates node adjacency information in the spatial dimension and models the evolution trend of node states in the temporal dimension, thereby outputting a prediction result representing the future operation of the construction site as the prediction data for future construction operations, including behavioral changes, equipment state trends, and potential risk paths.

[0057] First, abnormal behavioral trajectories or logical deviations in the predicted data of future construction operations are analyzed, and cross-modal consistency checks are performed to identify contradictory information between different sensing signals. If a potential anomaly is determined, the event path is backtracked based on the risk causal chain model in the knowledge graph, automatically generating safety evidence chain analysis data that includes behavioral paths, modal conflicts, physical states, and construction logic. This data includes spatial coordinates, image fragments, sensor data, and personnel identification information, and features traceability and compliance visualization.

[0058] Based on the analysis of the safety evidence chain data, the system automatically links the on-site control system and the BIM platform, and sends control commands to the PLC controller via the MQTT protocol to realize operations such as equipment stop, high-altitude operation lock, and on-site voice alarm. At the same time, the system overlays and displays the heat map of the risk area and the three-dimensional location of the event node in real time in the BIM model, and finally forms a risk monitoring data stream of the construction site for subsequent safety statistics, dispatching and scheduling and operation and maintenance analysis.

[0059] In the aforementioned intelligent monitoring method for engineering management based on multimodal perception, personnel identification data and equipment sensor data from the construction site are comprehensively acquired. High-precision spatiotemporal synchronous fusion processing is then performed on these multi-source heterogeneous data to construct a fusion feature vector that comprehensively reflects the dynamic state of the construction site. Based on this, the fusion feature vector is used to intelligently infer and predict the future operating state of the construction site, forming forward-looking prediction data for future construction operations. Furthermore, by implementing alarm-adaptive analysis on the prediction data, a safety evidence chain analysis data is constructed, including potential risk factors, triggering conditions, and event chains, enabling in-depth exploration and tracing of safety hazards. Finally, based on this safety evidence chain, continuous construction safety monitoring is conducted at the construction site, outputting risk monitoring data and early warning information in real time. This significantly improves the risk perception capability and safety response efficiency of the construction site, effectively ensuring project quality and guaranteeing safe, controllable, and scientific management throughout the entire construction process.

[0060] In one exemplary embodiment, such as Figure 3 As shown, the step of fusing feature vectors based on the site conditions to infer the future operation of the construction site and obtain predictive data for future construction operations includes steps 302 to 306. Wherein:

[0061] Step 302: Identify construction personnel information, construction equipment information, and construction material information from personnel identification data and equipment sensor data.

[0062] Step 304: Using construction personnel information, construction equipment information, and construction material information as nodes, construct the spatiotemporal diagram structure corresponding to the construction site.

[0063] Step 306: Use the fusion feature vector of the site conditions as the input feature of the spatiotemporal graph structure, perform graph structure dynamic reasoning on the spatiotemporal graph structure, and obtain the prediction data of the future operation of construction.

[0064] Among them, the spatiotemporal graph structure can model key entities with behavioral attributes (such as construction personnel, construction equipment and construction materials) in the construction site as nodes in the graph structure, and establish edge connections between nodes through spatial proximity, operation interaction frequency, process dependency relationship, etc. At the same time, the time dimension is introduced to express the dynamic evolution characteristics of the graph structure in continuous time steps in the form of sequence. This structure can simultaneously reflect the spatial topology layout and behavioral change trend of the construction site.

[0065] Among them, graph structure dynamic reasoning can take the constructed spatiotemporal graph structure as input and use graph neural network models such as spatiotemporal graph convolutional network (ST-GCN) to model and calculate the feature evolution of nodes in the graph at continuous time steps, thereby predicting the state changes, behavioral trends or potential anomalies of each node (such as personnel, equipment, materials) in the future. This process integrates the topological information and time series information of the graph, and can capture the potential safety risk paths and behavioral evolution patterns of the construction site, so as to realize intelligent prediction of future operating status.

[0066] Specifically, the data streams from personnel identification devices and equipment sensing data from various sensors are first analyzed to extract structured information related to construction personnel, equipment, and materials. For construction personnel, identity identifiers, current location, and behavioral status are generated based on facial recognition images and work badge data to serve as personnel information. For equipment, equipment type, operating status, and historical maintenance records are extracted from sensor data such as temperature, vibration, and current to serve as construction equipment information. For materials, features such as material number, stacking location, and usage status are obtained by combining RFID data to serve as construction material information, thereby constructing semantic information representations of the three types of entity elements in the construction site.

[0067] Based on the three types of entity information identified above, construction personnel information, construction equipment information, and construction material information are mapped to nodes in a graph structure. Then, based on real-time on-site location information, operation frequency data, and process dependencies, edge connections are established between nodes to form a spatiotemporal graph structure that dynamically reflects the construction site status. The edge weights of the spatiotemporal graph structure can be dynamically assigned based on spatial distance, interaction frequency, or physical connection strength, supporting continuous updates and topological evolution of the graph structure.

[0068] The site condition fusion feature vector generated by integrating multimodal perception information is used as the initial input feature for each node in the spatiotemporal graph structure. At continuous time steps, this spatiotemporal graph structure is input into the Spatiotemporal Graph Convolutional Network (ST-GCN) model for inference. This model aggregates features of adjacent nodes in the spatial dimension and models the trend of node state changes in the temporal dimension, enabling predictive outputs for events such as personnel behavior, equipment failure, and material movement. Ultimately, it generates predictive data for the future operation of construction, covering the overall construction status.

[0069] In this embodiment, semantic information of construction personnel, equipment and materials is accurately extracted from personnel identification data and equipment sensing data, and a spatiotemporal graph structure with spatial topology and temporal evolution characteristics is constructed based on this. The fused multimodal feature vector is introduced into the graph structure as input, and intelligent prediction of the future operating status of the construction site is realized through dynamic reasoning of the graph structure. This enables comprehensive perception of the construction environment, dynamic identification of potential risk paths and early warning of abnormal behavior, significantly improving the foresight, robustness and automation level of on-site monitoring.

[0070] In one exemplary embodiment, such as Figure 4 As shown, the step of using the fused feature vector of the site conditions as the input feature of the spatiotemporal graph structure, and performing graph structure dynamic reasoning on the spatiotemporal graph structure to obtain the predicted data for future construction operations includes steps 402 to 406. Wherein:

[0071] Step 402: The fusion feature vector of the on-site situation is assigned to each node in the spatiotemporal graph structure as the initial feature of each node.

[0072] Step 404: Based on the spatial distance, interaction frequency, and physical connection relationship between the construction personnel information, construction equipment information, and construction material information, construct the initial edge connection between the initial features of any node and all other nodes.

[0073] Step 406: Input the initial features of each node and the initial edge connections into the spatiotemporal graph convolutional inference model corresponding to the construction site to obtain the prediction data for the future operation of the construction.

[0074] Among them, the initial node feature can be the basic input vector assigned to each node in the graph when constructing the spatiotemporal graph structure. This feature is formed by the fusion of multimodal data from the construction site and includes information such as image recognition results, infrared temperature values, vibration signal patterns, and RFID trajectory codes. It is used to comprehensively describe the current state and historical behavior of the corresponding entity (such as personnel, equipment, or materials).

[0075] Spatial distance can be the three-dimensional coordinate distance between two entities (such as personnel and equipment, equipment and materials) represented by two nodes in the construction site. It is usually provided by positioning systems such as UWB, GPS or laser ranging and is used to measure the degree of spatial association between nodes. In graph structures, it is often used to determine whether to establish edge connections or set edge weights.

[0076] Interaction frequency can be the number or intensity of physical operations, signal communication or process linkage between two nodes within a certain time range, such as the frequency of personnel operating equipment or the frequency of material being moved to a designated area. This indicator can be obtained through sensor logs, RFID tracks or historical operation records, and is used to reflect the dynamic correlation strength between nodes. It is one of the important bases for building edges in graph structures.

[0077] Among them, physical connection relationships can be physical coupling or structural dependence relationships between two node entities in the construction site, such as the hoisting connection between tower crane and hanging materials, the binding relationship between personnel and the equipment they wear, and the interface connection between pipelines and valves. This relationship usually has clear engineering attributes and fixed structural paths, which are used to establish fixed edges with high confidence in the graph structure.

[0078] The initial edge connection can be a set of edges in the graph initially determined when constructing the spatiotemporal graph structure based on the spatial distance, interaction frequency and physical connection relationship between nodes. It is used to reflect the direct correlation between entities at the current moment. These edge connections are assigned certain edge weights and can be dynamically updated over time. They are the basic structure for the spatiotemporal graph convolution model to perform graph computation and propagation operations.

[0079] Among them, the spatio-temporal graph convolutional inference model can be a deep learning model that integrates graph neural networks and temporal modeling mechanisms to process graph structure information that changes in structure and evolves dynamically in continuous time steps. This model uses graph convolution to aggregate node neighbor information in the spatial dimension and captures the temporal evolution trend of node state in the temporal dimension, thereby realizing the prediction of the future state and potential risks of each node at the construction site. A representative structure is ST-GCN (Spatio-Temporal Graph Convolutional Network).

[0080] Specifically, the spatiotemporal graph structure is traversed node by node, and based on the entity category (construction personnel, equipment or materials) corresponding to each node, the corresponding feature fragments (such as multimodal information such as image recognition, infrared temperature measurement, vibration signal and positioning data) are extracted from the feature vector fused from the site conditions, and these fragments are used as the initial input features of the node to bind them, thereby obtaining the initial node features corresponding to each node.

[0081] Based on UWB positioning data, the spatial distance between nodes is calculated. Simultaneously, by combining construction logs, sensor communication interaction frequencies, process flow documents, and the physical coupling or structural dependencies between any two node entities, the operational behaviors and dependencies between construction personnel information and construction equipment information, and between construction equipment information and construction material information, are identified. Based on this, weighted edges are generated for each pair of actually related nodes in the graph structure. The weights can represent operation frequency, physical coupling degree, or risk propagation potential. This constructs initial edge connections between any node's initial features and the other nodes, reflecting the actual operational relationships, for modeling dynamic interactions between nodes in the graph neural network.

[0082] The graph structure, consisting of a node feature matrix constructed from the initial features of each node and an adjacency matrix constructed from the initial edges, is input into a Spatiotemporal Graph Convolutional Network (ST-GCN). This model aggregates node neighbor information in the spatial dimension through graph convolution and captures the evolution trend of node states in the temporal dimension through convolution or recurrent structures. The model dynamically infers the behavior of construction personnel, changes in equipment state, and material displacement at continuous time steps, and outputs future state estimates and risk predictions for each node, thereby generating construction future operation prediction data that reflects the future operation trend of the construction site.

[0083] In this embodiment, the fused multimodal site perception feature vectors are precisely assigned to various nodes in the spatiotemporal graph structure as their initial node features. Combined with the spatial distance, interaction frequency, and physical connection relationship between construction personnel, equipment, and materials, initial edge connections reflecting the association of real construction behaviors are constructed. This makes the entire graph structure not only rich in entity semantics but also possess high-fidelity structural relationships. Furthermore, by dynamically analyzing the structure through the spatiotemporal graph convolutional inference model, accurate prediction of the future operating status of the construction site can be achieved. This significantly improves the foresight of risk discovery, the sensitivity of complex process identification, and the automation level of safety monitoring, which is superior to traditional static rules or single-modal monitoring schemes.

[0084] In one exemplary embodiment, such as Figure 5 As shown, the step of inputting the initial features of each node and the initial edges into the spatiotemporal graph convolutional inference model corresponding to the construction site to obtain the prediction data for the future operation of the construction includes steps 502 to 510. Wherein:

[0085] Step 502: Based on the current continuous time step, perform graph convolution calculation on the initial features of each node and the initial edge connections to obtain the spatiotemporal dynamic features corresponding to each node.

[0086] Step 504: Based on the spatiotemporal dynamic characteristics, predict the operating status of the construction site at the target future time to obtain initial future operating prediction data.

[0087] Step 506: Update the connection relationships of each node based on the initial future operation prediction data to obtain the updated edge connections.

[0088] Step 508: Connect each updated edge as the initial edge, and take the next continuous time step as the current continuous time step.

[0089] Step 510: Return to the execution step of performing graph convolution calculation on the initial features of each node and each initial edge connection based on the current continuous time step to obtain the spatiotemporal dynamic features corresponding to each node, until the next continuous time step triggers the preset continuous time step, and the initial future operation prediction data is used as the construction future operation prediction data.

[0090] The current continuous time step can be a time window selected by the system during the graph neural network inference process, which includes multiple adjacent moments. It is used to model the continuous evolution behavior of node features in a short period of time. The graph structure within this time period is updated synchronously, enabling the model to capture both spatial interaction relationships and time series change trends, thus forming the basic unit of dynamic spatiotemporal modeling.

[0091] Among them, the spatiotemporal dynamic features can be dynamic representation vectors obtained by inferring the initial features and adjacency relationships of nodes in consecutive time steps after calculation through spatiotemporal graph convolution. These features encode the local interaction information of nodes in the spatial structure and the trajectory of state changes in the time dimension, and are used to predict the future behavior trends and potential risks of nodes.

[0092] Among them, the initial future operation prediction data can be the state prediction results of the construction site at the target future time point, which are derived from the current spatiotemporal dynamic characteristics. It usually includes the behavior prediction, state classification or anomaly probability of each node, which is used to characterize the event evolution trend that may occur in the construction system in the subsequent time.

[0093] Among them, updating edge connections can be new connection relationships or edge weight configurations between nodes derived from the initial future operation prediction data. This is used to reflect new interaction patterns or risk transmission paths that may occur after node state changes. Operations include adding connections, adjusting edge weights, or deleting edges, which are used to dynamically adjust the graph structure to adapt to the evolution of the field state in the next time step.

[0094] Among them, the preset continuous time step can be the termination condition of the iteration time window set in advance by the system when performing dynamic inference of graph structure. It usually represents the time range or the maximum number of sliding steps covered by the entire inference process. When the system slides to the termination time step, it stops the graph convolution iteration and outputs the complete future running prediction results, thereby controlling the model calculation depth and prediction cycle length.

[0095] Specifically, within the current consecutive time step, the initial features of each node and the edge connections between them are input into the spatiotemporal graph convolutional network. The spatiotemporal graph convolutional network aggregates the adjacency features of each node in the spatial dimension through graph convolution mechanism, and captures the dynamic evolution of the node state in the temporal dimension through temporal convolution or recurrent mechanism, thereby outputting the spatiotemporal dynamic features of each node in the current consecutive time step.

[0096] Based on the spatiotemporal dynamic characteristics of each node, the operational status of personnel, equipment and materials at the construction site at the target future time point is analyzed, including whether there are potential abnormal situations such as personnel violations, equipment overload, and material dislocation, and then the initial future operation prediction data is output.

[0097] Based on the evolution of node states and potential behavioral changes of each node in the initial future operation prediction data, the interaction strength and spatial relationship between nodes are reassessed, and the edge connection weights or topology are adjusted. For example, some high-frequency operation behavior connection edges are added, and edges that have lost spatial association are weakened, thereby generating updated edge connections that reflect future dynamic interaction patterns.

[0098] The updated edge connections obtained in the previous step are used as the structural basis for the next round of graph convolutional inference. Simultaneously, the time window is shifted forward by one time step, making the next time step the new current time step, ensuring that the graph inference process is updated synchronously in both structural and temporal dimensions. Graph convolutional inference and edge connection update operations are executed iteratively until the set termination time step is reached. Finally, the accumulated set of node prediction states and risk assessment results within this period are output as prediction data for future construction operations.

[0099] In this embodiment, by performing graph convolution calculation on the initial features and initial edge connections of nodes within the current continuous time step, spatiotemporal dynamic features with time evolution capability are extracted. Based on these features, the operating state of the construction site at a future target time is predicted. At the same time, the edge connection relationship in the graph structure is dynamically updated according to the prediction results, and the reasoning process is iteratively executed until a preset time range is reached. This enables continuous modeling of the construction site state and high-precision prediction of future situations, which can significantly improve the system's adaptability to complex working conditions, its ability to capture risk transmission paths, and its ability to perceive and control dynamic safety scenarios.

[0100] In one exemplary embodiment, such as Figure 6 As shown, the step of performing alarm adaptive analysis on the predicted future construction operation data to obtain the safety evidence chain analysis data of the construction site includes steps 602 to 606. Wherein:

[0101] Step 602: Identify the node behavior paths that deviate from the construction plan from the construction future operation prediction data, and extract potential abnormal trajectory sequences.

[0102] Step 604: Perform cross-modal consistency analysis on the potential abnormal trajectory sequence to obtain abnormal trajectory consistency analysis data.

[0103] Step 606: Perform risk causal chain analysis on the abnormal trajectory consistency analysis data to obtain safety evidence chain analysis data.

[0104] Among them, the node behavior path can be the trajectory of the state change of a certain node (such as construction personnel, equipment or materials) in a spatiotemporal graph structure within a continuous time step. It is usually a directed path sequence formed by the change of information such as the node's behavior state, spatial location, and interactive objects in the graph over time, and is used to describe the dynamic behavior process of the node in the construction cycle.

[0105] Among them, the potential abnormal trajectory sequence can be a sequence of nodes that deviate from the construction plan or safety specifications from the node behavior path. It usually includes nodes that exhibit unplanned operations, boundary crossing activities, abnormal stays, or sudden changes in the operation trajectory. By analyzing its evolution pattern, it is possible to make a preliminary judgment on possible abnormal events or potential risks in the construction site.

[0106] Cross-modal consistency analysis can compare the same trajectory point or behavior segment from different sensing channels (such as image recognition, thermal imaging, RFID trajectory, acoustic signals, etc.) to evaluate whether their descriptions are consistent in different modalities. If multiple modalities show the same or highly correlated abnormal signals for the same event, they are considered to be anomalies with high confidence, thereby improving the accuracy and robustness of anomaly detection.

[0107] Among them, the abnormal trajectory consistency analysis data can be structured result data generated through cross-modal consistency analysis. It records the response characteristics, consistency scores, conflict degree and time and space location of the abnormal trajectory in multiple modal dimensions. It is used for subsequent causal chain modeling and safety evidence chain generation, and is an important intermediate result for intelligent alarm and accident interpretation.

[0108] Among them, risk causal chain analysis can be based on the identified high-confidence abnormal trajectory, combined with the construction knowledge graph and experience rule base, to construct the causal reasoning relationship between abnormal events and their causes, triggering factors, propagation paths and possible consequences, forming a logically closed-loop risk event chain, thereby assisting the system in tracking the cause of events, predicting the risk spread trend and generating a traceable safety explanation path.

[0109] Specifically, the project analyzes the future operation prediction data of construction node by node to identify the sequence of nodes that deviate from the preset construction plan in the behavioral evolution path, such as personnel entering unauthorized areas, equipment operating in unplanned states, or materials having abnormal flow locations. Then, it extracts the potential abnormal trajectory sequence formed by these deviation behaviors.

[0110] For the identified potential abnormal trajectory sequences, information sources from multiple modalities, such as image recognition, thermal imaging, RFID trajectory, and vibration data, are invoked to perform cross-modal comparison and verification on each trajectory point to determine whether the behavioral states reflected by each modality are consistent. When a certain behavior is identified as a violation in the image and also shows an abnormal signal in thermal imaging or RFID, it is identified as an abnormal trajectory with high consistency, and structured abnormal trajectory consistency analysis data is generated.

[0111] By combining abnormal trajectory consistency analysis data with construction knowledge graphs and risk causal models, we can trace the preceding nodes, upstream and downstream impact paths, and possible related events that led to abnormal behavior, and construct a complete risk causal chain. Based on the risk causal chain, we can express the occurrence logic of abnormal events in the construction site through entity-relationship triples, and synchronously associate information such as video clips, sensor data snapshots, personnel identities, and work locations, and finally generate safety evidence chain analysis data containing causal reasoning results and multimodal evidence.

[0112] In this embodiment, by accurately identifying the node behavior paths that deviate from the construction plan from the future construction operation prediction data, potential abnormal trajectory sequences are extracted, and the consistency of the sequences is further verified under multiple perception modalities, which effectively improves the accuracy and confidence of anomaly detection. Based on this, combined with the consistency analysis results of abnormal trajectories, risk causal chain modeling is used to deeply explore the causes of events and potential propagation paths, and finally generate safety evidence chain analysis data with causal logic, modal correlation and spatiotemporal positioning capabilities. This can significantly enhance the system's cognitive, interpretive and traceability capabilities for complex construction anomalies, and improve the effectiveness and auditability of intelligent monitoring of engineering safety.

[0113] In one exemplary embodiment, such as Figure 7As shown, the step of performing risk causal chain analysis on the abnormal trajectory consistency analysis data to obtain the security evidence chain analysis data includes steps 702 to 706. Wherein:

[0114] Step 702: Perform conflict check on the abnormal trajectory consistency analysis data. If the check results indicate that there is a conflict in the abnormal trajectory consistency analysis data, perform causal path backtracking analysis on the abnormal trajectory consistency analysis data to obtain risk causal source data.

[0115] Step 704: Model the event environment relationship of the risk causal tracing data to obtain the abnormal event chain.

[0116] Step 706: Based on the abnormal event chain, logically deduce the interruption points of the process dependency path at the construction site to obtain safety evidence chain analysis data.

[0117] Among them, conflict checking can compare the results from different modalities or data sources in the consistency analysis data of abnormal trajectories to determine whether there are contradictions or inconsistencies in terms of time, space, behavior category, etc. For example, image recognition judges that the operation is illegal but the RFID trajectory shows that the person is not present, or thermal imaging is abnormal but the equipment vibration signal is normal. This process is used to detect abnormal discrepancies in multimodal information fusion as a prerequisite for triggering causal tracing analysis.

[0118] Among them, risk causal tracing data can be a structured data set obtained by the system inferring the cause, evolution process and potential impact relationship of risk events after tracing back the starting node, propagation path and triggering conditions of abnormal trajectories in the case of multimodal conflict detection, combined with knowledge graph and on-site process data.

[0119] Among them, event environment relationship modeling can be based on risk causal tracing data. The system further constructs the relationship between abnormal events and environmental elements of the construction site, including the time point, spatial area, participating personnel, operating equipment, weather conditions and operation stage of the event, etc. The event is embedded in its dynamic environmental context in the form of graph structure or causal network to form a complete and traceable semantic structure of abnormal behavior.

[0120] Among them, the abnormal event chain can be an ordered relational chain composed of one or more key event nodes with abnormal behavior. This chain connects the event nodes through causal relationships and environmental dependencies, reflecting the causes, triggering conditions, participating elements, spatial paths and propagation effects of abnormal events.

[0121] Among them, the interruption point of the process-dependent path can be a critical node in the construction process where an abnormal event causes a certain process to be unable to be transmitted or executed normally to downstream nodes. It usually manifests as a problem point such as incomplete work, undelivered resources, or unauthorized operation. This interruption point represents the actual degree of interference of the abnormality on the normal construction plan and is a reference for analyzing construction progress risks and safety hazards.

[0122] Logical deduction can be performed by identifying abnormal event chains and process interruption points, and then using a predefined process dependency model and rule engine to conduct a logical-level recursive analysis of the subsequent operation status to determine whether the abnormality will trigger a chain reaction, process conflict or safety accident, and generate a structured reasoning chain to build the final safety evidence chain and generate early warning response strategies.

[0123] Specifically, conflict checks are performed on the multimodal results in the abnormal trajectory consistency analysis data to identify whether different data sources give contradictory or inconsistent judgments on the same abnormal trajectory. When an information conflict is detected between the image recognition results and sensor data, RFID trajectory or thermal imaging information, causal path backtracking analysis is immediately initiated. Based on the operation records, historical process flow and risk knowledge graph of the construction site, the starting node, time point and related work unit that caused the conflict are gradually traced to generate risk causal traceability data that reflects the cause and propagation path of the event.

[0124] Based on risk causal tracing data, we further integrate on-site environmental parameters, personnel distribution, equipment status, and material flow paths to construct the correlation between abnormal events and the surrounding environment. Through graphical models or spatiotemporal data structures, we describe the environmental constraints, collaborative operation conditions, and potential impact range of the events, generating a complete abnormal event chain. This abnormal event chain records in detail the development process of the event, participating entities, related procedures, and their environmental adaptation conditions, providing a clear context for subsequent safety analysis.

[0125] By combining the abnormal event chain and the process dependency path at the construction site, the system automatically identifies work interruption nodes or potential process conflicts caused by abnormal events. It further uses process flow diagrams and construction schedule plans to conduct logical deductions, analyze the possible process chain break points and impact ranges caused by abnormalities, and integrates abnormal event chains, related process nodes, work conflict paths and environmental evidence to form structured safety evidence chain analysis data.

[0126] In this embodiment, conflict checks are performed on the consistency analysis data of abnormal trajectories to proactively identify information contradictions between multimodal data, thereby triggering causal path backtracking analysis to deeply explore the causes, triggering conditions, and propagation paths of abnormal events, generating risk causal tracing data. Furthermore, through event environment relationship modeling, risk events are embedded into the personnel, equipment, time, and space context of real construction scenarios to construct a logically clear and causally defined abnormal event chain. Finally, based on this event chain, logical deduction is performed on the interruption points in the construction process dependency path to identify the possible chain effects of abnormalities on subsequent work processes, generating safety evidence chain analysis data with traceability and upstream and downstream explanatory capabilities. This significantly improves the system's intelligent analysis capabilities for identifying the root causes of risks, attributing responsibility, and tracing accident sources in complex construction environments, enhancing the accuracy of safety supervision and decision support value.

[0127] In one exemplary embodiment, such as Figure 8 As shown, the process of performing spatiotemporal synchronization processing on the personnel identification data and the equipment sensor data to obtain a fusion feature vector of the on-site situation includes steps 802 to 806. Wherein:

[0128] Step 802: Perform time synchronization processing on personnel identification data and equipment sensor data to obtain personnel time synchronization data and equipment time synchronization data.

[0129] Step 804: Perform spatial registration processing on the personnel time synchronization data and the equipment time synchronization data to obtain personnel spatial registration data and equipment spatial registration data.

[0130] Step 806: Perform feature fusion on personnel spatial registration data and equipment spatial registration data to obtain the site situation fusion feature vector.

[0131] Time synchronization processing can be a process of uniformly calibrating the time axis of data collected from different sensing sources (such as facial recognition devices, sensors, cameras, etc.). By applying high-precision time synchronization protocols (such as PTP) and sliding window interpolation algorithms, time deviations caused by differences in sampling frequency, network transmission delays, or internal clock drift of devices are eliminated, thereby ensuring that all data are aligned under the same time reference and have time consistency.

[0132] Among them, personnel time synchronization data can be the alignment result of all sensory data (such as image recognition results, location information, behavior recognition sequences, etc.) related to construction personnel on a unified time axis after time synchronization processing.

[0133] Among them, the equipment time synchronization data can be the result of adjusting the sensor data (such as temperature, vibration, current, etc.) collected by various equipment (such as lifting machinery, conveying devices, and detection terminals) at the construction site to a unified time reference after time synchronization processing.

[0134] Spatial registration processing can map data collected by multiple sensors or sensing systems into a unified three-dimensional spatial coordinate system, so that spatial information from different sources has a consistent physical reference frame. Registration is usually achieved through positioning technology (such as UWB), image geometry inverse calculation or BIM model calibration, so as to realize the spatial alignment of data and accurate modeling of spatial relationships between entities.

[0135] Among them, personnel spatial registration data can be structured spatial data formed by uniformly mapping the behavior trajectory, location points, recognition images and other information of construction personnel to the three-dimensional coordinate system of the construction site after the spatial registration process is completed. This data reflects the real-time spatial distribution and movement trajectory of personnel in the scene.

[0136] Among them, equipment spatial registration data can be the unified expression of the spatial location information, installation area, operation boundary and other data of various construction equipment in the three-dimensional model coordinate system of the construction site after spatial registration processing, so as to ensure that the equipment status and its spatial location have a one-to-one correspondence.

[0137] Specifically, the timestamps of each data stream are uniformly calibrated using a precision network clock protocol (such as PTP) for personnel identification data; at the same time, in order to cope with the time difference caused by different device sampling frequencies and network latency, sliding window interpolation or resampling technology is used to time-align the device sensor data, thereby obtaining personnel time synchronization data and device time synchronization data respectively.

[0138] Personnel time synchronization data and equipment time synchronization data are mapped to the three-dimensional spatial coordinate system of the construction site. Specifically, spatial transformation and coordinate alignment of the data are achieved through UWB positioning system, BIM model coordinates, and camera intrinsic and extrinsic parameter matrices. Information such as personnel movement trajectories and equipment spatial distribution is re-encoded into position descriptions under a unified spatial reference, and output as personnel spatial registration data and equipment spatial registration data.

[0139] Based on the spatiotemporally aligned personnel and equipment spatial registration data, a joint feature representation of personnel and equipment is constructed at the semantic feature level. The deep feature extraction module performs vector encoding on modalities such as images, trajectories, vibrations, and temperatures. A cross-modal attention mechanism is introduced to dynamically weight and fuse multi-source information, generating a high-dimensional embedded representation vector as a fusion feature vector for the on-site situation.

[0140] In this embodiment, by performing time synchronization processing on personnel identification data and equipment sensing data, the inconsistency problem in the collection frequency and timestamp of multi-source data is effectively eliminated, ensuring that the information has comparability and temporal continuity under a unified time dimension. Furthermore, through spatial registration processing, the synchronized personnel and equipment data are mapped to a unified three-dimensional coordinate system of the construction site, achieving accurate alignment of data at the spatial level. Finally, through deep feature extraction and multimodal fusion mechanisms, the registered personnel and equipment data are efficiently integrated to generate a fusion feature vector that can comprehensively express the dynamic state of the construction site, significantly improving the spatiotemporal consistency and semantic expression capability of the sensing data, and providing a high-quality input foundation for downstream behavior modeling, risk prediction, and intelligent monitoring.

[0141] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0142] Based on the same inventive concept, this application also provides an intelligent monitoring device for engineering management based on multimodal perception, used to implement the aforementioned intelligent monitoring method for engineering management based on multimodal perception. For example... Figure 9 As shown, a multimodal perception-based intelligent monitoring device for engineering management is provided, including: a construction data acquisition module 902, a construction data fusion module 904, a construction status prediction module 906, a construction safety prediction module 908, and a construction safety monitoring module 910. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the multimodal perception-based intelligent monitoring device for engineering management provided below can be found in the limitations of the multimodal perception-based intelligent monitoring method for engineering management described above, and will not be repeated here.

[0143] The modules in the aforementioned intelligent monitoring device for engineering management based on multimodal perception can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0144] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores server data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When the computer program is executed by the processor, it implements an intelligent monitoring method for engineering management based on multimodal perception.

[0145] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0146] In one embodiment, an intelligent monitoring system for engineering management based on multimodal perception is also provided, the system including: a terminal and computer equipment;

[0147] The terminal is used to acquire personnel identification data and equipment sensor data corresponding to the construction site, and send the personnel identification data and equipment sensor data to the computer device;

[0148] Computer equipment is used to perform spatiotemporal synchronization processing on personnel identification data and equipment sensor data to obtain a fusion feature vector of the on-site situation.

[0149] Computer equipment is used to fuse feature vectors based on on-site conditions, infer the future operation of the construction site, and obtain predictive data for the future operation of the construction.

[0150] Computer equipment is used to perform alarm-adaptive analysis on prediction data of future construction operations to obtain safety evidence chain analysis data at the construction site.

[0151] Computer equipment is used to analyze data based on the security evidence chain to monitor construction safety at the construction site and obtain risk monitoring data for the construction site.

[0152] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0153] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.

[0154] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0155] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0156] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0157] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for intelligent monitoring of engineering management based on multimodal perception, characterized in that, The method includes: Acquire personnel identification data and equipment sensor data corresponding to the construction site; The personnel identification data and the equipment sensor data are processed in a spatiotemporal synchronization manner to obtain a fusion feature vector of the on-site situation. Based on the fusion of feature vectors according to the site conditions, the future operation of the construction site is inferred to obtain prediction data of the future operation of the construction. An alarm-adaptive analysis is performed on the predicted data of future construction operations to obtain safety evidence chain analysis data of the construction site. Based on the safety evidence chain analysis data, construction safety monitoring is carried out at the construction site to obtain risk monitoring data for the construction site.

2. The method according to claim 1, characterized in that, The step of fusing feature vectors based on the site conditions to infer the future operation of the construction site and obtain future construction operation prediction data includes: Information on construction personnel, construction equipment, and construction materials is identified from the personnel identification data and the equipment sensor data. Using the construction personnel information, construction equipment information, and construction material information as nodes, a spatiotemporal diagram structure corresponding to the construction site is constructed. The fusion feature vector of the site conditions is used as the input feature of the spatiotemporal graph structure, and graph structure dynamic reasoning is performed on the spatiotemporal graph structure to obtain the prediction data of the future operation of the construction.

3. The method according to claim 2, characterized in that, The step of using the fused feature vector of the site conditions as input features of the spatiotemporal graph structure, and performing graph structure dynamic reasoning on the spatiotemporal graph structure to obtain the predicted data for future construction operations includes: The fused feature vectors of the on-site situation are respectively assigned to each node in the spatiotemporal graph structure, serving as the initial features of each node; Based on the spatial distance, interaction frequency, and physical connection relationship between the construction personnel information, the construction equipment information, and the construction material information, an initial edge connection is constructed between the initial features of any node and the other nodes. The initial features of each node and the initial edges are connected and input into the spatiotemporal graph convolutional inference model corresponding to the construction site to obtain the prediction data for the future operation of the construction.

4. The method according to claim 3, characterized in that, The step of inputting the initial features of each node and the initial edges into the spatiotemporal graph convolutional inference model corresponding to the construction site to obtain the predicted data for the future operation of the construction includes: Based on the current continuous time step, graph convolution calculation is performed on the initial features of each node and the initial edge connections to obtain the spatiotemporal dynamic features corresponding to each node; Based on the aforementioned spatiotemporal dynamic characteristics, the operating status of the construction site at the target future time is predicted to obtain initial future operating prediction data; Based on the initial future operation prediction data, the connection relationship of each node is updated to obtain each updated edge connection; Each of the updated edge connections is used as the initial edge connection, and the next consecutive time step is used as the current consecutive time step; Return to the step of performing graph convolution calculation on the initial features of each node and the initial edge connections based on the current continuous time step to obtain the spatiotemporal dynamic features corresponding to each node, until the next continuous time step triggers the preset continuous time step, and use the initial future operation prediction data as the construction future operation prediction data.

5. The method according to claim 1, characterized in that, The alarm adaptive analysis of the predicted future operation data of the construction site is performed to obtain the safety evidence chain analysis data of the construction site, including: Identify the node behavior paths that deviate from the construction plan from the predicted data of future construction operations, and extract potential abnormal trajectory sequences; Cross-modal consistency analysis was performed on the potential abnormal trajectory sequence to obtain abnormal trajectory consistency analysis data; Risk causal chain analysis is performed on the abnormal trajectory consistency analysis data to obtain the security evidence chain analysis data.

6. The method according to claim 5, characterized in that, The step of performing risk causal chain analysis on the abnormal trajectory consistency analysis data to obtain the security evidence chain analysis data includes: The abnormal trajectory consistency analysis data is subjected to conflict checking. If the check results indicate that there is a conflict in the abnormal trajectory consistency analysis data, causal path backtracking analysis is performed on the abnormal trajectory consistency analysis data to obtain risk causal source tracing data. The event environment relationship model is performed on the aforementioned risk causal tracing data to obtain an abnormal event chain; Based on the abnormal event chain, logical deduction is performed on the interruption points of the process dependency path at the construction site to obtain the safety evidence chain analysis data.

7. The method according to claim 1, characterized in that, The process of performing spatiotemporal synchronization processing on the personnel identification data and the equipment sensor data to obtain a fusion feature vector of the on-site situation includes: The personnel identification data and the device sensor data are processed for time synchronization to obtain personnel time synchronization data and device time synchronization data. Spatial registration processing is performed on the personnel time synchronization data and the equipment time synchronization data to obtain personnel spatial registration data and equipment spatial registration data; The spatial registration data of personnel and the spatial registration data of equipment are fused to obtain the fused feature vector of the field situation.

8. An intelligent monitoring system for engineering management based on multimodal perception, characterized in that, The system includes: a terminal and computer equipment; The terminal is used to acquire personnel identification data and equipment sensor data corresponding to the construction site, and send the personnel identification data and equipment sensor data to the computer device. The computer device is used to perform spatiotemporal synchronization processing on the personnel identification data and the device sensor data to obtain a fusion feature vector of the scene situation. The computer device is used to fuse feature vectors based on the site conditions, infer the future operation of the construction site, and obtain prediction data for the future operation of the construction site. The computer equipment is used to perform alarm adaptive analysis on the construction future operation prediction data to obtain the safety evidence chain analysis data of the construction site. The computer equipment is used to perform construction safety monitoring on the construction site based on the security evidence chain analysis data, and to obtain risk monitoring data of the construction site.

9. The system according to claim 8, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. An intelligent monitoring device for engineering management based on multimodal perception, characterized in that, The device includes: The construction data acquisition module is used to acquire personnel identification data and equipment sensor data corresponding to the construction site. The construction data fusion module is used to perform spatiotemporal synchronization processing on the personnel identification data and the equipment sensor data to obtain a fusion feature vector of the site conditions. The construction status prediction module is used to fuse feature vectors based on the site conditions, infer the future operation of the construction site, and obtain construction future operation prediction data. The construction safety prediction module is used to perform alarm adaptive analysis on the construction future operation prediction data to obtain the safety evidence chain analysis data of the construction site. The construction safety monitoring module is used to monitor the construction site based on the safety evidence chain analysis data, and obtain the risk monitoring data of the construction site.

Images