Job risk real-time monitoring system based on multi-angle video and AI recognition

Abstract

The application discloses a job risk real-time monitoring system based on multi-angle video and AI recognition and belongs to the technical field of video monitoring, comprising a job data extraction module, an interaction prediction module, a posture prediction module, a risk decision module and an early warning execution module, the job data extraction module collects and pre-processes construction operation data, the interaction prediction module predicts the interaction risk probability between dynamic entities, the posture prediction module predicts the motion trajectory and posture instability probability of personnel, the risk decision module outputs a risk grade determination result, and the early warning execution module adopts a distributed computing architecture, executes a multi-channel early warning mechanism at an edge computing node and determines the release condition of the risk grade in real time. Through the extraction of three-dimensional space pose sequences, motion trajectory sequences, three-dimensional human skeleton key point time series data and attribute features and the adoption of edge computing, low-delay closed-loop early warning is realized, and the problems of perception loss, prediction blank, early warning lag and open loop are solved.

Description

Technical Field

[0001] This invention belongs to the field of video monitoring technology, specifically relating to a real-time monitoring system for operational risks based on multi-angle video and AI recognition. Background Technology

[0002] In high-risk environments such as industrial, energy, and construction operations, ensuring the safety of operators and the site environment is crucial for project management. In recent years, to meet the increasingly stringent requirements for on-site safety management, real-time operational risk monitoring systems have been widely applied and developed. These systems aim to utilize modern information technology to continuously collect and monitor data on the work environment, personnel status, and operational procedures by integrating various sensors and monitoring equipment deployed on-site. The core function is to efficiently and proactively identify potential violations that could lead to safety accidents based on safety regulations and risk models. After identifying a risk, the system generates and sends alarm signals to achieve early warning, rapid response, and effective control of the risk, thereby comprehensively improving the efficiency of on-site safety management and accident prevention capabilities.

[0003] To address the issue that existing monitoring systems can only identify static violations and cannot predict dynamic interaction and posture risks in real time, current technologies use two-dimensional video image analysis to identify the static violation status of a single target. However, this approach suffers from a lack of perception of the three-dimensional spatial structure of the construction site, an inability to obtain the precise spatial location, motion trajectory, and attribute characteristics of dynamic entities, an inability to analyze the complex spatiotemporal interactions between personnel and machinery, and between personnel and the environment, and an inability to predict the future motion trajectory and premonitions of posture instability of personnel. Furthermore, data analysis and decision-making heavily rely on cloud computing, resulting in high overall system latency and a lack of an automatic closed-loop confirmation mechanism after early warning. Consequently, the system may miss dynamic interaction risks, falsely report static information caused by occlusion, and experience severe delays in early warning information, making it difficult to effectively intervene in advance of impending collisions and falls. Summary of the Invention

[0004] The purpose of this invention is to provide a real-time monitoring system for operational risks based on multi-angle video and AI recognition. Through multi-angle video acquisition and AI modeling, it achieves accurate three-dimensional perception, interaction, and early prediction of posture risks of dynamic entities. Furthermore, it relies on edge computing to achieve low-latency hierarchical early warning and closed-loop confirmation, thus solving the problems of missing perception, insufficient prediction, and delayed early warning in traditional monitoring systems.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A real-time operational risk monitoring system based on multi-angle video and AI recognition includes:

[0007] The operation data extraction module is used to collect and preprocess construction operation data, and extract the three-dimensional spatial pose sequence, motion trajectory sequence, three-dimensional human skeleton key point time series data and attribute features of dynamic entities in the construction site.

[0008] The interaction prediction module is used to input three-dimensional spatial pose sequences, motion trajectory sequences and attribute features into a spatiotemporal interaction risk prediction model with a spatiotemporal graph neural network structure to predict the probability of interaction risk between dynamic entities.

[0009] The posture prediction module is used to input the time-series data of the key points of the three-dimensional human skeleton into the human posture intention prediction model to predict the human's motion trajectory and the probability of posture instability.

[0010] The risk decision module is used to construct a comprehensive risk decision model based on a multilayer perceptron, based on the probability of interaction risk, the probability of motion trajectory and attitude instability, and to train and optimize it using a labeled dataset containing historical risk events, set the operation risk judgment logic, and output the risk level judgment result.

[0011] The early warning execution module is used to execute a multi-channel early warning mechanism on edge computing nodes based on the risk level determination results, using a distributed computing architecture, and to determine the risk level clearance status in real time.

[0012] Preferably, in the work data extraction module, the process of collecting and preprocessing construction work data includes:

[0013] Construction operation data includes spatiotemporal video data of the construction site, mechanical operation parameter data, and real-time location status data of personnel;

[0014] Real-time spatiotemporal video data is collected using multi-angle fixed cameras, personnel wearable cameras, and mobile device cameras deployed at the construction site.

[0015] By using sensors deployed on construction machinery and position sensors on personnel's wearable devices, the system collects operational parameter data of the machinery and real-time location status data of the personnel.

[0016] This ensures that multi-angle fixed cameras, wearable cameras, mobile device cameras and sensors are synchronized with the central time server, providing a unified time reference for the collected spatiotemporal video data, operational parameter data and real-time location status data;

[0017] Perform spatial coordinate system registration. Through camera calibration methods, obtain the internal parameter matrix and external parameter matrix of the multi-angle fixed camera. Use this matrix to establish the mapping relationship between world coordinate system points and pixel coordinate system points. Unify the spatiotemporal video data to the same world coordinate system of the construction site, generate preprocessed spatiotemporal video data, and convert the real-time position status data of personnel to the same world coordinate system of the construction site.

[0018] Perform sensor data cleaning, which involves denoising and interpolating the machine's operating parameter data and the personnel's real-time position status data to obtain pre-processed operating parameter data and real-time position status data.

[0019] Preferably, in the work data extraction module, the process of extracting the three-dimensional spatial pose sequence, motion trajectory sequence, and three-dimensional human skeleton key point time series data and attribute features of dynamic entities in the construction site includes:

[0020] Dynamic entities include personnel entities, machine entities, and material entities;

[0021] Based on the preprocessed spatiotemporal video data, a multi-view geometric reconstruction algorithm is used to extract the three-dimensional spatial pose sequence and motion trajectory sequence of dynamic entities in the construction site.

[0022] In spatiotemporal video data, run an instance segmentation model to obtain dynamic entities in multiview. Figure 2 Bounding boxes and segmentation masks in 3D images;

[0023] By using segmentation masks, internal parameter matrices, and external parameter matrices, a 3D point cloud model of a dynamic entity is generated through a multi-view contour reconstruction method.

[0024] Perform 3D bounding box fitting and pose estimation on the 3D point cloud model of a dynamic entity to obtain the 3D spatial pose of the dynamic entity.

[0025] A multi-target tracking algorithm is used to temporally correlate the three-dimensional spatial poses in consecutive frames to generate a three-dimensional spatial pose sequence and a motion trajectory sequence of dynamic entities.

[0026] For human entities, extract the temporal data of key points of the three-dimensional human skeleton from the preprocessed spatiotemporal video data;

[0027] In spatiotemporal video data, a two-dimensional human pose estimation algorithm is run to extract human entities in multi-view... Figure 2 Key points of the two-dimensional human skeleton in a 3D image;

[0028] Using the internal parameter matrix and the external parameter matrix, triangulation reconstruction of key points of the two-dimensional human skeleton is performed, and the coordinates of key points of the three-dimensional human skeleton are calculated.

[0029] A multi-target tracking algorithm is used to temporally correlate the coordinates of key points of the three-dimensional human skeleton in consecutive frames to generate temporal data of key points of the three-dimensional human skeleton of a person.

[0030] The pre-processed machine operation parameter data and the real-time location status data of personnel are integrated as attribute features of dynamic entities.

[0031] Preferably, in the interaction prediction module, the process of predicting the probability of interaction risk between dynamic entities includes:

[0032] Construct a time series graph structure, which includes graph nodes, graph edges, node features, and edge features at each time step;

[0033] Define the extracted dynamic entities as graph nodes;

[0034] Assign the 3D spatial pose sequence, motion trajectory sequence, and attribute features as node features of the graph nodes;

[0035] The dynamic spatiotemporal relationships between dynamic entities are defined as edges of a graph;

[0036] Calculate the relative spatial distance and relative motion velocity between dynamic entities, and assign the calculated relative spatial distance and relative motion velocity as edge features of the graph.

[0037] The temporal graph structure is input into the spatiotemporal interaction risk prediction model. The spatiotemporal interaction risk prediction model adopts a spatiotemporal graph neural network structure. By learning the temporal changes of graph node and edge features, it calculates and outputs the probability of interaction risk between dynamic entities within a future preset time window.

[0038] Spatial graph convolutional layers are used to aggregate node features and edge features at each time step to generate spatial aggregated features;

[0039] Temporal convolutional layers are used to learn the changing patterns of spatial aggregate features at consecutive time steps;

[0040] Based on the change pattern, the probability of interaction risk is calculated and output through a fully connected layer.

[0041] Preferably, in the attitude prediction module, the process of predicting the motion trajectory and the probability of attitude instability of a person includes:

[0042] At each time step of the three-dimensional human skeleton key point time series data, a preset root node is selected as the reference point, and the three-dimensional relative coordinates of each key point in the three-dimensional human skeleton key point relative to the reference point are calculated to generate normalized time series data.

[0043] Normalized time-series data is input into the human posture and intention prediction model. The human posture and intention prediction model adopts a long short-term memory network structure. The long short-term memory network structure learns the temporal dependence of normalized time-series data through internal gating units and outputs posture feature vectors.

[0044] The posture feature vector is input into the trajectory prediction layer, which is used to calculate and output the coordinates of the three-dimensional human skeleton key points of the person within a preset time window in the future, as the person's motion trajectory.

[0045] The posture feature vector is input into the instability classification layer, which calculates and outputs the probability of posture instability of the person.

[0046] Preferably, in the risk decision-making module, the process of constructing a comprehensive risk decision-making model based on a multilayer perceptron includes:

[0047] The probability of interactive risks, the trajectory of personnel and the probability of posture instability are spliced ​​and flattened to generate the input feature vector of the risk comprehensive decision model.

[0048] A risk comprehensive decision-making model with a multilayer perceptron structure is constructed, which includes an input layer, a hidden layer, and an output layer.

[0049] The input layer receives the input feature vector;

[0050] Hidden layers perform high-dimensional feature mapping on the input feature vector using non-linear activation functions;

[0051] The output layer outputs a linear output value corresponding to the risk level, which is divided into emergency, warning, alert, and normal levels.

[0052] Preferably, in the risk decision-making module, the process of training and optimizing using a labeled dataset containing historical risk events includes:

[0053] Historical risk events include violations, near-accidents, and accidents;

[0054] The historical interaction risk probability, historical personnel movement trajectory and historical posture instability probability in the labeled dataset are concatenated and flattened to generate training feature vectors.

[0055] Input the trained feature vectors into the risk comprehensive decision-making model to obtain linear output values;

[0056] Calculate the loss function value between the linear output value and the corresponding true risk level label in the labeled dataset;

[0057] The backpropagation algorithm is used to update the network weight parameters in the multilayer perceptron structure until the loss function value converges, thus obtaining the trained and optimized risk comprehensive decision model.

[0058] Preferably, in the risk decision-making module, the process of setting the operational risk assessment logic and outputting the risk level assessment result includes:

[0059] The real-time generated input feature vector is imported into the trained and optimized risk comprehensive decision-making model to obtain a linear output value.

[0060] A multi-class classification function is used to calculate the predicted score for each risk level.

[0061] The risk level with the highest predicted score is selected as the final risk level determination.

[0062] Preferably, in the early warning execution module, the process of executing a multi-channel early warning mechanism on edge computing nodes using a distributed computing architecture based on the risk level determination result includes:

[0063] When the risk level is determined to be emergency, targeted early warning information is sent to the wearable devices of the personnel involved in the risk, the control console of the control room of the relevant machinery, and the automatic control system of the machinery.

[0064] When the risk level assessment result is warning level, targeted early warning information is sent to the wearable devices of the personnel involved in the risk and the terminals of the safety management personnel.

[0065] When the risk level assessment result is at the warning level, a targeted early warning message is sent to the safety management personnel's terminal.

[0066] Preferably, in the early warning execution module, the process of determining the clearance status of the risk level in real time includes:

[0067] After sending targeted early warning information, the risk decision module continuously outputs the risk level determination results for subsequent time steps;

[0068] If the risk level assessment result of the subsequent time step is normal, then the risk level is determined to be lifted, and the sending of targeted early warning information will be stopped.

[0069] If the risk level assessment result in subsequent time steps is not at the normal level, the risk level is determined to be not resolved, and targeted early warning information will continue to be sent.

[0070] The beneficial effects of this invention are:

[0071] This invention solves the problems of lack of three-dimensional spatial structure perception and inability to obtain accurate position information caused by existing technologies relying on two-dimensional image analysis by collecting and extracting three-dimensional spatial pose sequences, motion trajectory sequences, three-dimensional human skeleton key point time sequence data and attribute features of dynamic entities, and realizes high-precision three-dimensional reconstruction and continuous tracking of dynamic entities at construction sites.

[0072] This invention utilizes a spatiotemporal graph neural network structure and a temporal prediction network structure, combined with a multilayer perceptron for risk decision-making. It solves the problem that existing technologies can only identify static violations and cannot predict dynamic risks. It achieves accurate prediction of the risk of interaction between dynamic entities, the probability of future movement trajectory and posture instability of personnel, and has the ability to proactively intervene before accidents occur.

[0073] This invention employs a distributed computing architecture to perform real-time judgment and early warning on edge computing nodes, and has the function of real-time judgment of risk level clearance. It solves the problems of serious delay in early warning information and lack of early warning closed-loop confirmation mechanism caused by existing technologies relying on cloud computing, and ensures extremely low latency of system early warning and effectiveness of closed-loop management. Attached Figure Description

[0074] Figure 1 This is a structural block diagram of the system of the present invention;

[0075] Figure 2 This is a simulation test diagram of the normal monitoring stage during the verification process of this invention;

[0076] Figure 3 This is a simulation test diagram of the prompt-level early warning stage during the verification process of this invention;

[0077] Figure 4 This is a simulation test diagram of the warning level early warning stage during the verification process of this invention;

[0078] Figure 5 This is a simulation test diagram of the emergency warning stage during the verification process of this invention. Detailed Implementation

[0079] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0080] Example 1: As Figure 1 As shown, the real-time operational risk monitoring system based on multi-angle video and AI recognition includes:

[0081] The operation data extraction module is used to collect and preprocess construction operation data, and extract the three-dimensional spatial pose sequence, motion trajectory sequence, three-dimensional human skeleton key point time sequence data and attribute features of dynamic entities in the construction site.

[0082] Collect and preprocess construction operation data, which includes spatiotemporal video data of the construction site, mechanical operation parameter data, and real-time location status data of personnel.

[0083] Multi-angle fixed cameras, personnel wearable cameras, and mobile device cameras deployed at the construction site are used to collect multi-source heterogeneous spatiotemporal video data in real time.

[0084] By using sensors deployed on construction machinery and position sensors on personnel's wearable devices, data on the machinery's operating parameters and the personnel's real-time location status are collected.

[0085] This ensures that multi-angle fixed cameras, wearable cameras, mobile device cameras and sensors are synchronized with the central time server, providing a unified time reference for the collected spatiotemporal video data, operational parameter data and real-time location status data.

[0086] Perform spatial coordinate system registration and obtain the internal parameter matrix K and external parameter matrix of the multi-angle fixed camera using camera calibration methods. Used to establish world coordinate system points The mapping relationship between the data and the pixel coordinate system point p unifies the spatiotemporal video data to the same world coordinate system of the construction site, generating preprocessed spatiotemporal video data, and converting the real-time location status data of personnel to the same world coordinate system of the construction site. The calculation formula for the mapping relationship is as follows:

[0087] ;

[0088] in, Representing a two-dimensional point in pixel coordinates, u and v are the column and row coordinates of the pixel in the image, respectively. Represents a three-dimensional point in the world coordinate system. These are the three-dimensional coordinates of a point in space within the world coordinate system. is the scale factor, K is a 3×3 internal parameter matrix containing the focal length and principal point, R is a 3×3 rotation matrix, and T is a 3×1 translation vector.

[0089] Perform sensor data cleaning, which involves denoising and interpolating the machine's operating parameter data and the personnel's real-time position status data to obtain pre-processed operating parameter data and real-time position status data.

[0090] Dynamic entities include personnel entities, machine entities, and material entities.

[0091] Based on the preprocessed spatiotemporal video data, a multi-view geometric reconstruction algorithm is used to extract the three-dimensional spatial pose sequence and motion trajectory sequence of dynamic entities in the construction site.

[0092] In spatiotemporal video data, run an instance segmentation model to obtain dynamic entities in multiview. Figure 2 Bounding boxes and segmentation masks in 3D images.

[0093] By utilizing segmentation masks, internal parameter matrices, and external parameter matrices, a 3D point cloud model of a dynamic entity is generated through a multi-view contour reconstruction method.

[0094] 3D bounding box fitting and pose estimation are performed on the 3D point cloud model of a dynamic entity to obtain the 3D spatial pose of the dynamic entity.

[0095] A multi-target tracking algorithm is used to temporally correlate the three-dimensional spatial poses in consecutive frames to generate a three-dimensional spatial pose sequence and a motion trajectory sequence of dynamic entities.

[0096] For human entities, the temporal data of key points of the three-dimensional human skeleton is extracted from the preprocessed spatiotemporal video data.

[0097] In spatiotemporal video data, a two-dimensional human pose estimation algorithm is run to extract human entities in multi-view... Figure 2 Key points of a two-dimensional human skeleton in a 3D image.

[0098] Using internal and external parameter matrices, triangulation reconstruction of key points of the two-dimensional human skeleton is performed, and the coordinates of key points of the three-dimensional human skeleton are calculated.

[0099] A multi-target tracking algorithm is used to temporally correlate the coordinates of key points of the three-dimensional human skeleton in consecutive frames, generating temporal data of key points of the three-dimensional human skeleton of a person.

[0100] The pre-processed machine operation parameter data and the real-time location status data of personnel are integrated as attribute features of dynamic entities.

[0101] The machine's operating parameter data is associated with the corresponding machine entity based on the machine's unique identifier.

[0102] The real-time location status data of personnel is associated with the corresponding personnel entity based on the personnel's unique identifier.

[0103] The associated operation parameter data and real-time location status data are unified as attribute features of the dynamic entity.

[0104] The interaction prediction module is used to input three-dimensional spatial pose sequences, motion trajectory sequences and attribute features into a spatiotemporal interaction risk prediction model with a spatiotemporal graph neural network structure to predict the probability of interaction risks between dynamic entities.

[0105] Construct a time series graph structure, which includes graph nodes, graph edges, node features, and edge features at each time step.

[0106] The extracted dynamic entities are defined as graph nodes.

[0107] The three-dimensional spatial pose sequence, motion trajectory sequence, and attribute features are assigned as node features of the graph nodes.

[0108] The dynamic spatiotemporal relationships between dynamic entities are defined as edges of a graph.

[0109] Calculate the relative spatial distance and relative motion velocity between dynamic entities, and assign the calculated relative spatial distance and relative motion velocity as edge features of the graph.

[0110] The temporal graph structure is input into the spatiotemporal interaction risk prediction model. The spatiotemporal interaction risk prediction model adopts a spatiotemporal graph neural network structure. By learning the temporal changes of graph node and edge features, it calculates and outputs the probability of interaction risk between dynamic entities within a future preset time window.

[0111] Spatial graph convolutional layers are used to aggregate node features and edge features at each time step to generate spatial aggregated features.

[0112] Temporal convolutional layers are used to learn the variation patterns of spatial aggregated features over consecutive time steps.

[0113] Based on the change pattern, the probability of interaction risk is calculated and output through a fully connected layer.

[0114] The spatiotemporal interaction risk prediction model is based on a spatiotemporal graph neural network (AI deep learning architecture). It first processes input data such as 3D spatial pose sequences and motion trajectory sequences using AI recognition technology. For example, it uses instance segmentation AI models to identify the boundary features of dynamic entities (personnel, machinery, materials) and multi-target tracking AI algorithms to identify the continuous motion trajectories of entities. Then, through the AI ​​logic of the graph neural network, it defines dynamic entities as graph nodes and the spatiotemporal relationships between entities as graph edges, learning the temporal change patterns of nodes and edges. Finally, it identifies and predicts the probability of interaction risks between entities within a preset time window, representing AI-powered intelligent identification and risk prediction of dynamic entity interaction relationships.

[0115] The posture prediction module is used to input the time-series data of key points of the three-dimensional human skeleton into the human posture intention prediction model to predict the human's motion trajectory and the probability of posture instability.

[0116] At each time step of the 3D human skeleton key point time series data, a preset root node is selected as the reference point, and the 3D relative coordinates of each key point in the 3D human skeleton key point with respect to the reference point are calculated to generate normalized time series data.

[0117] Normalized time-series data is input into the human posture and intention prediction model. The human posture and intention prediction model adopts a long short-term memory network structure. The long short-term memory network structure learns the temporal dependence of normalized time-series data through internal gating units and outputs posture feature vectors.

[0118] The posture feature vector is input into the trajectory prediction layer, which calculates and outputs the coordinates of the three-dimensional human skeleton key points within a preset time window in the future, as the movement trajectory of the person.

[0119] The posture feature vector is input into the instability classification layer, which calculates and outputs the probability of posture instability of the person. , where z is the linear output of the unstable classification layer and e is the natural constant.

[0120] The risk decision module is used to construct a comprehensive risk decision model based on a multilayer perceptron, based on the probability of interaction risk, the probability of motion trajectory and attitude instability, and to train and optimize it using a labeled dataset containing historical risk events. It sets the logic for judging the operation risk and outputs the risk level judgment result.

[0121] The human posture intention prediction model is based on a Long Short-Term Memory Network (LSTM, an AI temporal prediction model). It first extracts the two-dimensional skeletal key points of a person through a two-dimensional human posture estimation algorithm (AI visual recognition technology), and then obtains the three-dimensional skeletal key point temporal data through triangulation reconstruction (AI spatial geometry calculation logic). Subsequently, the LSTM model learns the temporal dependence of this temporal data through AI algorithms, identifies the dynamic change pattern of human posture, and then predicts the future movement trajectory of the person (AI recognition of posture intention) and the probability of posture instability (AI recognition of risk posture features), realizing the AI ​​intelligent transformation from posture data to intention and risk.

[0122] The probability of interactive risks, the trajectory of movement of personnel, and the probability of posture instability are spliced ​​and flattened to generate the input feature vector of the risk comprehensive decision-making model.

[0123] A risk comprehensive decision-making model with a multilayer perceptron structure is constructed, which includes an input layer, a hidden layer, and an output layer.

[0124] The input layer receives the input feature vector.

[0125] Hidden layers perform high-dimensional feature mapping on the input feature vector using non-linear activation functions.

[0126] The output layer outputs a linear output value corresponding to the risk level, which is divided into emergency, warning, alert, and normal levels.

[0127] Historical risk events include violations, near-accidents, and accidents.

[0128] The historical interaction risk probability, historical personnel movement trajectory, and historical posture instability probability in the labeled dataset are concatenated and flattened to generate training feature vectors.

[0129] The trained feature vectors are input into the risk comprehensive decision model to obtain linear output values.

[0130] Calculate the loss function value between the linear output value and the corresponding true risk level label in the labeled dataset.

[0131] The backpropagation algorithm is used to update the network weight parameters in the multilayer perceptron structure until the loss function value converges, thus obtaining the trained and optimized risk comprehensive decision model.

[0132] The real-time generated input feature vector is imported into the trained and optimized risk comprehensive decision model to obtain a linear output value.

[0133] A multi-class classification function is used to calculate the predicted score for each risk level. Where j is the index of the current risk level, and C is the total number of risk level categories. This represents the linear output value of the output layer for the j-th risk level. The linear output value for the k-th risk level.

[0134] Select predicted score The risk level with the highest score is used as the final risk level determination result.

[0135] The early warning execution module is used to execute a multi-channel early warning mechanism on edge computing nodes based on the risk level determination results, using a distributed computing architecture, and to determine the risk level clearance status in real time.

[0136] When the risk level is determined to be emergency, targeted early warning information is sent to the wearable devices of the personnel involved in the risk, the control consoles in the operating rooms of the relevant machinery, and the automatic control systems of the machinery.

[0137] When the risk level is determined to be warning level, targeted early warning information is sent to the wearable devices of the personnel involved in the risk and the terminals of the safety management personnel.

[0138] When the risk level assessment result is at the warning level, a targeted early warning message is sent to the safety management personnel's terminal.

[0139] After sending targeted early warning information, the risk decision module continuously outputs the risk level determination results for subsequent time steps.

[0140] If the risk level assessment result of the subsequent time step is normal, then the risk level is determined to be lifted, and the sending of targeted early warning information will be stopped.

[0141] If the risk level assessment result in subsequent time steps is not at the normal level, the risk level is determined to be not resolved, and targeted early warning information will continue to be sent.

[0142] The extracted dynamic entity's 3D spatial pose sequence, motion trajectory sequence, and risk level determination results are presented in real time in the 3D visualization interface.

[0143] First, hardware, including cameras, sensors, a central time server, and edge computing nodes, is deployed at the construction site, and a distributed computing architecture is built. Second, a labeled dataset containing historical risk events is used to train and optimize the risk comprehensive decision-making model, the spatiotemporal interaction risk prediction model, and the personnel posture and intention prediction model, so as to obtain the trained and optimized model.

[0144] The operation data extraction module begins to collect and preprocess construction operation data in real time, as well as extract the three-dimensional spatial pose sequence, motion trajectory sequence, and three-dimensional human skeleton key point time series data and attribute features of dynamic entities;

[0145] The system performs real-time risk assessment at edge computing nodes. The interactive prediction module takes into account the 3D spatial pose sequence, motion trajectory sequence and attribute features to predict the probability of interaction risk between dynamic entities. The posture prediction module takes into account the time series data of key points of the 3D human skeleton to predict the motion trajectory and posture instability probability of the person. The risk decision module, based on the interaction risk probability, motion trajectory and posture instability probability, imports the trained and optimized risk comprehensive decision model and outputs the risk level assessment result.

[0146] The early warning execution module receives the risk level determination result at the edge computing node. If the determination is at the emergency, warning, or alert level, a multi-channel early warning mechanism is triggered. If the determination is at the normal level, no early warning is executed. At the same time, the relevant 3D sequence and risk level are displayed in real time on the 3D visualization interface. Finally, the early warning execution module performs an early warning closed-loop determination. After an early warning is triggered, the system continues to determine the subsequent risk level. If the subsequent risk level determination result is at the normal level, the risk level is determined to be lifted and the early warning stops. If the subsequent risk level is not lifted, early warning information continues to be sent.

[0147] Through simulation experiments, the effectiveness of the system in this embodiment is verified. In a simulated construction site, under typical scenarios where workers approach and enter the dangerous area of ​​mechanical operation, the system has the ability to predict dynamic interaction risks and attitude instability risks in real time and execute multi-level early warnings.

[0148] The construction site was simulated using the MATLAB simulation environment. The simulation scenario consisted of one worker (simulated as a 3D skeleton), one central machine (a black cube), three material entities (triangles), and a core danger zone marked by a dashed circle. The simulation experiment diagram is shown below. Figures 2-5 As shown.

[0149] Figures 2-5 In the middle, the left side is a schematic diagram of the simulation scenario, and the right side is the data dashboard output by the system in real time in this embodiment. The dashboard dynamically displays five core indicators: interaction risk probability, predicted trajectory, attitude instability probability, comprehensive risk, and early warning channel.

[0150] Figure 2 The diagram shows the normal monitoring stage, where the operator is located in a safe area outside the danger zone. At this time, the interactive risk probability output by the interactive prediction module of this invention is 0.02, the attitude instability probability output by the attitude prediction module is 0.23, and the risk decision module determines the overall risk to be at the normal level, and does not issue a warning.

[0151] Figure 3 The image shows the warning level stage. As the worker continues to walk and approaches the danger zone, the interactive prediction module predicts that the probability of interactive risk increases to 0.14, and the attitude prediction module predicts that the worker's trajectory will lead to instability risk, outputting an attitude instability probability of 0.41. The risk decision module comprehensively determines that the overall risk is at the warning level, and the warning execution module is activated to send targeted warning information to the management terminal.

[0152] Figure 4 The diagram shows the warning level early warning stage. The operator has entered the edge of the danger zone. Both the system's interaction prediction module and attitude prediction module output high probabilities. The interaction risk probability rises to 0.45, and the attitude instability probability rises to 0.65. The risk decision module determines that the risk has been upgraded to the warning level. The early warning execution module adds an early warning channel and sends targeted early warning information to the wearable device and management terminal at the same time.

[0153] Figure 5 The diagram shows the emergency warning stage. The workers have entered the center of the danger zone. At this time, the probability of interactive risk (0.98) and the probability of attitude instability (0.91) are close to saturation. The risk decision module determines that the comprehensive risk is emergency level. The warning execution module triggers the highest level warning, displays "!!!Emergency Risk!!!" in the scene, and sends targeted warnings to "wearable devices, mechanical control console, and automatic control" to achieve closed-loop intervention before the accident occurs.

[0154] This simulation example verifies that, by combining spatiotemporal graph neural networks and long short-term memory networks, this embodiment effectively solves the problem that traditional monitoring systems can only identify static violations and cannot predict dynamic interactive risks and attitude instability. By using a multilayer perceptron to perform multi-source risk fusion and four-level (normal, alert, warning, and emergency) risk determination, the accuracy and timeliness of the early warning are guaranteed.

[0155] In summary, the system in this embodiment demonstrates powerful dynamic risk prediction and graded early warning capabilities in dynamic work scenarios where workers approach and enter hazardous sources. Furthermore, by combining a multimodal AI prediction model with a comprehensive risk decision-making model, a closed-loop management system from pre-prediction to real-time early warning is constructed.

Claims

1. A real-time operational risk monitoring system based on multi-angle video and AI recognition, characterized in that: include: The operation data extraction module is used to collect and preprocess construction operation data, and extract the three-dimensional spatial pose sequence, motion trajectory sequence, three-dimensional human skeleton key point time series data and attribute features of dynamic entities in the construction site. The interaction prediction module is used to input 3D spatial pose sequences, motion trajectory sequences, and attribute features into a spatiotemporal interaction risk prediction model with a spatiotemporal graph neural network structure to predict the probability of interaction risks between dynamic entities. The process includes: Construct a time series graph structure, which includes graph nodes, graph edges, node features, and edge features at each time step; Define the extracted dynamic entities as graph nodes; Assign the 3D spatial pose sequence, motion trajectory sequence, and attribute features as node features of the graph nodes; The dynamic spatiotemporal relationships between dynamic entities are defined as edges of a graph; Calculate the relative spatial distance and relative motion velocity between dynamic entities, and assign the calculated relative spatial distance and relative motion velocity as edge features of the graph. The temporal graph structure is input into the spatiotemporal interaction risk prediction model. The spatiotemporal interaction risk prediction model adopts a spatiotemporal graph neural network structure. By learning the temporal changes of graph node and edge features, it calculates and outputs the probability of interaction risk between dynamic entities within a future preset time window. Spatial graph convolutional layers are used to aggregate node features and edge features at each time step to generate spatial aggregated features; Temporal convolutional layers are used to learn the changing patterns of spatial aggregate features at consecutive time steps; Based on the change pattern, the probability of interaction risk is calculated and output through a fully connected layer; The posture prediction module is used to input the temporal data of key points of the 3D human skeleton into the person's posture intention prediction model, and predict the person's motion trajectory and the probability of posture instability. The process includes: At each time step of the three-dimensional human skeleton key point time series data, a preset root node is selected as the reference point, and the three-dimensional relative coordinates of each key point in the three-dimensional human skeleton key point relative to the reference point are calculated to generate normalized time series data. Normalized time-series data is input into the human posture and intention prediction model. The human posture and intention prediction model adopts a long short-term memory network structure. The long short-term memory network structure learns the temporal dependence of normalized time-series data through internal gating units and outputs posture feature vectors. The posture feature vector is input into the trajectory prediction layer, which is used to calculate and output the coordinates of the three-dimensional human skeleton key points of the person within a preset time window in the future, as the person's motion trajectory. The posture feature vector is input into the instability classification layer, which calculates and outputs the probability of posture instability of the person. Where z is the linear output of the unstable classification layer, and e is the natural constant; The risk decision module is used to construct a comprehensive risk decision model based on a multilayer perceptron, based on the probability of interaction risk, the probability of motion trajectory and attitude instability, and to train and optimize it using a labeled dataset containing historical risk events, set the operation risk judgment logic, and output the risk level judgment result. The early warning execution module is used to execute a multi-channel early warning mechanism on edge computing nodes based on the risk level determination results, using a distributed computing architecture, and to determine the risk level clearance status in real time.

2. The real-time operational risk monitoring system based on multi-angle video and AI recognition according to claim 1, characterized in that, The process of collecting and preprocessing construction operation data in the operation data extraction module includes: Construction operation data includes spatiotemporal video data of the construction site, mechanical operation parameter data, and real-time location status data of personnel; Real-time spatiotemporal video data is collected using multi-angle fixed cameras, personnel wearable cameras, and mobile device cameras deployed at the construction site. By using sensors deployed on construction machinery and position sensors on personnel's wearable devices, the system collects operational parameter data of the machinery and real-time location status data of the personnel. This ensures that multi-angle fixed cameras, wearable cameras, mobile device cameras and sensors are synchronized with the central time server, providing a unified time reference for the collected spatiotemporal video data, operational parameter data and real-time location status data; Perform spatial coordinate system registration. Through camera calibration methods, obtain the internal parameter matrix and external parameter matrix of the multi-angle fixed camera. Use this matrix to establish the mapping relationship between world coordinate system points and pixel coordinate system points. Unify the spatiotemporal video data to the same world coordinate system of the construction site, generate preprocessed spatiotemporal video data, and convert the real-time position status data of personnel to the same world coordinate system of the construction site. Perform sensor data cleaning, which involves denoising and interpolating the machine's operating parameter data and the personnel's real-time position status data to obtain pre-processed operating parameter data and real-time position status data.

3. The real-time operational risk monitoring system based on multi-angle video and AI recognition according to claim 2, characterized in that, The process of extracting the three-dimensional spatial pose sequence, motion trajectory sequence, and three-dimensional human skeleton key point time series data and attribute features of dynamic entities in the construction site in the operation data extraction module includes: Dynamic entities include personnel entities, machine entities, and material entities; Based on the preprocessed spatiotemporal video data, a multi-view geometric reconstruction algorithm is used to extract the three-dimensional spatial pose sequence and motion trajectory sequence of dynamic entities in the construction site. In spatiotemporal video data, run an instance segmentation model to obtain the bounding boxes and segmentation masks of dynamic entities in multi-view 2D images; By using segmentation masks, internal parameter matrices, and external parameter matrices, a 3D point cloud model of a dynamic entity is generated through a multi-view contour reconstruction method. Perform 3D bounding box fitting and pose estimation on the 3D point cloud model of a dynamic entity to obtain the 3D spatial pose of the dynamic entity. A multi-target tracking algorithm is used to temporally correlate the three-dimensional spatial poses in consecutive frames to generate a three-dimensional spatial pose sequence and a motion trajectory sequence of dynamic entities. For human entities, extract the temporal data of key points of the three-dimensional human skeleton from the preprocessed spatiotemporal video data; In spatiotemporal video data, a two-dimensional human pose estimation algorithm is run to extract the two-dimensional human skeleton key points of the human entity in multi-view two-dimensional images; Using the internal parameter matrix and the external parameter matrix, triangulation reconstruction of key points of the two-dimensional human skeleton is performed, and the coordinates of key points of the three-dimensional human skeleton are calculated. A multi-target tracking algorithm is used to temporally correlate the coordinates of key points of the three-dimensional human skeleton in consecutive frames to generate temporal data of key points of the three-dimensional human skeleton of a person. The pre-processed machine operation parameter data and the real-time location status data of personnel are integrated as attribute features of dynamic entities.

4. The real-time operational risk monitoring system based on multi-angle video and AI recognition according to claim 1, characterized in that, In the risk decision-making module, the process of constructing a comprehensive risk decision-making model based on a multilayer perceptron includes: The probability of interactive risks, the trajectory of personnel and the probability of posture instability are spliced ​​and flattened to generate the input feature vector of the risk comprehensive decision model. A risk comprehensive decision-making model with a multilayer perceptron structure is constructed, which includes an input layer, a hidden layer, and an output layer. The input layer receives the input feature vector; Hidden layers perform high-dimensional feature mapping on the input feature vector using non-linear activation functions; The output layer outputs a linear output value corresponding to the risk level, which is divided into emergency, warning, alert, and normal levels.

5. The real-time operational risk monitoring system based on multi-angle video and AI recognition according to claim 1, characterized in that, In the risk decision-making module, the process of training and optimizing using a labeled dataset containing historical risk events includes: Historical risk events include violations, near-accidents, and accidents; The historical interaction risk probability, historical personnel movement trajectory and historical posture instability probability in the labeled dataset are concatenated and flattened to generate training feature vectors. Input the trained feature vectors into the risk comprehensive decision-making model to obtain linear output values; Calculate the loss function value between the linear output value and the corresponding true risk level label in the labeled dataset; The backpropagation algorithm is used to update the network weight parameters in the multilayer perceptron structure until the loss function value converges, thus obtaining the trained and optimized risk comprehensive decision model.

6. The real-time operational risk monitoring system based on multi-angle video and AI recognition according to claim 4, characterized in that, In the risk decision-making module, the process of setting up the operational risk assessment logic and outputting the risk level assessment result includes: The real-time generated input feature vector is imported into the trained and optimized risk comprehensive decision-making model to obtain a linear output value. A multi-class classification function is used to calculate the predicted score for each risk level. The risk level with the highest predicted score is selected as the final risk level determination.

7. The real-time operational risk monitoring system based on multi-angle video and AI recognition according to claim 2, characterized in that, In the early warning execution module, based on the risk level determination results, the process of executing a multi-channel early warning mechanism on edge computing nodes using a distributed computing architecture includes: When the risk level is determined to be emergency, targeted early warning information is sent to the wearable devices of the personnel involved in the risk, the control console of the control room of the relevant machinery, and the automatic control system of the machinery. When the risk level assessment result is warning level, targeted early warning information is sent to the wearable devices of the personnel involved in the risk and the terminals of the safety management personnel. When the risk level assessment result is at the warning level, a targeted early warning message is sent to the safety management personnel's terminal.

8. The real-time operational risk monitoring system based on multi-angle video and AI recognition according to claim 7, characterized in that, The process of determining the clearance status of risk levels in real time within the early warning execution module includes: After sending targeted early warning information, the risk decision module continuously outputs the risk level determination results for subsequent time steps; If the risk level assessment result of the subsequent time step is normal, then the risk level is determined to be lifted, and the sending of targeted early warning information will be stopped. If the risk level assessment result in subsequent time steps is not at the normal level, the risk level is determined to be not resolved, and targeted early warning information will continue to be sent.

Images